6

I'm running a propietary program which generates text output. Kind of like a logger.

logger

I'd like to both view this output on the terminal to watch the server breathe, and send it to a text file. So normally I'd do:

logger | tee /tmp/log.txt

But these log files can become large enough to overwhelm my VMs disk space. Instead of sending the text to an uncompressed text file, I'd like it to be compressed immediately. So I might do:

logger

in one terminal and

logger | gzip > /tmp/log.gz

in another terminal. But this doesn't feel very *nixy.

Is there a way I can accomplish this in one command, similar to using tee? Here is what I'm going for, which obviously won't work, but maybe you'll get the idea:

logger | tee gzip > /tmp/log.txt
17
+50

With ksh93, zsh or bash:

logger | tee >(gzip > /tmp/log.txt.gz)

That uses that ksh feature called process substitution. >(gzip > /tmp/log.txt.gz) is substituted with the path of a file (typically something like /dev/fd/something, but that could also be a temporary named pipe) that refers to the writing end of a pipe. At the other (reading) end of that pipe, the shell connects the standard input of a new process (run in background) that executes the gzip command.

So, when tee writes to that file, it's actually feeding the data to gzip.

On systems with /dev/fd/n, you can do it manually (with any Bourne-like shell but ksh93) like:

{ logger | tee -a /dev/fd/3 | gzip > /tmp/log.txt.gz; } 3>&1

(though in that case /dev/fd/3 refers to the original stdin which we've made available on the file descriptor 3 with 3>&1, not to the pipe to gzip which here is just connected to tee's stdout)

With zsh:

logger >&1 > >(gzip > /tmp/log.txt.gz)

That uses zsh multios feature whereby zsh implements a sort of tee internally to redirect the same output to more than one file (here the original stdout (>&1) and the pipe to gzip using process substitution again.

logger's stdout will actually be the writing end of a pipe. At the other end of the pipe is a shell process that reads it and distributes it both outputs like tee would.

  • Brilliant, thanks. I don't totally get the syntax of the bash version. Can I offer you a bribe for explaining it? I'll pay a 50 pt bounty as soon as I can. – John Dibling Aug 15 '13 at 13:45
  • @JohnDibling, I'm not that venal. Answer updated. – Stéphane Chazelas Aug 15 '13 at 14:01
  • Let's not call it a bribe, then. I'll be back in a couple of days. :) Thanks again. – John Dibling Aug 15 '13 at 14:09
  • @JohnDibling, You can also write the output to STDERR as follows: logger | tee -a /dev/fd/2 | gzip > /tmp/log.txt.gz – webKnjaZ Aug 23 '13 at 5:54
  • Not completely on topic but, why can't you view logs via 'logger' and gzip it via size-based logrotate rule? – user1293137 Aug 23 '13 at 7:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.