38

I'm trying to tail a log file on multiple remote machines and forward the output to my local workstation. I want connections to close when pressing Ctrl-C.

At the moment I have the following function that almost works as intended.

function dogfight_tail() {
 logfile=/var/log/server.log
 pids=""
 for box in 02 03; do
   ssh server-$box tail -f $logfile | grep $1 &
   pids="$pids $!"
 done
 trap 'kill -9 $pids' SIGINT
 trap  wait
}

The connections close and I receive the output from tail. BUT, there is some kind of buffering going on because the output come in batches.

And here's the fun part…

I can see the same buffering behaviour when executing the following and append "test" to the file /var/log/server.log on the remote machines 4-5 times…

ssh server-01 "tail -f /var/log/server.log | grep test"

…and found two ways of disabling it…

  1. Add -t flag to ssh.

    ssh -t server-01 "tail -f /var/log/server.log | grep test"
    
  2. Remove quotation from the remote command.

    ssh server-01 tail -f /var/log/server.log | grep test
    

However, neither of these approaches work for the function that execute on multiple machines mentioned above.

I have tried dsh, which have the same buffering behaviour when executing.

dsh -m server-01,server-02 -c "tail -f /var/log/server.log | grep test"

Same here, if I remove the quotation, the buffering goes away and everything works fine.

dsh -m server-01,server-02 -c tail -f /var/log/server.log | grep test

Also tried parallel-ssh which works exactly the same as dsh. Can somebody explain what's going on here?

How do I fix this problem? Would be ideal to go with straight ssh if possible.

P.S. I do not want to use multitail or similar since I want to be able to execute arbitrary commands.

  • You can checkout dbitail and download it from here. – user176409 Jun 23 '16 at 8:57
37

What you see is effect of a standard stdout buffer in grep provided by Glibc. The best solution is to disable it by using --line-buffered (GNU grep, I'm not sure what other implementations might support it or something similar).

As for why this only happens in some cases:

ssh server "tail -f /var/log/server.log | grep test"

runs the whole command in the quotes on the server - thus grep waits to fill its buffer.

ssh server tail -f /var/log/server.log | grep test

runs grep on your local machine on the output tail sent through the ssh channel.

The key part here is, that grep adjusts its behaviour depending on whether its stdin is a terminal or not. When you run ssh -t, the remote command is running with a controlling terminal and thus the remote grep behaves like your local one.

  • Thanks a lot for the detailed explanation. It make sense to me now and the script works as expected with --line-buffered. – deephacks Oct 3 '14 at 11:41
  • @deephacks In that case please consider accepting the answer - it gives a clue to others having the same problem. – peterph Oct 3 '14 at 23:34
  • 2
    grep/glibc's buffering depends on its stdout. ssh tail | grep outputs to the local terminal, unbuffered. ssh -t "tail|grep" outputs to a pty, unbuffered. ssh "tail|grep" outputs to a pipe (to sshd), buffered (unless --line-buffered). – dave_thompson_085 Jun 25 '16 at 16:55
2

check this out: multitail

MultiTail allows you to monitor logfiles and command output in multiple windows in a terminal, colorize, filter and merge.

To tail logs in multiple servers use:

multitail -l 'ssh user@host1 "tail -f /path/to/log/file"' -l 'ssh user@host2 "tail -f /path/to/log/file"'
  • 3
    But it doesn't let you do it over ssh, which is a condition of this question. (And, also, the question specifically says "don't want to use multitail".) – bishop Sep 29 '16 at 16:42
  • 1
    @bishop: I think this criticism is unfair in part because while the question may have specified not using multitail, it seems to be due to a misunderstanding. The example above shows how to use arbitrary commands and the regular shell expansions also work — multitail <(ssh …) <(ssh …) — allowing the desired outcome even if it's not how they originally thought the question could be answered. – Chris Adams Nov 8 '17 at 20:25
0

You can checkout in'side log.

A Java tool I created, able to read local and distant log files using SSH. It is fairly simple to use.

Some more explanations: https://github.com/pschweitz/insidelog/wiki

Just download version corresponding to your operating system, of native jar release executable within your Java Runtime (requires java 8_40 or higher):

https://github.com/pschweitz/insidelog/releases

You can find a complete documentation (embedded with and in Github's page as well)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.