Best way to follow a log and execute a command when some text appears in the log

Question

I have a server log that outputs a specific line of text into its log file when the server is up. I want to execute a command once the server is up, and hence do something like the following:

tail -f /path/to/serverLog | grep "server is up" ...(now, e.g., wget on server)?

What is the best way to do this?

Answer 1

A simple way would be awk.

tail -f /path/to/serverLog | awk '/Printer is on fire!/ { system("shutdown -h now") }
                                  /new USB high speed/ { system("echo New USB" | mail admin")'

And yes, both of those are real message from a kernel log. Perl might be a little more elegant to use for this and can also replace the need for tail. If using perl, it will look something like this:

open(my $fd, "<", "/path/to/serverLog") or die "Can't open log";
while(1) {
    if(eof $dataFd) {
        sleep 1;
        $fd->clearerr;
        next;
    }
    my $line = <$fd>;
    chomp($line);
    if($line =~ /Printer is on fire!/) {
        system("shutdown -h now");
    } elsif($line =~ /new USB high speed/) {
        system("echo New USB" | mail admin");
    }
}

Answer 2

That's how i started doing this too but have become much more sophisticated with it. A couple things to be concerned with:

1. If the tail of the log already contains "server is up".
2. Automatically ending the tail process once it's found.

---

I use something along the lines of this:

RELEASE=/tmp/${RANDOM}$$
(
  trap 'false' 1
  trap "rm -f ${RELEASE}" 0
  while ! [ -s ${RELEASE} ]; do sleep 3; done
  # You can put code here if you want to do something
  # once the grep succeeds.
) & wait_pid=$!
tail --pid=${wait_pid} -F /path/to/serverLog \
| sed "1,10d" \
| grep "server is up" > ${RELEASE}

It works by holding tail open until the ${RELEASE} file contains data.

Once the grep succeeds it:

1. writes the output to ${RELEASE} which will
2. terminate the ${wait_pid} process to
3. exit the tail

Note: The sed can be more sophisticated to actually determine the number of lines tail will produce at startup and the remove that number. But generally, it's 10.

Answer 3

It is strange that no one mentioned about multitail utility which has this functionality out-of-box. One of usage example:

Show the output of a ping-command and if it displays a timeout, send a message to all users currently logged in

multitail -ex timeout "echo timeout | wall" -l "ping 192.168.0.1"

See also another examples of multitail usage.

Answer 4

This question appears to be answered already, but I think there's a better solution.

Rather than tail | whatever, I think what you really want is swatch. Swatch is a program designed explicitly for doing what you're asking, watching a log file and executing actions based on log lines. Using tail|foo will require that you've got a terminal actively running to do this. Swatch on the other hand runs as a daemon and will always be watching your logs. Swatch is available in all Linux distros,

I encourage you to try it out. While you can pound a nail in with the back side of a screwdriver does not mean you should.

The best 30-second tutorial on swatch I could find is here: http://www.campin.net/newlogcheck.html

Answer 5

If you're only looking for one possibility and want to stay mostly in the shell rather than using awk or perl, you could do something like:

tail -F /path/to/serverLog | grep 'server is up' | while read ; do my_command ; done

...which will run my_command every time "server is up" appears in the log file. For multiple possibilities, you could maybe drop the grep and instead use a case within the while.

The capital -F tells tail to watch for the log file to be rotated; i.e. if the current file gets renamed and another file with the same name takes its place, tail will switch over to the new file.