Best way to follow a log and execute a command when some text appears in the log

Question

I have a server log that outputs a specific line of text into its log file when the server is up. I want to execute a command once the server is up, and hence do something like the following:

tail -f /path/to/serverLog | grep "server is up" ...(now, e.g., wget on server)?

What is the best way to do this?

Answer 1

A simple way would be awk.

tail -f /path/to/serverLog | awk '
                    /Printer is on fire!/ { system("shutdown -h now") }
                    /new USB high speed/  { system("echo \"New USB\" | mail admin") }'

And yes, both of those are real messages from a kernel log. Perl might be a little more elegant to use for this and can also replace the need for tail. If using perl, it will look something like this:

open(my $fd, "<", "/path/to/serverLog") or die "Can't open log";
while(1) {
    if(eof $fd) {
        sleep 1;
        $fd->clearerr;
        next;
    }
    my $line = <$fd>;
    chomp($line);
    if($line =~ /Printer is on fire!/) {
        system("shutdown -h now");
    } elsif($line =~ /new USB high speed/) {
        system("echo \"New USB\" | mail admin");
    }
}

Answer 2

If you're only looking for one possibility and want to stay mostly in the shell rather than using awk or perl, you could do something like:

tail -F /path/to/serverLog | 
grep --line-buffered 'server is up' | 
while read ; do my_command ; done

...which will run my_command every time "server is up" appears in the log file. For multiple possibilities, you could maybe drop the grep and instead use a case within the while.

The capital -F tells tail to watch for the log file to be rotated; i.e. if the current file gets renamed and another file with the same name takes its place, tail will switch over to the new file.

The --line-buffered option tells grep to flush its buffer after every line; otherwise, my_command may not be reached in a timely fashion (assuming the logs have reasonably sized lines).

Answer 3

It is strange that no one mentioned about multitail utility which has this functionality out-of-box. One of usage example:

Show the output of a ping-command and if it displays a timeout, send a message to all users currently logged in

multitail -ex timeout "echo timeout | wall" -l "ping 192.168.0.1"

See also another examples of multitail usage.

Answer 4

This question appears to be answered already, but I think there's a better solution.

Rather than tail | whatever, I think what you really want is swatch. Swatch is a program designed explicitly for doing what you're asking, watching a log file and executing actions based on log lines. Using tail|foo will require that you've got a terminal actively running to do this. Swatch on the other hand runs as a daemon and will always be watching your logs. Swatch is available in all Linux distros,

I encourage you to try it out. While you can pound a nail in with the back side of a screwdriver does not mean you should.

The best 30-second tutorial on swatch I could find is here.

Answer 5

bash could do the job by himself

Let see how simple and readable it could be:

mylog() {
    echo >>/path/to/myscriptLog "$@"
}

while read line;do
    case "$line" in
        *"Printer on fire"* )
            mylog Halting immediately
            shutdown -h now
            ;;
        *DHCPREQUEST* )
            [[ "$line" =~ DHCPREQUEST\ for\ ([^\ ]*)\  ]]
            mylog Incomming or refresh for ${BASH_REMATCH[1]}
            $HOME/SomethingWithNewClient ${BASH_REMATCH[1]}
            ;;
        * )
            mylog "untrapped entry: $line"
            ;;
    esac
  done < <(tail -f /path/to/logfile)

While you don't use bash's regex, this could stay very quick!

But bash + sed is a very efficient and interesting tandem

But for high load server, and as I like sed because it's very quick and very scalable, I often use this:

while read event target lost ; do
    case $event in
        NEW )
            ip2int $target intTarget
            ((count[intTarget]++))
        ...

    esac
done < <(tail -f /path/logfile | sed -une '
  s/^.*New incom.*from ip \([0-9.]\+\) .*$/NEW \1/p;
  s/^.*Auth.*ip \([0-9.]\+\) failed./FAIL \1/p;
  ...
')

Answer 6

That's how i started doing this too but have become much more sophisticated with it. A couple things to be concerned with:

1. If the tail of the log already contains "server is up".
2. Automatically ending the tail process once it's found.

---

I use something along the lines of this:

RELEASE=/tmp/${RANDOM}$$
(
  trap 'false' 1
  trap "rm -f ${RELEASE}" 0
  while ! [ -s ${RELEASE} ]; do sleep 3; done
  # You can put code here if you want to do something
  # once the grep succeeds.
) & wait_pid=$!
tail --pid=${wait_pid} -F /path/to/serverLog \
| sed "1,10d" \
| grep "server is up" > ${RELEASE}

It works by holding tail open until the ${RELEASE} file contains data.

Once the grep succeeds it:

1. writes the output to ${RELEASE} which will
2. terminate the ${wait_pid} process to
3. exit the tail

Note: The sed can be more sophisticated to actually determine the number of lines tail will produce at startup and the remove that number. But generally, it's 10.

Answer 7

I use fail2ban to monitor and execute a command