4

I've got a log file that I process as follows:

grep pattern /var/log/whatever.log | \
    cut ... | sort | uniq -c | sort -rn | \
    etc....

However, the log file is quite big, and records events from the beginning of the day. It's also appended to constantly. I'd like to only process the last next 10 minutes of the file.

So I'm looking for something like the following:

killafter 600 tail -f /var/log/whatever.log | stuff

Or, better yet (wait however long it takes to capture 1000 matching lines):

tail -f /var/log/whatever.log | grep pattern | stopafter -lines 1000 | stuff

Are there any tools that'll let me do this?

  • 1
    Look for the timeout command. – roaima Sep 9 '16 at 9:16
6

roaima rightly points you at the timeout command, and head actually terminates after it has read the desired number of lines, so I would hope that with

timeout 600s tail -f ‹logfile› | ‹filter› | head -n 1000

you'd get there.

1

By default, timeout kills the process and the pipe gets killed.

timeout --foreground 600s tail -n 0 -f logfile | ...filter | head -n 1000| ...
  • timeout --foreground otherwise we would see just "Terminated"
  • tail -n 0 -f to show only the new loglines
  • head -n 1000 to "stop after 1000 lines"

\thanks{roaima and Ulrich Schwarz}

  • 2
    How is this different to Ulrich Schwarz's answer? – roaima Sep 9 '16 at 10:16
  • @roaima, Ulrich Schwartz's answer, gets "terminated" and prints no output in case of timeout. – JJoao Sep 9 '16 at 10:36
  • @roaima and Ulrich, unfortunately my answer is not working properly too: in case of obtaining "1000 lines", it prints the answer but the "timout" process continues :( – JJoao Sep 9 '16 at 10:41
0

This perl code will give you a counted tail

#!/usr/bin/perl
#
# Usage tailmax.pl <filename> [<num_lines>]
#
use strict;
use warnings;

use File::Tail;

my $logfile = shift @ARGV;
my $pattern = shift @ARGV || '';
my $count = shift @ARGV || -1;

my $fh = File::Tail->new(name => $logfile, tail => 0, interval => 0, maxinterval => 2) or
    die "Cannot open $logfile: $!\n";

while ($count != 0 and defined (my $line = $fh->read)) {
    if ($pattern eq '' or $line =~ /$pattern/) {
        print "$line";
        $count-- if $count > 0;
    }
}

If you save it to a file such as /usr/local/bin/tailmax and make it executable, you can call it like this, which will give you the next 100 messages from the log file /var/log/messages that contain the RE somepattern. (You may need to be root to read this file.)

tailmax /var/log/messages 'somepattern' 100

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.