3

I have a log file which has a date and time at the start of every line.

I need to search the log file starting from a specific time to the end of the file.

For example:

Starting point: July 29 2018 21:00:00
End point     : end of file

My concern is even if the pattern of July 29 2018 21:00:00 does not exist, I still get the lines between for example July 29 2018 21:05:11 since this is still beyond July 29 2018 21:00:00.

Is awk or sed work for this?

  • can you share the sample file.. – msp9011 Aug 14 '18 at 10:33
  • Is this something ausearch can perform? – SPITFINGERS Aug 14 '18 at 17:21
2

I'd use perl for this, to parse the timestamp on each line:

$ cat file
June 5 2018 00:00:00 do not print
July 29 2018 20:59:59 do not print
July 29 2018 21:00:00 print me
July 29 2018 21:00:01 print me

$ perl -MTime::Piece -sane '
    BEGIN {
        $start = Time::Piece->strptime($startdate, "%B %e %Y %T");
    }
    # the string "@F[0..3]" is the first 4 words on the line
    $time = Time::Piece->strptime("@F[0..3]", "%B %e %Y %T");
    print if $time >= $start;
' -- -startdate="July 29 2018 21:00:00" file
July 29 2018 21:00:00 print me
July 29 2018 21:00:01 print me

This version is somewhat more efficient, as it stops parsing the timestamp once the start date has been seen (assumes the file is in increasing cronological order):

perl -MTime::Piece -sane '
    BEGIN {
        $start = Time::Piece->strptime($startdate, "%B %e %Y %T");
    }
    unless ($go) {
        $time = Time::Piece->strptime("@F[0..3]", "%B %e %Y %T");
        $go = $time >= $start;
    }
    print if $go;
' -- -startdate="July 29 2018 21:00:00" file
  • +1 compared to my solution this is just superfast. awk with a system call for date was even slower. – pLumo Aug 14 '18 at 14:35
  • GNU awk (gawk) has builtin time functions, but parsing times is still quite manual. – glenn jackman Aug 14 '18 at 14:41
1

Try this:

grepfromdate() {
    readarray f < $1
    fromdate=$(date +%s -d "$2")
    for (( lineno=${#f[@]}-1 ; lineno>=0; lineno-- )) ; do
        line=${f[$lineno]}
        time_from_line=$(echo "$line" | grep -o "^[A-Z][a-z]* [0-9][0-9] [0-9][0-9][0-9][0-9] [0-9][0-9]:[0-9][0-9]:[0-9][0-9]")
        [[ $(date +%s -d "$time_from_line") -gt $fromdate ]] && echo "$line" || break
    done | tac
}

Usage:
grepfromdate "filename" "July 29 2018 21:00:00"

You can pass any date format that date can read to it, e.g. 2018-07-01. If the format of the date changes, you can change the grep pattern according to that.

  • this is slow as it needs to open the file for each line – pLumo Aug 14 '18 at 14:40
  • it also needs to invoke sed, grep and date (twice) for each line. – glenn jackman Aug 14 '18 at 14:40
  • 1
    I changed the script and now read the file into an array to remove invocation of sed, changed date to be invoked only once. But still one date and one grep for each line. Still that doubled the performance ;-) – pLumo Aug 14 '18 at 14:53
0

You could search for the first line matching a defined String (i.e. July 29 2018 21: for everything after 9 pm). If you have this line number, you can tail the file starting with the found line number.

   $ man tail
   -n, --lines=[+]NUM
          output the last NUM lines, instead of the last 10; or use -n +NUM to output starting with line NUM

my example:

$ log=/var/log/syslog

# get line number
$ first_line=$(grep -no "Aug 14 08:" $log | tail -n1 | cut -d: -f1)

# count the lines from $first_line to EOF
$ tail -n +$first_line $log | wc -l
24071

# output the content starting with $first_line
$ tail -n +$first_line $log

# line count of the whole file:
$ wc -l $log
70896 /var/log/syslog
0

With sed you can do

sed -n '/July 29 2018 21:/,/$!d/p' file

That will get you all lines between July 29 2018 21:** and the last line of file

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.