resume reading a log file from the point I left it last time [duplicate]

Question

I have a log file which is continuously updating(new line added) after few time period.

I am fetching only error messages from the file in every 10 minutes.

Initially, at 1st time I fetched all line into a new file with a matching pattern "ERROR FOUND" using awk.

But after 10 min more new line has been added to a log file, so I want to read that log file where I left. I don't want to start from the beginning again.

Can any body suggest me best code or script for this?

Answer 1

If you open the file on a file descriptor like:

exec 3< /path/to/log/file

You can then process it:

awk '...' <&3

After which fd 3 will point to where awk left it.

10 minutes later, from the same shell invocation, you can run that

awk '...' <&3

command again to process the new data.

If you want to save the position you were at, so you can resume reading from a different shell invocation, with ksh93, you can do:

#! /usr/bin/env ksh93
file=/path/to/some-file
offset_file=$file.offset

exec 3< "$file"
[ -f "$offset_file" ] && exec 3<#(($(<"$offset_file")))

awk '...' <&3

echo "$(3<#((CUR)))" > "$offset_file"

Or with zsh:

#! /usr/bin/env zsh

zmodload zsh/system
file=/path/to/some-file
offset_file=$file.offset

exec 3< $file
[ -f "$offset_file" ] && sysseek -u 3 "$(<$offset_file)"

awk '...' <&3

echo $((systell(3))) > $offset_file

Answer 2

I like Stéphane's answer because it doesn't read the whole file again and again, so I add here the bash (on Linux) equivalent of his solution (bash has no builtin seek or tell ability). I would have used a comment but my reputation is too low.

LASTPOS=/tmp/saved_pos

exec 3< "$1"
test -f "$LASTPOS" && STARTPOS=$(($(<$LASTPOS)+1))
tail -c "+${STARTPOS:-1}" <&3 | grep "ERROR FOUND"
grep '^pos:' /proc/self/fdinfo/3 | cut -f2 > "$LASTPOS"

I also replaced the awk command with a grep because it is usually faster. You can pipe the output to a awk command if you need further processing.

Answer 3

I would give a try with wc -l and tail.
If you are using bash, this should work:

#!/bin/bash
LASTLNFILE=/tmp/lastline     # replace with a suitable path
test -f $LASTLNFILE && LASTLN=$(<$LASTLNFILE)
CURLN=$(wc -l $1 | cut -d' ' -f1)

if ((CURLN-LASTLN > 0)); then
  tail -n $((CURLN-LASTLN)) $1
fi
echo $CURLN > $LASTLNFILE

P.S. use it as a filter before your awk program, e.g. (assuming you named it 'newlines.sh'):

./newlines.sh <log_file> | awk -f <your_awk_program>`

---

I am leaving the above script as an example of how to not do it. Just after writing it I realized it is vulnerable to a race condition, whenever the log file is updated while the script is running.

A pure AWK approach is preferable:

#!/bin/awk

BEGIN { 
  lastlinefile = "/tmp/lastlinefile"
  getline lastline < lastlinefile
}

NR > lastline && /ERROR FOUND/ {
  # do your stuff...
  print
}

END { print NR > lastlinefile }