0

This question already has an answer here:

I have a log file which is continuously updating(new line added) after few time period.

I am fetching only error messages from the file in every 10 minutes.

Initially, at 1st time I fetched all line into a new file with a matching pattern "ERROR FOUND" using awk.

But after 10 min more new line has been added to a log file, so I want to read that log file where I left. I don't want to start from the beginning again.

Can any body suggest me best code or script for this?

marked as duplicate by msp9011, DarkHeart, countermode, schily, Thomas Aug 29 '18 at 17:31

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • Yes it is back end system file which updates in every second. In this I'm trying to store last line no. also in another file. SO i Can start reading file again after that line no. But some how I'm not able to querying it. – Vipin Sahu Aug 29 '18 at 7:08
2

If you open the file on a file descriptor like:

exec 3< /path/to/log/file

You can then process it:

awk '...' <&3

After which fd 3 will point to where awk left it.

10 minutes later, from the same shell invocation, you can run that

awk '...' <&3

command again to process the new data.

If you want to save the position you were at, so you can resume reading from a different shell invocation, with ksh93, you can do:

#! /usr/bin/env ksh93
file=/path/to/some-file
offset_file=$file.offset

exec 3< "$file"
[ -f "$offset_file" ] && exec 3<#(($(<"$offset_file")))

awk '...' <&3

echo "$(3<#((CUR)))" > "$offset_file"

Or with zsh:

#! /usr/bin/env zsh

zmodload zsh/system
file=/path/to/some-file
offset_file=$file.offset

exec 3< $file
[ -f "$offset_file" ] && sysseek -u 3 "$(<$offset_file)"

awk '...' <&3

echo $((systell(3))) > $offset_file
1

I like Stéphane's answer because it doesn't read the whole file again and again, so I add here the bash (on Linux) equivalent of his solution (bash has no builtin seek or tell ability). I would have used a comment but my reputation is too low.

LASTPOS=/tmp/saved_pos

exec 3< "$1"
test -f "$LASTPOS" && STARTPOS=$(($(<$LASTPOS)+1))
tail -c "+${STARTPOS:-1}" <&3 | grep "ERROR FOUND"
grep '^pos:' /proc/self/fdinfo/3 | cut -f2 > "$LASTPOS"

I also replaced the awk command with a grep because it is usually faster. You can pipe the output to a awk command if you need further processing.

0

I would give a try with wc -l and tail.
If you are using bash, this should work:

#!/bin/bash
LASTLNFILE=/tmp/lastline     # replace with a suitable path
test -f $LASTLNFILE && LASTLN=$(<$LASTLNFILE)
CURLN=$(wc -l $1 | cut -d' ' -f1)

if ((CURLN-LASTLN > 0)); then
  tail -n $((CURLN-LASTLN)) $1
fi
echo $CURLN > $LASTLNFILE

P.S. use it as a filter before your awk program, e.g. (assuming you named it 'newlines.sh'):

./newlines.sh <log_file> | awk -f <your_awk_program>`

I am leaving the above script as an example of how to not do it. Just after writing it I realized it is vulnerable to a race condition, whenever the log file is updated while the script is running.

A pure AWK approach is preferable:

#!/bin/awk

BEGIN { 
  lastlinefile = "/tmp/lastlinefile"
  getline lastline < lastlinefile
}

NR > lastline && /ERROR FOUND/ {
  # do your stuff...
  print
}

END { print NR > lastlinefile }

Not the answer you're looking for? Browse other questions tagged or ask your own question.