8

I have long-running programs that can restart their internal state. I want to see the log file entries only for the most recent state (to load into vim's quickfix). How can I show all lines after the last occurrence of the string STARTING SESSION?

My current solution (log files are sometimes gigabytes long, so I never look at more than the last 5000 lines):

tail -n5000 logfile.log | grep -B5000 -v -e 'STARTING SESSION'> shortened.log

This works well when sessions produce a lot of logging, but if I have shorter logs with many restarts, it includes multiple sessions.

Essentially, I want something like a --reverse flag that would make grep search from the end of the file instead of the start:

grep --reverse --after-context=5000 --max-count=1 'STARTING SESSION' logfile.log

Notes:

The question is similar to Print line after nth occurrence of a match, but I want the last occurrence.

The problem is almost the same as Getting text from last marker to EOF in POSIX.2 except that I don't have a POSIX requirement and my files are large. I'd prefer efficient solutions with GNU utils (I'm using mingw64).

1

2 Answers 2

18

Reverse the file, display it until the first occurrence, and reverse the output again:

tac logfile.log | sed '/STARTING SESSION/q' | tac

tac is efficient when given a regular (seekable) file to process, and since sed exits as soon as it sees the start line, the whole pipeline will only process the end of the log file as far as necessary (rounded up to tac’s, sed’s, and the kernel’s buffer sizes). This should scale well to large files.

tac is a GNU utility. On non-GNU systems, you can often use tail -r to do the same.

If the log file doesn’t have a “STARTING SESSION” line at all, this won’t produce the same behaviour as your grep: it will output the complete log file. To avoid this, a variant of Kusalananda’s approach can be used instead:

tac logfile.log | sed -n '/STARTING SESSION/{H;x;p;q;};H' | tail -n +2 | tac

The sed expression looks for “STARTING SESSION”, and when matched, append the current line to the hold space, swaps the hold space with the pattern space, outputs it and exits; any other line is appended to the hold space. tail -n +2 is used to skip the first blank line (appending the pattern space to the hold space adds a leading newline).

0
5

Using sed without tac:

sed \
    -e '/STARTING SESSION/h' \
    -e '//,$ { //!H; }' \
    -e '$!d' \
    -e x logfile.log

or, using ; between expressions on a single line,

sed '/STARTING SESSION/h; //,$ { //!H; }; $!d; x' logfile.log

Annotated variant:

# If this line matches our trigger, save buffer in hold-space (overwrites).
/STARTING SESSION/ h

# In the range from the trigger to the end, append buffer to hold-space,
# but only if the current line isn't the trigger.
# (// re-uses the most recent expression)
//,$ { //!H; }

# If we're not at the end, restart with the next line without outputting anything.
$! d

# At the end, swap the hold-space into the buffer.
x

# (buffer is implicitly printed)

Summary: This sed script saves all lines between the trigger and the end of the document in the hold-space of sed. Whenever the trigger is found, the hold-space is cleared. At the end, the hold-space is outputted.

If the trigger is not found, no output will be produced.

Note too that this will by necessity read through the entire file.


A similar approach with awk:

awk '
    /STARTING SESSION/ { delete hold; i = 1 } 
    i { hold[i++] = $0  } 
    END { for (j = 1; j < i; j++) print hold[j] }' logfile.log

Here we'll start collecting data in the hold array once we have found our trigger (when we set i to one the first time). We delete the collected data and reset i down to one each time we hit the trigger.

At the end, all collected lines are outputted.

The delete hold statement is not strictly needed.

3
  • 2
    It might be worth pointing out that both these approaches involve reading the complete log file, which might be a problem given that “log files are sometimes gigabytes long”. Commented Aug 25, 2021 at 9:40
  • @StephenKitt Ah, correct. I presume that tac is smarter and does some sort of chunking of the input.
    – Kusalananda
    Commented Aug 25, 2021 at 10:12
  • 1
    Yes, if it can seek in its input, it reads from the end, 8KiB at a time. Commented Aug 25, 2021 at 10:16

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .