6

I have a log file that looks like this,

Another thousand lines above this
I 10/03/15 12:04AM 42 [Important] 4th to last
I 10/03/15 04:31AM 42 (534642712) [1974,2106,258605080,0,0,32817,30711]
I 10/03/15 04:33AM 42 (2966927) [91,0,2966927,0,0,291,291]
I 10/03/15 04:52AM 42 (3026559) [93,0,3026559,0,0,314,314]
I 10/03/15 04:55AM 42 (3065494) [94,0,3065494,0,0,301,301]
I 10/03/15 05:04AM 42 [Important] 3rd to last
I 10/04/15 12:04AM 42 [Important] 2nd to last occurence
I 10/04/15 04:31AM 42  (7,30711]55
I 10/04/15 04:33AM 42 dfsadfs,0,0,291,291]
I 10/04/15 04:52AM 42 (30,0,314,314]
I 10/04/15 04:55AM 42 (30,301]
I 10/04/15 05:04AM 42 [Important] - last occurence

The only pattern that remains the same throughout this file is [Important], everything else changes, including the number of lines between each occurrence of [Important]

I'm trying to take the end of this file, ignore the last occurrence and find the 2nd to last, then extract the remaining contents of the file into another.

This is what I have been trying but haven't been able to single out the 2nd to last occurrence with tac. What I'm trying..

<logfile tac | sed '/Important/q' | tac >  output_file

This is what the output should look like..

I 10/04/15 12:04AM 42 [Important] 2nd to last occurence
I 10/04/15 04:31AM 42  (7,30711]55
I 10/04/15 04:33AM 42 dfsadfs,0,0,291,291]
I 10/04/15 04:52AM 42 (30,0,314,314]
I 10/04/15 04:55AM 42 (30,301]
I 10/04/15 05:04AM 42 [Important] - last occurence
  • Depending on how important performance is and how often you are going to do this, you could also use vi. Open up the log and type: :1,$?Important??Important?-1d, then :w outputfile, then :q! – Wildcard Oct 4 '15 at 22:20
  • Or just :$?Important??Important?,$w outputfile and then :q. – Wildcard Oct 4 '15 at 22:22
  • @don_crissti Either would work with what I want done. They both capture exactly what I need. – Pi4All Oct 5 '15 at 1:41
6

Find all lines with "Important", pick the last two, take the line numbers, print the range:

sed -n `grep -n Important log | tail -n 2 | cut -d : -f 1 | tr '\n' ',' | sed -e 's#,$#p#'` log

Outputs as expected:

I 10/04/15 12:04AM 42 [Important] 2nd to last occurence
I 10/04/15 04:31AM 42  (7,30711]55
I 10/04/15 04:33AM 42 dfsadfs,0,0,291,291]
I 10/04/15 04:52AM 42 (30,0,314,314]
I 10/04/15 04:55AM 42 (30,301]
I 10/04/15 05:04AM 42 [Important] - last occurence

As a script:

#!/bin/bash
lines=`grep -n Important log | tail -n 2 | cut -d : -f 1`
range=`echo "${lines}" | tr '\n' ',' | sed -e 's#,$#p#'`
sed -n "${range}" log
  • Oooh, I like this one. set -- $(sed -n /Important/=; tail -2) $file; sed -n $1,$2p $file On a big file two-pass might be a problem. – jthill Oct 4 '15 at 21:42
  • @jthill - you mean with a big file single-pass might be a problem ? – don_crissti Oct 4 '15 at 22:18
  • Wow. I'm claiming actual genetic defect. I can't seem to learn not to keyboard-to-editbox waaay too often. Naaah, I leave "exercises for the reader", yeah, that's it. . . . anyway, set -- $(sed -n /Important/= $file | tail -2); sed -n $1,$2p $file works. @don_crissti I'm thinking about pipeline use, adding an extra writeitall-rereaditall trip through a filesystem is generally worse than maybe having to buffer it all in the first place. But I see your point. If it gets really big, the filesystem cache might be more capacious and perform better than swap. – jthill Oct 5 '15 at 23:40
  • @jthill - it's going to be setup dependent I think but anyway, with a two-pass approach you could ditch the second sed and do something like set --.....; N=$(($1 -1)); { head -n $N >/dev/null; head -n $(($2 -$N)); } <$file That should be faster... – don_crissti Oct 6 '15 at 8:02
5
$ awk '/Important/{pen=s; s=$0;next} s{s=s"\n"$0} END{print pen "\n" s}' logfile
I 10/04/15 12:04AM 42 [Important] 2nd to last occurence
I 10/04/15 04:31AM 42  (7,30711]55
I 10/04/15 04:33AM 42 dfsadfs,0,0,291,291]
I 10/04/15 04:52AM 42 (30,0,314,314]
I 10/04/15 04:55AM 42 (30,301]
I 10/04/15 05:04AM 42 [Important] - last occurence

How it works

awk implicitly loops through all lines in the input file. After each appearance of Important, we save the lines in variable s. When we reach a new line with Important in it, the old set of important lines is transferred to the variable pen and we start saving the new lines in s.

pen has the penultimate (second to last) Important section. s has the ultimate (last) Important section. At the end, we print pen and s.

In more detail:

  • /Important/{pen=s; s=$0;next}

    If this line contains Important, then move the contents of variable s to pen, save the current line in s. Then, skip the rest of the commands and jump to the next line.

  • s{s=s"\n"$0}

    If we get here, then the current line does not contain Important.

    If s has been set to a value, then append the current line to it.

  • END{print pen "\n" s}

    After we reach the end of the file, print pen and s.

3

If ed is an option:

ed -s file <<EOF
1
?Important
?
;w output_file
Q
EOF
  • @don_crissti 1: go to beg of file; ?Important: go to previous occurrence of Important: e.g., to last occurrence. ?: repeat last search (so here we're at 2nd to last occurrence). Finally, ;w output_file write from current address till the end of the file to output_file. – gniourf_gniourf Oct 4 '15 at 20:33
1

If sed can just buffer the whole file (if you're on GNU/anything, it can),
(last edit: I've fixed multiple brainos here)

sed -En 'H;$!d
     g;s/.*[\n](.*Important.*\n.*Important[^\n]*).*/\1/p
'    

The H;$!d buffers ("holds") each line with a leading \n up to the end of the file. What follows the $!d runs only after the last line has been buffered. g gets the buffer.

To understand the regex, remember that regex's are leftmost-longest. A leading .* finds the last match for what follows. Since H unconditionally appends a \n to the front, the .*\n matches every complete line before two "Important"s separated by at least one newline and followed by everything up to any following line.

If there aren't two Important lines, nothing gets printed.

It might be at least aesthetically better to incrementally discard lines as you find they're unwanted

sed -En 'H
        /Important/    {x; s/.*[\n](.*Important.*\n.*Important[^\n]*)/\1/; H}
        $              {g; s/.*[\n](.*Important.*\n.*Important[^\n]*).*/\1/p }
'

The /Important/ match exchanges the pattern and hold buffers, keeps only the last interesting block, and puts the result back into the hold buffer.

I put the [\n] in brackets just to highlight it and visually match it with the trailing non-newline class, a one-character class can of course be written without the brackets.

  • This answer doesn't satisfy your constraints as stated, but I think what you're after is really "The last two "Important" lines and everything between them, i.e. only the last "Important" block, not that block and everything that follows. It's easy enough to fix up the regex's here if I got that wrong, change the [^\n]*s to .*. Because of leftmost-longest, that change will also leave nothing for any trailing .* to match. – jthill Oct 4 '15 at 21:36
  • To avoid buffering lines you know immediately won't be needed, if you've got GNU sed you can put in a leading '0,/Important/ //!d' The 0, range starter (treating the block as already started) is GNU. – jthill Oct 4 '15 at 21:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.