2

I am fairly sure this has been asked before but I cannot locate an exact dupe.

My input looks like this:

Compiling File1
... commands ...

Compiling File2
... commands ...

Compiling File3
... commands ...
In file included from ...
In file included from ...
In file included from ...
error: could not find A

Compiling File4
... commands ...

Compiling File5
... commands ...
In file included from ...
In file included from ...
In file included from ...
error: could not find B

Expected output:

Compiling File3
... commands ...
In file included from ...
In file included from ...
In file included from ...
error: could not find A
---separator---
Compiling File5
... commands ...
In file included from ...
In file included from ...
In file included from ...
error: could not find B
---separator---

I would like a shell command to print, for every match of error:, from the closest preceding match of Compiling to the current line, i.e. the full context of the error message. All other files that succeeded compilation can be skipped.

I suppose I can achieve this easily with awk or sed by keeping a pattern space comprising all text since the last match "Compiling", but there can be thousands of lines without an error. Would it be very inefficient?

0

3 Answers 3

2

Regarding keeping a pattern space comprising all text since the last match "Compiling", but there can be thousands of lines without an error. Would it be very inefficient? - it probably wouldn't any less efficient than any alternative approach such as doing 2 passes of the input file to identify the matching pairs of delimiters before starting printing and it'd have the advantage that it will work whether the input is stored in a file or coming from a pipe.

Probably most efficient is just to use 2 calls to tac with awk in between if you're on a system that has tac:

$ tac file |
    awk '/^error:/{f=1; print "---separator---"} f; /^Compiling/{f=0}' |
        tac
Compiling File3
... commands ...
In file included from ...
In file included from ...
In file included from ...
error: could not find A
---separator---
Compiling File5
... commands ...
In file included from ...
In file included from ...
In file included from ...
error: could not find B
---separator---

Otherwise, just using any awk in any shell on every Unix box:

$ awk '
    /^Compiling/ { buf="" }
    { buf = buf $0 "\n" }
    /^error:/ { print buf "---separator---" }
' file
Compiling File3
... commands ...
In file included from ...
In file included from ...
In file included from ...
error: could not find A
---separator---
Compiling File5
... commands ...
In file included from ...
In file included from ...
In file included from ...
error: could not find B
---separator---

Alternatively, using GNU awk for multi-char RS and RT:

$ awk -v RS='\nerror:[^\n]+' -v ORS='\n---separator---\n' '
    sub(/(^|.*\n)Compiling/,"Compiling") { print $0 RT }
' file
Compiling File3
... commands ...
In file included from ...
In file included from ...
In file included from ...
error: could not find A
---separator---
Compiling File5
... commands ...
In file included from ...
In file included from ...
In file included from ...
error: could not find B
---separator---
5
  • I like the tac idea. I would definitely go install gtac if the other solutions turn out to be too inefficient when I put it against gigabytes of logs.
    – Alan Tam
    Sep 30, 2021 at 19:34
  • tac is not POSIX compliant, use sed instead. Oct 1, 2021 at 17:32
  • @Iyachtharwanambarek that would be vastly (i.e. several orders of magnitude) slower than any other method. If you don't have tac and want to reverse the order of lines in a file using mandatory POSIX tools then it's awk -v OFS='\t' '{print NR, $0}' | sort -k1,1rn | cut -f2-.
    – Ed Morton
    Oct 1, 2021 at 18:20
  • @EdMorton: Could you please add that to this question: unix.stackexchange.com/q/9356/493431 Oct 1, 2021 at 18:43
  • @Iyachtharwanambarek There's already 2 variations of it on there but I added a comment to one of them, unix.stackexchange.com/questions/9356/…, to include that specific implementation.
    – Ed Morton
    Oct 1, 2021 at 21:03
2

Using perl it is very easy as it has a paragraph mode with -00:

perl -00 -ne 'print if /\nerror:/' file

Output:

Compiling File3
... commands ...
In file included from ...
In file included from ...
In file included from ...
error: could not find A

Compiling File5
... commands ...
In file included from ...
In file included from ...
In file included from ...
error: could not find B

If you add | sed 's/^$/----separator----/', you can also add your own separator instead of empty lines if you need that.

2
  • I've never heard of -00. Is it using an empty line as input record separator? In that case it will work with my specific input, but not according to my problem description. It cuts the records apart when empty lines occur between Compiling and error:. But it would be an interesting solution if -00 can be instrumented to use "empty line before Compiling" as record separator.
    – Alan Tam
    Sep 30, 2021 at 19:17
  • @AlanTam awk has paragraph mode too (RS="") if you can't have empty lines within Compiling sections. It won't use any less memory than the buffering after Compiling solutions though. You can also set GNU awk (and I assume perl or whatever else) to use \n\nCompiling as the record separator if that was useful but again, it won't help with memory.
    – Ed Morton
    Sep 30, 2021 at 19:40
0

Using Raku (formerly known as Perl_6)

raku -e 'my @array; for slurp.split("\n\n") {@array.push($_)}; for @array {.put if /^Compiling .* \n error/};' 

Sample Input:

Compiling File1
... commands ...

Compiling File2
... commands ...

Compiling File3
... commands ...
In file included from ...
In file included from ...
In file included from ...
error: could not find A

Compiling File4
... commands ...

Compiling File5
... commands ...
In file included from ...
In file included from ...
In file included from ...
error: could not find B

Sample Output (1):

Compiling File3
... commands ...
In file included from ...
In file included from ...
In file included from ...
error: could not find A
Compiling File5
... commands ...
In file included from ...
In file included from ...
In file included from ...
error: could not find B

Briefly, "Compiling..." sections are broken on \n\n separators and each element pushed onto the @array (via the $_ 'topic' variable). Resulting elements of the @array are printed only if they start with Compiling... and have error... starting their last line.

It's no clear why the OP requests a ---separator--- line (since the starting- and ending- lines are fairly well-specified), however it's easy enough to add:

raku -e 'my @array; for slurp.split("\n\n") {@array.push($_)}; for @array {put($_,"\n---separator---") if /^Compiling .* \n error/};'

Sample Output (2) :

Compiling File3
... commands ...
In file included from ...
In file included from ...
In file included from ...
error: could not find A
---separator---
Compiling File5
... commands ...
In file included from ...
In file included from ...
In file included from ...
error: could not find B
---separator---

ADDENDUM: The OP mentions in a comment that memory efficiency is key. In Raku the lines routine is lazy, so here's a crude approach (each 'Compiling ... error' block returns on one line, for now):

raku -e 'for lines.split( "Compiling ") {say "ERROR Compiling "~$_ if m/error/};'

OR

raku -e 'say "ERROR Compiling $_" if m/error/ for lines.split( "Compiling ");' 

Sample Output (3):

ERROR Compiling File3 ... commands ... In file included from ... In file included from ... In file included from ... error: could not find A  
ERROR Compiling File5 ... commands ... In file included from ... In file included from ... In file included from ... error: could not find B

https://speakerdeck.com/util/reading-files-cant-be-this-simple
https://raku.org

2
  • Thanks for the intro to raku/perl6. I probably should have said this out loud (although it's mildly implied when I asked about memory efficiency), but I have in mind gigabytes of compiler logs, and slurping everything into memory sounds like a bad plan.
    – Alan Tam
    Sep 30, 2021 at 19:32
  • @AlanTam Edited! Oct 1, 2021 at 13:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.