I have an application that outputs a set of log files to a central directory like this:

├── node01.log
├── node02.log
├── node03.log
├── node04.log
├── node05.log
├── node06.log

Inside each file, different measures are taken during the lifetime of each log's process, so the lines look like this:

prop1=5, ts=X, node01
prop2=3, ts=X, node01
prop1=7, ts=Y, node01

I'm struggling to write some commands that can process all files and output the LAST reading of a giving property, ideally outputting something like this:

node01, prop1=7, ts=...
node02, prop1=9, ts=...
node03, prop1=3, ts=...

Any suggestions? I started using a combination of grep, cut, sort, uniq like this:

$ grep -sirh "prop1" /tmp/experiment/log/ | \
   cut --delimiter=, --fields=1,4 | uniq | sort | \
   tail -n 14`  --this example had 14 log files

but it only worked partially as in some experiments it would end up printing multiple records of the same log and exclude some other logs.

I moved on to awk with this:

$ awk -F":" '/prop1/ { print $NF $2}' /tmp/experiment/log/node*.log | \
   awk 'END { print }'

and had the problem that when I pass multiple input files, it only gives me the last line of the last log file instead of 1 output line per log file.

Any suggestions on how to accomplish this?

  • regarding your awk , i wouldn't see any : in the input file. can you share the exact context of node01.log and expected result. – msp9011 Jul 31 '18 at 11:32

Take a look at ENDFILE blocks (GNU awk specific). You could run something along the lines of

awk     'BEGINFILE { a = ""}
         /prop1/   { a=$NF $2 $1}    ## Change this if necessary
         ENDFILE   { if (a != "") print FILENAME, a}' ./node*.log
  • 1
    And there are 3 fields in PO's expected result, but we printing only 2. – msp9011 Jul 31 '18 at 12:00
  • 2
    This works like a charm! Thank you very much. I was trying to use some begin/end code to make it happen but I think that applied to the entire awk program, where I really wanted it to apply to each file separately. – Dash83 Jul 31 '18 at 12:11

With GNU's grep & sed you could this do this as follows:

grep -zoPhr '(.*\n)+\Kprop1=[^\n]*' /tmp/experiment/log/ | sed 's/\(.*\),\s\(.*\)/\2, \1/'


  • -z option makes grep treat the file as one long string, delimited by \0.
  • -r option will make grep recursive
  • -P option turns on the Perl regex flavor.
  • -o option will select the matched portion.
  • -h option will suppress the printing of filenames.
  • Thank you for your answer, always good to have alternative ways to do things. I gotta say though, holy shit, that's difficult to read! – Dash83 Jul 31 '18 at 14:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.