4

I have an application that outputs a set of log files to a central directory like this:

/tmp/experiment/log/    
├── node01.log
├── node02.log
├── node03.log
├── node04.log
├── node05.log
├── node06.log

Inside each file, different measures are taken during the lifetime of each log's process, so the lines look like this:

prop1=5, ts=X, node01
prop2=3, ts=X, node01
prop1=7, ts=Y, node01
...

I'm struggling to write some commands that can process all files and output the LAST reading of a giving property, ideally outputting something like this:

node01, prop1=7, ts=...
node02, prop1=9, ts=...
node03, prop1=3, ts=...

Any suggestions? I started using a combination of grep, cut, sort, uniq like this:

$ grep -sirh "prop1" /tmp/experiment/log/ | \
   cut --delimiter=, --fields=1,4 | uniq | sort | \
   tail -n 14`  --this example had 14 log files

but it only worked partially as in some experiments it would end up printing multiple records of the same log and exclude some other logs.

I moved on to awk with this:

$ awk -F":" '/prop1/ { print $NF $2}' /tmp/experiment/log/node*.log | \
   awk 'END { print }'

and had the problem that when I pass multiple input files, it only gives me the last line of the last log file instead of 1 output line per log file.

Any suggestions on how to accomplish this?

1
  • regarding your awk , i wouldn't see any : in the input file. can you share the exact context of node01.log and expected result.
    – Siva
    Commented Jul 31, 2018 at 11:32

2 Answers 2

8

Take a look at ENDFILE blocks (GNU awk specific). You could run something along the lines of

awk     'BEGINFILE { a = ""}
         /prop1/   { a=$NF $2 $1}    ## Change this if necessary
         ENDFILE   { if (a != "") print FILENAME, a}' ./node*.log
2
  • 1
    And there are 3 fields in PO's expected result, but we printing only 2.
    – Siva
    Commented Jul 31, 2018 at 12:00
  • 2
    This works like a charm! Thank you very much. I was trying to use some begin/end code to make it happen but I think that applied to the entire awk program, where I really wanted it to apply to each file separately.
    – Dash83
    Commented Jul 31, 2018 at 12:11
2

With GNU's grep & sed you could this do this as follows:

grep -zoPhr '(.*\n)+\Kprop1=[^\n]*' /tmp/experiment/log/ | sed 's/\(.*\),\s\(.*\)/\2, \1/'

Explanation:

  • -z option makes grep treat the file as one long string, delimited by \0.
  • -r option will make grep recursive
  • -P option turns on the Perl regex flavor.
  • -o option will select the matched portion.
  • -h option will suppress the printing of filenames.
1
  • Thank you for your answer, always good to have alternative ways to do things. I gotta say though, holy shit, that's difficult to read!
    – Dash83
    Commented Jul 31, 2018 at 14:52

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .