I have been trying to figure out on how to sort through a log file (format pasted below) using linux commands for sometime and I just can't get it to work. The file has a setup as follows;

columnA | colB | colC | colD  
name_1| data | 1   | data
name_1| data | 2   | data
name_1| data | 3   | data
name_2| data | 1   | data
name_2| data | 2   | data

I would like to have a file of only one entry of each name in columnA based on the highest value in column C with the corresponding data from the other columns at the row with highest value in colC.

If anyone knows any commands or pipes that would be greatly appreciated.

Thank you

  • 6
    You will get a much more friendly reception and much better help here if you show what code you have tried so far and describe what problems you were having with it. Without code, your question looks like a request for free consulting and many people don't like that. – John1024 Mar 14 '18 at 22:25
  • Does the file have all that space,or did you add it for readability? – RonJohn Mar 15 '18 at 2:13
$ awk -F'|' '
  NR==1 { header=$0 };
  NR>1 && ($3 > colC[$1]) { colC[$1] = $3 ; line[$1] = $0};

  END {
    print header;
    for (i in line) { print line[i] }
  }' file.log 
columnA | colB | colC | colD
name_1| data | 3   | data
name_2| data | 2   | data

Using | as the field separator, this awk script saves the first line in variable header, then it uses the array colC to keep, for each value of column A ($1), the highest value seen in column C ($3). The array line is also used to store the entire matching input line ($0).

Finally, when all the input has been read and processed, it prints each saved line. The output will probably need to be sorted, as associative arrays are not stored in any given order - pipe the output to sort. Depending on the input file, the header line may end up being sorted into the middle of the output, or the end...if that's the case, it's probably better to not save or print it in the awk script but to print it in the shell script that runs the awk script.

NOTE: There are various ways to optimise this algorithm, I've chosen to optimise for readability and simplicity. Unless your input files are millions or billions of lines long, performance and memory consumption are unlikely to be a problem worth optimising for.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.