2

I'm grepping a log file with the following command:

grep "System has completed" my_log.log

and getting something like

2019-12-07 17:03:09.527   System has completed 0 of 15778 files
2019-12-07 17:03:20.936   System has completed 4 of 15778 files
2019-12-07 17:03:32.381   System has completed 5 of 15778 files
2019-12-07 17:03:44.053   System has completed 5 of 15778 files
2019-12-07 17:03:55.753   System has completed 21 of 15778 files
2019-12-07 17:04:07.252   System has completed 22 of 15778 files
2019-12-07 17:04:18.728   System has completed 28 of 15778 files
2019-12-07 17:04:30.181   System has completed 28 of 15778 files
2019-12-07 17:04:41.627   System has completed 28 of 15778 files

I want to further process these results to cut out the lines with repeated numbers of completed files so that the output is

2019-12-07 17:03:09.527   System has completed 0 of 15778 files
2019-12-07 17:03:20.936   System has completed 4 of 15778 files
2019-12-07 17:03:32.381   System has completed 5 of 15778 files
2019-12-07 17:03:55.753   System has completed 21 of 15778 files
2019-12-07 17:04:07.252   System has completed 22 of 15778 files
2019-12-07 17:04:18.728   System has completed 28 of 15778 files

Keeping only the first line when multiple lines repeat the same number. Simply filtering all unique lines is not possible because of the timestamps. What is the best way to do this?

1

Assuming the number will always be in the same position you can use sort:

grep "System has completed" my_log.log | sort -unk6,6

Or uniq:

grep "System has completed" my_log.log | uniq -f2

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.