1

Current logs:

18:56:54 Info: Starting
18:56:55 Error: timed out
18:56:56 Error: timed out
18:56:57 Error: timed out
18:56:58 Info: reconnected
18:56:59 Error: timed out

Desired output:

18:56:54 Info: Starting
18:56:55 Error: timed out (3)
18:56:57 Info: reconnected
18:56:58 Error: timed out

I have log files that can have thousands of repeated lines, I want to copy the behaviour of chrome logs using bash/linux commands.

I found this, which is close: Remove partial duplicates consecutive lines but keep first and last

It gives this magic awk command:

awk '{n=$2$3$4$5$6$7}l1!=n{if(p)print l0; print; p=0}l1==n{p=1}{l0=$0; l1=n}END{print}' file

(Crucially having n=$1 excluded allows the timestamp to be different, which is needed. Exact timestamp shown for compressed lines isn't important.)

But I need a counter added as well, so I have clear idea of what was eliminated, giving a decent compromise between readability and accuracy (the only info lost will be the exact timing of repeated messages, having the first or last timestamp is enough.)

Thanks, I'm bad at awk and just learned about uniq, hopefully someone can link me to a solution, or sees this as a fun exercise. Cheers.

5
  • Are you logging directly to a file from you application, or via syslog (or some systemd equivalence, if there is one)? Logging facilities often have options for supressing consecutive duplicate output lines.
    – Kusalananda
    May 7 at 22:02
  • 1
    You mention something about timestamps partway through your question, but your examples show no timestamps. It would be extremely helpful you you were able to show an actual example, or you would receive answers that work with your example data, but not with your real data.
    – Kusalananda
    May 7 at 22:08
  • @Kusalananda Thanks, edited. If the answer works for the general question and not mine i'll just modify it, my philosophy is that I might use this answer once, but if it's clear and simple, thousands of people might find this from google in the future, and that's how I give back. But timestamps is a common problem people will have, so probably worth including. Re: syslog, this is just written to a file. Any way I can steal this functionality from an existing program is appreciated, syslog is used on my system. May 7 at 22:23
  • @ParanoidAltoidto be clear, the answer from the question you quoted doesn't work in general, it only works if the fields in each column have a fixed width, and the answer to handle duplicate lines is not the same as the answer to handle lines with some duplicate fields. So saying "If the answer works for the general question and not mine i'll just modify it` is like saying "If I get the answer for how to change the wheel on a vehicle I'll just apply that to my bike" - you can't turn your car upside down to replace it's wheel so the "general" answer may not be the best starting point for you.
    – Ed Morton
    May 8 at 12:43
  • Do you care which timestamp you get output for each unique string (first or last or any)?
    – Ed Morton
    May 8 at 12:46
6

No need for awk, just use uniq directly,

uniq -c -f 1 file

The -c option gives the count for the number of times a line was found consecutively in the input, and you can skip the timestamp in the first space or tab-delimited field with -f 1.

Example given the data in the question:

$ uniq -c -f 1 file
   1 18:56:54 Info: Starting
   3 18:56:55 Error: timed out
   1 18:56:58 Info: reconnected
   1 18:56:59 Error: timed out
3
  • Awesome, that's super easy. I've updated the question to include timestamps though, which uniq doesn't automatically account for. I'll start thinking of a way to make uniq ignore timestamps. Possibly just trim them, pipe them into uniq, then add the timestamps back in somehow. May 7 at 22:27
  • 1
    @ParanoidAltoid at least with the GNU Coreutils implementation of uniq, you can ignore the timestamps using the -f, --skip-fields=N parameter ex. uniq -f1 -c logfile May 8 at 1:11
  • @steeldriver -f is a standard option to uniq.
    – Kusalananda
    May 8 at 20:32
2

If you're happy with the count at the beginning of the line, uniq is all you need:

$ uniq -f 1 -c log.txt 
      1 18:56:54 Info: Starting
      3 18:56:55 Error: timed out
      1 18:56:58 Info: reconnected
      1 18:56:59 Error: timed outhet

-f 1 skips the first field, -c counts the lines.

To get the exact format you asked for, you need to process uniq's output, e.g. with sed:

$ uniq -f1 -c log.txt | 
    sed -E -e 's/^[[:space:]]+//;
               s/^([[:digit:]]+)[[:space:]]+(.*)/\2 (\1)/;
               s/ \(1\)$//' 
18:56:54 Info: Starting
18:56:55 Error: timed out (3)
18:56:58 Info: reconnected
18:56:59 Error: timed out

The sed script uses Extended Regular Expressions (-E option) aka ERE instead of the default Basic Regular Expressions (BRE) and:

  • strips leading spaces from each line
  • moves all digits at the start of each line to the end of the line, surrounded by parentheses
  • removes " (1)" from the end of each line if it's there.
1

Using the awk utility we can arrive at the unique count of events as follows:

$ awk '
{t=substr($0,1+index($0,FS))}
prev != t {
  if (NR>1) {fx(line,knt)}
  knt = 0
  prev = t
}{
  line = $0
  knt++
}
END {fx(line,knt)}
function fx(a,b) {
  print a (b>1?sprintf("%s(%d)",OFS,b):"")
}
' file
18:56:54 Info: Starting
18:56:57 Error: timed out (3)
18:56:58 Info: reconnected
18:56:59 Error: timed out

Notes:-

  • Assuming no leading whitespace/no consecutive whitespace.
1

command

uniq -f 1 -c filename| awk '{print $2,$3,$4,$5,$1}'| awk '{if($NF > 1){$NF="("$NF")";print }else{$NF="";print }}'

output

18:56:54 Info: Starting 
18:56:55 Error: timed out (3)
18:56:58 Info: reconnected 
18:56:59 Error: timed out 

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.