2

I've got a file of ~1-2 million lines that I'm trying to reduce down by counting duplicate groups of lines, preserving order.

uniq -c works okay :

$ perl -E 'say for (("foo") x 4, ("bar") x 4, "baz", ("foo", "bar", "baz") x 4)' | uniq -c
      4 foo
      4 bar
      1 baz
      1 foo
      1 bar
      1 baz
      1 foo
      1 bar
      1 baz
      1 foo
      1 bar
      1 baz
      1 foo
      1 bar
      1 baz

In my use-case (but not in the following foo-bar-baz example), counting pairs of lines is ~20% more efficient, and looks like :

$ perl -E 'say for (("foo") x 4, ("bar") x 4, "baz", ("foo", "bar", "baz") x 4)' \
  | sed 's/^/__STARTOFSTRINGDELIMITER__/' \
  | paste - - \
  | uniq -c \
  | sed -r 's/__STARTOFSTRINGDELIMITER__//; s/__STARTOFSTRINGDELIMITER__/\n\t/;'
      2 foo
        foo
      2 bar
        bar
      1 baz
        foo
      1 bar
        baz
      1 foo
        bar
      1 baz
        foo
      1 bar
        baz
      1 foo
        bar
      1 baz

(That format is acceptable to me.)

How can I reduce duplicate groups of arbitrary numbers of lines (well, keeping a sane buffer count like 2-10 lines) down to a single copy + count of them ?

Following the above example, I would want output similar to :

4 foo
4 bar
1 baz
4 foo
  bar
  baz
  • That's similar to what some compression algorithms do. Maybe some avenue worth exploring. – Stéphane Chazelas Mar 28 at 18:32
  • The issue seems to be finding the groups of lines. Your output may as well say that the combination of foo followed by bar occur 5 times. – Kusalananda Mar 29 at 6:34
  • @Kusalananda Do you mean foo followed by bar 4 times ? (The first two sets of four each). You would be correct then, yes, and either output would be acceptable for me (either foo x4 then bar x4, or (foo, bar) x4). I assume it would depend on the buffer length - 10 lines of buffer would produce the latter, less than 8 lines of buffer would produce the former. It's not really an issue as you say, just a consideration. – robut Mar 29 at 12:25
0

I don't have such a huge dataset for benchmarking. Give this a try:

$ perl -E 'say for (("foo") x 4, ("bar") x 4, "baz", ("foo", "bar", "baz") x 4)' | awk 'NR == 1 {word=$0; count=1; next} $0 != word {print count,word; word=$0; count=1; next} { count++ } END { print count,word }'
4 foo
4 bar
1 baz
1 foo
1 bar
1 baz
1 foo
1 bar
1 baz
1 foo
1 bar
1 baz
1 foo
1 bar
1 baz

Using mawk instead of awk may improve performance.

  • Can this be adapted to work with multi-word lines ? echo -e 'a b c \n a b c \n a b c' | awk 'NR == 1 {word=$1; count=1; next} $1 != word {print count,word; word=$1; count=1; next} { count++ } END { print count,word }' for example only counts a. Sorry that it wasn't clear in my original question that my actual lines are multi-word with non-alphanumeric characters too. – robut Mar 29 at 13:02
  • Just replace the $1 with $0 to compare whole lines. I've edited my answer. – finswimmer Mar 29 at 15:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.