2

I've got a file of ~1-2 million lines that I'm trying to reduce down by counting duplicate groups of lines, preserving order.

uniq -c works okay : 

$ perl -E 'say for (("foo") x 4, ("bar") x 4, "baz", ("foo", "bar", "baz") x 4)' | uniq -c
      4 foo
      4 bar
      1 baz
      1 foo
      1 bar
      1 baz
      1 foo
      1 bar
      1 baz
      1 foo
      1 bar
      1 baz
      1 foo
      1 bar
      1 baz

In my use-case (but not in the following foo-bar-baz example), counting pairs of lines is ~20% more efficient, and looks like : 

$ perl -E 'say for (("foo") x 4, ("bar") x 4, "baz", ("foo", "bar", "baz") x 4)' \
  | sed 's/^/__STARTOFSTRINGDELIMITER__/' \
  | paste - - \
  | uniq -c \
  | sed -r 's/__STARTOFSTRINGDELIMITER__//; s/__STARTOFSTRINGDELIMITER__/\n\t/;'
      2 foo
        foo
      2 bar
        bar
      1 baz
        foo
      1 bar
        baz
      1 foo
        bar
      1 baz
        foo
      1 bar
        baz
      1 foo
        bar
      1 baz

(That format is acceptable to me.)

How can I reduce duplicate groups of arbitrary numbers of lines (well, keeping a sane buffer count like 2-10 lines) down to a single copy + count of them ?

Following the above example, I would want output similar to : 

4 foo
4 bar
1 baz
4 foo
  bar
  baz
Improve this question
3
  • That's similar to what some compression algorithms do. Maybe some avenue worth exploring.
    – Stéphane Chazelas
    Mar 28, 2019 at 18:32
  • The issue seems to be finding the groups of lines. Your output may as well say that the combination of foo followed by bar occur 5 times.
    – Kusalananda ♦
    Mar 29, 2019 at 6:34
  • @Kusalananda Do you mean foo followed by bar 4 times ? (The first two sets of four each). You would be correct then, yes, and either output would be acceptable for me (either foo x4 then bar x4, or (foo, bar) x4). I assume it would depend on the buffer length - 10 lines of buffer would produce the latter, less than 8 lines of buffer would produce the former. It's not really an issue as you say, just a consideration.
    – robut
    Mar 29, 2019 at 12:25

1 Answer 1

Reset to default
0

I don't have such a huge dataset for benchmarking. Give this a try:

$ perl -E 'say for (("foo") x 4, ("bar") x 4, "baz", ("foo", "bar", "baz") x 4)' | awk 'NR == 1 {word=$0; count=1; next} $0 != word {print count,word; word=$0; count=1; next} { count++ } END { print count,word }'
4 foo
4 bar
1 baz
1 foo
1 bar
1 baz
1 foo
1 bar
1 baz
1 foo
1 bar
1 baz
1 foo
1 bar
1 baz

Using mawk instead of awk may improve performance.

Improve this answer
2
  • Can this be adapted to work with multi-word lines ? echo -e 'a b c \n a b c \n a b c' | awk 'NR == 1 {word=$1; count=1; next} $1 != word {print count,word; word=$1; count=1; next} { count++ } END { print count,word }' for example only counts a. Sorry that it wasn't clear in my original question that my actual lines are multi-word with non-alphanumeric characters too.
    – robut
    Mar 29, 2019 at 13:02
  • Just replace the $1 with $0 to compare whole lines. I've edited my answer.
    – finswimmer
    Mar 29, 2019 at 15:22

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .