3

Running

printf "lol\nlol\nfoo\n\n\n\n\nbar\nlol\nlol\nfoo\nlol\nfoo" | uniq --unique

prints

foo
bar
foo
lol
foo

Why is foo printed three times? Shouldn't uniq --unique remove them?

Also, notably, it seems all duplicates of lol were removed. Why were lol duplicates removed, but not foo duplicates?

1
11

uniq requires the input to be sorted (from man uniq) if you want it to remove all duplicate lines:

DESCRIPTION

Filter adjacent matching lines from INPUT (or standard input), writing to OUTPUT (or standard output).

As you can see above, it only filters adjacent matching lines. This is why the lols were removed. So sort your data before passing to uniq:

$ printf "lol\nlol\nfoo\n\n\n\n\nbar\nlol\nlol\nfoo\nlol\nfoo" | sort | uniq 

bar
foo
lol

Or, with GNU sort, skip uniq:

$ printf "lol\nlol\nfoo\n\n\n\n\nbar\nlol\nlol\nfoo\nlol\nfoo" | sort --unique

bar
foo
lol

Finally, if you want to completely remove lines that were present more than once (instead of keeping one copy, the default behavior), use uniq -u or --unique as in your question:

$ printf "lol\nlol\nfoo\n\n\n\n\nbar\nlol\nlol\nfoo\nlol\nfoo" | sort | uniq -u
bar

In all cases, however, the sorting is necessary.

5
  • thanks, somehow missed the meaning of adjacent in the docs; fwiw i ended up using command | awk '!a[$0]++' instead (because i wanted the first of each unique line without sorting it, seems that's what that odd-loking awk invocation does per unix.stackexchange.com/questions/159695/how-does-awk-a0-work )
    – hanshenrik
    Jul 16 at 10:29
  • 2
    @hanshenrik you're welcome. And yes, that awk works fine for small datasets. However, since it keeps an array entry for every non-duplicate line, that can cause problems if you try it on a huge dataset where the array might fill up your RAM. More than fine for regular sized datasets though, absolutely.
    – terdon
    Jul 16 at 10:32
  • good point. if i need to do this on datasets so large that i don't want it all in ram, i guess... sqlite to the rescue? maybe (luckily im not in that situation now)
    – hanshenrik
    Jul 16 at 10:34
  • You can use sort | uniq on arbitrarily sized datasets. The only limit will be disk space, if I remember correctly, sort will create some temp files so that might fail if it fills up your disk, but apart from that you should be fine.
    – terdon
    Jul 16 at 10:37
  • strictly speaking, uniq doesn't require the input to be sorted...but it only removes consecutive duplicate lines. This can still be useful without sorting, e.g. if you want to reduce multiple blank lines between paragraphs down to just one blank line. As @terdon says, though, if you want to get rid of dupes anywhere in the input, you'll need to sort it so that the all become adjacent to each other.
    – cas
    Jul 16 at 11:52
4

Are you asking why foo is still in the list? uniq only removes adjacent repeated lines, "squeezing" the adjacent repeated lines into single lines. With -u (or --unique with GNU uniq), it also removes lines that had adjacent repeated lines.

In you example, none of the three lines saying foo were ever adjacent to any other line saying foo. This is why they are outputted.

The lol lines that were removed were removed because they were adjacent to another line saying lol. The last lol line was not duplicated in this sense, so it was kept.

1
  • thanks, somehow missed the meaning of adjacent in the docs, my bad
    – hanshenrik
    Jul 16 at 10:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.