Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It only takes a minute to sign up.
Sign up to join this community
Running
printf "lol\nlol\nfoo\n\n\n\n\nbar\nlol\nlol\nfoo\nlol\nfoo" | uniq --unique
prints
foo
bar
foo
lol
foo
Why is foo printed three times? Shouldn't uniq --unique remove them?
Also, notably, it seems all duplicates of lol were removed. Why were lol duplicates removed, but not foo duplicates?
uniq requires the input to be sorted (from man uniq) if you want it to remove all duplicate lines:
DESCRIPTION
Filter adjacent matching lines from INPUT (or standard input), writing to OUTPUT (or standard output).
As you can see above, it only filters adjacent matching lines. This is why the lols were removed. So sort your data before passing to uniq:
$ printf "lol\nlol\nfoo\n\n\n\n\nbar\nlol\nlol\nfoo\nlol\nfoo" | sort | uniq
bar
foo
lol
Or, with GNU sort, skip uniq:
$ printf "lol\nlol\nfoo\n\n\n\n\nbar\nlol\nlol\nfoo\nlol\nfoo" | sort --unique
bar
foo
lol
Finally, if you want to completely remove lines that were present more than once (instead of keeping one copy, the default behavior), use uniq -u or --unique as in your question:
$ printf "lol\nlol\nfoo\n\n\n\n\nbar\nlol\nlol\nfoo\nlol\nfoo" | sort | uniq -u
bar
In all cases, however, the sorting is necessary.
adjacent in the docs; fwiw i ended up using command | awk '!a[$0]++' instead (because i wanted the first of each unique line without sorting it, seems that's what that odd-loking awk invocation does per unix.stackexchange.com/questions/159695/how-does-awk-a0-work )
Jul 16, 2021 at 10:29
awk works fine for small datasets. However, since it keeps an array entry for every non-duplicate line, that can cause problems if you try it on a huge dataset where the array might fill up your RAM. More than fine for regular sized datasets though, absolutely.
sort | uniq on arbitrarily sized datasets. The only limit will be disk space, if I remember correctly, sort will create some temp files so that might fail if it fills up your disk, but apart from that you should be fine.
uniq doesn't require the input to be sorted...but it only removes consecutive duplicate lines. This can still be useful without sorting, e.g. if you want to reduce multiple blank lines between paragraphs down to just one blank line. As @terdon says, though, if you want to get rid of dupes anywhere in the input, you'll need to sort it so that the all become adjacent to each other.
Are you asking why foo is still in the list? uniq only removes adjacent repeated lines, "squeezing" the adjacent repeated lines into single lines. With -u (or --unique with GNU uniq), it also removes lines that had adjacent repeated lines.
In you example, none of the three lines saying foo were ever adjacent to any other line saying foo. This is why they are outputted.
The lol lines that were removed were removed because they were adjacent to another line saying lol. The last lol line was not duplicated in this sense, so it was kept.
adjacent in the docs, my bad
Jul 16, 2021 at 10:31