Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It only takes a minute to sign up.
Sign up to join this community
Here are commands on a random file from pastebin:
wget -qO - http://pastebin.com/0cSPs9LR | wc -l
350
wget -qO - http://pastebin.com/0cSPs9LR | sort -u | wc -l
287
wget -qO - http://pastebin.com/0cSPs9LR | sort | uniq | wc -l
287
wget -qO - http://pastebin.com/0cSPs9LR | sort | uniq -u | wc -l
258
The man pages are not clear on what the -u flag is doing. Any advice?
uniq with -u skips any lines that have duplicates. Thus:
$ printf "%s\n" 1 1 2 3 | uniq
1
2
3
$ printf "%s\n" 1 1 2 3 | uniq -u
2
3
Usually, uniq prints lines at most once (assuming sorted input). This option actually prints lines which are truly unique (having not appeared again).
uniq could be called distinct, since it prints all distinct lines, whereas uniq -u prints all unique lines.
Jun 18, 2015 at 12:41
uniq, without -u, makes every line of the output unique.uniq -u only prints every unique line from the input.uniq is for dealing with files that have lines duplicated, and only when those lines appear successively in the input. So, for its purposes, a unique line is one that is not duplicated immediately.
(uniq has a very limited short-term memory; it will never remember whether a line appeared earlier in the input, unless it was the immediately previous line -- this is why uniq is very often paired with sort.)
When it encounters a run of duplicate lines, uniq, without the -u arg, prints one copy of that line. (It makes every line of the output unique).
With the -u argument, it prints zero copies of that line -- runs of duplicates just get omitted from the output.
sort -u instead of sort | uniq.
uniq does normalisation and collation, yes. But even then it's only a local consideration -- you know where in the sorted output the line will appear, and just have to select which of several adjacent lines to keep. If the input isn't sorted then the decision affects the whole operation of uniqifying, for example if you're going to keep the last duplicate then you can't output anything until you've read the last line of the input...
Jun 19, 2015 at 15:00
uniq POSIX spec described it clearly:
-u
Suppress the writing of lines that are repeated in the input.
-u option make uniq not to print repeated lines.
Most uniq implementations used bytes comparison, while GNU uniq used collation order to filter duplicated lines. So it can produce wrong result in some locales, example in en_US.UTF-8 locale:
$ printf '%b\n' '\U2460' '\U2461' | uniq
①
and -u gave you no lines:
$ printf '%b\n' '\U2460' '\U2461' | uniq -u
<blank>
So you should set locale to C to get bytes comparison:
$ printf '%b\n' '\U2460' '\U2461' | LC_ALL=C uniq
①
②
uniq (though apparently the intent of POSIX was that it should do byte comparison instead of strcoll() comparison as in sort -u) as those locales that erroneously have ① sorting the same as ②. At least GNU uniq is consistent with sort -u.
Jun 18, 2015 at 16:27
uniq required to do memcmp/strcmp as opposed to strcoll, that is not very apparent to me but that was to Geoff. About the GNU locales having ① sorting the same as ②, that's clearly a bug as there's no reason why they should sort the same. That's allowed by POSIX but there's some change coming.
Jun 20, 2015 at 7:29
normal:
echo "a b a b c c c" | tr ' ' '\n'
a
b
a
b
c
c
c
uniq : no two subsequent repeating lines
echo "a b a b c c c" | tr ' ' '\n' | uniq
a
b
a
b
c
sorted
echo "a b a b c c c" | tr ' ' '\n' | sort
a
a
b
b
c
c
c
sort -u : no two repeating lines
echo "a b a b c c c" | tr ' ' '\n' | sort -u
a
b
c
sort / uniq: all distinct
echo "a b a b c c c" | tr ' ' '\n' | sort | uniq
a
b
c
counts distinct occurrences
echo "a b a b c c c" | tr ' ' '\n' | sort | uniq -c
2 a
2 b
3 c
only lines which are not repeated (not sorted first)
echo "a b a b c c c" | tr ' ' '\n' | uniq -u
a
b
a
b
only lines which are not repeated (after sorting)
echo "a b a b c c c Z" | tr ' ' '\n' | sort | uniq -u
Z
uniq -d : only print duplicate lines, one for each group
echo "a b a b c c c" | tr ' ' '\n' | uniq -d
c
.. counted
echo "a b a b c c c" | tr ' ' '\n' | uniq -dc
3 c
