sort command will probably be the fastest option.
But you'll probably want to fix the locale to C.
sort -u doesn't report unique lines, but one of each set of lines that sort the same. In the C locale, 2 different lines necessarily don't sort the same, but that's not the case in most UTF-8 based locales.
Also using the C locale avoid the overhead of having to parse UTF-8 and processing complex sort orders so would improve performance dramatically.
LC_ALL=C sort -u file
You can also improve performance by using a faster drive (or a different drive from the one where the input and/or output files are) for the temporary files (using
$TMPDIR environment variable), or by fiddling with the
-S option supported by some
For some type of input or for slow storage, using the
--compress-program option of GNU
sort (for instance with
lzop) might improve performance in addition to storage usage.
Now just a note for those objecting (rightly to some extent) that it will not be the correct order:
I agree that as a human, I'd like to see Stéphane sort in between Stefan and Stephanie, but:
So IMO, chances are you always want
sort -u with LC_ALL=C, if you want unique lines. And if you want that resulting list to be sorted in the user's sort order, pipe it to
LC_ALL=C sort -u | sort
LC_ALL=C sort | LC_ALL=C uniq -c | sort -k2