sort command will probably be the fastest option.
But you'll probably want to fix the locale to C.
sort -u doesn't report unique lines, but one of each set of lines that sort the same. In the C locale, 2 different lines necessarily don't sort the same, but that's not the case in most UTF-8 based locales on GNU systems.
Also, using the C locale avoids the overhead of having to parse UTF-8 and processing complex sort orders so would improve performance dramatically.
LC_ALL=C sort -u file
You can also improve performance by using a faster drive (or a different drive from the one where the input and/or output files are) for the temporary files (using
$TMPDIR environment variable), or by fiddling with the
-S option supported by some
For some type of input or for slow storage, using the
--compress-program option of GNU
sort (for instance with
lzop) might improve performance in addition to storage usage.
Now just a note for those objecting (rightly to some extent) that it will not be the correct order:
I agree that as a human, I'd like to see Stéphane sort in between Stefan and Stephanie, but:
So IMO, chances are you always want
sort -u with LC_ALL=C, if you want unique lines. And if you want that resulting list to be sorted in the user's sort order, pipe it to
LC_ALL=C sort -u | sort
LC_ALL=C sort | LC_ALL=C uniq -c | sort -k2
¹ 2019 edit. the order of ① ② ③ ④ ⑤... has since been fixed in newer versions of the GNU libc, but as of 2.30, over 95% of characters still don't have a defined order, you can replace ① ② ③ ④ ⑤ with 🧙 🧚 🧛 🧜 🧝 for instance. Hopefully, GNU locales will eventually be fixed completely (they will have to if they want to comply to the next revision of the standard) and the problem will then be limited to user-defined locales