That can be just:
find . -type f -size +1c -exec sort -uo {} {} ';'
(here skipping the files that are less than 2 byte large as you need at least 3 bytes to make two different lines, or possibly 2 for "\nx"
which is an empty line followed by an undelimited line¹).
Beware the default sort order in sort
is based on the locale's collation algorithm.
Two lines can sort the same even if they're not byte-to-byte identical, especially if the lines contain sequences of bytes that don't form valid characters, and also on GNU systems such as Debian on characters whose collation order is not defined.
You can do:
LC_ALL=C find . -type f -exec sort -uo {} {} ';'
Instead which on ASCII based systems (such as Debian on all architectures and kernels it's available for) will sort the lines by byte value instead of locale collation order (or IOW, the collation order of the C locale is based on byte value) and should guarantee that two byte-to-byte-different lines don't sort the same.
That runs one invocation of sort
per file. To speed things up if the files are reasonably short, you could instead do with zsh
:
zmodload zsh/mapfile
for f (**/*(N.)) print -rC1 -v 'mapfile[$f]' - ${(fou)mapfile[$file]}
Which avoids running an external sort
command so many times and uses its o
and u
parameter expansion flag to sort and unique the lines instead. Note that it strips empty lines in the input if any, and skips hidden files (add the D
glob qualifier if you want them).
Contrary to GNU sort -u
, zsh
won't consider as duplicate two strings that are not byte-to-byte identical (even if they sort the same), so you don't need to fix the locale to C there.
$ locale title charmap
English locale for Britain
UTF-8
$ a=(🧚 🧛)
$ print -rC1 - $a | sort -u
🧚 (oops, the vampire vanished as it sorted the same as the fairy)
$ print -rC1 - ${(ou)a}
🧚
🧛
As to your question about uniq -u
, that's what it's for, uniq -u
reports lines that are unique in the input. To remove duplicates, it's just sort | uniq
. The complement of uniq -u
would be uniq -D
(report all duplicated lines).
GNU uniq
used to report the first of sequences of lines that sort the same (so sort -u
would be the same as sort | uniq
). Newer versions report the first of sequences of identical lines, so sort | uniq
cannot be used anymore if there can be different lines that sort the same.
Here, with newer GNU versions:
$ print -rC1 🧚 🧛 🧛 🧚 🧛 | sort | uniq
🧚
🧛
🧚
🧛
Since fairies and vampires sort the same in my locale, the sort
result will be left as in the input (as GNU sort
uses a stable sort algorithm), and uniq
will only remove duplicates if the input happened to original contain adjacent duplicates.
What we can do though is first sort
by byte value, and then by collation order before calling uniq
:
LC_ALL=C sort | sort | LC_ALL=C uniq
Calling uniq
under LC_ALL=C
will make older and newer versions work the same.
sort
does add back that newline character if it was missing in the input, in effect making a text file out of that non-text file. By skipping the 1-byte size files that contain non-newline bytes, that means those files are not fixed, so you may want to skip that optimisation if you still want the benefit of that text-file fixing, or extend the optimisation to -size +2c
if you know all your files are properly formed text files.