What sed/awk command can I use? Just
sort -u will remove all instances
abc abc def abc abc def
abc def abc def
That's what the
uniq standard command is for.
Note that some
uniq implementations like GNU
uniq will give you the first of a sequence of lines that sort the same (where
strcoll() returns 0) as opposed to are byte-to-byte identical (where
strcmp() returns 0). To force a byte to byte comparison regardless of the
uniq implementation, you can force the locale to
LC_ALL=C uniq your-file
Vim can achieve this nicely:
Or if you'd rather use vim as a command line tool, you could do this as
vim file -c "g/\v^(.*\n)\1/d" -c "wq"
This way you don't have to wrestle with exiting vim later ;)
On all lines that match this regex...
Any line followed by itself...
run the delete command (delete the current line). The
-c "wq" is to save the changes and exit.