15

What sed/awk command can I use? Just sort -u will remove all instances

Input:

abc
abc
def
abc
abc
def

Expected output:

abc
def
abc
def
36

That's what the uniq standard command is for.

uniq your-file

Note that some uniq implementations like GNU uniq will give you the first of a sequence of lines that sort the same (where strcoll() returns 0) as opposed to are byte-to-byte identical (where memcmp() or strcmp() returns 0). To force a byte to byte comparison regardless of the uniq implementation, you can force the locale to C with:

LC_ALL=C uniq your-file
7

Vim can achieve this nicely:

:g/\v^(.*\n)\1/d

Or if you'd rather use vim as a command line tool, you could do this as

vim file -c "g/\v^(.*\n)\1/d" -c "wq"

This way you don't have to wrestle with exiting vim later ;)

Explanation:

:g/

On all lines that match this regex...

\v^(.*\n)\1

Any line followed by itself...

/d

run the delete command (delete the current line). The -c "wq" is to save the changes and exit.

  • Note that with vim 8.1.2112 at least, it doesn't work for duplicated lines that are the last two lines of the file. Also, it only removes one duplicate from sequences of 3 duplicate lines. – Stéphane Chazelas Oct 4 '19 at 22:02
  • 1
    @StéphaneChazelas except that we currently lack specification for desired behavior in the 3-line case—I could see desired behavior both ways. – D. Ben Knoble Oct 6 '19 at 5:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.