Remove duplicate lines from a file but leave 1 occurrence

Question

I'm looking to remove duplicate lines from a file but leave 1 occurrence in the file.

Example of the file:

this is a string
test line
test line 2
this is a string

From the above example, I would want to remove 1 occurrence of "this is a string".

Best way to do this?

Answer 1

Demo file stuff.txt contains:

one
two
three
one
two
four
five

Remove duplicate lines from a file assuming you don't mind that lines are sorted

$ sort -u stuff.txt 
five
four
one
three
two

Explanation: the u flag sent to sort says sort the lines of the file and force unique.

Remove duplicate lines from a file, preserve original ordering, keep the first:

$ cat -n stuff.txt | sort -uk2 | sort -nk1 | cut -f2-
one
two
three
four
five

Explanation: The n flag passed to cat appends line numbers to left of every line, plus space, then the first sort says sort by unique and but only after the first word, the second sort command says use the line numbers we stored in step 1 to resort by the original ordering, finally cut off the first word.

Remove duplicate lines from a file, preserve order, keep last.

tac stuff.txt > stuff2.txt; cat -n stuff2.txt | sort -uk2 | sort -nk1 | cut -f2- > stuff3.txt; tac stuff3.txt > stuff4.txt; cat stuff4.txt
three
one
two
four
five

Explanation: Same as before, but tac reverse the file, achieving the desired result.

Answer 2

As removing all but the last occurrence is the inverse of removing all but the first, there is also this solution:

tac file | awk '! seen[$0]++' | tac

tac reverses lines in the file, and awk will only output the first occurrence of a duplicate line.

Answer 3

Based on your comments you want the result to be the same output file without having to create another output or append into the new file, you could use the following:

example:

gawk -i inplace '!a[$0]++' $file

Answer 4

This leaves the first occurrence:

awk '! a[$0]++' inputfile

start cmd:> echo 'this is a string
cont. cmd:> test line
cont. cmd:> test line 2
cont. cmd:> this is a string' | awk '! a[$0]++'
    this is a string
    test line
    test line 2

Answer 5

If you use vim, could try the following code:

g/./if(temp == getline('.')) | let temp = getline('.') | else | :norm dd | endif