16

I'm looking to remove duplicate lines from a file but leave 1 occurrence in the file.

Example of the file:

this is a string
test line
test line 2
this is a string

From the above example, I would want to remove 1 occurrence of "this is a string".

Best way to do this?

9
  • 2
    With such questions you should always provide example input and output. May 19, 2018 at 13:12
  • 1
    Possibly related: Remove duplicate lines while keeping the order of the lines May 19, 2018 at 13:12
  • Are the duplicated lines adjacent to one another? Is the output to remain in the same order or would it be ok to sort the data?
    – Kusalananda
    May 19, 2018 at 13:14
  • 1
    Keep one occurrence of a duplicate (ie two identical lines per match) or simply "remove all duplicate lines, leaving only one line per set of duplicates"? Does the final order matter? May 19, 2018 at 13:17
  • 1
    it is not a problem for you that the lines will be sorted, then a sort file|uniq will do what you want.
    – peterh
    May 19, 2018 at 19:03

5 Answers 5

14

Demo file stuff.txt contains:

one
two
three
one
two
four
five

Remove duplicate lines from a file assuming you don't mind that lines are sorted

$ sort -u stuff.txt 
five
four
one
three
two

Explanation: the u flag sent to sort says sort the lines of the file and force unique.

Remove duplicate lines from a file, preserve original ordering, keep the first:

$ cat -n stuff.txt | sort -uk2 | sort -nk1 | cut -f2-
one
two
three
four
five

Explanation: The n flag passed to cat appends line numbers to left of every line, plus space, then the first sort says sort by unique and but only after the first word, the second sort command says use the line numbers we stored in step 1 to resort by the original ordering, finally cut off the first word.

Remove duplicate lines from a file, preserve order, keep last.

tac stuff.txt > stuff2.txt; cat -n stuff2.txt | sort -uk2 | sort -nk1 | cut -f2- > stuff3.txt; tac stuff3.txt > stuff4.txt; cat stuff4.txt
three
one
two
four
five

Explanation: Same as before, but tac reverse the file, achieving the desired result.

3
  • 1
    > Remove duplicate lines from a file, preserve original ordering, keep the first: cat -n stuff.txt | sort -uk2 | sort -nk1 | cut -f2- The command you gave doesn't preserve at least one occurrence. I tested it out with a file which contained only duplicates, and only the last line remained after running the command (because it contained a newline, and was therefore unique) Jun 17, 2021 at 13:20
  • Surprise left jab: Whitespace and newlines at end of file, relative to end of lines, are both different and significant. I diagnose your operating system as defective. Bazinga. Jun 17, 2021 at 15:52
  • I can't seem to be able to edit my previous comment, but slight correction - the last line is the only line that didn't contain a newline, and was therefore unique. I used Ilya Bobyr's solution below, which worked fine and preserved one occurrence, as required. The problem with your solution was that if there were two non-unique entries, then neither of them was preserved. Jun 18, 2021 at 10:48
5

As removing all but the last occurrence is the inverse of removing all but the first, there is also this solution:

tac file | awk '! seen[$0]++' | tac

tac reverses lines in the file, and awk will only output the first occurrence of a duplicate line.

3

Based on your comments you want the result to be the same output file without having to create another output or append into the new file, you could use the following:

example:

gawk -i inplace '!a[$0]++' $file
2

This leaves the first occurrence:

awk '! a[$0]++' inputfile

start cmd:> echo 'this is a string
cont. cmd:> test line
cont. cmd:> test line 2
cont. cmd:> this is a string' | awk '! a[$0]++'
    this is a string
    test line
    test line 2
5
  • It seems to just print out and not actually make in changes in the file.
    – Tom Bailey
    May 19, 2018 at 15:49
  • @TomBailey That's why I told you to provide example input and output. I did test it and it works fine for me. May 19, 2018 at 16:49
  • I have edited it now.
    – Tom Bailey
    May 19, 2018 at 19:29
  • @TomBailey works fine for me. May 19, 2018 at 20:16
  • See my answer without creating another output to store the result
    – MaXi32
    Jun 6, 2020 at 8:46
0

If you use vim, could try the following code:

g/./if(temp == getline('.')) | let temp = getline('.') | else | :norm dd | endif

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .