1

I'm looking to remove duplicate lines from a file but leave 1 occurrence in the file.

Example of the file:

this is a string
test line
test line 2
this is a string

From the above example, I would want to remove 1 occurrence of "this is a string".

Best way to do this?

  • 2
    With such questions you should always provide example input and output. – Hauke Laging May 19 '18 at 13:12
  • 1
  • Are the duplicated lines adjacent to one another? Is the output to remain in the same order or would it be ok to sort the data? – Kusalananda May 19 '18 at 13:14
  • 1
    Keep one occurrence of a duplicate (ie two identical lines per match) or simply "remove all duplicate lines, leaving only one line per set of duplicates"? Does the final order matter? – roaima May 19 '18 at 13:17
  • 1
    it is not a problem for you that the lines will be sorted, then a sort file|uniq will do what you want. – peterh says reinstate Monica May 19 '18 at 19:03
3

Demo file stuff.txt contains:

one
two
three
one
two
four
five

Remove duplicate lines from a file assuming you don't mind that lines are sorted

$ sort -u stuff.txt 
five
four
one
three
two

Explanation: the u flag sent to sort says sort the lines of the file and force unique.

Remove duplicate lines from a file, preserve original ordering, keep the first:

$ cat -n stuff.txt | sort -uk2 | sort -nk1 | cut -f2-
one
two
three
four
five

Explanation: The n flag passed to cat appends line numbers to left of every line, plus space, then the first sort says sort by unique and but only after the first word, the second sort command says use the line numbers we stored in step 1 to resort by the original ordering, finally cut off the first word.

Remove duplicate lines from a file, preserve order, keep last.

tac stuff.txt > stuff2.txt; cat -n stuff2.txt | sort -uk2 | sort -nk1 | cut -f2- > stuff3.txt; tac stuff3.txt > stuff4.txt; cat stuff4.txt
three
one
two
four
five

Explanation: Same as before, but tac reverse the file, achieving the desired result.

0

This leaves the first occurrence:

awk '! a[$0]++' inputfile

start cmd:> echo 'this is a string
cont. cmd:> test line
cont. cmd:> test line 2
cont. cmd:> this is a string' | awk '! a[$0]++'
    this is a string
    test line
    test line 2
  • It seems to just print out and not actually make in changes in the file. – Tom Bailey May 19 '18 at 15:49
  • @TomBailey That's why I told you to provide example input and output. I did test it and it works fine for me. – Hauke Laging May 19 '18 at 16:49
  • I have edited it now. – Tom Bailey May 19 '18 at 19:29
  • @TomBailey works fine for me. – Hauke Laging May 19 '18 at 20:16
0

As removing all but the last occurrence is the inverse of removing all but the first, there is also this solution:

tac file | awk '! seen[$0]++' | tac

tac reverses lines in the file, and awk will only output the first occurrence of a duplicate line.

0

If you use vim, could try the following code:

g/./if(temp == getline('.')) | let temp = getline('.') | else | :norm dd | endif

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.