I have 12 text files all in one folder, each with about 5 million lines, each file has no duplicate line on its own but there are duplicated across multiple files, I want to remove the duplicate lines in each file but still save them separately, I have tried many Linux sort command and it keep merging the file together, I have Windows, Linus, and Mac, Is there any code or application that can do this?

  • 1
    You say "remove duplicates in each file" -> Does that mean that the line should still be present once, or should all lines that appear more than once be removed.
    – FelixJN
    Jan 14 at 19:19

to remove all duplicates but keep the first one which seen first in a file and remove from the other files, you could use awk as following, this also preserve the order of lines.

awk '!seen[$0]++ { print >FILENAME".new" }' file1 file2 ... file12

not much necessary here to close the open files by awk as it would be maximum 12 files, but you can still close() them after each file processed and finished.

awk '!seen[$0]++ { 
    if(prev!=FILENAME) close(prev".new");
    print >FILENAME".new";
    prev=FILENAME }
' file1 file2 ... file12

you can use shell and external command mv then to rename the output files to original name (you could use rename command as well):

for file in ./*.new; do echo mv -v -- "$file" "${file%.new}"; done

Note: remove the echo if you were happy with the result.

if you have GNU awk, you can use -i inplace option to modify the files inplace and simplify the whole command as following:

gawk -i inplace '!seen[$0]++' file1 file2 ... file12
  • 1
    I want to keep line that appear once, this code works well, thanks, but now i need to remove .new in front of my files. Jan 14 at 21:08
  • @SurpriseAwofemi see the update; also please add this clarification into the body of your question Jan 15 at 6:25

Not the answer you're looking for? Browse other questions tagged or ask your own question.