0

I know that with

awk '!seen[$0]++' filename > output.txt

it is possible to remove all duplicates from a text file.But how could I get a list of the removed lines in order to use this information to remove the same lines in another file.

I need this, because I want to remove all duplicates in a large textfile written in one language and then remove the same lines in the file with the translation of this language without losing the matching of the translation.

To make it more clear: (file 1)

line A
line B
line A
line C
...

(file 2):

line 1
line 2
line 3
line 4
...

Remove "line A" from file 1 and remove "line 3" from file 2. Needed for translation:

line A -> line 1
line B -> line 2
line C -> line 4
... 

file 1B (with removed duplicates) would then look like:

line A
line B
line C
...

file 2B (with removed duplicates) would then look like:

line 1
line 2
line 4
... 
  • Do you want to remove the duplicated lines in the second file by contents ("the line saying hello is duplicated in the 1st file so remove all lines that are hello in the 2nd file") or by position ("line 2 is duplicated in the 1st file, so remove line 2 in the 2nd file")? – Kusalananda Jun 17 at 19:11
  • Ah, by position. Good. – Kusalananda Jun 17 at 19:11
  • yes, by position – user3776738 Jun 17 at 19:20
  • What is the output? Two new files? – Kusalananda Jun 17 at 19:20
  • yes, like described above (new edit) – user3776738 Jun 17 at 19:20
0
$ awk 'FNR==NR && !seen[$0]++ { keep[FNR]=$0; next } (FNR in keep) { printf("%s -> %s\n", keep[FNR], $0) }' file1 file2
line A -> line 1
line B -> line 2
line C -> line 4

This reproduces the output in the middle of the question with an awk program.

The program reads the first file (file1) and finds the duplicated lines in the first block. That block also saves the lines to keep in the array keep, indexed by line number.

When reading the second file (the second block in the code), if the line number is one that we'd like to keep, output the saved line from the first file together with the line from the second file.

Variation that saves two new files:

awk 'FNR==NR && !seen[$0]++ { keep[FNR]=$0; print >FILENAME ".new"; next } (FNR in keep) { print >FILENAME ".new" }' file1 file2

This will write file1.new with only the unique lines in file1, and it will write file2.new with the same lines, but from file2.

The only downside with this solution is that it requires 2x memory, where x is the number of unique lines in the first file, i.e., it will store two copies of each unique line in file1 (as an index in seen and as values in the keep array).

0

Here's a 2 step process:

  1. an awk script to generate a sed script to delete the unwanted lines

    awk 'seen[$0]++ {print NR "d"}' file1 > remove.sed 
    
  2. then use the sed script to remove the lines in the needed files

    sed -i -f remove.sed file1 file2
    

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.