How to remove duplicate lines inside a text file & get the number of the removed lines?

Question

I know that with

awk '!seen[$0]++' filename > output.txt

it is possible to remove all duplicates from a text file.But how could I get a list of the removed lines in order to use this information to remove the same lines in another file.

I need this, because I want to remove all duplicates in a large textfile written in one language and then remove the same lines in the file with the translation of this language without losing the matching of the translation.

To make it more clear: (file 1)

line A
line B
line A
line C
...

(file 2):

line 1
line 2
line 3
line 4
...

Remove "line A" from file 1 and remove "line 3" from file 2. Needed for translation:

line A -> line 1
line B -> line 2
line C -> line 4
... 

file 1B (with removed duplicates) would then look like:

line A
line B
line C
...

file 2B (with removed duplicates) would then look like:

line 1
line 2
line 4
... 

Answer 1

$ awk 'FNR==NR && !seen[$0]++ { keep[FNR]=$0; next } (FNR in keep) { printf("%s -> %s\n", keep[FNR], $0) }' file1 file2
line A -> line 1
line B -> line 2
line C -> line 4

This reproduces the output in the middle of the question with an awk program.

The program reads the first file (file1) and finds the duplicated lines in the first block. That block also saves the lines to keep in the array keep, indexed by line number.

When reading the second file (the second block in the code), if the line number is one that we'd like to keep, output the saved line from the first file together with the line from the second file.

Variation that saves two new files:

awk 'FNR==NR && !seen[$0]++ { keep[FNR]=$0; print >FILENAME ".new"; next } (FNR in keep) { print >FILENAME ".new" }' file1 file2

This will write file1.new with only the unique lines in file1, and it will write file2.new with the same lines, but from file2.

The only downside with this solution is that it requires 2x memory, where x is the number of unique lines in the first file, i.e., it will store two copies of each unique line in file1 (as an index in seen and as values in the keep array).

Answer 2

Here's a 2 step process:

1. an awk script to generate a sed script to delete the unwanted lines

   awk 'seen[$0]++ {print NR "d"}' file1 > remove.sed 
   

2. then use the sed script to remove the lines in the needed files

   sed -i -f remove.sed file1 file2