I know that with
awk '!seen[$0]++' filename > output.txt
it is possible to remove all duplicates from a text file.But how could I get a list of the removed lines in order to use this information to remove the same lines in another file.
I need this, because I want to remove all duplicates in a large textfile written in one language and then remove the same lines in the file with the translation of this language without losing the matching of the translation.
To make it more clear: (file 1)
line A
line B
line A
line C
...
(file 2):
line 1
line 2
line 3
line 4
...
Remove "line A" from file 1 and remove "line 3" from file 2. Needed for translation:
line A -> line 1
line B -> line 2
line C -> line 4
...
file 1B (with removed duplicates) would then look like:
line A
line B
line C
...
file 2B (with removed duplicates) would then look like:
line 1
line 2
line 4
...
hello
is duplicated in the 1st file so remove all lines that arehello
in the 2nd file") or by position ("line 2 is duplicated in the 1st file, so remove line 2 in the 2nd file")? – Kusalananda♦ Jun 17 '19 at 19:11