I have many folders and folders contain files. The same line might appear multiple times in single file and/or in multiple files. Files are not sorted. So there are some lines duplicated across multiple files and those files are in different folders.
I want to remove duplicate lines and keep only one of them across all files. Also file structure and names should stay same.
I've tried but made only unique in each single file not in all files. This code makes lines unique in each file and keeps file name:
for i in $(find . -type f); do awk '!seen[$0]++' "$i" > tmp_file mv ./tmp_file "$i" done
Question: how can I make lines unique across all files in all subfolders while keeping files structure and name?
Here is a sample of my files. To simplify, I'm listing only files here, but files are located in same or different folders.
$ cat File-1 1 2 3 1 $ cat File-2 2 3 4 1 $ cat File-3 2 4 5 6
$ cat File-1 1 2 3 $ cat File-2 4 $ cat File-3 5 6
In my case, retaining first occurrence of line is preferred but not required (retained line can be in any file).