I have many folders and folders contain files. The same line might appear multiple times in single file and/or in multiple files. Files are not sorted. So there are some lines duplicated across multiple files and those files are in different folders.
I want to remove duplicate lines and keep only one of them across all files. Also file structure and names should stay same.
I've tried but made only unique in each single file not in all files. This code makes lines unique in each file and keeps file name:
for i in $(find . -type f); do awk '!seen[$0]++' "$i" > tmp_file mv ./tmp_file "$i" done
Question: how can I make lines unique across all files in all subfolders while keeping files structure and name?
Here is a sample of my files. To simplify, I'm listing only files here, but files are located in same or different folders.
$ cat File-1 1 2 3 1 $ cat File-2 2 3 4 1 $ cat File-3 2 4 5 6
$ cat File-1 1 2 3 $ cat File-2 4 $ cat File-3 5 6
In my case, retaining first occurrence of line is preferred but not required (retained line can be in any file).
findmay find them in this order),
File-3ends up containing
File-1ends up containing
3, and nothing is left in
File-2just stay, empty?