3

I have many folders and folders contain files. The same line might appear multiple times in single file and/or in multiple files. Files are not sorted. So there are some lines duplicated across multiple files and those files are in different folders.

I want to remove duplicate lines and keep only one of them across all files. Also file structure and names should stay same.

I've tried but made only unique in each single file not in all files. This code makes lines unique in each file and keeps file name:

for i in $(find . -type f); do
    awk '!seen[$0]++' "$i" > tmp_file
    mv ./tmp_file "$i"
done

Question: how can I make lines unique across all files in all subfolders while keeping files structure and name?

Here is a sample of my files. To simplify, I'm listing only files here, but files are located in same or different folders.

Input:

$ cat File-1
1
2
3
1

$ cat File-2
2
3
4
1

$ cat File-3
2
4
5
6

Output:

$ cat File-1
1
2
3

$ cat File-2
4

$ cat File-3
5
6

In my case, retaining first occurrence of line is preferred but not required (retained line can be in any file).

  • Which file should retain the unique copy of a line that appeared originally in more than one file? It would probably help if you provided a simplified, practical example (a few file names with a few lines of content for each one, before and after). – fra-san Jul 14 at 22:02
  • @fra-san Only in one file. Added sample. – Ikrom Jul 15 at 7:03
  • What should be done if no line is left for a file? E.g., in your sample, if the order of file processing is File-3File-1File-2 (find may find them in this order), File-3 ends up containing 2, 4, 5, 6, File-1 ends up containing 1, 3, and nothing is left in File-2. Should File-2 just stay, empty? – fra-san Jul 15 at 7:44
  • output file can be empty. – Ikrom Jul 15 at 7:48
2
#!/usr/bin/perl
use File::Find;
my $headdir="/some/path";
my @files=();
my $lines={};
find( { wanted => sub { push @files, $_ }, no_chdir => 1 }, $headdir );
foreach my $file (@files) {
  next unless(-f $file);
  system "cp $file $file". ".old";
  open(my $fhin, "$file".".old");
  open(my $fhout, ">$file");
  while(<$fhin>) {
    if(not defined $lines->{$_}) {
      print $fhout $_;
      $lines->{$_} = 1;
    }
  }
  close($fhin);
  close($fhout);
  #optional: system("rm $file".".old");
}

EDIT: (Only) tested with files mentioned in the question, tiny change in the code was necessary

| improve this answer | |
  • Thanks! Worked fine. Also I tried with mv command instead of cp to make it faster. Like this: system "mv $file $file". ".old"; This worked too. I deleted .old files like this find . -name "*.old" -delete – Ikrom Jul 15 at 20:19
1

What follows will only work if the number of files to process is small enough to make find run awk exactly once. It also assumes you can make a copy of the entire file tree (i.e. you are not storage-constrained).

Assuming your file tree is in the orig directory:

$ cp -pr orig tmp
$ cd tmp
$ find . -type f -exec awk '
  BEGIN { print ARGC }
  FILENAME != fn {
    close( "../orig/"fn )
    printf "" > ( "../orig/"FILENAME )
  }
  !seen[$0]++ { print > ( "../orig/"FILENAME ) }
  { fn = FILENAME; }' {} +

Once you are satisfied with the result, you can rm -r tmp.

print ARGC is used to show how many times awk was invoked. ARGC is the number of elements in the array of command line arguments (including the script itself); seeing it printed more than once means that global line deduplication failed.
(Indeed, if you can compute the total number of files to process, you may change that block into if ( (ARGC - 1) < total_number_of_files) exit to make sure no file is modified if awk is going to be invoked more than once).

| improve this answer | |
  • Thanks, it worked well. I had to install gawk, otherwise it didn't work with default awk in macos. In my case it printed 77 for 17Mb data. – Ikrom Jul 15 at 21:44
  • @Ikrom Glad to hear that! Did your awk (the one from MacOS) give you any error message? I try to only use portable features, I'd like to know what is not working cross-platform. – fra-san Jul 15 at 21:50
  • yes, I copied your code which starts with find and replaced the orig folder name with mine and run the code. This is the error: awk: syntax error at source line 3 context is FILENAME != fn { printf "" >>> >"../13/"FILENAME <<< } awk: illegal statement at source line 4 awk: illegal statement at source line 4 My awk --version is awk version 20070501 – Ikrom Jul 15 at 21:56
  • @Ikrom Edited. I can't test it on MacOS, though. – fra-san Jul 15 at 22:58
  • I tried with edited code, it gives another error. This is the output of run code (77 is number of files, 13 is original folder name): 77 awk: ../13/./18/filename1 makes too many open files input record number 1, file ./18/filename1 source line number 3. If I use 'gawk' instead of 'awk' in the last changed code, it works. – Ikrom Jul 16 at 17:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.