I have a folder with thousands of json files. The contents of each folder look something like this:


Some of the files contain lines that are duplicated in other files in the folder. For example, the single line


May occur in


or in some other files.

I would like to run a script that goes through all of the files, records which lines are duplicated in any of the files, then removes all the duplicate occurrences from the files (retaining the first occurrence).

I tried

sort -u *.json > newfile

and created a massive single file with the unique lines in all of the files, but this isn't useful to me. I'd like to preserve the existing file structure. Thanks for any tips!

  • This feels like a programming task beyond the 'easy' capabilities of shell scripting. As you want to preserve the json structure you probably want a json parsing library, most readily available in Perl or Python. – slowko Nov 23 '16 at 13:24

Assuming your filenames do not have spaces or special characters this should work for you. You may have to tweak the first command to get your desired sort order for which files get worked first.

for file_to_dedupe in $(echo *.json|sort)
   for file_to_strip in *.json
      [ "$file_to_dedupe" == "$file_to_strip" ] && continue
      grep -w -Ff ${file_to_dedupe} -v ${file_to_strip} > ${temp}
      mv ${temp} ${file_to_strip}


  • temp=$(mktemp) create a tmp file which to work with
  • for file_to_dedupe in $(echo *.json|sort) start looping for files to de-duplicate.
  • for file_to_strip in *.json start looping through the files to remove the duplicates from.
  • [ "$file_to_dedupe" == "$file_to_strip" ] && continue skip our current file.
  • grep -w -Ff ${file_to_dedupe} -v ${file_to_strip} > ${temp} remove the exact dupes using each line as a pattern from file_to_dedupe
  • mv ${temp} ${file_to_strip} put the new file in place.
  • Thanks for this suggestion. I've run this as a script in the folder where all the .json files are, and it has been running for 4 hours so far. It appears to access every file in the folder, but not make any changes (I can follow it using FileZilla ftp). It seems it is simply looping through every .json file in the folder over and over again, without changing anything. Is there something I need to change in the script you provided? I know almost nothing about shell scripting. If it matters, this is running on a remote server. – Zwentibold Nov 23 '16 at 18:35
  • UPDATE: I tested the script on a much smaller folder and it works! For my folder (15GB, 10k files) I will probably simply have to leave it running for a day or two! – Zwentibold Nov 24 '16 at 6:25
  • @Zwentibold yeah that's a lot of processing, there may be ways to improve this script for performance. – Zachary Brady Nov 24 '16 at 7:07
perl -i.bak -ne 'print $_ unless $a{$_}++ '  *.json

and remove files.bak if it worked.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.