I have created two .txt files that look like:

5fba026dcf271c6fa174b9fc2fc1fefadf1370f9 ./.changeFiles/change/addImage1.change  
bb352fba5807f4d03ea11abb00e411b00ce38673  ./.changeFiles/change/addImage2.change  
999eea42e4e72431695815456a3ac16796df5eed  ./.changeFiles/change/addSurface1.change

One is a list of all files that end with .change or .tstwf in the given directory and the other is a sorted version of that list with duplicate entries (based on the shasum) removed.

When I use a colordiff -u on the two files all the changes are the files I want removed. There are hundreds of files to be removed and I would prefer not to have to do it manually...

So my question is: How do I remove the files from the directory that are not included in the sorted.txt and show up in the diff?


workflows.txt (original file):

23ded53098d5d233c0a54af482542ba7a42860d3  ./encryption/AFPtoPDFwEncryption.tstwf
24a37e00de73a58736be1aee56d4b073e75a994f  ./workflows-from-will/workflow/AFPtoPDF.tstwf
2532934d268bb6eddc6a018e3469b0a53a794e10  ./workflows-from-will/workflow/change/table.without.emptycells.rotation.270.mark-content.change
264a08fef59d9b9c18a857e2532f0b7b5a92c60b  ./pageModification/AFPtoPDFwFillRectMod1.tstwf
2680a37bb22512bf3119e3991bb2c814cb23fb77  ./workflows-from-will/workflow/AFPtoPDFwHyperlinks.tstwf
268635261e91a4caf1a09979ffdb8163d4d22c8a  ./AFPmicr.tstwf
268635261e91a4caf1a09979ffdb8163d4d22c8a  ./workflows-from-will/eds-0923/AFPmicr.tstwf

workflows-sorted.txt (list of all files of this type to be kept):

0bdbcbed279e2df9149de319253609f11b19d689  ./textModification/AFPtoPDFwReplaceTextMod2.tstwf
0d0ebfb8d519ff84e3dd86771d067bdeea835df6  ./pageModification/AFPtoPDFwSoftClipMod1.tstwf
0dcafe876ea8574619664920ec5b1fdbff3ca218  ./workflows-from-will/AFPtoPDF_CE003A01.tstwf
0e55506bf5d7694d8cda9467a5809ea75c486fb1  ./workflows-from-will/workflow/change/pdf.change
0ec03992e19a12520f131f89cd9060c2e4ba2be2  ./workflows-from-will/AFPNOPIndexCallback.tstwf
0f1dca3e5d9b17f11c194b250096fc15891b9998  ./stapling/AFPtoPDFwStapling1.tstwf

Also, further in my research I came across the command "comm" it seems like this could be used to isolate the files I want removed.

  • Could you provide a list (with 5 or so lines) of each file, So we could run local tests?. – user79743 Dec 31 '15 at 18:13
  • Surely, you will get a more meaningful diff if you compare the sorted version of the original list, against the sorted list with the dupes removed. Diff sorted against sorted. – Kaz Jan 4 '16 at 7:26

So you want to remove the files listed in workflows.txt, except for those that are listed in workflows-sorted.txt. You can obtain the list of files by stripping off the checksums, sorting the names and running comm to extract the lines that are only present in workflows.txt. In a shell that supports process substitution (ksh93, bash, zsh):

comm -23 <(<workflows.txt sed 's/^[^ ]*[ ][ ]*//' | sort) \
         <(workflows-sorted.txt sed 's/^[^ ]*[ ][ ]*//' | sort)

comm -23 removes the lines that are only present in the second argument (-2) and the lines that are present in both files (-3), thus keeping only the lines that are present in the first argument but not the second argument. Keep in mind that comm requires the input files to be sorted.

To delete them:

comm -23 <(<workflows.txt sed 's/^[^ ]*[ ][ ]*//' | sort) \
         <(workflows-sorted.txt sed 's/^[^ ]*[ ][ ]*//' | sort) |
xargs -I rm -- {}

You can make the last line xargs rm to go slightly faster (by grouping calls to rm) if the file names don't contain any whitespace or \'". Alternatively you can make the last line tr '\n' '\0' | xargs -0 rm -- or xargs -d '\n' rm -- if your xargs supports these options. You don't need the -- if all your file names begin with / or ./ (or anything that's guaranteed not to begin with -).

  • Thanks for the help. I ended up editing the sed command to more accurately strip the checksum. From: (<workflows.txt 'sed/^[^ *][ ][ ]*//' | sort) To: (sed 's/^[^ ]* *//' workflows.txt | sort) Also, the last line of tr '\n' '\0' | xargs -0 rm -- worked the best for my case. – Zak Jan 5 '16 at 17:27

If your filenames do not have spaces, this should give you a list of the repeated files you want to erase:

awk 'c[$1]++' workflows.txt > erasefiles.txt

Then, it is just a matter of erasing all of them (use echo first till you are sure this does exactly what you need) (in bash):

rm $(awk '{print $2}' erasefiles.txt)

If your files could have spaces (or other odd chars, except new line - \n and zero \0 ):

awk '{$1=""; print}' erasefiles.txt | tr '\n' '\0' | xargs -0 -r rm --

For a one off job, I'd start with the diff output, and edit that into a list of files to delete (if bored, leave that task to sed(1), a Perl script, or some such). After double-checking the list, run it off by e.g. (bash(1)!):

rm $(< list-of-files-to-axe)

Might season with -f or -i flags to taste.

Automating this will be a lot more work (and introduce too many gotchas) to be worth it for occasional use. For repetitive use, write it all in a capable scripting language, like Perl or Python. Stay away from special-purpose languages like awk(1).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.