0

Thanks to this forum I was able to process a large text file containing filenames and hashes and end up with a list of files with identical hashes as shown below:

file_35.txt 8208ad321576b521b23b07b9ba598e5c43b03ec4172c96fdbd35a858ec205ae6
file_87.txt 8208ad321576b521b23b07b9ba598e5c43b03ec4172c96fdbd35a858ec205ae6
file_32.txt aaf6b8c4a95d0e8f191784943ba1ea5c0b4d4baab733efe8ceb8b35478b6afd2
file_58.txt aaf6b8c4a95d0e8f191784943ba1ea5c0b4d4baab733efe8ceb8b35478b6afd2
file_89.txt aaf6b8c4a95d0e8f191784943ba1ea5c0b4d4baab733efe8ceb8b35478b6afd2

I want to DELETE lines that contain duplicate hashes so that I will end up with a text file containing unique hashes as below:

file_35.txt 8208ad321576b521b23b07b9ba598e5c43b03ec4172c96fdbd35a858ec205ae6
file_32.txt aaf6b8c4a95d0e8f191784943ba1ea5c0b4d4baab733efe8ceb8b35478b6afd2

Once this is done I will be able to create scripts to move and delete the files I want to save and those I want to delete.

  • 1
    Easy to understand is this awk one. awk '!seen[$2]++' filename – Valentin Bajrami Nov 23 '16 at 20:10
1

If your file is already sorted, how about:

uniq --skip-field=1 file

otherwise you can sort your file first and then use uniq:

sort -k2 file | uniq --skip-field=1

Output:

file_35.txt 8208ad321576b521b23b07b9ba598e5c43b03ec4172c96fdbd35a858ec205ae6
file_32.txt aaf6b8c4a95d0e8f191784943ba1ea5c0b4d4baab733efe8ceb8b35478b6afd2
  • I don't think skipping the first 11 characters will work - the filename length varies from short to very long - the above is a simplification. Any suggestions for variable filename length? – speld_rwong Nov 23 '16 at 19:23
  • @speld_rwong You are right! I edited my answer. Thanks! – andreatsh Nov 23 '16 at 19:55
  • Just wondering, what defines a field? If it is an empty space this may be a problem. My example may have been too simple - some filenames contain spaces. e.g. hd12_picture of giant trees 005.jpg – speld_rwong Nov 24 '16 at 2:09
  • From man uniq: A field is a run of blanks (usually spaces and/or TABs), then non-blank characters.. If your filenames contain spaces, assuming the hash is the last field you can use a awk solution, i.e. awk '!z[$NF]++' file. I did a quick test and this should work. – andreatsh Nov 24 '16 at 10:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.