1
[user@notebook foobar]$ ls *.jpg|wc -l
1959
[user@notebook foobar]$ cksum * | cut -d' ' -f-2 | sort | uniq -di | wc -l
698
[user@notebook foobar]$ 

There are many jpg files in a directory. Many of the files are duplicates, but I can spot them out if the cksum is the same for them. Sometimes there are 2 or 3 file of the same picture.

Q: How can I delete the unneeded duplicates?

I need to leave 1 from every picture, so if there are 3 exactly same pictures with different filenames, only one of them should remain, so there will be no duplicates of the picture, how to do this?

  • do you care which filename remains out of a duplicate set? – Jeff Schaller Nov 14 '16 at 21:55
1

Use fdupes:

fdupes -dN .

man fdupes:

   -d --delete
          prompt user for files to  preserve,  deleting  all  others  (see
          CAVEATS below)

   -N --noprompt
          when  used  together  with  --delete, preserve the first file in
          each set of duplicates and delete the others  without  prompting
          the user
0

This script uses an associative array in bash to save checksums, then report duplicates; change the echo to be rm if it looks good (or an rm -i for more paranoia):

#!/usr/bin/env bash
declare -A sums
for f in *
do
  if [[ ! -f "$f" ]]; then continue; fi
  c=$(cksum "$f" | awk '{print $1}')
  [[ -n "${sums[$c]}" ]] && echo "# rm \"$f\" -- duplicate of ${sums[$c]}"
  sums[$c]="$f"
done

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.