Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I'm working with graphic design. I've downloaded many files (EPS files, PSD files, etc) from various websites.

Because it come from various websites, after downloaded from more than 10 different websites, I got many same files with same size, and same everything but different file name (2 to 4 copies for same file). To remove the duplication by manually open one by one is very time consuming

I hope there is a way to rename all downloaded files to be unique name for different files (I don't mind if the new name is not descriptive).

For example, 2 same file: file nice-sun.eps downloaded from site 1, while 678.eps downloaded from site 2. It will become same file name after renamed.

share|improve this question
    
You could use md5sum or sha1sum to identify duplicates. –  Mikel Apr 23 '12 at 5:03
    
This could help: unix.stackexchange.com/questions/22085/finding-duplicate-files –  Mat Apr 23 '12 at 5:29
add comment

3 Answers 3

up vote 6 down vote accepted

This command will rename all files to the md5sum of their content. That means files with the same content will get the same name.

for f in *; do mv $f $(md5sum $f | cut -d " " -f 1); done

You can replace md5sum with sha1sum in the command.

You can also safely run this command in a directory where some files have unified filename while other have not.

For this demonstration I added -v to mv so we can see what is being renamed.

$ echo 1 > a
$ echo 2 > b
$ echo 1 > c
$ ls -1
a
b
c
$ for f in *; do mv -v $f $(md5sum $f | cut -d " " -f 1); done
`a' -> `b026324c6904b2a9cb4b88d6d61c81d1'
`b' -> `26ab0db90d72e28ad0ba1e22ee510510'
`c' -> `b026324c6904b2a9cb4b88d6d61c81d1'
$ echo 1 > d
$ echo 2 > e
$ ls -1
26ab0db90d72e28ad0ba1e22ee510510
b026324c6904b2a9cb4b88d6d61c81d1
d
e
$ for f in *; do mv -v $f $(md5sum $f | cut -d " " -f 1); done
mv: `26ab0db90d72e28ad0ba1e22ee510510' and `26ab0db90d72e28ad0ba1e22ee510510' are the same file
mv: `b026324c6904b2a9cb4b88d6d61c81d1' and `b026324c6904b2a9cb4b88d6d61c81d1' are the same file
`d' -> `b026324c6904b2a9cb4b88d6d61c81d1'
`e' -> `26ab0db90d72e28ad0ba1e22ee510510'
$ ls -1
26ab0db90d72e28ad0ba1e22ee510510
b026324c6904b2a9cb4b88d6d61c81d1
share|improve this answer
add comment

This is the rename-hash script I use to rename files to their hashsum, keeping the extension unchanged. Takes a list of files to be renamed as parameters. Use -n as first parameter for dry-run.

#!/bin/sh

unset DRY
[ "$1" == "-n" ] && { DRY=1; shift; }

for i in $@; do

if [ -f "$i" ]; then

HASHSUM=$(md5sum "$i" | sed 's/\(..........\).*/\1/')
DIR=$(dirname "$i")
BASE=$(basename "$i")
EXT=$(echo "$BASE" | sed -n 's/[^.]*\(\..*\)/\1/p')
TARGET="$DIR"/"$HASHSUM""$EXT"
if [ "$TARGET" != "$DIR/$BASE" ]; then
   if [ -n "$DRY" ]; then
       echo will rename "$i" to "$TARGET"
   else
       mv "$i" "$TARGET"
   fi
fi

fi

done

Example:

$ rename-hash -n *
will rename test.pdf to ./f604d0d6ad.pdf
will rename images.tar.gz to ./d41d8cd91b.tar.gz
share|improve this answer
add comment

There are some utilities to find duplicates. You can use find and md5sum, but it can take a pretty long time.

I usually use fdupes for that. If can find all duplicates and output their names to stdout. After that you can parse output and rename all files as you want or even remove them.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.