Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm working with graphic design. I've downloaded many files (EPS files, PSD files, etc) from various websites.

Because it come from various websites, after downloaded from more than 10 different websites, I got many same files with same size, and same everything but different file name (2 to 4 copies for same file). To remove the duplication by manually open one by one is very time consuming

I hope there is a way to rename all downloaded files to be unique name for different files (I don't mind if the new name is not descriptive).

For example, 2 same file: file nice-sun.eps downloaded from site 1, while 678.eps downloaded from site 2. It will become same file name after renamed.

share|improve this question
You could use md5sum or sha1sum to identify duplicates. – Mikel Apr 23 '12 at 5:03
This could help: unix.stackexchange.com/questions/22085/finding-duplicate-files – Mat Apr 23 '12 at 5:29
up vote 6 down vote accepted

This command will rename all files to the md5sum of their content. That means files with the same content will get the same name.

for f in *; do mv $f $(md5sum $f | cut -d " " -f 1); done

You can replace md5sum with sha1sum in the command.

You can also safely run this command in a directory where some files have unified filename while other have not.

For this demonstration I added -v to mv so we can see what is being renamed.

$ echo 1 > a
$ echo 2 > b
$ echo 1 > c
$ ls -1
$ for f in *; do mv -v $f $(md5sum $f | cut -d " " -f 1); done
`a' -> `b026324c6904b2a9cb4b88d6d61c81d1'
`b' -> `26ab0db90d72e28ad0ba1e22ee510510'
`c' -> `b026324c6904b2a9cb4b88d6d61c81d1'
$ echo 1 > d
$ echo 2 > e
$ ls -1
$ for f in *; do mv -v $f $(md5sum $f | cut -d " " -f 1); done
mv: `26ab0db90d72e28ad0ba1e22ee510510' and `26ab0db90d72e28ad0ba1e22ee510510' are the same file
mv: `b026324c6904b2a9cb4b88d6d61c81d1' and `b026324c6904b2a9cb4b88d6d61c81d1' are the same file
`d' -> `b026324c6904b2a9cb4b88d6d61c81d1'
`e' -> `26ab0db90d72e28ad0ba1e22ee510510'
$ ls -1
share|improve this answer

This is the rename-hash script I use to rename files to their hashsum, keeping the extension unchanged. Takes a list of files to be renamed as parameters. Use -n as first parameter for dry-run.


unset DRY
[ "$1" == "-n" ] && { DRY=1; shift; }

for i in $@; do

if [ -f "$i" ]; then

HASHSUM=$(md5sum "$i" | sed 's/\(..........\).*/\1/')
DIR=$(dirname "$i")
BASE=$(basename "$i")
EXT=$(echo "$BASE" | sed -n 's/[^.]*\(\..*\)/\1/p')
if [ "$TARGET" != "$DIR/$BASE" ]; then
   if [ -n "$DRY" ]; then
       echo will rename "$i" to "$TARGET"
       mv "$i" "$TARGET"




$ rename-hash -n *
will rename test.pdf to ./f604d0d6ad.pdf
will rename images.tar.gz to ./d41d8cd91b.tar.gz
share|improve this answer

There are some utilities to find duplicates. You can use find and md5sum, but it can take a pretty long time.

I usually use fdupes for that. If can find all duplicates and output their names to stdout. After that you can parse output and rename all files as you want or even remove them.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.