rename all files in a directory to the md5 hash of their filename (not content)

Question

I'm very new to linux / command line and need to encrypt the names of 10K+ files (unique names) so they match the MD5 encrypted name in the mySQL database.
I've seen how you can rename a directory of files and how to get the hash of a file (mdsum?) but I'm stuck on how to get the hash of the file name and then rename that file to the generated hash retaining the extension i.e.

mynicepicture.jpg > fba8255e8e9ce687522455f3e1561e53.jpg 

It seems like it should be a simple rename or mv line but I can't get my head around it.
Many thanks for your insights

PS I've seen the use of Perl functions in a few examples close to what I'm looking for but have no idea where / how to use those.

Answer 1

You didn't say which shell you want to use, so I'm just assuming Bash – the answer needs adjustments to work with other shells.

for i in *; do sum=$(echo -n "$i"|md5sum); echo -- "$i" "${sum%% *}.${i##*.}"; done

Script version:

for i in *; do
  sum=$(echo -n "$i" | md5sum)
  echo -- "$i" "${sum%% *}.${i##*.}"
done

This simple for loop takes every file in the current directory, computes the md5 sum of its name and outputs it. Use this to check the functionality, if you want to start renaming replace the second echo by mv.

Explanations

• echo -n "$i" | md5sum – calculate md5 sum of the full file name including the file extension (Piping), to strip the extension change echo -n "$i" to one of the following:

  ${i%%.*}
  sed 's/\..*//' <<< "$i"
  echo "$i" | sed 's/\..*//'
  

• sum=$(…) – execute … and save the output in $sum (Command Substitution)

• ${sum%% *} – output everything until the first space (Parameter Substitution), the same as one of the following:

  $(sed 's/ .*//' <<< "$sum")
  $(echo "$sum" | sed 's/ .*//')
  

• ${i##*.} – output everything after the last dot (Parameter Substitution), the same as one of the following:

  $(sed 's/.*\.//' <<< "$i")
  $(echo "$i" | sed 's/.*\.//')
  

If you need to rename files recursively in different folders, use find with the -exec option.

Answer 2

#!/bin/bash

md5name () {
    local base=${1##*/}
    local ext=${base##*.}
    local dir=${1%/*}

    printf '%s' "${base%.$ext}" | md5sum |
    awk -v dir="$dir" -v ext="$ext" '{ printf("%s/%s.%s\n", dir, $1, ext) }'
}

dir=$HOME  # where your files are

for pathname in "$dir"/*; do
    test -f "$pathname" || continue
    echo mv "$pathname" "$( md5name "$pathname" )"
done

This bash script uses the md5sum utility from GNU coreutils to compute the MD5 hash from the base name (sans extension) of any given pathname. The helper function md5name does the actual computation and will output the new name with complete path and extension.

The md5name function uses awk to assemble the new name from the parts of the given pathname and the result from md5sum.

Examples of the function in use by itself:

$ md5name '/some/path/file name here.extension'
/some/path/c9e89fa443d16da4b96ea858881320c9.extension

... where c9e89fa443d16da4b96ea858881320c9 is the MD5 hash of the string file name here.

Remove the echo from the script at the top to actually rename the files. You may want to save the output of the original script to file (with the echo in place) if you at some point need to restore the file names to their originals.

Note that running this twice on a set of files will compute the MD5 hash of MD5 hashes, and that the original filename then becomes unrecoverable unless you make careful notes about what files are called what after each run of the script.

Answer 3

With perl's rename:

find . -name '*.jpg' -type f -exec rename -n '
  BEGIN{use Digest::MD5 qw(md5_hex)}
  my ($dir, $name, $ext) = m{(.*)/(.*)\.(.*)}s;
  $_ = "$dir/" . md5_hex($name) . ".$ext"' {} +

(remove -n when happy).

Answer 4

This in approach I often like using.

ls | sed "s|^\(.*\)\.\([^\.]*\)$|mv \1.\2 \\`echo \1 \| md5sum \| cut -d' ' -f 1\\`.\2|" | sh -

The "ls" command produces a stream of text lines. The "sed" command tranforms each line with pattern-matching rules. The "sed" command outputs a "mv" command which is then piped though a shell "sh" for execution. The "mv" command's parameters are like "mv oldfilename newfilename", which renames the file. I construct the new file-name with a sed command that takes the part before the last dot, and echoes it into the input of the "md5sum" command, and then takes just the hash from it's output.

Walking through my process, first list files ('head -n 3' to just see the first 3 lines):

ls | head -n 3
    1000-26092016.xml
    1000-27092016.xml
    12312-28092016.xml

Then think about transforming with sed (not yet piping any generated commands through a shell)

ls | sed "s|^\(.*\)\.\([^\.]*\)$|mv \1.\2 \1.\2|" | head -n 3
    mv 1000-26092016.xml 1000-26092016.xml
    mv 1000-27092016.xml 1000-27092016.xml
    mv 12312-28092016.xml 12312-28092016.xml

There are three match patterns:

^\(.*\)      = match from start-of-line up to a dot
\.           = matches a single dot
\([^\.]*\)$  = match 0-or-more non-dot chars from end of line

I want to use sed to replace an input filename with "mv filename NEWfilename", but as I'm piping commands through a shell, I can generate commands that get the md5sum, like this

echo "1000-26092016" | md5sum
    55b18a6b0add4a318b0079e18512b4e8  -

to get just the hash

echo "1000-26092016" | md5sum | cut -d' ' -f 1
    55b18a6b0add4a318b0079e18512b4e8

In a unix shell, we can use backtick operators (`some_command`) to run a sub-command, so for example

echo "howdy date there"
    howdy date there
echo "howdy `date` there"
    howdy Fri Sep 15 18:39:00 IST 2017 there

Back to the mv command, I want sed to produce "mv here there" with "there" replaced with a backtick command to get the md5sum. The string inside the sed replace-string starts like this

ls | sed "s|^\(.*\)\.\([^\.]*\)$|mv \1.\2 `echo \1 | md5sum | cut -d' ' -f 1`.\2|" | head -n 3
    mv 1000-26092016.xml     b026324c6904b2a9cb4b88d6d61c81d1.xml
    mv 1000-27092016.xml     b026324c6904b2a9cb4b88d6d61c81d1.xml
    mv 12312-28092016.xml    b026324c6904b2a9cb4b88d6d61c81d1.xml

But is clearly making the same hash for each filename, as the backticked-command is being run before sed sees the string. To stop the shell running the backtick command so sed will output the backticks, we have to prepend slashes (also to the pipe-character), so again:

ls | sed "s|^\(.*\)\.\([^\.]*\)$|mv \1.\2 \`echo \1 \| md5sum \| cut -d' ' -f 1\`.\2|" | head -n 3
    mv 1000-26092016.xml     `echo 1000-26092016 | md5sum | cut -d' ' -f 1`.xml
    mv 1000-27092016.xml     `echo 1000-27092016 | md5sum | cut -d' ' -f 1`.xml
    mv 12312-28092016.xml    `echo 12312-28092016 | md5sum | cut -d' ' -f 1`.xml

The output also needs filenames to be quoted in case of spaces, so

ls | sed "s|^\(.*\)\.\([^\.]*\)$|mv \"\1.\2\" \"\`echo \1 \| md5sum \| cut -d' ' -f 1\`.\2\"|" | grep trick
    mv "a trick€€ fíle nÁme.xml" "`echo a trick€€ fíle nÁme | md5sum | cut -d' ' -f 1`.xml"

So lets try this one out, by piping it through a shell:

ls | sed "s|^\(.*\)\.\([^\.]*\)$|mv \"\1.\2\" \"\`echo \1 \| md5sum \| cut -d' ' -f 1\`.\2\"|" | grep trick | sh -

Did it work ? i guess:

echo "a trick€€ fíle nÁme" | md5sum
    629db9c3071928ba0746f18444713b65  -
ls 629db9c3071928ba0746f18444713b65*
    629db9c3071928ba0746f18444713b65.xml

Here's an approach to cross-check; use "ls" option "-i" to output the unix filesystem i-node (which doesn't change with "mv"):

ls -1i | sort -n > .before
ls | sed "s|^\(.*\)\.\([^\.]*\)$|mv \"\1.\2\" \"\`echo \1 \| md5sum \| cut -d' ' -f 1\`.\2\"|" | sh -
ls -1i | sort -n > .after
cut -d' ' -f 1 .before | while read I ; do echo "mv'd \"`grep ${I} .before`\" to \"`grep ${I} .after`\"" | sed "s| *$I *||g" ; done | head -n 3
    mv'd "1000-26092016.xml" to "55b18a6b0add4a318b0079e18512b4e8.xml"
    mv'd "1000-27092016.xml" to "b1baa80d99d5edf85c8aeb98185dd440.xml"
    mv'd "12312-28092016.xml" to "2b2d692bd047b64c99f7b9161349d430.xml"

Or, using the "paste" command ('coreutils' package)

paste .before .after | head -n 3
    36703389 1000-26092016.xml  36703389 55b18a6b0add4a318b0079e18512b4e8.xml
    36703390 1000-27092016.xml  36703390 b1baa80d99d5edf85c8aeb98185dd440.xml
    36703391 12312-28092016.xml 36703391 2b2d692bd047b64c99f7b9161349d430.xml

Answer 5

For an AWK approach:

find [Directory] -type f [various other find options] | 
     awk '{orig=$0; 
           match($0,/^.*\//,path); sub("^"path[0], "");
           match($0, /.[[^.]+$/,ext); sub(ext[0]"$", "");
           ("echo \"" $0 "\"|md5sum") | getline;
           com=sprintf("mv \"%s\" \"%s%s%s\"", orig, p[0], $1, ext[0]);
           print(com)
           }'

Modern find commands don't require a directory for input . is assumed, so the [Directory] could be left blank. The -type f only finds files, which is handy since md5sum isn't fond of directories and changing the directory name while running wouldn't be a good idea. Use -iname pattern if you only want to use some files, e.g. -iname \*.dat, if case is important, use -name instead of -iname.

The match(...); sub(...) pieces are extracting portions of the filename and replacing them in the input string. Note that "^" and "$" are [pre/ap]pended to prevent replacing a string that may repeat the path/extension.

Replace print(com) with system(com) to actually perform the rename, e.g.:

find -type f | 
     awk '{orig=$0; 
           match($0,/^.*\//,path); sub("^"path[0], "");
           match($0, /.[[^.]+$/,ext); sub(ext[0]"$", "");
           ("echo \"" $0 "\"|md5sum") | getline;
           com=sprintf("mv \"%s\" \"%s%s%s\"", orig, p[0], $1, ext[0]);
           system(com)
           }'

If you want to use the md5sum of the actual file as a name you could use the fact that md5sum outputs the sum and input filename to do something like:

 find -type f -exec md5sum '{}' ';' | 
     while read sum file ; do 
       [echo] mv "$file" "`dirname $file`/$sum".extension ; 
     done

The while read sum file will take 2 arguments, the results of the md5sum command, and assign sum and file variables with them. Since the sum shouldn't have spaces in it, the read should work fine.

Obviously the [echo] should be removed when actually running, but it's always a good idea when testing any scripted changes to test the search before running.

This all assumes you are running bash. Also, this can be typed as one longish line:

find -iname \*.jpg -exec md5sum '{}' ';' | while read sum file ; do mv "$file" "`dirname $file`/$sum".jpg ; done

Answer 6

I like that one line answer, but it breaks because it parses the filename. I also bumped it up a bit with sha hashes.

find -iname "*.jpg" -exec sha1sum '{}' ';' | while read sum file ; do mv -v "$file" "`dirname '$file'`/$sum".jpg ; done

I think it pulls the files out too and puts them at the base of where the command was entered.

Thanks.