Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I'd like to find an md5sum (or similar calculation) of a folder without compressing it into an archive.

For example, if in the folder MyFolder we have the files 1.txt, 2.txt and 3.txt, containing:


1.txt

The text into 1.txt

2.txt

The text into 2.txt

3.txt

The text into 3.txt


How can I calculate the md5sum of this entire MyFolder? Is there a way? I want to use this to verify if two folders have the same contents.

This can be util to verify if two cds or folders are equal. I'd like a easy way to do it.

share|improve this question

4 Answers 4

up vote 13 down vote accepted

The md5deep tool was developed for precisely this purpose. Many Linux distributions offer it in package form.

share|improve this answer
    
Thx! It's worked fine to me. To other users with the same problem to like this: sudo apt-get install md5deep md5deep -rel /path/to/your/directory/ > directory_hash.md5 md5deep -X directory_hash.md5 -r /path/to/your/second/direcotory Thx a lot. –  GarouDan Oct 17 '11 at 1:15
    
@GarouDan. The command you have shown will follow symbolic links. You can use the -o option to handle file types. –  Peter.O Oct 17 '11 at 7:16
    
Oh thx...don't know that...really helpfull. I'll remember! –  GarouDan Oct 18 '11 at 10:00

If you don't want to archive it, maybe you could do something like this

diff <(find folder1) <(find folder2)

You may have to adapt the find commands to be more accurate.

EDIT You could add -exec to your find call to compare the content of files. Something similar to this:

diff <(find folder1 -type f -exec md5sum {} \; | sort) <(find folder2 -type f -exec md5sum {} \; | sort)

Remember that you may want to adapt this.

share|improve this answer
    
It's an interesting point. This lists all files of each folder...but if I have a really great amount of archives... how could verify there are the same files (with the data - maybe using a md5sum tool) in each folder? –  GarouDan Oct 16 '11 at 23:49
1  
Then do a diff of the MD5SUM output of each file. You just need to string together the find, md5sum and diff commands differently. –  sybreon Oct 17 '11 at 0:01
    
Thx about your ideas too, cya...I will try interesting things with these find comand after...thanks. –  GarouDan Oct 17 '11 at 1:16

One way to test could be to generate an md5sum based on the concatenation of all of the files in the folder and its subfolders. Bear in mind that this also requires that the files have the same names (as they must be in the same sort order). The following code should work:

#!/bin/bash

shopt -s nullglob
shopt -s globstar || { printf '%s\n' 'Bash 4 is required for globstar.' ; exit 1 ; }
(( $# == 2 )) || { printf '%s\n' "Usage: ${0##*/} olddir newdir" ; exit 2 ; }

for _file in "$1"/**/*; do [[ -f ${_file} && ! -L ${_file} ]] && _files_in_old_dir+=( "${_file}" ); done
for _file in "$2"/**/*; do [[ -f ${_file} && ! -L ${_file} ]] && _files_in_new_dir+=( "${_file}" ); done

(( ${#_files_in_old_dir[@]} )) || { printf '%s\n' 'No files in old dir.' ; exit 3 ; }
(( ${#_files_in_new_dir[@]} )) || { printf '%s\n' 'No files in new dir.' ; exit 4 ; }

_md5_old_dir=$(cat "${_files_in_old_dir[@]}" | md5sum)
_md5_new_dir=$(cat "${_files_in_new_dir[@]}" | md5sum)

{ [[ ${_md5_old_dir} == "${_md5_new_dir}" ]] && (( ${#_files_in_old_dir[@]} == ${#_files_in_new_dir[@]} )) ; } && printf '%s\n' 'Folders are identical.' || { printf '%s\n' 'Folders are not identical.' ; exit 3 ; }

If you truly care about the file names, etc, you could use a loop to compare what is in ${_files_in_old_dir} and ${_files_in_new_dir}. This should work for most cases (it at least checks the number of files in the dir and its subdirectories).

share|improve this answer
    
This is a nice script...thx @Chris . But it have a problem...using cat, subfolders crashes with errors... Do you have a idea to solve it? Thx a lot. –  GarouDan Oct 17 '11 at 1:29
    
It works fine for me. Make sure your shell supports globstar. What is the error? –  Chris Down Oct 17 '11 at 1:59
1  
It "works" (+1)... but globstar in bash 4 follows directory symlinks, but that's only a problem if either directory contains a symlink. –  Peter.O Oct 17 '11 at 7:14
    
@fered Good call, I added in a test. –  Chris Down Oct 17 '11 at 11:04

I noticed the original post is rather old, however, I think this information can still be valuable to those looking for a solution to verify that files are copied correctly. Rsync might be the best method to copy data and answers given in this thread are good, however for those not that experienced with Linux, I will try to give a more detailed explanation.

Scenario: You just copied data from a disk to another, with lots of sub-directories and files. You want to verify that all the data is copied correctly.

First check that md5deep is installed by issuing the command md5deep -v.

If you get a message saying something like 'command not found', then install md5deep by apt-get install md5deep.

It's assumed you only want to deal with regular files. If you want to deal with other types of files, refer to the -o flag in the md5deep manual. (man md5deep)

Now you are good to go, and we assume that you copied files from /mnt/orginal to /mnt/backup, substitute these for any directories you are using.

First change to the source directory, this is the original source for the files you copied or backed up:

cd /mnt/orginal

Then make a checksum of each file:

md5deep -rel -o f . >> /tmp/checksums.md5

This command explained:

-r enables recursive mode

-e displays progress indicator

-l enables relative file paths.

-o f only work on regular files (not block devices, named pipes etc.)

. tells md5deep to start in the current directory.

>> /tmp/checksums.md5 tells md5deep to redirect all output to /tmp/checksums.md5.

Note, if you want to overwrite content in previous versions of /tmp/checksums.md5, use > and not >>

Note that this command could take quite a while, depending on the io-speed and the size of the data. You could experiment with nice and/or ionice to increase performance of md5deep, but that's outside the scope of this answer.

When the creation of the check sums has finished, you now have a file that has entries similar to:

69c0a826b29c8f40b7ca5e56e53d7f83 ./oldconfig-11-09-2013/etc2/apm/event.d/20hdparm 651f3c7f79a14332f9fa7bb368039210 ./oldconfig-11-09-2013/etc2/apm/event.d/anacron 50d89784c1e201f68ff978b95ff4bdfb ./oldconfig-11-09-2013/etc2/apm/scripts.d/alsa e9b9131660a8013983bc5e19d7d669eb ./oldconfig-11-09-2013/etc2/ld.so.cache

The first column is the md5 check sum, and the second column is the relative path to the file the checksum belongs to.

If you want to see how many files exists in the checksum file, issue the command:

wc /tmp/checksums.md5 -l

Now, you want to check that the copied data is correct:

cd /mnt/backup

md5deep -o f -reX /tmp/checksums.md5 . >> /tmp/compare.result

The only difference from when we created the checksums is -X which displays the current hash of a file if the entry in the checksums.md5 file does not match. So by the end of the test, if /tmp/compare.result is blank, you can trust that all the files are copied correctly since the checksums matches.

Note that only files listed in the /tmp/checksums.md5 file will be checked for a correct checksum, if there's any additional files in the /mnt/backup directory, md5deep will not notify you about these.

Notes:

  • You don't necessarily have to use redirection to store output files. Refer to the md5deep manual for further information.

  • You might have to run md5deep commands as root, depending on the permissions of the files you're handling.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.