Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

The md5sum program does not provide checksums for directories. I want to get a single MD5 checksum for the entire contents of a directory, including files in sub-directories. That is, one combined checksum made out of all the files. Is there a way to do this?

share|improve this question

9 Answers 9

The right way depends on exactly why you're asking:

Option 1: Compare Data Only

If you just need a hash of the tree's file contents, this will do the trick:

$ find -s somedir -type f -exec md5sum {} \; | md5sum

This first summarizes all of the file contents individually, in a predictable order, then passes that list of file names and MD5 hashes to be hashed itself, giving a single value that should only change when the content of one of the files in the tree changes.

Unfortunately, find -s only works with BSD find(1), used in Mac OS X, FreeBSD, NetBSD and OpenBSD. To get something comparable on a system with GNU or SUS find(1), you need something a bit uglier:

$ find somedir -type f -exec md5sum {} \; | sort -k 34 | md5sum

We've replaced find -s with a call to sort. The -k 34 bit tells it to skip over the MD5 hash, so it only sorts the file names.

There's a weakness with this version of the command, which is that it's liable to become confused if you have any filenames with newlines in them, because it'll look like multiple lines to the sort call. The find -s variant doesn't have that problem, because the tree traversal and sorting happen within the same program, find.

In either case, the sorting is necessary to avoid false positives. *ix filesystems don't maintain the directory listings in a stable, predictable order; you might not realize this from using ls and such, which silently sort the directory contents for you. find without -s or a sort call is going to print out files in whatever order the underlying filesystem returns them, which could cause this command to give a changed hash value when all that's changed is the order of files in a directory.

You might need to change the md5sum commands to md5 or some other hash function. If you choose another hash function and need the second form of the command for your system, you might need to adjust the -k 34 part of the command. It would be -k 42 for sha1sum, for example.

Don't use the old Unix sum program for this. Its output doesn't include the file name, so it won't work properly with the external sorting in the second form of the command.

This method is somewhat inefficient, calling md5sum N+1 times, where N is the number of files in the tree, but that's a necessary cost to avoid hashing file and directory metadata.

Option 2: Compare Data and Metadata

If you need to be able to detect that anything in a tree has changed, not just file contents, ask tar to pack the directory contents up for you, then send it to md5sum:

$ tar -cf - somedir | md5sum

Because tar also sees file permissions, ownership, etc., this will also detect changes to those things, not just changes to file contents.

This method is considerably faster, since it makes only one pass over the tree and runs the hash program only once.

As with the find based method above, tar is going to process file names in the order the underlying filesystem returns them. It may well be that in your application, you can be sure you won't cause this to happen. I can think of at least three different usage patterns where that is likely to be the case. (I'm not going to list them, because we're getting into unspecified behavior territory. Each filesystem can be different here, even from one version of the OS to the next.)

If you find yourself getting false positives, I'd recommend going with the find | cpio option in Gilles' answer.

share|improve this answer
    
I think it is best to navigate to the directory being compared and use find . instead of find somedir. This way the file names are the same when providing different path-specs to find; this can be tricky :-) –  Abbafei Jun 24 at 6:50

The checksum needs to be of a deterministic and unambiguous representation of the files as a string. Deterministic means that if you put the same files at the same locations, you'll get the same result. Unambiguous means that two different sets of files have different representations.

Data and metadata

Making an archive containing the files is a good start. This is an unambiguous representation (obviously, since you can recover the files by extracting the archive). It may include file metadata such as dates and ownership. However, this isn't quite right yet: an archive is ambiguous, because its representation depends on the order in which the files are stored, and if applicable on the compression.

A solution is to sort the file names before archiving them. If your file names don't contain newlines, you can run find | sort to list them, and add them to the archive in this order. Take care to tell the archiver not to recurse into directories. Here are examples with POSIX pax, GNU tar and cpio:

find | LC_ALL=C sort | pax -w -d | md5sum
find | LC_ALL=C sort | tar -cf - -T - --no-recursion | md5sum
find | LC_ALL=C sort | cpio -o - | md5sum

Names and contents only, the low-tech way

If you only want to take the file data into account and not metadata, you can make an archive that includes only the file contents, but there are no standard tools for that. Instead of including the file contents, you can include the hash of the files. If the file names contain no newlines, and there are only regular files and directories (no symbolic links or special files), this is fairly easy, but you do need to take care of a few things:

{ export LC_ALL=C;
  find -type f -exec wc -c {} \; | sort; echo;
  find -type f -exec md5sum {} + | sort; echo;
  find . -type d | sort; find . -type d | sort | md5sum;
} | md5sum

We include a directory listing in addition to the list of checksums, as otherwise empty directories would be invisible. The file list is sorted (in a specific, reproducible locale — thanks to Peter.O for reminding me of that). echo separates the two parts (without this, you could make some empty directories whose name look like md5sum output that could also pass for ordinary files). We also include a listing of file sizes, to avoid length-extension attacks.

By the way, MD5 is deprecated. If it's available, consider using SHA-2, or at least SHA-1.

Names and data, supporting newlines in names

Here is a variant of the code above that relies on GNU tools to separate the file names with null bytes. This allows file names to contain newlines. The GNU digest utilities quote special characters in their output, so there won't be ambiguous newlines.

{ export LC_ALL=C;
  du -0ab | sort -z; # file lengths, including directories (with length 0)
  echo | tr '\n' '\000'; # separator
  find -type f -exec sha256sum {} + | sort -z; # file hashes
  echo | tr '\n' '\000'; # separator
  echo "End of hashed data."; # End of input marker
} | sha256sum

A more robust approach

Here's a minimally tested Python script that builds a hash describing a hierarchy of files. It takes directories and file contents into accounts and ignores symbolic links and other files, and returns a fatal error if any file can't be read.

#! /usr/bin/env python
import hashlib, hmac, os, stat, sys
## Return the hash of the contents of the specified file, as a hex string
def file_hash(name):
    f = open(name)
    h = hashlib.sha256()
    while True:
        buf = f.read(16384)
        if len(buf) == 0: break
        h.update(buf)
    f.close()
    return h.hexdigest()
## Traverse the specified path and update the hash with a description of its
## name and contents
def traverse(h, path):
    rs = os.lstat(path)
    quoted_name = repr(path)
    if stat.S_ISDIR(rs.st_mode):
        h.update('dir ' + quoted_name + '\n')
        for entry in sorted(os.listdir(path)):
            traverse(h, os.path.join(path, entry))
    elif stat.S_ISREG(rs.st_mode):
        h.update('reg ' + quoted_name + ' ')
        h.update(str(rs.st_size) + ' ')
        h.update(file_hash(path) + '\n')
    else: pass # silently symlinks and other special files
h = hashlib.sha256()
for root in sys.argv[1:]: traverse(h, root)
h.update('end\n')
print h.hexdigest()
share|improve this answer
    
OK, this works, thanks. But is there any way to do it without including any metadata? Right now I need it for just the actual contents. –  user17429 Apr 6 '12 at 1:12
    
How about LC_ALL=C sort for checking from different environments...(+1 btw) –  Peter.O Apr 6 '12 at 6:16
    
You made a whole Python program for this? Thanks! This is really more than what I had expected. :-) Anyway, I will check these methods as well as the new option 1 by Warren. –  user17429 Apr 6 '12 at 17:33

Best solution is this one offered above.

However, a super quick & sloppy solution:

find <path_to_directory>  -type f -print0 | xargs -0 cat | md5sum

See comments below - this method is not as reliable as it could be, because it doesn't discern shifting of bytes from the end of one file to the beginning of another, and relies on stable ordering of find's output (which on Linux at least, seems to be stable, but may not always be - especially if you need something cross platform).

share|improve this answer
    
Perfect. Thanks! :-) –  user17429 Apr 5 '12 at 20:01
3  
This doesn't work for many reasons. It allows collisions: it can't distinguish between a file that contains foobar followed by a file that contains qux and a file that contains foo followed by a file that contains barqux. Furthermore, it doesn't take file names and locations into account. And it produces different checksums depending on the order in which find lists the file names. –  Gilles Apr 6 '12 at 0:27
    
Hmm.... Problem. –  user17429 Apr 6 '12 at 1:01
    
Does find not always list filenames in ASCIIbetical order? –  user1024 Apr 6 '12 at 3:42
    
@user1024 No, it lists files in the order it finds them, which on most filesystems is completely arbitrary. –  Gilles Apr 6 '12 at 7:13

Have a look at md5deep. Some of the features of md5deep that may interest you:

Recursive operation - md5deep is able to recursive examine an entire directory tree. That is, compute the MD5 for every file in a directory and for every file in every subdirectory.

Comparison mode - md5deep can accept a list of known hashes and compare them to a set of input files. The program can display either those input files that match the list of known hashes or those that do not match.

...

share|improve this answer
    
Nice, but can't get it to work, it says .../foo: Is a directory, what gives? –  Camilo Martin Oct 2 at 1:21

If your goal is just to find differences between two directories, consider using diff.

Try this:

diff -qr dir1 dir2
share|improve this answer
    
Yes, this is useful as well. I think you meant dir1 dir2 in that command. –  user17429 Apr 6 '12 at 17:35
    
I don't usually use GUIs when I can avoid them, but for directory diffing kdiff3 is great and also works on many platforms. –  sinelaw Apr 17 '12 at 2:21
    
Differing files are reported as well with this command. –  Serge Stroobandt Apr 2 at 15:02

A good tree check-sum is the tree-id of Git.

There is unfortunately no stand-alone tool available which can do that (at least I dont know it), but if you have Git handy you can just pretend to set up a new repository and add the files you want to check to the index.

This allows you to produce the (reproducible) tree hash - which includes only content, file names and some reduced file modes (executable).

share|improve this answer

I use this my snippet for moderate volumes:

find . -xdev -type f -print0 | LC_COLLATE=C sort -z | xargs -0 cat | md5sum -

and this one for XXXL:

find . -xdev -type f -print0 | LC_COLLATE=C sort -z | xargs -0 tail -qc100 | md5sum -

share|improve this answer

File contents only, excluding filenames

I needed a version that only checked the filenames because the contents reside in different directories.

This version (Warren Young's answer) helped a lot, but my version of md5sum outputs the filename (relative to the path I ran the command from), and the folder names were different, therefore even though the individual file checksums matched, the final checksum didn't.

To fix that, in my case, I just needed to strip off the filename from each line of the find output (select only the first word as separated by spaces using cut):

find -s somedir -type f -exec md5sum {} \; | cut -d" " -f1 | md5sum
share|improve this answer

I didn't want new executables nor clunky solutions so here's my take:

#!/bin/sh
# md5dir.sh by Camilo Martin, 2014-10-01.
# Give this a parameter and it will calculate an md5 of the directory's contents.
# It only takes into account file contents and paths relative to the directory's root.
# This means that two dirs with different names and locations can hash equally.

if [[ ! -d "$1" ]]; then
    echo "Usage: md5dir.sh <dir_name>"
    exit
fi

d="$(tr '\\' / <<< "$1" | tr -s / | sed 's-/$--')"
c=$((${#d} + 35))
find "$d" -type f -exec md5sum {} \; | cut -c 1-33,$c- | sort | md5sum | cut -c 1-32

Hope it helps you :)

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.