The md5sum program does not provide checksums for directories. I want to get a single MD5 checksum for the entire contents of a directory, including files in sub-directories. That is, one combined checksum made out of all the files. Is there a way to do this?
The right way depends on exactly why you're asking:
Option 1: Compare Data Only
If you just need a hash of the tree's file contents, this will do the trick:
This first summarizes all of the file contents individually, in a predictable order, then passes that list of file names and MD5 hashes to be hashed itself, giving a single value that should only change when the content of one of the files in the tree changes.
There's a weakness with this version of the command, which is that it's liable to become confused if you have any filenames with newlines in them, because it'll look like multiple lines to the
In either case, the sorting is necessary to avoid false positives. *ix filesystems don't maintain the directory listings in a stable, predictable order; you might not realize this from using
You might need to change the
Don't use the old Unix
This method is somewhat inefficient, calling
Option 2: Compare Data and Metadata
If you need to be able to detect that anything in a tree has changed, not just file contents, ask
This method is considerably faster, since it makes only one pass over the tree and runs the hash program only once.
As with the
If you find yourself getting false positives, I'd recommend going with the
The checksum needs to be of a deterministic and unambiguous representation of the files as a string. Deterministic means that if you put the same files at the same locations, you'll get the same result. Unambiguous means that two different sets of files have different representations.
Data and metadata
Making an archive containing the files is a good start. This is an unambiguous representation (obviously, since you can recover the files by extracting the archive). It may include file metadata such as dates and ownership. However, this isn't quite right yet: an archive is ambiguous, because its representation depends on the order in which the files are stored, and if applicable on the compression.
A solution is to sort the file names before archiving them. If your file names don't contain newlines, you can run
Names and contents only, the low-tech way
If you only want to take the file data into account and not metadata, you can make an archive that includes only the file contents, but there are no standard tools for that. Instead of including the file contents, you can include the hash of the files. If the file names contain no newlines, and there are only regular files and directories (no symbolic links or special files), this is fairly easy, but you do need to take care of a few things:
We include a directory listing in addition to the list of checksums, as otherwise empty directories would be invisible. The file list is sorted (in a specific, reproducible locale — thanks to Peter.O for reminding me of that).
By the way, MD5 is deprecated. If it's available, consider using SHA-2, or at least SHA-1.
Names and data, supporting newlines in names
Here is a variant of the code above that relies on GNU tools to separate the file names with null bytes. This allows file names to contain newlines. The GNU digest utilities quote special characters in their output, so there won't be ambiguous newlines.
A more robust approach
Here's a minimally tested Python script that builds a hash describing a hierarchy of files. It takes directories and file contents into accounts and ignores symbolic links and other files, and returns a fatal error if any file can't be read.
Have a look at md5deep. Some of the features of md5deep that may interest you:
Best solution is this one offered above.
However, a super quick & sloppy solution:
See comments below - this method is not as reliable as it could be, because it doesn't discern shifting of bytes from the end of one file to the beginning of another, and relies on stable ordering of
If your goal is just to find differences between two directories, consider using diff.
I use this my snippet for moderate volumes:
and this one for XXXL:
File contents only, excluding filenames
I needed a version that only checked the filenames because the contents reside in different directories.
This version (Warren Young's answer) helped a lot, but my version of
To fix that, in my case, I just needed to strip off the filename from each line of the
A good tree check-sum is the tree-id of Git.
There is unfortunately no stand-alone tool available which can do that (at least I dont know it), but if you have Git handy you can just pretend to set up a new repository and add the files you want to check to the index.
This allows you to produce the (reproducible) tree hash - which includes only content, file names and some reduced file modes (executable).
I didn't want new executables nor clunky solutions so here's my take:
Hope it helps you :)