How do I get the MD5 sum of a directory's contents as one sum?

Question

The md5sum program does not provide checksums for directories. I want to get a single MD5 checksum for the entire contents of a directory, including files in sub-directories. That is, one combined checksum made out of all the files. Is there a way to do this?

Answer 1

The right way depends on exactly why you're asking:

Option 1: Compare Data Only

If you just need a hash of the tree's file contents, this will do the trick:

$ find -s somedir -type f -exec md5sum {} \; | md5sum

This first summarizes all of the file contents individually, in a predictable order, then passes that list of file names and MD5 hashes to be hashed itself, giving a single value that only changes when the content of one of the files in the tree changes.

Unfortunately, find -s only works with BSD find(1), used in macOS, FreeBSD, NetBSD and OpenBSD. To get something comparable on a system with GNU or SUS find(1), you need something a bit uglier:

$ find somedir -type f -exec md5sum {} \; | sort -k 2 | md5sum

We've mimicked the behavior of BSD find -s by adding a call to sort. The -k 2 bit tells it to skip over the MD5 hash, so it only sorts the file names, which are in field 2 through end-of-line by sort's reckoning.

There's a weakness with this version of the command, which is that it's liable to become confused if you have any filenames with newlines in them, because it'll look like multiple lines to the sort call. The find -s variant doesn't have that problem, because the tree traversal and sorting happen within the same program, find.

In either case, the sorting is necessary to avoid false positives: the most common Unix/Linux filesystems don't maintain the directory listings in a stable, predictable order. You might not realize this from using ls and such, which silently sort the directory contents for you. Calling find without sorting its output in some way will cause the order of lines in the output to match whatever order the underlying filesystem returns them, which will cause this command to give a changed hash value if the order of files given to it as input changes, even if the data remain identical.

You may well ask whether the -k 2 bit in the GNU sort command above is necessary. Given that the hash of the file's data is an adequate proxy for the file's name as long as the contents have not changed, we will not get false positives if we drop this option, allowing us to use the same command with both GNU and BSD sort. However, realize that there is a small chance (1:2¹²⁸ with MD5) that the exact ordering of file names does not match the partial order that doing without -k 2 can give if there is ever a hash collision. Keep in mind, however, if such small chances of a mismatch matter to your application, this whole approach is probably out of the question for you.

You might need to change the md5sum commands to md5 or some other hash function. If you choose another hash function and need the second form of the command for your system, you might need to adjust the sort command accordingly. Another trap is that some data summing programs don't write out a file name at all, a prime example being the old Unix sum program.

This method is somewhat inefficient, calling md5sum N+1 times, where N is the number of files in the tree, but that's a necessary cost to avoid hashing file and directory metadata.

Option 2: Compare Data and Metadata

If you need to be able to detect that anything in a tree has changed, not just file contents, ask tar to pack the directory contents up for you, then send it to md5sum:

$ tar -cf - somedir | md5sum

Because tar also sees file permissions, ownership, etc., this will also detect changes to those things, not just changes to file contents.

This method is considerably faster, since it makes only one pass over the tree and runs the hash program only once.

As with the find based method above, tar is going to process file names in the order the underlying filesystem returns them. It may well be that in your application, you can be sure you won't cause this to happen. I can think of at least three different usage patterns where that is likely to be the case. (I'm not going to list them, because we're getting into unspecified behavior territory. Each filesystem can be different here, even from one version of the OS to the next.)

If you find yourself getting false positives, I'd recommend going with the find | cpio option in Gilles' answer.

Answer 2

The checksum needs to be of a deterministic and unambiguous representation of the files as a string. Deterministic means that if you put the same files at the same locations, you'll get the same result. Unambiguous means that two different sets of files have different representations.

Data and metadata

Making an archive containing the files is a good start. This is an unambiguous representation (obviously, since you can recover the files by extracting the archive). It may include file metadata such as dates and ownership. However, this isn't quite right yet: an archive is ambiguous, because its representation depends on the order in which the files are stored, and if applicable on the compression.

A solution is to sort the file names before archiving them. If your file names don't contain newlines, you can run find | sort to list them, and add them to the archive in this order. Take care to tell the archiver not to recurse into directories. Here are examples with POSIX pax, GNU tar and cpio:

find | LC_ALL=C sort | pax -w -d | md5sum
find | LC_ALL=C sort | tar -cf - -T - --no-recursion | md5sum
find | LC_ALL=C sort | cpio -o | md5sum

Names and contents only, the low-tech way

If you only want to take the file data into account and not metadata, you can make an archive that includes only the file contents, but there are no standard tools for that. Instead of including the file contents, you can include the hash of the files. If the file names contain no newlines, and there are only regular files and directories (no symbolic links or special files), this is fairly easy, but you do need to take care of a few things:

{ export LC_ALL=C;
  find -type f -exec wc -c {} \; | sort; echo;
  find -type f -exec md5sum {} + | sort; echo;
  find . -type d | sort; find . -type d | sort | md5sum;
} | md5sum

We include a directory listing in addition to the list of checksums, as otherwise empty directories would be invisible. The file list is sorted (in a specific, reproducible locale — thanks to Peter.O for reminding me of that). echo separates the two parts (without this, you could make some empty directories whose name look like md5sum output that could also pass for ordinary files). We also include a listing of file sizes, to avoid length-extension attacks.

By the way, MD5 is deprecated. If it's available, consider using SHA-2, or at least SHA-1.

Names and data, supporting newlines in names

Here is a variant of the code above that relies on GNU tools to separate the file names with null bytes. This allows file names to contain newlines. The GNU digest utilities quote special characters in their output, so there won't be ambiguous newlines.

{ export LC_ALL=C;
  du -0ab | sort -z; # file lengths, including directories (with length 0)
  echo | tr '\n' '\000'; # separator
  find -type f -exec sha256sum {} + | sort -z; # file hashes
  echo | tr '\n' '\000'; # separator
  echo "End of hashed data."; # End of input marker
} | sha256sum

A more robust approach

Here's a minimally tested Python script that builds a hash describing a hierarchy of files. It takes directories and file contents into accounts and ignores symbolic links and other files, and returns a fatal error if any file can't be read.

#! /usr/bin/env python
import hashlib, hmac, os, stat, sys
## Return the hash of the contents of the specified file, as a hex string
def file_hash(name):
    f = open(name)
    h = hashlib.sha256()
    while True:
        buf = f.read(16384)
        if len(buf) == 0: break
        h.update(buf)
    f.close()
    return h.hexdigest()
## Traverse the specified path and update the hash with a description of its
## name and contents
def traverse(h, path):
    rs = os.lstat(path)
    quoted_name = repr(path)
    if stat.S_ISDIR(rs.st_mode):
        h.update('dir ' + quoted_name + '\n')
        for entry in sorted(os.listdir(path)):
            traverse(h, os.path.join(path, entry))
    elif stat.S_ISREG(rs.st_mode):
        h.update('reg ' + quoted_name + ' ')
        h.update(str(rs.st_size) + ' ')
        h.update(file_hash(path) + '\n')
    else: pass # silently symlinks and other special files
h = hashlib.sha256()
for root in sys.argv[1:]: traverse(h, root)
h.update('end\n')
print h.hexdigest()

Answer 3

If your goal is just to find differences between two directories, consider using diff.

Try this:

diff -qr dir1 dir2

Answer 4

Have a look at md5deep. Some of the features of md5deep that may interest you:

Recursive operation - md5deep is able to recursive examine an entire directory tree. That is, compute the MD5 for every file in a directory and for every file in every subdirectory.

Comparison mode - md5deep can accept a list of known hashes and compare them to a set of input files. The program can display either those input files that match the list of known hashes or those that do not match.

...

Answer 5

Use checksumdir:

$ pip install checksumdir
$ checksumdir -a md5 assets/js
981ac0bc890de594a9f2f40e00f13872
$ checksumdir -a sha1 assets/js
88cd20f115e31a1e1ae381f7291d0c8cd3b92fad

Faster and easier than the other bash solutions.

Answer 6

You can hash every file recursively and then hash the resulting text:

> md5deep -r -l . | sort | md5sum
d43417958e47758c6405b5098f151074 *-

md5deep is required.

Answer 7

nix-hash from the Nix package manager

The command nix-hash computes the cryptographic hash of the contents of each path and prints it on standard output. By default, it computes an MD5 hash, but other hash algorithms are available as well. The hash is printed in hexadecimal.

The hash is computed over a serialisation of each path: a dump of the file system tree rooted at the path. This allows directories and symlinks to be hashed as well as regular files. The dump is in the NAR format produced by nix-store --dump. Thus, nix-hash path yields the same cryptographic hash as nix-store --dump path | md5sum.

Answer 8

I use this my snippet for moderate volumes:

find . -xdev -type f -print0 | LC_COLLATE=C sort -z | xargs -0 cat | md5sum -

and this one for XXXL:

find . -xdev -type f -print0 | LC_COLLATE=C sort -z | xargs -0 tail -qc100 | md5sum -

Answer 9

File contents only, excluding filenames

I needed a version that only checked the filenames because the contents reside in different directories.

This version (Warren Young's answer) helped a lot, but my version of md5sum outputs the filename (relative to the path I ran the command from), and the folder names were different, therefore even though the individual file checksums matched, the final checksum didn't.

To fix that, in my case, I just needed to strip off the filename from each line of the find output (select only the first word as separated by spaces using cut):

find -s somedir -type f -exec md5sum {} \; | cut -d" " -f1 | md5sum

Answer 10

As a follow-up to this excellent answer, if you find yourself wanting to speed up the calculation of the checksum for a large directory, try GNU Parallel:

find -s somedir -type f | parallel -k -n 100 md5 {} | md5

(This is using a Mac with md5, replace as needed.)

The -k flag is important, that instructs parallel to maintain order, otherwise the overall sum can change run to run even if the files are all the same. -n 100 says to run each instance of md5 with 100 arguments, this is a parameter you can tweak for best run time. See also -X flag of parallel (though in my personal case that caused an error.)

Answer 11

So far the fastest way to do it is still with tar. And with several additional parameters we can also get rid of the difference caused by metadata.

To use tar for hash the dir, one need to make sure you sort the path during tar, otherwise it is always different.

tar -C <root-dir> -cf - --sort=name <dir> | sha256sum

ignore time

If you do not care about the access time or modify time also use something like --mtime='UTC 2019-01-01' to make sure all timestamp is the same.

ignore ownership

Usually we need to add --group=0 --owner=0 --numeric-owner to unify the owner metadata.

ignore some files

use --exclude=PATTERN

Answer 12

A good tree check-sum is the tree-id of Git.

There is unfortunately no stand-alone tool available which can do that (at least I dont know it), but if you have Git handy you can just pretend to set up a new repository and add the files you want to check to the index.

This allows you to produce the (reproducible) tree hash - which includes only content, file names and some reduced file modes (executable).

Answer 13

A script which is well tested and supports a number of operations including finding duplicates, doing comparisons on both data and metadata, showing additions as well as changes and removals, you might like Fingerprint.

Fingerprint right now doesn't produce a single checksum for a directory, but a transcript file which includes checksums for all files in that directory.

fingerprint analyze

This will generate index.fingerprint in the current directory which includes checksums, filenames and file sizes. By default it uses both MD5 and SHA1.256.

In the future, I hope to add support for Merkle Trees into Fingerprint which will give you a single top-level checksum. Right now, you need to retain that file for doing verification.

Answer 14

A robust and clean approach

• First things first, don't hog the available memory! Hash a file in chunks rather than feeding the entire file.
• Different approaches for different needs/purpose (all of the below or pick what ever applies):
  • Hash only the entry name of all entries in the directory tree
  • Hash the file contents of all entries (leaving the meta like, inode number, ctime, atime, mtime, size, etc., you get the idea)
  • For a symbolic link, its content is the referent name. Hash it or choose to skip
  • Follow or not to follow(resolved name) the symlink while hashing the contents of the entry
  • If it's a directory, its contents are just directory entries. While traversing recursively they will be hashed eventually but should the directory entry names of that level be hashed to tag this directory? Helpful in use cases where the hash is required to identify a change quickly without having to traverse deeply to hash the contents. An example would be a file's name changes but the rest of the contents remain the same and they are all fairly large files
  • Handle large files well(again, mind the RAM)
  • Handle very deep directory trees (mind the open file descriptors)
  • Handle non standard file names
  • How to proceed with files that are sockets, pipes/FIFOs, block devices, char devices? Must hash them as well?
  • Don't update the access time of any entry while traversing because this will be a side effect and counter-productive(intuitive?) for certain use cases.

This is what I have on top my head, any one who has spent some time working on this practically would have caught other gotchas and corner cases.

Here's a tool(disclaimer: I'm a contributor to it) dtreetrawl, very light on memory, which addresses most cases, might be a bit rough around the edges but has been quite helpful.

Usage:
  dtreetrawl [OPTION...] "/trawl/me" [path2,...]

Help Options:
  -h, --help                Show help options

Application Options:
  -t, --terse               Produce a terse output; parsable.
  -d, --delim=:             Character or string delimiter/separator for terse output(default ':')
  -l, --max-level=N         Do not traverse tree beyond N level(s)
  --hash                    Hash the files to produce checksums(default is MD5).
  -c, --checksum=md5        Valid hashing algorithms: md5, sha1, sha256, sha512.
  -s, --hash-symlink        Include symbolic links' referent name while calculating the root checksum
  -R, --only-root-hash      Output only the root hash. Blank line if --hash is not set
  -N, --no-name-hash        Exclude path name while calculating the root checksum
  -F, --no-content-hash     Do not hash the contents of the file

An example human friendly output:

...
... //clipped
...
/home/lab/linux-4.14-rc8/CREDITS
        Base name                    : CREDITS
        Level                        : 1
        Type                         : regular file
        Referent name                :
        File size                    : 98443 bytes
        I-node number                : 290850
        No. directory entries        : 0
        Permission (octal)           : 0644
        Link count                   : 1
        Ownership                    : UID=0, GID=0
        Preferred I/O block size     : 4096 bytes
        Blocks allocated             : 200
        Last status change           : Tue, 21 Nov 17 21:28:18 +0530
        Last file access             : Thu, 28 Dec 17 00:53:27 +0530
        Last file modification       : Tue, 21 Nov 17 21:28:18 +0530
        Hash                         : 9f0312d130016d103aa5fc9d16a2437e

Stats for /home/lab/linux-4.14-rc8:
        Elapsed time     : 1.305767 s
        Start time       : Sun, 07 Jan 18 03:42:39 +0530
        Root hash        : 434e93111ad6f9335bb4954bc8f4eca4
        Hash type        : md5
        Depth            : 8
        Total,
                size           : 66850916 bytes
                entries        : 12484
                directories    : 763
                regular files  : 11715
                symlinks       : 6
                block devices  : 0
                char devices   : 0
                sockets        : 0
                FIFOs/pipes    : 0

Answer 15

I didn't want new executables nor clunky solutions so here's my take:

#!/bin/sh
# md5dir.sh by Camilo Martin, 2014-10-01.
# Give this a parameter and it will calculate an md5 of the directory's contents.
# It only takes into account file contents and paths relative to the directory's root.
# This means that two dirs with different names and locations can hash equally.

if [[ ! -d "$1" ]]; then
    echo "Usage: md5dir.sh <dir_name>"
    exit
fi

d="$(tr '\\' / <<< "$1" | tr -s / | sed 's-/$--')"
c=$((${#d} + 35))
find "$d" -type f -exec md5sum {} \; | cut -c 1-33,$c- | sort | md5sum | cut -c 1-32

Answer 16

Migration to POSIX archive format affects GNU Tar based checksums

This answer is intended to be a supplementary update to the approach of using Tar output to hash the contents of directories, as it was proposed (among other things) in the excellent answers of Warren Young and Gilles some time ago.

Since then, at least openSUSE (since its release 12.2) changed their default GNU Tar format from "GNU tar 1.13.x format" to the (slightly) superior "POSIX 1003.1-2001 (pax) format". Also upstream (among the developers of GNU Tar) they discuss to perform the same migration, see for example the last paragraph on this page of the GNU Tar manual:

The default format for GNU tar is defined at compilation time. You may check it by running tar --help, and examining the last lines of its output. Usually, GNU tar is configured to create archives in gnu format, however, future version will switch to posix.

(This page also gives a nice review on the different archive formats that are available with GNU Tar.)

In our case, where we tar the directory contents and hash the result, and without taking specific measures, a change from GNU to POSIX format has the following consequences:

• In spite of identical directory contents, the resulting checksum will be different.

• In spite of identical directory contents, the resulting checksum will be different from run to run if the default pax headers are used.

The latter comes from the fact, that the POSIX (pax) format includes extended pax headers which are determined by a format string that defaults to %d/PaxHeaders.%p/%f in GNU Tar. Within this string, the specifier %p is replaced by the process ID of the generating Tar process, which of course is different from run to run. See this section of the GNU Tar manual and in particular this one for details.

Just now, dating from 2019-03-28, there is a commit accepted upstream that defuses this issue.

So, to be able to continue using GNU Tar in the given use case, I can recommend the following alternative options:

• Use the Tar option --format=gnu to explicitly tell Tar to generate the archive in the "old" format. This is mandatory to validate "old" checksums.

• Use the newer POSIX format, but explicitly specify a suitable pax header, for example by --pax-option="exthdr.name=%d/PaxHeaders/%f". However, this breaks the backward compatibility to "old" checksums.

Here is a Bash code fragment that I use on a regular basis to compute checksums of directory contents including metadata:

( export LC_ALL=C
  find <paths> ! -type s -print0 |
  sort -z |
  tar cp --format=gnu --numeric-owner \
         --atime-preserve \
         --no-recursion --null --files-from - |
  md5sum --binary; )

Herein, <paths> is replaced by a space separated list of the paths of all directories that I want to be covered by the checksum. The purpose of using the C locale, the null byte separation of filenames, and of using find and sort to get a filesystem independent order of the files in the archive is already sufficiently discussed in other answers.

The surrounding parentheses keep the LC_ALL setting local in a subshell.

In addition, I use the expression ! -type s with find to avoid warnings from Tar that occur if socket files are part of the directory contents: GNU Tar does not archive sockets. If you prefer to be notified about skipped sockets, leave that expression away.

I use --numeric-owner with Tar, to be able to verify the checksums later even on systems, where not all of the file owners are known.

The --atime-preserve option for Tar is better omitted if any of the <paths> lies on a read-only mounted device. Otherwise you will be warned for each single file whose access timestamp Tar was not able to restore. For write enabled <paths>, I use this option, well, to preserve the access timestamps in the hashed directories.

The Tar option --no-recursion, which was already used in Gilles proposal, prevents Tar from recursively descent into directories by itself, and to operate instead file by file on whatever it gets fed from the sorted find output.

And finally, it is not true that I use md5sum: I actually use sha256sum.

Answer 17

If you have Python 3 with pip installed, you could use my pyfstools package for that. Quick usage:

$ pip install git+https://github.com/sorgloomer/pyfstools.git@main
...
$ python -m pyfstools hash --algo md5 .
dir 88c17b149c1d9fef50f642b698cef9e6

Answer 18

Doing individually for all files in each directory.

# Calculating
find dir1 | xargs md5sum > dir1.md5
find dir2 | xargs md5sum > dir2.md5
# Comparing (and showing the difference)
paste <(sort -k2 dir1.md5) <(sort -k2 dir2.md5) | awk '$1 != $3'

Answer 19

I use the following approach to determine if directory contents/attributes have changed:

cd /path/to/dir; ls -lnAR --time-style=+%s . | md5sum

If you also want to track the changes in extended attributes:

cd /path/to/dir; (ls -lnAR --time-style=+%s .; getfacl -Rns .) | md5sum

Under normal circumstances there is no need to check contents of the files because file modification time already indicates a change in the file contents. This also speeds up the entire operation.

Notes:

• cd /path/to/dir - makes checksum resistant to directory relocation
• -n - makes checksum persistent across systems with different uid/gid mapping
• -A - includes all files including dotted (like -a) except . and ..
• --time-style=+%s - without it the changes made within 1 minute after checksum has made may left untracked, this output is also persistent between rsync copies (unlike --full-time option)

If you still want to check files contents as well: tar c -C /path/to/dir | md5sum or with extended attributes: cd /path/to/dir; (tar c .; getfacl -Rns .) | md5sum

Update:

In order to make it work on wider range of systems I had to set locale and also filter out hardlink count and directory sizes from ls output with awk:

cd /path;LC_ALL=C ls -lnAR --time-style=+%s .|awk '{$2=0;$1~/^d/&&$5=0;print}'|md5sum

Answer 20

One-liner:

find directory -exec md5sum {} \; 2>&1 | sort -k 2 | md5sum

This lists all files and directories and gets md5sum for each. Then gets md5sum for everything.

Tricky bit solved here that md5sum is not capable to do the sum for a directory, but it tells this to us: md5sum: dir/sub_dir: Is a directory. We just move this message to a standard output.

Answer 21

If you dont need md5, you can try

find . -type f | xargs cksum | cksum