5

Background

I am about to migrate files from my old NAS to a new one, and want to verify the data integrity. The old NAS (Debian) is using Linux Ext3 file system, whilst the new one (FreeNAS) is based on ZFS. To speed up the integrity validation I am trying to use the triage approach:

  • first validate all file sizes
  • secondly md5 hash the first 512 bytes of each file
  • lastly md5 hash entire file

The idea being that the first two steps would filter out obviously corrupted files, and be much quicker to detect than running md5 in bulk for TB of files.

Question

I have constructed a bash command for performing a md5 hash of a directory structure, and sorting the output based on file name to ensure a deterministic order on my Linux NAS.

#find somedir -type f -exec md5sum {} \; | sort -k 34;
12e761f96223145aa63f4f48f252d7fb  /somedir/foo.txt
18409feb00b6519c891c751fe2541fdc  /somedir/bar.txt

But how to modify above if I want to md5 only the first 512 bytes of each file?

| improve this question | | | | |
  • This strategy is counterproductive unless you expect a significant number of corrupted files. Just calculate all the hashes, it'll be faster. – Gilles 'SO- stop being evil' Feb 29 '16 at 22:37
6

You can use dd to pipe only the first 512 bytes to md5sum. However this will cause md5sum to be oblivious of the filename, so in addition replace - with the filename again.

find . -type f -exec sh -c "dd if={} bs=512 count=1 2>/dev/null | md5sum | sed s\|-\|{}\|" \; | sort -k 34;
| improve this answer | | | | |
  • Tried it out and it works as intended. Thanks! – Jimmy1969 Feb 29 '16 at 7:01
  • This command fails with files having special characters. – shivams May 11 at 18:02
1

The accepted answer did not work for me. The {} inside the exec was failing for files with special characters. Hence, I instead used GNU parallel (install its latest version to make the following solution work).

Using parallel, the {} didn't misbehave for files with special characters. Also, parallel made the whole process much faster as it distributes the processes across multiple cores. Here is what worked for me using parallel:

find . -type f | parallel 'dd if={} bs=512 count=1 2>/dev/null | md5sum | tr -d "\n"; echo {};' | sort -k34;
| improve this answer | | | | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.