2

Is there a way in any linux distro to perform a md5sum or sha1 check while the file is being transferred from a local partition to an NFS one?

Example:

I have a NFS mounted drive and a very big file on my local drive. I'd like to transfer that file to the NFS mounted drive and at the same time, do a md5 check. I found a lot of scripts that would do the md5 after the file was copied on the remote drive but since it's a very big file 100GB+ I'd like to know if there's a way to take advantage of the fact that the file is already being red while being transferred.

  • Just to be sure: A) you do have the md5sum for the file already stored somewhere; B) you don't care about the file arriving correctly over the network. – Anthon Oct 18 '13 at 8:26
  • @Anthon A) The md5sum is stored inside the folder structure which contains the big file(s) – Oktav Oct 18 '13 at 9:08
  • rsync always does this when it copies files but it doesn't tell you what the checksum is, or verify it against a prepared checksums file. – mr.spuratic Jul 2 '14 at 17:41
5

I am not aware of a standard Linux utility that can do this. If the file would fit in your memory based cache it would not be so inefficient to first do the md5sum and then copy (the copy would get the data from memory).

You can use the following combination:

cat filename | tee remote_output_name | md5sum

which you might be able to amend to directly check the sum printed by md5sum against the stored md5. This reads the file from disc only once.

Assuming you have generated a check.md5 with

cd dir_with_big_files
md5sum * > check.md5

, the following Python program would do the copying and checking for a single file, reading/writing in 64Mb at a time. Save it as /usr/local/bin/chkcopy, chmod +x /usr/local/chkcopyand call it with chkcopy file_name check.md5 destination_filename_or_dir

#! /usr/bin/env python

import sys, os, hashlib

m = hashlib.md5()
file_name = sys.argv[1]
md5_name = sys.argv[2]
out_name = sys.argv[3]
if os.path.isdir(out_name):
    out_name = os.path.join(out_name, file_name)
BUF_SIZE = 64 * (1024 ** 2)

with open(file_name, 'rb') as ifp:
    with open(out_name, 'wb') as ofp:
        buf = ifp.read(BUF_SIZE)
        while buf:
            m.update(buf)
            ofp.write(buf)
            buf = ifp.read(BUF_SIZE)
with open(md5_name) as fp:
    for line in fp:
        md5, fn = line.rstrip().split('  ', 1)
        if fn == file_name:
            assert m.hexdigest() == md5
            break
    else:
        print('no md5 found for ' + file_name)
  • Any idea about whether the use of tee instead of cp/dd/rsync will noticeably affect the speed of large copies? I googled my way here because I thought of the same trick to save time on verification and can't think of any other reason to use it. – ndemou Jun 30 '16 at 8:47
1

You can use external program (crcsum), which extends cp and mv with checksum:

https://sourceforge.net/projects/crcsum/

  • 1
    Welcome to Unix & Linux! Whilst this may theoretically answer the question, it would be preferable to include the essential parts of the answer here, and provide the link for reference. – slm Oct 7 '14 at 18:24
  • there seeems to be little documentations which hash-types are supported by this tool. the linked sf-project page now also lists securecopytools which may contain the crccp and crcmv utilities (not tested/verified). – antiplex Feb 12 '18 at 8:36
0

There exists a fork of the well-known dd with extended functionality called dcfldd that I've been using for years or a patched dd-version called dc3dd with somewhat similar functionality.

Both tools can perform hashing (even with multiple hash-types simultaneously if wanted) during copying. Hashes can be calculated over chunks and/or the whole data-steam.

some distros such as debian directly offer packages in their repositories, packages for fedora are available through the external cert-repositories for example.

To copy a file in 8MiB-chunks and calculating the whole data's MD5sum which is printed to STDERR:

dcfldd if=/path/to/input bs=8M hash=md5 of=/path/to/outputfile

To copy a file in 8MiB-chunks calculating the whole data's SHA256-hash plus the SHA256sum over each 64MiB-block:

dcfldd if=/path/to/input bs=8M hash=SHA256 hashwindow=64M of=/path/to/outputfile

An output-file for the calculated hash can also be supplied by specifying a file through the hashlog parameter. When calculating multiple hashes, separate outputs can be specified through e.g. md5log=FILE1.log sha256log=FILE2.log .

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.