How can I copy and verify a file as I do the copy?

Question

(Yet another "copy and verify" question...)

I have a reasonable amount of data I want to copy between several HDDs, and I want to make absolutely sure the data is copied correctly.

My current copying approach has been fairly standard, copying then verifying (with sha256). I don't like the extra steps and time involved - 30-60 minutes to checksum is enough flow disruption that I do have to work harder making sure I haven't missed any steps and verified everything correctly.

So, I am specifically looking for a single-step copy-and-verify process.

However, I don't want to reread the copied file out of the disk cache, as that would negate the point of verification!!

I'm prepared to write my own tool for the job, if absolutely needed, but I would absolutely love to hear of someone who echoes the views here and who has already written a tool like what I'm describing.

On Linux I also know I can kill the disk cache with echo 3 > /proc/sys/vm/drop_caches. But doing this every few kilobytes or even megabytes is absolutely certain to kill I/O performance. This specific technique is also not generally portable, so I don't really want to file it away as my solution to this problem.

I know fsync(file); fdatasync(file); fsync(dir); fdatasync(dir) is probably a good starting point (overkill is good here), but I fear it'll kill sequential streaming throughput?

My current idea is to do the read/write copy while I also have the target file open 256MB behind where the read/write is up to, so it's well outside the range of the HDD's onboard controller and cache RAM, and then sequentially read and verify from that $copypoint-256MB location.

Could I do some fancy footwork with O_SYNC | O_DIRECT and get both acceptable speed and good guarantees that my read-back is getting data off the disk platters?

Or... is write-then-verify just the best way to deal with the very sub-optimal status quo that is "modern" operating systems, HDD controllers and good safe file I/O design? :/

Answer 1

You could cat the source file and pipe it to tee to save it to the destination, and simultaneously pipe the output to sha256sum. That way you don't have to re-read the source to compute its hash. If you want to make sure the destination is re-read from the disk instead of cache, you can use dd with the iflag=direct option.