When I connect the external hard drives to my computer(with FreeBSD or other Unix systems) and copy files from the first external hard drive to the second hard drive, are the files on the second hard drive the same as the files from the source (first external hard drive)?

I know there is a hash (checksum). I read somewhere that copying from different volumes will result in different files(since they are 2 different volumes).

Only when I copy a file to the same volume, I can guarantee it is the same file.

What is the recommendation for copying, and will my files stay the same?

  • 2
    After the copy you have two different files, but the contents should be the same, apart from I/O errors. You can run a checksum on both to verify. – Gerard H. Pille May 21 at 13:18
  • 4
    If I buy two copies of "War and Peace", are they the same book? Yes -- they have identical words, typeface, paper, bindings, etc. No -- I can burn one and keep the other. I can annotate one and never even open the covers of the other. "The same" is too broad a condition: ask a pair of twins for an answer. – Paul_Pedant May 21 at 13:18
  • 1
    What do you mean by "the same"? If we are talking about the streams of bits they must be. Otherwise they are not the same. – Artem S. Tashkinov May 21 at 13:20
  • 2
    If they are not the same data content (even considering disk-to-tape and disk-to-optical copies, and download from repositories), then 50 years of nightly backups were all wasted, and every distro is unreliable. Of course there are observable differences: inode numbers, block numbers, even sparse blocks, but the logical contents are independent of the physical mechanisms. Can you identify your source which asserted "copying from different volumes will result in different files" ? – Paul_Pedant May 21 at 13:58
  • 1
    Because you have the "forensics" tag, I'll point out that you're copying the data, not the metadata. cp has some options to help, but the best forensic move is to start with a bit-for-bit copy of the whole volume, with dd or similar. – waltinator May 21 at 16:36

When a copy of a file is made, the copy is exactly the same as the original (assuming that there were no errors during the copy). This is true whether the file is copied to another location on the same device or to a different device.

The copy might have a different filename, or maybe a different timestamp or different permissions, but the contents are identical.

This can be verified by running some sort of checksum or hashing algorithm (e.g. md5sum) on the original and the copy.

For example:

$ cp original /tmp/thecopy
$ md5sum original /tmp/thecopy 
93d9d61139ff5f1287764f1c1994cbe3  original
93d9d61139ff5f1287764f1c1994cbe3  /tmp/thecopy

Both files have the exact same md5sum. original was stored on an NVME. /tmp/ is a ramdisk.

It is technically possible for two different files to have the same md5sum. The probability of that happening is extremely low. md5sum is mostly "good enough" for many simple purposes, but most people use & recommend stronger hashing methods these days to reduce the probability even further. Here's what sha512sum has to say about the same-ness of the files.

$ sha512sum original /tmp/thecopy 
5ba61d6f2a883c3afebc949b0f0d0a1c020498a1052771de98e6e1bbb42d438a0a53f49f381a2e1311c1bdf82a0cea9de646fc03c529fcb6fca0ab6476badf35  original
5ba61d6f2a883c3afebc949b0f0d0a1c020498a1052771de98e6e1bbb42d438a0a53f49f381a2e1311c1bdf82a0cea9de646fc03c529fcb6fca0ab6476badf35  /tmp/thecopy

again, identical.

  • 2
    You might want cp -p to maintain permissions, ownerships, and timestamps. – roaima May 21 at 13:39
  • @roaima i'm already giving the copy a different filename, differences in other metadata don't affect the fact that the original and the copy are identical. – cas May 21 at 13:42
  • btw, original was just a copy of /bin/true to begin with :-) – cas May 21 at 13:43
  • 1
    @johnf yes, it is a perfect clone. You misunderstood Gerard H. Pille's comment. The others seem to be waxing philosophical on the nature of identity. – cas May 21 at 13:54
  • 1
    The algorithms are NOT the original file or the copy. They are tools for verifying that the files are identical. The tools are 99.99999999+ % accurate, not 100%. There is always a tiny chance that they get it wrong. You can use cmp to do a comparison of every single byte in the files - if it detects any difference, it will say so. There's still a tiny chance of a HW fault, or a cosmic ray hitting the wrong bit of non-ECC memory at the wrong time causing a false result. Nothing's 100% reliable or perfect. For all practical purposes, cmp or one of the SHA* algorithms is good enough. – cas May 21 at 14:37

What is the recommendation for copying, and will my files stay the same?

cp -a will copy the data but it will not copy the creation time, and you may lose the last access time. If your source system uses SeLinux context, the cp the other system which doesn't know about SeLinux will not copy it.

As for the crtime please refer to Copying or restoring crtime for files/directories on ext4fs filesystem

Lastly, running md5sum right after copying especially for small files is not advisable as files at the time could not have physically been dumped to the storage and the kernel will instead work with dirty buffers which are stored in RAM.

In order to be sure you've copied everything precisely you need to:

  • Copy all the files
  • Create their checksums (if your CPU is modern enough sha256sum is more reliable and faster than md5sum)
  • Drop caches echo 3 | sudo tee /proc/sys/vm/drop_caches
  • Create checksums for newly copied files and compare them to the old ones.

Alternatively for ext2/ext3/ext4 filesystems you could use e2image which copies everything (all the time stamps, including crtime and last access times) except free space. I normally use it for copying partitions.

For NTFS I've used ntfsclone on multiple occasions - it's fast and reliable.

I presume there are similar utilities for other filesystems but I only use ext4 and NTFS on my devices.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.