du gives two different results for the same file

Question

I am a graduate student of computational chemistry with access to a Linux cluster. The cluster consists of a very large (25 TB) fileserver, to which several dozen compute nodes are connected. Each compute node consists of 8 to 24 Intel Xeon cores. Each compute node also contains a local disk of about 365 TB.

Since the fileserver is routinely accessed by a dozen or so users in the research group, the fileserver is mainly used for long term file storage (it is backed up nightly, whereas the compute nodes' local disks are never backed up). Thus, the system administrator has instructed us to run simulations on the local disks -- which have faster I/O than the fileserver -- so as to not slow down the fileserver for the other users.

So, I run simulations on the local disks and then, after they are finished, I copy the trajectory files -- I am running molecular dynamics (MD) simulations -- to the fileserver for storage. Suppose I have a trajectory file called traj.trr in a directory on the local disk of a node, /home/myusername/mysimulation1/traj.trr. For long term storage, I always copy traj.trr to a directory in the fileserver, ~/mysimulation1/traj.trr, where ~ represents my directory in the fileserver, /export/home/myusername. After copying it, then I habitually use du -h to verify that /home/myusername/mysimulation1/traj.trr has the same file size as ~/mysimulation1/traj.trr. This way, I can be at least reasonably sure that the transfer to the fileserver was successful. For example:

cd /home/myusername/mysimulation1/
cp -v traj.trr ~/mysimulation1/
du /home/myusername/mysimulation1/traj.trr -h
du ~/mysimulation1/traj.trr -h

If the two calls to du -h give the same human-readable file size, then I can be reasonably sure that the transfer/copy was successful. (My typical traj.trr files range in size from about 15 to 20 GB, depending on the exact simulation I have run.) If I run du (i.e., without the -h switch) on the two traj.trr files, their sizes in bytes are usually very, very similar -- usually within just a few bytes. I have been using this overall method for the past year and a half, with no problems.

However, recently I have run into the following problem: sometimes du -h reports that the two traj.trr files are different in size by several GB. Here is an example:

cd /home/myusername/mysimulation1/            # this is the local disk
cp -v traj.trr ~/mysimulation1/
du traj.trr -h
cd ~/mysimulation1/                           # this is the fileserver
du traj.trr -h

The output from the two calls to du -h is as follows, respectively:

20G     traj.trr
28G     traj.trr

I believe that the former (i.e., the traj.trr in the local disk, /home/myusername/mysimulation1/) is the correct file size, since my simulation trajectories are expected to be about 15 to 20 GB each. But then how could the file on the fileserver actually be larger? I could see how it could be smaller, if somehow the cp transfer failed. But I don't see how it could actually be larger.

I get similar output when I execute the same commands as above, but without the -h switch given to du:

20717480        traj.trr
28666688        traj.trr

Can you think of any reason for the difference?

If, by some unlikely chance, du is somehow malfunctioning, I can be okay with that. But I just really need to make sure that the copy of traj.trr on the fileserver is complete and identical to its source version on the local disk. I need to delete the local file so that I have enough local disk space to run new simulations, but I can't afford to have the version of traj.trr on the fileserver be corrupted.

The .trr file format (from the Gromacs molecular dynamics package) is a binary format, not text. Thus, I am not sure if the files can be reliably compared by a program such as diff.

Answer 1

You really should use something like md5sum or sha1sum to check integrity.

If you really want to use the size use ls -l or du -b.

The du utility normally only shows the disk usage of the file, i.e. how much of the file system is used by it. This value totally depends on the backing file system and other factors like sparse files.

Example:

$ truncate -s 512M foo
$ cat foo >bar
$ ls -l foo bar
-rw-r--r-- 1 michas users 536870912 23. Dez 00:06 bar
-rw-r--r-- 1 michas users 536870912 23. Dez 00:03 foo
$ du foo bar
0       foo
524288  bar
$ du -b foo bar
536870912       foo
536870912       bar

We have two files both containing 512MB of zeros. The first one is stored sparse and does not use any disk space, while the second stores each byte explicitly on disk. -- Same file, but completely different disk usage.

The -b option might be good for you:

   -b, --bytes
          equivalent to '--apparent-size --block-size=1'

   --apparent-size
          print apparent sizes, rather than disk usage; although the apparent
          size is  usually  smaller,  it  may  be  larger  due  to  holes  in
          ('sparse')  files, internal fragmentation, indirect blocks, and the
          like

Answer 2

This is a common problem when you put the same data on 2 different HDDs. You'll want to run the du command with and additional switch, assuming it has it - which it should given these are Linux nodes.

The switch?

   --apparent-size
          print  apparent  sizes,  rather  than  disk  usage;  although the 
          apparent size is usually smaller, it may be larger due to holes in
          ('sparse') files, internal fragmentation, indirect blocks, and the 
          like

Example

$ du -sh --apparent-size /home/sam/scsconfig.log ~/scsconfig.log 
93K /home/sam/scsconfig.log
93K /root/scsconfig.log

The above filesystems are a local disk (/root) while the other /home/sam is a NFS share from my NAS.

$ df -h . /home/sam
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      222G  118G   92G  57% /
mulder:/export/raid1/home/sam
                      917G  566G  305G  65% /home/sam

So what's up?

This confuses a lot of people but remember that when files are stored to a disk they consume blocks of space even if they're only using a portion of those blocks. When you run du without the --apparent-size you're getting the size based on the amount of disk's block space used, not the actual space consumed by the file(s).

using a checksum instead?

This is likely a better option if you're concerned about comparing 2 trees of files. You can use this command to calculate a checksum for all the files, and then calculate a final checksum of checksums. This example uses sha1sum but you could just as easily use md5sum instead.

$ cd /some/dir
$ find . -type f \( -exec sha1sum "{}" \; \) | sort -k2,2 | sha1sum

Example

$ cd ~/dir1
$ find . -type f \( -exec sha1sum "{}" \; \) | sort -k2,2 | sha1sum
55e2672f8d6fccff6d83f0bffba1b67aeab87911  -

$ cd ~/dir2
$ find . -type f \( -exec sha1sum "{}" \; \) | sort -k2,2 | sha1sum
55e2672f8d6fccff6d83f0bffba1b67aeab87911  -

So we can see that the 2 trees are identical.

(Note: find command will list files as they appeared in the file system. So, if you are comparing two directories from the different file system (e.g. Ext3 vs. APFS), you need to sort first before the final sha1sum. (added by Xianjun Dong)

Answer 3

The short answer: don't test the file size, test the return status of the command. The return status the only a reliable indication of whether the copy succeeded (short of comparing the two files byte by byte, directly of indirectly — which is redundant if the copy succeeded).

Checking the file size is not a very useful way of checking whether a copy succeeded. In some cases, it may be a useful sanity check, for example when you download a file from the web. But here there's a better way.

All Unix commands return a status to indicate whether they succeeded: 0 for success, 1 or more for errors. So check the exit status of cp. cp will normally have printed an error message if it failed, indicating what the error is. In a script, the exit status of the last command is in the magic variable $?.

cp -v traj.trr ~/mysimulation1/
if [ $? -ne 0 ]; then
  echo 1>&2 "cp failed due to the error above"
  exit 2
 fi

Instead of checking whether $? is zero, you can use boolean operators.

cp -v traj.trr ~/mysimulation1/ || exit 2

If you're running a script and want the script to stop if any command fails, run set -e. If any command fails (i.e. returns a non-zero status), the script will exit immediately with the same status as the command.

set -e
…
cp -v traj.trr ~/mysimulation1/

---

As for the reason your copied file was larger, it must be because it was a sparse file. Sparse file are a crude form of compression where blocks containing only null bytes are not stored. When you copy a file, the cp command reads and writes null bytes, so where the original had missing blocks, the copy has blocks full of null bytes. Under Linux, the cp command tries to detect sparse files, but it doesn't always succeed; cp --sparse=always makes it try harder at the expense of a very slight increase in CPU time.

More generally, du could return different results due to other forms of compression. Compressed filesystems are rare, though. If you want to know the size of a file as in the number of bytes in the file, as opposed to the number of disk blocks it uses, use ls -l instead of du.