I am a graduate student of computational chemistry with access to a Linux cluster. The cluster consists of a very large (25 TB) fileserver, to which several dozen compute nodes are connected. Each compute node consists of 8 to 24 Intel Xeon cores. Each compute node also contains a local disk of about 365 TB.
Since the fileserver is routinely accessed by a dozen or so users in the research group, the fileserver is mainly used for long term file storage (it is backed up nightly, whereas the compute nodes' local disks are never backed up). Thus, the system administrator has instructed us to run simulations on the local disks -- which have faster I/O than the fileserver -- so as to not slow down the fileserver for the other users.
So, I run simulations on the local disks and then, after they are finished, I copy the trajectory files -- I am running molecular dynamics (MD) simulations -- to the fileserver for storage. Suppose I have a trajectory file called
traj.trr in a directory on the local disk of a node,
/home/myusername/mysimulation1/traj.trr. For long term storage, I always copy
traj.trr to a directory in the fileserver,
~ represents my directory in the fileserver,
/export/home/myusername. After copying it, then I habitually use
du -h to verify that
/home/myusername/mysimulation1/traj.trr has the same file size as
~/mysimulation1/traj.trr. This way, I can be at least reasonably sure that the transfer to the fileserver was successful. For example:
cd /home/myusername/mysimulation1/ cp -v traj.trr ~/mysimulation1/ du /home/myusername/mysimulation1/traj.trr -h du ~/mysimulation1/traj.trr -h
If the two calls to
du -h give the same human-readable file size, then I can be reasonably sure that the transfer/copy was successful. (My typical
traj.trr files range in size from about 15 to 20 GB, depending on the exact simulation I have run.) If I run
du (i.e., without the
-h switch) on the two
traj.trr files, their sizes in bytes are usually very, very similar -- usually within just a few bytes. I have been using this overall method for the past year and a half, with no problems.
However, recently I have run into the following problem: sometimes
du -h reports that the two
traj.trr files are different in size by several GB. Here is an example:
cd /home/myusername/mysimulation1/ # this is the local disk cp -v traj.trr ~/mysimulation1/ du traj.trr -h cd ~/mysimulation1/ # this is the fileserver du traj.trr -h
The output from the two calls to
du -h is as follows, respectively:
20G traj.trr 28G traj.trr
I believe that the former (i.e., the
traj.trr in the local disk,
/home/myusername/mysimulation1/) is the correct file size, since my simulation trajectories are expected to be about 15 to 20 GB each. But then how could the file on the fileserver actually be larger? I could see how it could be smaller, if somehow the
cp transfer failed. But I don't see how it could actually be larger.
I get similar output when I execute the same commands as above, but without the
-h switch given to
20717480 traj.trr 28666688 traj.trr
Can you think of any reason for the difference?
If, by some unlikely chance,
du is somehow malfunctioning, I can be okay with that. But I just really need to make sure that the copy of
traj.trr on the fileserver is complete and identical to its source version on the local disk. I need to delete the local file so that I have enough local disk space to run new simulations, but I can't afford to have the version of
traj.trr on the fileserver be corrupted.
The .trr file format (from the Gromacs molecular dynamics package) is a binary format, not text. Thus, I am not sure if the files can be reliably compared by a program such as