I am a graduate student, and the group in which I work maintains a Linux cluster. Each node of the cluster has its own local disk, but these local disks are relatively small and are not equipped with automatic backup. So the group owns a fileserver with many TBs of storage space. I am a relative Linux novice, so I am not sure what are the specs of the fileserver in terms of speed, networking ability, etc. I do know from experience that the local disks are significantly faster than the fileserver in terms of I/O. About a dozen or so people use the fileserver.
cp to copy a ~20 GB file from the fileserver to one of the local disks takes about 11.5 minutes in real time on average (according to
time). I know that this
cp operation is not very efficient because (1)
time tells me that the system time for such a copy is only ~45 seconds; and because (2) when I examine
top during the copy, %CPU is quite low (by inspection, roughly 0-10% on average).
cp to copy the same ~20 GB file from one folder on the local disk to another folder on the same local disk takes less time -- about 9 minutes in real time (~51 seconds in system time, according to
time). So apparently the fileserver is somewhat slower than the local disk, as expected, but perhaps not significantly slower. I am surprised that copying from local to same local is not faster than 9 minutes.
I need to copy ~200 large files -- each ~20 GB -- from the fileserver to one of the local disks. So, my question is: Is there a faster alternative to
cp for copying large files in Linux? (Or are there any flags within
cp that I could use which would speed up copying?) Even if I could somehow shave a minute off this copying time, that would help immensely.
I am sure that buying new, faster hardware disks, but I don't have access to such resources. I am also not a system administrator -- I am only a (novice) user -- so I don't have access to more detailed information on the load that is on the disks. I do know that while about a dozen people use the fileserver daily, I am the only person using this particular node/local disk.