Is there a faster alternative to cp for copying large files (~20 GB)?

Question

I am a graduate student, and the group in which I work maintains a Linux cluster. Each node of the cluster has its own local disk, but these local disks are relatively small and are not equipped with automatic backup. So the group owns a fileserver with many TBs of storage space. I am a relative Linux novice, so I am not sure what are the specs of the fileserver in terms of speed, networking ability, etc. I do know from experience that the local disks are significantly faster than the fileserver in terms of I/O. About a dozen or so people use the fileserver.

Using cp to copy a ~20 GB file from the fileserver to one of the local disks takes about 11.5 minutes in real time on average (according to time). I know that this cp operation is not very efficient because (1) time tells me that the system time for such a copy is only ~45 seconds; and because (2) when I examine top during the copy, %CPU is quite low (by inspection, roughly 0-10% on average).

Using cp to copy the same ~20 GB file from one folder on the local disk to another folder on the same local disk takes less time -- about 9 minutes in real time (~51 seconds in system time, according to time). So apparently the fileserver is somewhat slower than the local disk, as expected, but perhaps not significantly slower. I am surprised that copying from local to same local is not faster than 9 minutes.

I need to copy ~200 large files -- each ~20 GB -- from the fileserver to one of the local disks. So, my question is: Is there a faster alternative to cp for copying large files in Linux? (Or are there any flags within cp that I could use which would speed up copying?) Even if I could somehow shave a minute off this copying time, that would help immensely.

I am sure that buying new, faster hardware disks, but I don't have access to such resources. I am also not a system administrator -- I am only a (novice) user -- so I don't have access to more detailed information on the load that is on the disks. I do know that while about a dozen people use the fileserver daily, I am the only person using this particular node/local disk.

Answer 1

%CPU should be low during a copy. The CPU tells the disk controller "grab data from sectors X–Y into memory buffer at Z". Then it goes and does something else (or sleep, if there is nothing else). The hardware triggers an interrupt when the data is in memory. Then the CPU has to copy it a few times, and tells the network card "transmit packets at memory locations A, B, and C". Then it goes back to doing something else.

You're pushing ~240mbps. On a gigabit LAN, you ought to be able to do at least 800mbps, but:

1. That's shared among everyone using the file server (and possibly a connection between switches, etc.)
2. That's limited by the speed the file server can handle the write, keeping in mind its disk I/O bandwidth is shared by everyone using it.
3. You didn't specify how you're accessing the file server (NFS, CIFS (Samba), AFS, etc.). You may need to tune your network mount, but on anything half-recent the defaults are usually pretty sane.

For tracking down the bottleneck, iostat -kx 10 is going to be a useful command. It'll show you the utilization on your local hard disks. If you can run that on the file server, it'll tell you how busy the file server is.

The general solution is going to be to speed up that bottleneck, which of course you don't have the budget for. But, there are a couple of special cases where you can find a faster approach:

• If the files are compressible, and you have a fast CPU, doing a minimal compress on-the-fly might be quicker. Something like lzop or maybe gzip --fastest.
• If you are only changing a few bits here and there, and then sending the file back, only sending deltas will be much faster. Unfortunately, rsync won't really help here, as it will need to read the file on both sides to find the delta. Instead, you need something that keeps track of the delta as you change the file... Most approaches here are app-specific. But its possible that you could rig something up with, e.g., device-mapper (see the brand new dm-era target) or btrfs.
• If you're copying the same data to multiple machines, you can use something like udpcast to send it to all the machines at once.

And, since you note you're not the sysadmin, I'm guessing that means you have a sysadmin. Or at least someone responsible for the file server & network. You should probably ask him/her/them, they should be much more familiar with the specifics of your setup. Your sysadmin(s) should at least be able to tell you what transfer rate you can reasonably expect.

Answer 2

This could, possibly, be a faster alternative, and you won't clog the network for two days: Take one or two large USB (USB 3 if you have it) or FireWire disks, connect it to the server and copy the files to the disk. Carry the disk to your local machine. Copy the files to the machine.

Answer 3

If you have direct SSH (or SFTP) access (ask your sysadmin), you can use scp with compression (-C):

scp -C you@server:/path/to/yourfile .

Of course, that's only useful if the file is compressible, and this will use more CPU time, since it will be using encryption (because it's over SSH), and compressing.

Answer 4

The cp implementation is most likely not a bottleneck. Try to observe IO usage via iotop on both server and cluster node. This will give you an idea where you can improve the performance.

Another tip, is to avoid copying same data from same host. For example, if you have identical 20G file to distribute from fileserver over the network to all cluster nodes, it will work much faster to if you copy files in peer-to-peer fashion rather than one-server-to-all-clients. It's bit more complicated to implement, but you can even try to use some commandline p2p like direct connect hub.

If within that 20G files, some part is common, and some are cluster node specific, consider splitting it into common, and specific parts, and then distribute common part in p2p way.

Answer 5

Your definition of efficient is backwards. A more efficient implementation wastes less cpu time. On the local copy you are averaging about 74 MB/s of throughput ( read + write ), which is about as good as a single hard disk is going to get.

Answer 6

The nature / contents of those files can make some difference. I understood that you need to copy 200 files, ~20 GB each, from one computer to another, is that it ?

If those files are compressible or with similar / identical pieces , you have two approaches:

• zip them before copying, or create a tunnel between the computers with zip enable on it. So, if the network is the bottleneck, it will be a bit faster

• if the files are very similar, or share some pieces of common content among them, try using rsync. It'll spend some time finding what is common among the files, and won't need to copy it literally, because it'll reconstruct it based on what is common.

edit

Will you need to copy those files many times ?? (like a copy -> use those files -> change something in the files in the computer A -> copy files again to computer B)

If so, rsync will be helpfull, because it'll try to detect what is equal among the versions and don't copy what is unchanged.

And a third method: if the above is correct (changes in file, then copy all files again to the second computer) you could try some binary diff to just change in the second computer what was changed in the first computer.

Answer 7

I see the following here, encryption is not a good idea as it might possibly INCREASE the amount of data to be transferred.

If you are copying between two systems, then the bottleneck is of course the connection between the servers.

If you are copying locally, look at how the process goes, it is SINGLE threaded, thus standard Linux utilities use:

- for all blocks in a file
      read a block
      write a block

There is NO concurrency to this operation.

To speed things up you can use something like this:

  buffer -i infile -o outfile -m size-of-shared-memory-default-1MByte

See the buffer(1) man page for more information.

The buffer command sets up two processes to run the copy process concurrently: one for reading, and the other for writing, and it uses a shared memory buffer to communicate the data between the two processes. The shared memory buffer is your classic circular buffer which prevents overwrite of unwritten data and writing of data already written. I have used this program to cut off about 10-20% of copy time in transfers from disk to tape.

Answer 8

Why not try a P2P propagation algorithm, if you need to update your entire cluster at the same time?

https://github.com/lg/murder is what twitter uses

There's BTSync that you can try as well.

Answer 9

If you're copying the same sets of files frequently from your local computer to the server with minor changes here and there. You can speed up the transfer by using rsync or a DVCS (e.g. hg or git).

git or hg can keep track and detect deltas and only transfer those deltas. In case of using a git, since both sides have full history of the repository, figuring out the delta is very cheap.

rsync uses a form of rolling checksumming algorithm to detect deltas without prior knowledge of what's on the other side. While it takes more work for rsync to calculate the deltas, it does not need to store the whole file history.

Answer 10

You might want to try packaging all the files into a single archive (need not be compressed). In my experience, copying that one archive is faster than copying a large number of individual files

Answer 11

Make sure the target files don't exist before copying.

Sometimes it is surprising how much time is spent even just copying on the same host (no network involved).

See my answer to another cp question here. Long story short, overwriting an existing file is much slower than truncating it or unlinking it first, and then copying. The latter is 8x faster for a 1.2GB file.

Answer 12

Try bbcp. Testing in our environment revealed that cp had some sort of built in governer. Just be careful because when you take off the governer, you can red-line your server and cause an outage. In our case we were taking the server offline to do the copy, so faster was better. This improved transfer time several hours.