I am currently copying a large number of directories and files recursively on the same disk using
Is there a way to do this more quickly? Would compressing the files first be better, or maybe using
I was recently puzzled by the sometimes slow speed of
cp. Specifically, how come
df = pandas.read_hdf('file1', 'df') (700ms for a 1.2GB file) followed by
df.to_hdf('file2') (530ms) could be so much faster than
cp file1 file2 (8s)?
Digging into this:
cat file1 > file2isn't any better (8.1s).
dd bs=1500000000 if=file1 of=file2neither (8.3s).
rsync file1 file2is worse (11.4s), because file2 existed already so it tries to do its rolling checksum and block update magic.
Oh, wait a second! How about unlinking (deleting)
file2 first if it exists?
Now we are talking:
rm -f file2: 0.2s (to add to any figure below).
cp file1 file2: 1.0s.
cat file1 > file2: 1.0s.
dd bs=1500000000 if=file1 of=file2: 1.2s.
rsync file1 file2: 4s.
So there you have it. Make sure the target files don't exist (or truncate them, which is presumably what
Edit: this was without emptying the cache before any of the commands, but as noted in the comments, doing so just consistently adds ~3.8s to all numbers above.
Also noteworthy: this was tried on various Linux versions (Centos w. 2.6.18-408.el5 kernel, and Ubuntu w. 3.13.0-77-generic kernel), and ext4 as well as ext3. Interestingly, on a MacBook with Darwin 10.12.6, there is no difference and both versions (with or without existing file at the destination) are fast.
On the same partition (and filesystem) you can use
-l to achieve hard links instead of copies. Hard link creation is much faster than copying things (but, of course, does not work across different disk partitions).
As a small example:
$ time cp -r mydir mydira real 0m1.999s user 0m0.000s sys 0m0.490s $ time cp -rl mydir mydirb real 0m0.072s user 0m0.000s sys 0m0.007s
That's a 28 times improvement. But that test used only ~300 (rather small) files. A couple of bigger files should perform faster, a lot of smaller files slower.
Copying a file on the local disk is 99% spent in reading and writing to the disk. If you try to compress data then you increase CPU load but don't reduce the read/write data... it will actually slow down your copy.
rsync will help if you already have a copy of the data and bring it "up to date".
But if you want to create a brand new copy of a tree then you can't really do much better than your