I am currently copying a large number of directories and files recursively on the same disk using cp -r
.
Is there a way to do this more quickly? Would compressing the files first be better, or maybe using rsync
?
I was recently puzzled by the sometimes slow speed of cp
. Specifically, how come df = pandas.read_hdf('file1', 'df')
(700ms for a 1.2GB file) followed by df.to_hdf('file2')
(530ms) could be so much faster than cp file1 file2
(8s)?
Digging into this:
cat file1 > file2
isn't any better (8.1s).dd bs=1500000000 if=file1 of=file2
neither (8.3s).rsync file1 file2
is worse (11.4s), because file2 existed already so it tries to do its rolling checksum and block update magic.Oh, wait a second! How about unlinking (deleting) file2
first if it exists?
Now we are talking:
rm -f file2
: 0.2s (to add to any figure below).cp file1 file2
: 1.0s.cat file1 > file2
: 1.0s.dd bs=1500000000 if=file1 of=file2
: 1.2s.rsync file1 file2
: 4s.So there you have it. Make sure the target files don't exist (or truncate them, which is presumably what pandas.to_hdf()
does).
Edit: this was without emptying the cache before any of the commands, but as noted in the comments, doing so just consistently adds ~3.8s to all numbers above.
Also noteworthy: this was tried on various Linux versions (Centos w. 2.6.18-408.el5 kernel, and Ubuntu w. 3.13.0-77-generic kernel), and ext4 as well as ext3. Interestingly, on a MacBook with Darwin 10.12.6, there is no difference and both versions (with or without existing file at the destination) are fast.
sudo sh -c 'sync && echo 3 > /proc/sys/vm/drop_caches'
) before every command. Just adding ~3.8s to all the numbers above. The delta between cp
to an existing file vs to an non-existent destination is as above: ~7s.
– Pierre D
Jul 9 '18 at 4:02
On the same partition (and filesystem) you can use -l
to achieve hard links instead of copies. Hard link creation is much faster than copying things (but, of course, does not work across different disk partitions).
As a small example:
$ time cp -r mydir mydira
real 0m1.999s
user 0m0.000s
sys 0m0.490s
$ time cp -rl mydir mydirb
real 0m0.072s
user 0m0.000s
sys 0m0.007s
That's a 28 times improvement. But that test used only ~300 (rather small) files. A couple of bigger files should perform faster, a lot of smaller files slower.
Copying a file on the local disk is 99% spent in reading and writing to the disk. If you try to compress data then you increase CPU load but don't reduce the read/write data... it will actually slow down your copy.
rsync will help if you already have a copy of the data and bring it "up to date".
But if you want to create a brand new copy of a tree then you can't really do much better than your cp
command.
iostat
while this copy operation is running, you might get more help from readers. Assuming you're running on Solaris from the/solaris
tag, post several lines fromiostat -sndzx 2
. That will emit an output line every 2 seconds, with the first line being not very useful. Again, that needs to be run while yourcp -r ...
command is running. – Andrew Henle Aug 27 '16 at 11:30