How can I best copy large numbers of small files over scp?

Question

I have a directory that's got several gigabytes and several thousand small files. I want to copy it over the network with scp more than once. CPU time on the source and destination machines is cheap, but the network overhead added by copying each file individually is huge. I would tar/gzip it up and ship it over, but the source machine is short on disk.

Is there a way for me to pipe the output of tar -czf <output> <directory> to scp? If not, is there another easy solution? My source machine is ancient (SunOS) so I'd rather not go installing things on it.

Answer 1

You can pipe tar across an ssh session:

$ tar czf - <files> | ssh user@host "cd /wherever && tar xvzf -"

Answer 2

Tar with bzip2 compression should take as much load off the network and on the cpu.

$ tar -C /path/to/src/dir -jcf - ./ | ssh user@server 'tar -C /path/to/dest/dir -jxf -'

Not using -v because screen output might slow down the process. But if you want a verbose output use it on the local side of tar (-jcvf), not on the remote part.

If you repeatedly copy over the same destination path, like updating a backup copy, your best choice is rsync with compression.

$ rsync -az -e ssh /path/to/src/dir/ user@server:/path/to/dest/dir/

Notice that both src and dest paths end with a /. Again, not using -v and -P flags on purpose, add them if you need verbose output.

Answer 3

use rsync, it uses SSH.

Usage:

rsync -aPz /source/path destination.server:remote/path

The rsync switches care about compression and I-Node information. -P displays progress of every file.

You can use scp -C, which enables compression, but if possible, use rsync.

Answer 4

@pdo's answer is good, but one can increase the speed with a buffer and good compression and add a progress bar.

Often the network is the bottleneck and the speed varies over time. Therefore, it helps to buffer the data before sending them over the network. This can be done with with pv.

Additionally, one can usually increase the speed with a proper compression algorithm. Gzip (like used above) is a fast compression algorithm, but in general zstandard (zstd) (and for high compression ratios LZMA/LZMA2 (xz) will compress better and be faster the same time. New xz and zstd have multi core support already built in. To use gzip with multiple cores pigz can be used.

Here is an example to send data with a progress bar, buffering and zstandard compression over a network:

tar cf - . | pv -perabs $(du -sk . | cut -f 1)K | zstd -14 --long=31 -T0 | pv -qCB 512M | ssh user@host "cd /wherever && pv -qCB 512M | zstd -cd -T0 --long=31 | tar xf -"

The first pv is to show the progress (p), estimated time (e), transfer rate (r), average rate (a), total transferred bytes (b). The total size is estimated with du and added to the size option (s). The progress is measured before compression and buffering, therefore it's not very accurate, but still helpful.

zstd is used with the compression setting 14. This number can be reduced or increased depending on the network and CPU speed so zstd is a little bit faster than the the network speed. With four cores on a Haswell 3.2 GHz CPU 14 gives a speed of around 120 MB/s. In the example, the long mode 31 (uses a 2 GB window, needs a lot of RAM, but very good e.g. to compress database dumps) is used. The T0 options sets the amount of threads to the number of cores. One should be aware that together with the long mode these settings use a lot of memory.

A problem with zstd is that most operating systems don't ship with version >= 1.3.4. This version is necessary for proper multi core and long support. If not available, it can be compiled and installed from https://github.com/facebook/zstd with just make -j4 && sudo make install. Instead of zstd, one can also use xz or pigz. xz is slow but compresses very well (good over slow connections), pigz/gzip is fast but compresses not so well. pv is then used again, but for buffering (q for quiet, C for the no splice mode [always needed for buffering] and B to set the buffer size).

In the example a buffer is also used on the receiver side. This is often unnecessary (because decompression and hard disk writing speed is most time higher than the network speed), but does usually not harm, either.

Answer 5

You can run tar on both ends using ssh. scp is part of the ssh family of goodness, so you probably have it on both ends.

 8:03AM 12 % tar cf - some_directory | ssh dest_host "tar xf -"

There may be a way to work gzip or bzip2 into the pipeline to lessen the network traffic, too.

Answer 6

If you have gzip on both ends: sourcehost$ cd sourcedir && tar cf - . | gzip -c - | ssh user@destinationhost "cd destinationdir && gzip -c -d | tar xf -"

If you don't have gzip on the source machine, make sure you have uncompress on the destination: sourcehost$ cd sourcedir && tar cf - . | compress | ssh user@destinationhost "cd destdir && uncompress | tar xf -"

This would be faster than first zipping it up, then sending, then unzipping, and it requires no extra disk space on either side. I sikpped the compression (z) flag on tar, because you probably dont have it on the ancient side.

Answer 7

Or you can do it the other way around if you need to. That is pull the tarball over the network rather than push it like has been suggested. This doesn't solve the repeating part of your question and rsync is best for that but there is probably tar switches to help.

So on the local machine:

ssh remote 'tar zcf - /etc/resolv.conf' | tar zxf -

Best to be in the right directory first or you have to use the -C switch on the untaring command at the end.

Just mentioning this in case this is needed. It is for me as in my situation my local server is behind nat, so would take some network futzing about to be able to do it the way that's previously been mentioned.

HTH

Answer 8

Or mount the remote filesystem via sshfs

sshfs user@remotehost:/path/on/remote /path/on/local

Answer 9

While not the most elegant, especially since it's not copying a single zip or tar file and doubly so as it doesn't help to reduce network ovehead, my only choice was to use scp -r:

-r

Recursively copy entire directories.  Note that scp follows symbolic links encountered in the tree traversal.

^{Source: scp(1)}

I was running into issues with running out of disk space with a 30 GB zipped tar file. I thought gunzip could do it inline, i.e., removing the original as it was being unzipped (and I may have missed a Google result) but I couldn’t find anything.

Finally, because I was tired of trying multiple times waiting for a new TAR or ZIP file to be finished tar'ing or zipping I finally just did:

1. From original server/PC/laptop, navigate to the directory where your folder with numerous files/folders are.
2. scp -r source_folder_name yourname@yourservername:destination_folder_name

Then just grab some beer, coffee or popcorn and wait. Good thing is, scp will retry if the network connection "stalls". Just hope it doesn’t go down completely.

Answer 10

I have destination that needs PEM to connect and it also has different SSH port. This worked for me like charm:

For e.g. Copy folder (and subfolders) from sourceserver to targetserver, run this on sourceserver

$ rsync -azu -e 'ssh -i ./pem_for_target_server.pem -p <port_number_for_target_server>' /folder/path/on/source username_on_target@target.server.name:/path/on/target/where/you/want/to/copy/folder/on/source/

(In case you get permissions are too open for pem file, run this: chmod 400 ./pem_for_target_server.pem)