88

I have a directory that's got several gigabytes and several thousand small files. I want to copy it over the network with scp more than once. CPU time on the source and destination machines is cheap, but the network overhead added by copying each file individually is huge. I would tar/gzip it up and ship it over, but the source machine is short on disk.

Is there a way for me to pipe the output of tar -czf <output> <directory> to scp? If not, is there another easy solution? My source machine is ancient (SunOS) so I'd rather not go installing things on it.

13 Answers 13

159

You can pipe tar across an ssh session:

$ tar czf - <files> | ssh user@host "cd /wherever && tar xvzf -"
15
  • 7
    +1 tar-pipe solution. If you have more bandwidth and less CPU you can remove the compression flag (although gzip is pretty lightweight).
    – snakehiss
    Mar 24, 2011 at 21:32
  • 3
    And you could drop the compression flag and instead activate it in SSH (ssh -C or Compression yes in ~/.ssh/config). Mar 25, 2011 at 1:06
  • 4
    Never thought of using tar like this. Well, thats why I come here! Mar 25, 2011 at 3:13
  • 4
    This command could be made slightly shorter: $ tar cz <files> | ssh user@host "cd /wherever; tar xvz"
    – carlodef
    Jul 28, 2015 at 18:14
  • 5
    When doing this keep in mind that using cd like this is not entirely safe and if it fails the files will be dumped in user's home dir. Using the -C option as described in @forcefsk's answer is safer. Alternatively use && instead of ;
    – Bram
    Jul 21, 2016 at 10:55
29

Tar with bzip2 compression should take as much load off the network and on the cpu.

$ tar -C /path/to/src/dir -jcf - ./ | ssh user@server 'tar -C /path/to/dest/dir -jxf -'

Not using -v because screen output might slow down the process. But if you want a verbose output use it on the local side of tar (-jcvf), not on the remote part.

If you repeatedly copy over the same destination path, like updating a backup copy, your best choice is rsync with compression.

$ rsync -az -e ssh /path/to/src/dir/ user@server:/path/to/dest/dir/

Notice that both src and dest paths end with a /. Again, not using -v and -P flags on purpose, add them if you need verbose output.

1
  • rsync command doesn't work for me. Just kept on waiting forever.
    – Atul
    Apr 16, 2020 at 6:31
27

use rsync, it uses SSH.

Usage:

rsync -aPz /source/path destination.server:remote/path

The rsync switches care about compression and I-Node information. -P displays progress of every file.

You can use scp -C, which enables compression, but if possible, use rsync.

2
  • Unfortunately, rsync isn't available on the source machine, and neither is sshd.
    – nmichaels
    Mar 24, 2011 at 15:12
  • 2
    sshd isn't necessary for those operations on the client machine.
    – polemon
    Mar 25, 2011 at 5:05
11

@pdo's answer is good, but one can increase the speed with a buffer and good compression and add a progress bar.

Often the network is the bottleneck and the speed varies over time. Therefore, it helps to buffer the data before sending them over the network. This can be done with with pv.

Additionally, one can usually increase the speed with a proper compression algorithm. Gzip (like used above) is a fast compression algorithm, but in general zstandard (zstd) (and for high compression ratios LZMA/LZMA2 (xz) will compress better and be faster the same time. New xz and zstd have multi core support already built in. To use gzip with multiple cores pigz can be used.

Here is an example to send data with a progress bar, buffering and zstandard compression over a network:

tar cf - . | pv -perabs $(du -sk . | cut -f 1)K | zstd -14 --long=31 -T0 | pv -qCB 512M | ssh user@host "cd /wherever && pv -qCB 512M | zstd -cd -T0 --long=31 | tar xf -"

The first pv is to show the progress (p), estimated time (e), transfer rate (r), average rate (a), total transferred bytes (b). The total size is estimated with du and added to the size option (s). The progress is measured before compression and buffering, therefore it's not very accurate, but still helpful.

zstd is used with the compression setting 14. This number can be reduced or increased depending on the network and CPU speed so zstd is a little bit faster than the the network speed. With four cores on a Haswell 3.2 GHz CPU 14 gives a speed of around 120 MB/s. In the example, the long mode 31 (uses a 2 GB window, needs a lot of RAM, but very good e.g. to compress database dumps) is used. The T0 options sets the amount of threads to the number of cores. One should be aware that together with the long mode these settings use a lot of memory.

A problem with zstd is that most operating systems don't ship with version >= 1.3.4. This version is necessary for proper multi core and long support. If not available, it can be compiled and installed from https://github.com/facebook/zstd with just make -j4 && sudo make install. Instead of zstd, one can also use xz or pigz. xz is slow but compresses very well (good over slow connections), pigz/gzip is fast but compresses not so well. pv is then used again, but for buffering (q for quiet, C for the no splice mode [always needed for buffering] and B to set the buffer size).

In the example a buffer is also used on the receiver side. This is often unnecessary (because decompression and hard disk writing speed is most time higher than the network speed), but does usually not harm, either.

5

Or you can do it the other way around if you need to. That is pull the tarball over the network rather than push it like has been suggested. This doesn't solve the repeating part of your question and rsync is best for that but there is probably tar switches to help.

So on the local machine:

ssh remote 'tar zcf - /etc/resolv.conf' | tar zxf -

Best to be in the right directory first or you have to use the -C switch on the untaring command at the end.

Just mentioning this in case this is needed. It is for me as in my situation my local server is behind nat, so would take some network futzing about to be able to do it the way that's previously been mentioned.

1
  • Shorter version: ssh remote 'tar zc /etc/resolv.conf' | tar xz. Remove both z if you don’t want to compress (e.g. with images).
    – bfontaine
    Oct 18, 2022 at 14:17
3

You can run tar on both ends using ssh. scp is part of the ssh family of goodness, so you probably have it on both ends.

 8:03AM 12 % tar cf - some_directory | ssh dest_host "tar xf -"

There may be a way to work gzip or bzip2 into the pipeline to lessen the network traffic, too.

3

If you have gzip on both ends: sourcehost$ cd sourcedir && tar cf - . | gzip -c - | ssh user@destinationhost "cd destinationdir && gzip -c -d | tar xf -"

If you don't have gzip on the source machine, make sure you have uncompress on the destination: sourcehost$ cd sourcedir && tar cf - . | compress | ssh user@destinationhost "cd destdir && uncompress | tar xf -"

This would be faster than first zipping it up, then sending, then unzipping, and it requires no extra disk space on either side. I sikpped the compression (z) flag on tar, because you probably dont have it on the ancient side.

1

Or mount the remote filesystem via sshfs

sshfs user@remotehost:/path/on/remote /path/on/local
0
1

While not the most elegant, especially since it's not copying a single zip or tar file and doubly so as it doesn't help to reduce network ovehead, my only choice was to use scp -r:

-r

      Recursively copy entire directories.  Note that scp follows symbolic links encountered in the tree traversal.
Source: scp(1)

I was running into issues with running out of disk space with a 30 GB zipped tar file. I thought gunzip could do it inline, i.e., removing the original as it was being unzipped (and I may have missed a Google result) but I couldn’t find anything.

Finally, because I was tired of trying multiple times waiting for a new TAR or ZIP file to be finished tar'ing or zipping I finally just did:

  1. From original server/PC/laptop, navigate to the directory where your folder with numerous files/folders are.
  2. scp -r source_folder_name yourname@yourservername:destination_folder_name

Then just grab some beer, coffee or popcorn and wait. Good thing is, scp will retry if the network connection "stalls". Just hope it doesn’t go down completely.

3
  • OK, this clearly takes less of your time than typing a thousand scp commands.  But the question asks about “network overhead”.  Does your solution use the network any less than copying each file individually?  Is your solution superior in any way to the seven that have already been posted? Feb 16, 2018 at 3:54
  • Snap, my bad - I totally missed the network overhead part - thanks for pointing that out @G-Man. I updated the answer, I still feel it might be useful if someone stumbles upon a similar problem like I had and when I stumbled upon this question.
    – JGlass
    Feb 16, 2018 at 15:08
  • 1
    Good thing is, scp will retry if the network connection "stalls"? Well. Even better thing is **rsync --partial ** continuing at a later point of time, as often as you need. Feb 26, 2022 at 11:04
1

This answer is almost 12 years old, however I am using ssh transfer a lot and I am going to share some experiences.

  1. It is possible to use scp for transfer whole directory, however it has disadvantages compare to rsync. I was using scp for short time before i replaced scp with rsync. Package rsync should be almost at every Linux computer or server.

  2. rsync work for synchronize whole directory with sub-directories. It can be used for synchronize two directories at different places on one computer-server or synchronize two directories, one directory is local and second directory at remote host. Source or destination can be remote or local (local = at server or computer, where you have accounts).

Command Structure is:

rsync -ahv -e ssh --delete --progress --bwlimit=2048k RemoteUser@RemoteHost:/Address_Of_Source_Directory/ /Address_Of_Destination_Directory

Address (full path) of source or target directory begin always with /

RemoteUser@RemoteHost: before path is for remote source-destination. This is not used for local source-destination.

Now lets look to commands:

rsync -ahv -e ssh --delete --progress --bwlimit=2048k Source_Full_Path Target_Full_Path

-e ssh it means that you use ssh protocol for transfering from-to remote host.

--progress showing, how rsync command is running (file by file)

--delete files and directories, that are at destination, but not at source, will be deleted.

--bwlimit=2048k it is limit of transfer speed in kB, to avoid traffic jam of internet network. Value can be changed and this do not need to be in command.

To make sure, that content of source and destination is the same, you can run rsync firs with -ahv (it test files and directories by date of change) and second time with -ahvc, it makes checksum for every file at source and destination (probably sha256sum), and you are sure that content of all files is the same. It can take longer time with plenty of files or many many gigabites of data.

rsync works good for lot of files, however you can compress whole directory with directory structure. Tar is not good if names of files are not only alphanumeric. Better is 7z (p7zip). Go to directory with data and then start 7zip. Best compression results are probably:

7z a -t7z -m0=lzma2 -mqs -mx=9 -mfb=273 -md=64m -ms=on -mmt=2 Name_of_archive.7z

-mqs it means that files in row are sorted by type (txt jpg pdf etc), not by directory structure and names. It can get better results.

-md=64m Something like size of blocks, dictationary. 64m is enough (64MB). You can add more, but it takes at least 11-12 times more RAM than size of blocks.

-mmt=2 numer of CPU treads. Use -mmt=off for one tread (if you have lot of small files) or -mmt=2. If you use more that 2 treads, it will takes more RAM. 7z takes ram something like this (numer of pairs of CPU treads * block-dictatonary size)*(around 11-12 times) or if you have lot of files (milions), it takes few more GiB of RAM.

1
  • 2
    Welcome to U&L. Your answer is a bit difficult to read. Could you please edit the question and add the code brackets { } around command, textual output etc. This preserves the output in its original format, which is essential for understanding it correctly, and keeps the answer readable. Another thing to help readability is using unnumbered lists, for example on your clarifications of the options. Feb 25, 2023 at 13:53
0

I have destination that needs PEM to connect and it also has different SSH port. This worked for me like charm:

For e.g. Copy folder (and subfolders) from sourceserver to targetserver, run this on sourceserver

$ rsync -azu -e 'ssh -i ./pem_for_target_server.pem -p <port_number_for_target_server>' /folder/path/on/source [email protected]:/path/on/target/where/you/want/to/copy/folder/on/source/

(In case you get permissions are too open for pem file, run this: chmod 400 ./pem_for_target_server.pem)

1
  • You know you can set the permissions on the files with rsync --chmod=F400, right?
    – Kusalananda
    Feb 26, 2022 at 11:23
0

These is what I got after having read all above answers:

tarscp_from_remote() {
    if [ "$#" -ne 3 ]; then echo "Usage: tarscp_from_remote {remote_host} {remote_dir_path} {local_dir_path}" >&2; return; fi
    ssh $1 "tar -C $2 -cf - ./" | tar -C $3 -xf -
}
tarscp_to_remote() {
    if [ "$#" -ne 3 ]; then echo "Usage: tarscp_to_remote {local_dir_path} {remote_host} {remote_dir_path}" >&2; return; fi
    tar -C $1 -cf - ./ | ssh $2 "tar -C $3 -xf -"
}

And the version with compression (xz):

tarscp_from_remote() {
    if [ "$#" -ne 3 ]; then echo "Usage: tarscp_from_remote {remote_host} {remote_dir_path} {local_dir_path}" >&2; return; fi
    ssh $1 "tar -C $2 -Jcf - ./" | tar -C $3 -Jxf -
}
tarscp_to_remote() {
    if [ "$#" -ne 3 ]; then echo "Usage: tarscp_to_remote {local_dir_path} {remote_host} {remote_dir_path}" >&2; return; fi
    tar -C $1 -Jcf - ./ | ssh $2 "tar -C $3 -Jxf -"
}
0

DIRECTORY you wrote - this is overlooked by many good answers. YES, you can copy a distinct directory including subsequent content with tar (NOT tampering with owners and rights in the directory):

  • become root if necessary
  • FROM: copy dir "gereric_horse" without path ("/mnt/piglet/")
  • TO: into directory "/mnt/moon/"
  • no need to change into a particular dir first

Example:

tar --one-file-system --atime-preserve --preserve-permissions --exclude-caches --exclude='.Trash*' -cf - -C /mnt/piglet/ gereric_horse | (cd /mnt/moon/ && tar --atime-preserve --preserve-permissions -xvf - ) 

The ( ) means: Subshell. Better results.

This way you copy a directory with tar. Works fine ;-)
BTW: rsync should be considered as well


I was hinted the "remote" part is missing; admitted ;-)

Here it comes as an example you may adopt to fit your env:
tar from local to remote

[*part before pipe*]    |  ssh  -l user4ssh  2a03:2260:6873:d1ff:4bf9:c9c4:d80d:5bde  'cd /home/targetman/Downloads/ && tar --atime-preserve --preserve-permissions  -xvf - '

Please take care to have matching access rights
No need for additional subshell "()" here,
neither "[" around IP address;
necessary to have single quotation marks around the "cd..." term.
SSH format needs separated target-user option behind the "-l"
The ip address is anonymized but the entire format is taken from a real working example.

Here comes the other way round:
local invoked tar from remote to local (w/o separating dir stuff)

sudo ssh remoteuser@2a03:2345:6873:d1ff:4bf9:c9c4:d80d:5bde  sudo -S tar --same-owner --one-file-system  --atime-preserve --preserve-permissions --exclude-caches --exclude='.Trash*'  -cf - /sourcedir/ | (cd /mnt/targetdir/ && sudo -S tar --atime-preserve --preserve-permissions --same-owner  -xvf - )


I did not invent the wheel. I borrowed it here:
https://docstore.mik.ua/orelly/unix3/upt/ch10_13.htm


PS: If this may be helpful please consider to upvote my contrib.

3
  • This seems to totally forget about the remote destination. It is unclear what "better results" means.
    – Kusalananda
    Feb 26, 2022 at 11:16
  • oh, right, you remind me to work on it again - Thanks for the hint! You're totally right, "remote" missing (implies access rights...) . Feb 27, 2022 at 20:19
  • amended. Hope you like it better now. Always needs some time and effort. Pls. consider to upvote my contribution if it is sufficient to help. Mar 12, 2022 at 16:32

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .