Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have a directory that's got several gigabytes and several thousand small files. I want to copy it over the network with scp more than once. CPU time on the source and destination machines is cheap, but the network overhead added by copying each file individually is huge. I would tar/gzip it up and ship it over, but the source machine is short on disk.

Is there a way for me to pipe the output of tar -czf <output> <directory> to scp? If not, is there another easy solution? My source machine is ancient (SunOS) so I'd rather not go installing things on it.

share|improve this question

5 Answers 5

up vote 30 down vote accepted

You can pipe tar across an ssh session:

$ tar czf - <files> | ssh user@host "cd /wherever; tar xvzf -"
share|improve this answer
1  
+1 tar-pipe solution. If you have more bandwidth and less CPU you can remove the compression flag (although gzip is pretty lightweight). –  dietbuddha Mar 24 '11 at 21:32
1  
And you could drop the compression flag and instead activate it in SSH (ssh -C or Compression yes in ~/.ssh/config). –  Sam Hocevar Mar 25 '11 at 1:06
2  
Never thought of using tar like this. Well, thats why I come here! –  Mr. Shickadance Mar 25 '11 at 3:13

Tar with bzip2 compression should take as much load off the network and on the cpu.

$ tar -C /path/to/src/dir -jcf - ./ | ssh user@server 'tar -C /path/to/dest/dir -jxf -'

Not using -v because screen output might slow down the process. But if you want a verbose output use it on the local side of tar (-jcvf), not on the remote part.

If you repeatedly copy over the same destination path, like updating a backup copy, your best choice is rsync with compression.

$ rsync -az -e ssh /path/to/src/dir/ user@server:/path/to/dest/dir/

Notice that both src and dest paths end with a /. Again, not using -v and -P flags on purpose, add them if you need verbose output.

share|improve this answer

use rsync, it uses SSH.

Usage:

rsync -aPz /source/path destination.server:remote/path

The rsync switches care about compression and I-Node information. -P displays progress of every file.

You can use scp -C, which enables compression, but if possible, use rsync.

share|improve this answer
    
Unfortunately, rsync isn't available on the source machine, and neither is sshd. –  nmichaels Mar 24 '11 at 15:12
    
sshd isn't necessary for those operations on the client machine. –  polemon Mar 25 '11 at 5:05

You can run tar on both ends using ssh. scp is part of the ssh family of goodness, so you probably have it on both ends.

 8:03AM 12 % tar cf - some_directory | ssh dest_host "tar xf -"

There may be a way to work gzip or bzip2 into the pipeline to lessen the network traffic, too.

share|improve this answer

If you have gzip on both ends: sourcehost$ cd sourcedir && tar cf - . | gzip -c - | ssh user@destinationhost "cd destinationdir && gzip -c -d | tar xf -"

If you don't have gzip on the source machine, make sure you have uncompress on the destination: sourcehost$ cd sourcedir && tar cf - . | compress | ssh user@destinationhost "cd destdir && uncompress | tar xf -"

This would be faster than first zipping it up, then sending, then unzipping, and it requires no extra disk space on either side. I sikpped the compression (z) flag on tar, because you probably dont have it on the ancient side.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.