Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

What is the canonical way to:

  • scp a file to a remote location
  • compress the file in transit (tar or not, single file or whole folder, 7za or something else even more efficient)
  • do the above without saving intermediate files

I am familiar with shell pipes like this:

tar cf - MyBackups | 7za a -si -mx=9 -ms=on MyBackups.tar.7z

essentially:

  • rolling a whole folder into a single tar
  • pass data through stdout to stdin of the compressing program
  • apply aggressive compression

What's the best way to do this over an ssh link, with the file landing on the remote filesystem?


I prefer not to sshfs mount.


This, does not work:

scp <(tar cvf - MyBackups | 7za a -si -mx=9 -so) localhost:/tmp/tmp.tar.7z

because:

/dev/fd/63: not a regular file
share|improve this question
add comment

1 Answer 1

There are many ways to do what you want. The simplest is to use a pìpe:

tar zcvf -  MyBackups | ssh user@server "cat > /path/to/backup/foo.tgz"

Here, the compression is being handled by tar which calls gzip (z flag). You can also use compress (Z) and bzip (j). For 7z, do this:

tar cf - MyBackups | 7za a -si -mx=9 -ms=on MyBackups.tar.7z | 
   ssh user@server "cat > /path/to/backup/foo.7z"

The best way, however, is probably rsync.

   Rsync is a fast and extraordinarily versatile  file  copying  tool.   It  can  copy
   locally, to/from another host over any remote shell, or to/from a remote rsync dae‐
   mon.  It offers a large number of options that control every aspect of its behavior
   and  permit  very  flexible  specification of the set of files to be copied.  It is
   famous for its delta-transfer algorithm, which reduces the amount of data sent over
   the network by sending only the differences between the source files and the exist‐
   ing files in the destination.  Rsync is widely used for backups and  mirroring  and
   as an improved copy command for everyday use.

rsync has way too many options. It really is worth reading through them but they are scary at first sight. The ones you care about in this context though is:

    -z, --compress              compress file data during the transfer
        --compress-level=NUM    explicitly set compression level

   -z, --compress
          With this option, rsync compresses the file data as it is sent to the desti‐
          nation machine, which reduces the amount of data being transmitted --  
          something that is useful over a slow connection.

          Note  that this option typically achieves better compression ratios than can
          be achieved by using a compressing remote shell or a  compressing  transport
          because  it takes advantage of the implicit information in the matching data
          blocks that are not explicitly sent over the connection.

So, in your case, you would want something like this:

rsync -z MyBackups user@server:/path/to/backup/

The files would be compressed while in transit and arrive decompressed at the destination.


Some more choices:

  • scp itself can compress the data

     -C      Compression enable.  Passes the -C flag to ssh(1) to
             enable compression.
    
    $ scp -C source user@server:/path/to/backup
    
  • There may be a way to get rsync and 7za to play nice but there is no point in doing so. The benefit of rsync is that it will only copy the bits that have changed between the local and remote files. However, a small local change can result in a very different compressed file so there is no point in using rsync for this. It just complicates matters with no benefit. Just use direct ssh as shown above. If you really want to do this, you can try by giving a subshell as an argument to rsync. On my system, I could not get this to work with 7za because it does not allow you to write compressed data to a terminal. Perhaps your implementation is different. Try something like (this does not work for me):

    rsync $(tar cf - MyBackups | 7za a -an -txz -si -so) \
      user@server:/path/to/backup
    
  • Another point is that 7z should not be used for backups on Linux. As stated on the 7z man page:

    DO NOT USE the 7-zip format for backup purpose on Linux/Unix because :
    - 7-zip does not store the owner/group of the file.

share|improve this answer
1  
A good note to add is that unless you're transferring over a generally slow network, say the internet, it's best to avoid compressing as it just slows down the transfer rate. On a LAN, -z is at least twice as slow. For even more speed than rsyncing over ssh, setup an rsync daemon and rsync using -W flag (copies files whole (w/o delta-xfer algorithm). –  laebshade Mar 30 '13 at 3:44
    
Thanks! I am going to accept this great answer but please, add a full, stand-alone command line that uses both rsync and 7za, with final output to the remote filesystem. I liked -z but I would like to decouple the compression stage so.. how would I use rsync in that case, please? –  Robottinosino Mar 30 '13 at 12:43
1  
@Robottinosino see updated answer. There is no point in using rsync with 7z. It should work with rsync and a subshel as shown but I could not figure out how anyway. –  terdon Mar 30 '13 at 15:13
    
I think that 7z and tar are a very powerful combination, I get with them better savings than with other compression algorithms. I am sure new and better ones will replace 7z.. but I don't understand why you would be against using the pair "in principle".. am I missing something? –  Robottinosino Apr 1 '13 at 20:10
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.