2

I need to copy large files over the network, but I am not sure about the command line to use for it. I have been using rsync, but I feel maybe I could zip the files first?

2
  • 1
    @Hermann Ingjaldsson for what reason should he prefer scp over rsync?
    – scai
    Aug 1, 2012 at 11:27
  • @HermannIngjaldsson so it is with rsync.
    – scai
    Aug 8, 2012 at 10:41

2 Answers 2

7

rsync already provides compression when you specify option -z. You should avoid applying this option for already compressed files or binaries as it will just introduce additional overhead.

When using rsync with the ssh protocol (the default) and you have only limited CPU resources and security is not an issue, you can also try specifying -e ssh -c arcfour to use a less expensive encryption algorithm (RC4). On modern CPUs this won't make any big difference, though.

Other interesting arguments you may want to use:

  • -vP for a nice progress bar and keep partially transfered files when interrupted, so you can resume from the partial file the next time you run rsync
  • -a to preserve most (but not all!) file meta data such as permissions and modification times
6
  • I already started copying files using the rsync command but I didnt specify the -z option, and I find that the copy is taking too long. Is it possible to pause it and then execute the rsync command again with -z?
    – cindy
    Aug 1, 2012 at 7:52
  • @cindy you can break rsync with ctrl-c and re-run it again with new options. It should continue from the place you stop it.
    – rush
    Aug 1, 2012 at 8:01
  • This may or may not work, I never figured out the conditions under which rsync doesn't delete half-transfered files. I wouldn't rely on it resuming the transfer.
    – scai
    Aug 1, 2012 at 8:01
  • thanks, I think I'll let this copy finish. It's 816MB folder, and it's already transferred almost 600MB. I will try the options -z another time.
    – cindy
    Aug 1, 2012 at 8:22
  • 1
    @scai: repeating and resuming transfers are a big part of the point of rsync. rsync won't delete files from the destination unless you specify one of the --delete options. If you do, then it will delete files that are in the dest but not the source.
    – cas
    Aug 1, 2012 at 10:03
1

I'm not sure I understand whether you have a problem compressing files or you have a problem transferring files, or you just have a general question. I'll go with the later.

If you have several files in a folder, you could tar them up in a single archive (.tar) file: tar -cvf myfile.tar myfolder

You may then compress this file using a very good compressor, such as lzma: lzma myfile.tar. Note that you can also instruct tar to compress with the -z (gzip), -j (bzip2), or --lzma options, in which case you execute a single command, for example: tar --lzma -cvf myfile.tar.lzma myfolder.

You may then copy your file using rsync or scp, and uncompress it (lzma -d myfile.tar.lzma) and then extract it (tar -xvf myfile.tar) once on the destination system. If you used a single command, the single command for extraction would be: tar --lzma -xvf myfile.tar.lzma.

You may also instruct rsync to compress using the -z option, and not worry about creating a single-file archive beforehand.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .