I need to copy large files over the network, but I am not sure about the command line to use for it. I have been using rsync, but I feel maybe I could zip the files first?

  • just use scp. asf – Hermann Ingjaldsson Aug 1 '12 at 10:25
  • 1
    @Hermann Ingjaldsson for what reason should he prefer scp over rsync? – scai Aug 1 '12 at 11:27
  • I don't know, I'm not familiar with rsync but I do know scp ing a file is just a single simple command. – Hermann Ingjaldsson Aug 1 '12 at 17:51
  • @HermannIngjaldsson so it is with rsync. – scai Aug 8 '12 at 10:41

rsync already provides compression when you specify option -z. You should avoid applying this option for already compressed files or binaries as it will just introduce additional overhead.

When using rsync with the ssh protocol (the default) and you have only limited CPU resources and security is not an issue, you can also try specifying -e ssh -c arcfour to use a less expensive encryption algorithm (RC4). On modern CPUs this won't make any big difference, though.

Other interesting arguments you may want to use:

  • -vP for a nice progress bar and keep partially transfered files when interrupted, so you can resume from the partial file the next time you run rsync
  • -a to preserve most (but not all!) file meta data such as permissions and modification times
  • I already started copying files using the rsync command but I didnt specify the -z option, and I find that the copy is taking too long. Is it possible to pause it and then execute the rsync command again with -z? – cindy Aug 1 '12 at 7:52
  • @cindy you can break rsync with ctrl-c and re-run it again with new options. It should continue from the place you stop it. – rush Aug 1 '12 at 8:01
  • This may or may not work, I never figured out the conditions under which rsync doesn't delete half-transfered files. I wouldn't rely on it resuming the transfer. – scai Aug 1 '12 at 8:01
  • thanks, I think I'll let this copy finish. It's 816MB folder, and it's already transferred almost 600MB. I will try the options -z another time. – cindy Aug 1 '12 at 8:22
  • 1
    @scai: repeating and resuming transfers are a big part of the point of rsync. rsync won't delete files from the destination unless you specify one of the --delete options. If you do, then it will delete files that are in the dest but not the source. – cas Aug 1 '12 at 10:03

I'm not sure I understand whether you have a problem compressing files or you have a problem transferring files, or you just have a general question. I'll go with the later.

If you have several files in a folder, you could tar them up in a single archive (.tar) file: tar -cvf myfile.tar myfolder

You may then compress this file using a very good compressor, such as lzma: lzma myfile.tar. Note that you can also instruct tar to compress with the -z (gzip), -j (bzip2), or --lzma options, in which case you execute a single command, for example: tar --lzma -cvf myfile.tar.lzma myfolder.

You may then copy your file using rsync or scp, and uncompress it (lzma -d myfile.tar.lzma) and then extract it (tar -xvf myfile.tar) once on the destination system. If you used a single command, the single command for extraction would be: tar --lzma -xvf myfile.tar.lzma.

You may also instruct rsync to compress using the -z option, and not worry about creating a single-file archive beforehand.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.