2

I need to transfer some files via rsync and can´t tar them before. So I thought maybe there is a mode which checks the folders content and if it is thousand of small files inside compress that folder for the transfer. However I couldn´t find something like that in the man (http://www.manpagez.com/man/1/rsync/). Is it possible and I didn´t find it or would I have to do this before I use rsync?

3
  • 1
    What do you mean by “compress them together”? Rsync will transfer them together, and it will compress them if you use the -z option. If you're thinking about performance, whatever tool you'd be using to “compress them together” (I guess you want to archive them together and compress the archive) would have to read all the files anyway, and that's what rsync is doing, I don't see what additional steps could make the transfer faster. Mar 19 '14 at 1:28
  • 1
    Unfortunately this is one of the weaknesses of rsync, it just won't perform well for large numbers of small files. Why can't you use a tar pipe over ssh?
    – Graeme
    Mar 19 '14 at 1:33
  • @Graeme the problem about tar is that i pull the files and don´t have write access (readonly). Mar 19 '14 at 2:31
2

You'll typically use one of the following methods when you need to push or pull files over SSH using tar.

Copy SOURCEDIR from localhost to remotehost

Conceptually I'll often refer to this as pushing files, since we're taking the output of a tar command and we're pushing the files to another system over SSH. The files are targeted for copying on the remote system into DESTDIR.

example 1
$ tar zcvf - SOURCEDIR | ssh user1@remotehost 'cd DESTDIR; tar zxvf - '
example 2
$ tar zcvf - SOURCEDIR | ssh user1@remotehost "(cd DESTDIR; tar zxvf -)"
example 3
$ tar zcvf - SOURCEDIR | ssh -l user1 remotehost 'cd DESTDIR ; tar zxvf -'

Copy SOURCEDIR from remotehost to localhost

Conceptually I'll often refer to this as pulling files, since we're making an SSH connection to a remote host, where we're kicking off a tar command. The output of this tar command is then targeted for copying on the local system into DESTDIR.

example 1
$ ssh remotehost 'tar zcvf - SOURCEDIR' | tar zxvf -
example 2
$ ssh -n remotehost 'tar zcvf - SOURCEDIR' | tar zxvf -
example 3
$ ssh remotehost "( cd SOURCEDIR ; tar zcvf - SOURCEFILES ) " | tar zxvf -

References

11
  • does that work in readonly mode as pull ? Mar 19 '14 at 2:33
  • Do you mean as in a dry run mode?
    – slm
    Mar 19 '14 at 2:41
  • well the acc I use to pull the data does not have write permission. What means dry run mode in this case? Mar 19 '14 at 2:42
  • 2
    @user2693017 - there is no tar file actually created, as the contents of tar is created, it's redirected via the pipe, | to another receiving tar which will uncompress it.
    – slm
    Mar 19 '14 at 3:05
  • 1
    @user2693017 - from my blog post that I referenced: tar zcvf - SOURCEDIR | ssh user1@remotehost "cat > /DESTDIR/DESTFILE.tar.gz".
    – slm
    Mar 26 '14 at 22:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.