Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I need to sync a directory to a directory on a remote server. at the moment i use rsync -ca to do this but the directory is quite large and the whole process fails with a timeout error.

Now i'm looking for a file by file sync instead of a 'all at once' approach.

like 'find' and for each found rsync to the server.

What would be the most easy way to do this ?

I get the following error :

receiving incremental file list

Write failed: Broken pipe

rsync: connection unexpectedly closed (1175057 bytes received so far) [receiver]

rsync error: error in rsync protocol data stream (code 12) at io.c(601)[receiver=3.0.7]

rsync: connection unexpectedly closed (287 bytes received so far) [generator]

rsync error: unexplained error (code 255) at io.c(601) [generator=3.0.7]

share|improve this question

4 Answers 4

Update: Since the timeout appears to happen when rsync is creating the checksums, here are a few different approaches that may work.

First, to keep only one rsync open, you can (in addition to the options mentioned by Michael Kjörling) play around with the option --sockopts. That gives you the ability to set the same kind of socket options that you can set when creating a socket in code, e.g. SO_RCVTIMEOUT=0 to disable timeouts for receiving data.

Second, the --progress-file option might be enough to not trigger the timeout, by continuously sending back progress information

Third, you could do a file-by-file sync, e.g. like this:

for i in ls /path/to/dir; do rsync -ca $i remoteserver:/path/to/dir; done

Fourth, you could use ssh as your transfer mechanism, and use its TCPKeepAlive and ServerAliveInterval/ServerAliveCountMax options in some suitable way to keep the connection alive.


Original answer: rsync already does a file-by-file approach - it will check each file in the list of files to be synced, and will only sync those that don't exist on the target system, or which are different from those files on the target system.

What you could do to reduce transfer time is to use the option -z which will compress during transport, and the flag --partial so that if the transfer is interrupted, you'll keep any partially synced file so that rsync can continue where it was. Also the --timeout and --contimeout options that Michael Kjörling mentioned would probably be useful.

share|improve this answer
    
it looks like it first calculates ALL the checksum before starting the transfer. is there a way it calculates the checksum of the first and if necessary transfers and then calculates the second ? –  k.mooijman Mar 22 '13 at 12:34
    
If you don't use the -c option it won't use checksums at all, so that'd be one way. –  Jenny D Mar 22 '13 at 12:59
    
you’re right but i intended to check the integrity of all the files so the -c is a must for me –  k.mooijman Mar 22 '13 at 17:00
    
OK, I've added some possible ways to handle it. –  Jenny D Mar 25 '13 at 10:10
up vote 1 down vote accepted

It looks like i found the answer

the server timed out on ssh level

i added

KeepAlive yes

ServerAliveInterval 20

to the /etc/ssh/ssh_config and now it looks promising

share|improve this answer
    
It was definitely the problem of my ssh client timing out. after adding this to /etc/ssh/ssh_config the rsync stayed stable. –  k.mooijman Mar 25 '13 at 14:00

Maybe something like this?

find . -type f -exec 'rsync' '{}' ';'

This will execute rsync once for each regular file under the current directory, passing the file name (represented by the {} token) as a command line parameter.

If you have symlinks, device files, etc. under the directory in question, you can try inverting the logic:

find . -not -type f -exec 'rsync' '{}' ';'

This should "work", in the sense of doing what you are asking (launching rsync once per file). But I get the feeling that you are going about it the wrong way.

The rsync man page lists --timeout (I/O timeout) and --contimeout (connection timeout), and that's just from grepping for timeout. Have you considered using those?

share|improve this answer
    
Thanks for the help and you were right! i managed to resolve it with the timeout of ssh –  k.mooijman Mar 22 '13 at 16:26
    
@k.mooijman Glad if I was able to help. If you feel the question has been answered, please accept the answer that you found most useful by clicking the checkmark outline next to it. –  Michael Kjörling Mar 23 '13 at 0:50

For me this kept happening on resuming a large rsync transfer on a somewhat slow connection. I was even using rsyncd to avoid all the problems with ssh.

I found that enabling compression (-z or --compress) fixed the issue. Transfer resumed almost immediately.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.