6

I have a XBian server (Raspberry Pi version of Debian) running rsync via inetd (not the native dæmon). I am serving a couple of directories on an ext4 filesystem (on a USB disk) as individual modules (the modules in question have in the order of 100-500 GB of data, and 1000-10000 files). I have noticed that lately, as I alter other parts of the filesystem (i.e. uploads, copies, etc. not necessarily in those aforementioned directories), rsync calls to these modules will time out.

For a routine rsync command like rsync -vrt rsync://host:port/module ./, where I would not expect to need any file transfer (i.e. both server and client locations would have the same data), in the rsync server log file, I see logs like these:

2014/12/15 22:59:59 [###] connect from UNKNOWN (1.1.1.1)
2014/12/15 22:59:59 [###] rsync on share/ from UNKNOWN (1.1.1.1)
2014/12/15 22:59:59 [###] building file list
2014/12/15 23:16:23 [###] rsync: read error: Connection timed out (110)
2014/12/15 23:16:23 [###] rsync error: error in socket IO (code 10) at io.c(785) [sender=3.1.1]

In the client logs, I see logs like these (yes, same transfer - the server reported timeout after 15 minutes while the client reported error after 30 minutes):

2014/12/15 23:00:01 [###] receiving file list
2014/12/15 23:29:26 [###] rsync: read error: Connection reset by peer (104)
2014/12/15 23:29:26 [###] rsync error: error in rsync protocol data stream (code 12) at /usr/src/ports/rsync/rsync-3.0.9-1/src/rsync-3.0.9/io.c(764) [Receiver=3.0.9]

Any number of issues could cause a situation like this, but after defragmenting a couple of files for other issues I noticed, I also noticed that my rsync transfers would begin to complete successfully again. Then, after I uploaded some more files (again, to a directory outside of the rsync module), I would see the timeouts return. Now, whenever I see my logs having timeout errors, I defragment (with e4defrag) my system and then can successfully run the rsync transfer again.

A few additional notes:

  • My ext4 partition uses less than 50% of its available space at the moment
  • My rsync calls to other, smaller modules do not time out
  • Even calls without data transfer (e.g. rsync -rt rsync://host:post/module) time out in this state
  • After further testing, it seems that after defragmentation, I can run the rsync call successfully once before I need to defragment again (does an rsync call actually cause file fragmentation?)

Why does my rsync setup require a defragmentation each time and what can I do to ensure my rsync doesn't break on such a minor inconvenience any more?

  • 1
    Have a look at dmesg | tail -n 30 when such a hang occurs. Are there hardware error messages? – Hauke Laging Dec 21 '14 at 15:08
  • 1
    Can you see whether the disk is actually busy during the "hang" or not? Doing defrag (or other activity) ma lead to metadata being cached so that a subsequent rsync run completes faster. I wouldn't be surprised that rsync takes more than 30 minutes to walk the module tree. Make sure you're using incremental transfers, also supplying a --timeout 600 option on both the client and server may help to avoid timeouts (sounds counterintuitive but it's true). – wurtel Dec 23 '14 at 14:11
  • An uncommon cause of ETIMEDOUT on a TCP socket is "The other end didn't acknowledge retransmitted data after some time." I think your server is too busy to either send or receive some packets on the rsync connection. The way to confirm it is by using tcpdump (or tshark) on both sides to collect the packets for the connection, and compare timestamps to find suspicious delays when the server is busy defragmenting. – aecolley Jan 23 '15 at 21:34
1

Try a tar to /dev/null of the directory instead of a defrag... that will definitely not modify the disk, but will get all of the inodes cached. With large directories containing lots of files, ext4 indexes them in a hash tree so readdir() returns them in essentially random order. Trying to stat() them in that same order causes a lot of seeks, making it very, very slow.

0

I have been gathering info about the journaling systems in ext3 and ext4, and a chapter in wikipedia on ext4 Delayed allocation and potential data loss, made me think of rsync as a potential cause for fragmentation. Googling that fraze sent me here, and I actually see described the result of the process I was going to ask about! The suggestion to tar to /dev/null seems like a good solution. Read the link, and find more info on delayed allocation.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.