Why does my rsync not allow block-size>128K?

Question

Running rsync with a large^* --block-size like this:

rsync -avvz --rsh 'ssh -c arcfour' --block-size 1048576 --inplace --progress example.com:/big.file /big.file

I get the following error:

Invalid block length 1048576 [sender]

Both ends are running 64bit CentOS 6.4. From Googling I've seen --block-size used with much higher values, why is this not working for me?

_{* I'm using a large block size because I'm trying to work around a bug where rsync just spins CPU forever 44% into a 300GB file}

Answer 1

From the source:

int32 max_blength = protocol_version < 30 ? OLD_MAX_BLOCK_SIZE : MAX_BLOCK_SIZE;

sum->blength = read_int(f);
if (sum->blength < 0 || sum->blength > max_blength) {
    rprintf(FERROR, "Invalid block length %ld [%s]\n",
        (long)sum->blength, who_am_i());
    exit_cleanup(RERR_PROTOCOL);
}

Where:

#define OLD_MAX_BLOCK_SIZE ((int32)1 << 29)
#define MAX_BLOCK_SIZE ((int32)1 << 17)

Which is 536870912 (512M) and 131072 (128k) respectively.

---

The change was made in version v3.0.0 and support for OLD_ was added in v3.0.3. (Links explain some of the rationale behind the changes.)

• [PATCH] A patch to try to make really large files get handled without bogging down in sender-side hashtable searching.

• [PATCH] Fixed the sending of large files with older rsync versions by handling the old block-size limit for protocols < 29.

Answer 2

The maximum block size depends on the rsync protocol version.

With protocol version less than 30, the max was 1 << 29, i.e. 536870912 bytes (512M). But with protocol version 30 or above, the max is 1 << 17, which is 128k bytes. So you'll need to use an older version if you want larger block sizes.

Source: rsync.h

#define MAX_BLOCK_SIZE ((int32)1 << 17)

/* For compatibility with older rsyncs */
#define OLD_MAX_BLOCK_SIZE ((int32)1 << 29)

And: io.c

// ...
int32 max_blength = protocol_version < 30 ? OLD_MAX_BLOCK_SIZE : MAX_BLOCK_SIZE;
// ...
if (sum->blength < 0 || sum->blength > max_blength) {
  rprintf(FERROR, "Invalid block length %ld [%s]\n",
                  (long)sum->blength, who_am_i());