15

Running rsync with a large* --block-size like this:

rsync -avvz --rsh 'ssh -c arcfour' --block-size 1048576 --inplace --progress example.com:/big.file /big.file

I get the following error:

Invalid block length 1048576 [sender]

Both ends are running 64bit CentOS 6.4. From Googling I've seen --block-size used with much higher values, why is this not working for me?

* I'm using a large block size because I'm trying to work around a bug where rsync just spins CPU forever 44% into a 300GB file

13

From the source:

int32 max_blength = protocol_version < 30 ? OLD_MAX_BLOCK_SIZE : MAX_BLOCK_SIZE;

sum->blength = read_int(f);
if (sum->blength < 0 || sum->blength > max_blength) {
    rprintf(FERROR, "Invalid block length %ld [%s]\n",
        (long)sum->blength, who_am_i());
    exit_cleanup(RERR_PROTOCOL);
}

Where:

#define OLD_MAX_BLOCK_SIZE ((int32)1 << 29)
#define MAX_BLOCK_SIZE ((int32)1 << 17)

Which is 536870912 (512M) and 131072 (128k) respectively.


The change was made in version v3.0.0 and support for OLD_ was added in v3.0.3. (Links explain some of the rationale behind the changes.)

  • [PATCH] A patch to try to make really large files get handled without bogging down in sender-side hashtable searching.

  • [PATCH] Fixed the sending of large files with older rsync versions by handling the old block-size limit for protocols < 29.

2
  • 1
    There still seems to be a bug in 3.0.x in block size handling under --dry-run. A command without --dry-run can work, but with it still results in the "Invalid block length NNN [sender]" message.
    – Paul Gear
    Jul 6 '16 at 5:09
  • 1
    Fascinating that I just encountered the nuance mentioned by @PaulGear, over 3 years later and this is still an issue in Ubuntu 18 with ? v3.1.2
    – TonyG
    Oct 15 '19 at 23:06
3

The maximum block size depends on the rsync protocol version.

With protocol version less than 30, the max was 1 << 29, i.e. 536870912 bytes (512M). But with protocol version 30 or above, the max is 1 << 17, which is 128k bytes. So you'll need to use an older version if you want larger block sizes.

Source: rsync.h

#define MAX_BLOCK_SIZE ((int32)1 << 17)

/* For compatibility with older rsyncs */
#define OLD_MAX_BLOCK_SIZE ((int32)1 << 29)

And: io.c

// ...
int32 max_blength = protocol_version < 30 ? OLD_MAX_BLOCK_SIZE : MAX_BLOCK_SIZE;
// ...
if (sum->blength < 0 || sum->blength > max_blength) {
  rprintf(FERROR, "Invalid block length %ld [%s]\n",
                  (long)sum->blength, who_am_i());

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy