1

I was testing the copy speed of files over a LAN. I have generated 1GB of /dev/urandom data and copied if over 1Gbps network to another machine.

The rsync transfer speed (just -a option) was around 17MB/s The rsync transfer speed with additional -W is a stable 110MB/s

My my particular case I expect that entire file is to be copied as I generate it each time before the copy. Without -a option the single out of total 4 cores shows 100% usage. I presume this is because rsync computes deltas to transfer and this seems to be the bottleneck.

The question: Is it possible to make the delta computation "multicore"? Now just 25% of the CPU is used.

Improve this question
1
  • 1
    If you expect each generated file to have a reasonable likeness to the one that's already been copied, then rsync -a should save you time. Otherwise use rsync -aW and just copy the file in its entirety
    – Chris Davies
    Commented Feb 7, 2021 at 9:35

2 Answers 2

Reset to default
0

For one file you don't gain any bandwidth optimization, neither much with calculation of using multicore, but when have more files, especially thousands with some just needing updates you could use xargs like following

ls /your/local/folder/ | xargs -n1 -P4 -I% rsync -Pa % ursrv.com:/remote/folder/

This command line will list all folders/files and then run 4 instances of rsync with input argument as a folder or file. This is not directly multicore, but will run multiple instances of rsync which will surelly load your CPU much more efficiently.

Improve this answer
0

Initially, the User creates a single file with random data on source server:

Delta Transfer uses an algorithm for updating a file on one machine to be identical to a file on another remote machine. The algorithm identifies parts of the source file which are identical to some parts of the destination file, and only sends those parts which cannot be matched in this way. Effectively, the algorithm computes a set of difference without having both files on the same machine. Typically, Delta Transfer works very well when the two machines are connected by a low-bandwidth, high-latency communications link. In this case, the connection between the two machines is the major bottleneck of the whole system. By dramatically reducing the amount of the bytes to be sent over the link, the efficiency of the operations would be greatly improved. -- SourceAnywhere

The User is transferring without a corresponding file on the remote server, therefore copying the file and not syncing, the initial transfer with --whole-file is more efficient.

To trade CPU cycles for network overhead I would suggest the -z compression option. The type and level of compression can also be adjusted:

--compress-choice= --compress-level=
zstd 0-9
lz4 0-9
zlibx 0-9
zlib 0-9

On supported filesystems, common data, unlike the User's random data, can be saved as a sparse file with the --sparse switch set, then on the subsequent rsync, set the --inplace switch to write in-place.

Adding the --update switch skips files which are newer on the receiver, saving wasted writes.

The --append switch lets the User append data onto shorter files.

The settings below can help increase overall verbosity during testing:

--verbose, -v            increase verbosity
--stats                  give some file-transfer stats
--human-readable, -h     output numbers in a human-readable format
--progress               show progress during transfer
Improve this answer

You must log in to answer this question.