I have a large log file (around 600 MB) that I am trying to transfer across a cellular network. Because it is a log file it is just appended to (although it is actually in an SQLite database with only INSERT being performed, so it isn't quite as simple as that, but with the exception of the last 4k page (or maybe a few) the file is identical each time. It is important that only the changes (and whatever checksums need to be transmitted) actually get sent, because the data connection is metered.
Yet when I perform a test across an unmetered connection (e.g. free wifi hotspot) I do not see an speed-up or reduced data transfer observed or reported. Over a slow WiFi connection I see on the order of 1MB/s or less, reporting that the transfer is going to take nearly 20 minutes. Over a fast WiFi connection I see a uniform faster speed, but no report of speedup, and a second attempt to transfer (which now should be faster because the two files are identical) does now show any difference.
The (sanitized to remove sensitive info) command I am using is:
rsync 'ssh -p 9999' --progress LogFile [email protected]:/home/michael/logs/LogFile
The output I get at the end looks like this:
640,856,064 100% 21.25MB/s 0:00:28 (xfr$1, to-chk=0/1)
There is no mention of any kind of speedup.
I suspect the problem may be one of the following:
- I am missing some command line option. However, re-reading the man page seems to suggest that delta transfers are enabled by default: I only see options for disabling them.
- I'm using rsync over ssh (on a non-standard port even) due to the server being behind a firewall that only allows ssh. I haven't seen anything explicitly saying delta transfers won't work if the rsync daemon isn't running, though. I tried using the "::" notation instead of ":" but the man page isn't very clear about what a "module" is, and my command is rejected for specifying an invalid module.
I have ruled out the following:
- delta transfers not performed on a local network. Ruled out because I am trying to perform the transfer across the internet
- overhead due to checksum calculation. I have seen this behavior both on a fast and slow Wifi connection and the transfer rate doesn't seem to be compute bound.