This is a situation I am frequently in:
- I have a source server with a 320GB hard-drive inside of it, and 16GB of ram (exact specs available here, but as this is an issue I run into frequently on other machines as well, I would prefer the answer to work on any "reasonable" Linux machine)
- I have a backup server with several terabytes of hard-drive space (exact specs here, see disclaimer above)
I want to transfer 320GB of data from the source server to the target server (specifically, the data from /dev/sda
).
- The two computers are physically next to each other, so I can run cables between them.
- I'm on a LAN, and I'm using a new-ish router, which means my network speeds should "ideally" be 1000Mbit, right?
- Security is not an issue. I am on a local network, and I trust all machines on the network, including the router.
- (optional) I don't necessarily need a signed checksum of the data, but basic error checking (such as dropped packets, or the drive becoming unreadable) should be detected rather than just disappear into the output.
I searched for this question online, and have tested several commands. The one that appears the most often is this:
ssh user@192.168.1.100 'dd bs=16M if=/dev/sda | gzip' > backup_sda.gz
This command has proven too slow (it ran for an hour, only got about 80GB through the data). It took about 1 minute and 22 seconds for the 1GB test packet, and ended up being twice as fast when not compressed. The results may also have been skewed by the fact that the transferred file is less than the amount of RAM on the source system.
Moreover (and this was tested on 1GB test pieces), I'm getting issues if I use the gzip
command and dd
; the resulting file has a different checksum when extracted on the target, than it does if piped directly. I'm still trying to figure out why this is happening.
/dev/sda
as an image or just the files. Why is rsync no option? Is/dev/sda
mounted while youdd
ed?