I'm using rsync to copy about 28TB of images to a 36TB RAID 5. The source has SSDs and the destination has 6 8TB 7200 SATA3 512e drives in a RAID 5 configuration.
Servers are connected over a 10G fiber connection. They are the only two machines on the switch.
Source is CentOS 6.8 destination is Ubuntu 18.04.
I understand that the HDDs are not going to get the full 600MB/s write speed but I am currently only getting 65MB/s when I was expecting somewhere in at least the 200MB/s range.
The speed started at about 72MB/s then gradually increased to 83MB/s before falling to and maintaining 65MB/s over the course of about an hour. Currently the transfer is on course for 5 days.
This seems extremely slow. I was hoping for any suggestions on speeding this up or an explanation as to why it is so slow. Command run:
rsync -a --info=progress2 user@sourceserver:/images/library/ /images/library
I tested a directory using ssh + tar. (rather than rsync)
I was able to transfer 24G in 55 seconds which is acceptable. I then applied to the entire data set. It quickly went back to the slow transfer speeds mentioned before.
I then stopped the transfer and tried the single directory test and achieved the 24G in 55 seconds.
So I wrote a script to use tar + ssh one directory at a time. The first two directories were fast but soon slowed.
I'm now taking 20min for 17G in the last dir checked.
Could this be RAID 5 issue?
Update: The fast speed I just noticed seems to be transfer of data from the page cache. (was retesting from same directory and deleting), once I used a new directory the speed slowed to about 3 min for 24G. But it seems to show the write potential.
I believe the issue may be from the source. I've tried running multiple processes (6) using ssh + tar but they slowed to a crawl. I tried netcat but it wasn't any faster than ssh + tar. Currently the most stable and fastest is to use ssh(arcfour) + tar from a script iterating over each directory with a 3 second pause in between. This method yielded 35G copies at about 6-7min.
Something that I noticed was immediately after midnight on both nights so far the transfer time almost doubled and stayed at that speed until I stopped the script and restarted.
BTW: The source file system is xfs and the target is ext4. Sorry for the lengthy posts but this seemed to be a good exercise to find the fastest way to transfer 28TB of small files.