1

I'm using rsync to copy about 28TB of images to a 36TB RAID 5. The source has SSDs and the destination has 6 8TB 7200 SATA3 512e drives in a RAID 5 configuration.

Servers are connected over a 10G fiber connection. They are the only two machines on the switch.

Source is CentOS 6.8 destination is Ubuntu 18.04.

I understand that the HDDs are not going to get the full 600MB/s write speed but I am currently only getting 65MB/s when I was expecting somewhere in at least the 200MB/s range.

The speed started at about 72MB/s then gradually increased to 83MB/s before falling to and maintaining 65MB/s over the course of about an hour. Currently the transfer is on course for 5 days.

This seems extremely slow. I was hoping for any suggestions on speeding this up or an explanation as to why it is so slow. Command run:

 rsync -a --info=progress2 user@sourceserver:/images/library/ /images/library

Update:
I tested a directory using ssh + tar. (rather than rsync)
I was able to transfer 24G in 55 seconds which is acceptable. I then applied to the entire data set. It quickly went back to the slow transfer speeds mentioned before.
I then stopped the transfer and tried the single directory test and achieved the 24G in 55 seconds.
So I wrote a script to use tar + ssh one directory at a time. The first two directories were fast but soon slowed.
I'm now taking 20min for 17G in the last dir checked.

Could this be RAID 5 issue?

Update: The fast speed I just noticed seems to be transfer of data from the page cache. (was retesting from same directory and deleting), once I used a new directory the speed slowed to about 3 min for 24G. But it seems to show the write potential.

I believe the issue may be from the source. I've tried running multiple processes (6) using ssh + tar but they slowed to a crawl. I tried netcat but it wasn't any faster than ssh + tar. Currently the most stable and fastest is to use ssh(arcfour) + tar from a script iterating over each directory with a 3 second pause in between. This method yielded 35G copies at about 6-7min.
Something that I noticed was immediately after midnight on both nights so far the transfer time almost doubled and stayed at that speed until I stopped the script and restarted.

BTW: The source file system is xfs and the target is ext4. Sorry for the lengthy posts but this seemed to be a good exercise to find the fastest way to transfer 28TB of small files.

  • Is it RAID5 software based?Have you check the load of target machine? – Romeo Ninov Mar 5 at 7:50
  • And do you have a lot of small (less than few megabytes) files? – Romeo Ninov Mar 5 at 7:57
  • Raid 5 is not software. Yes, a lot of small files. – Loom Mar 5 at 13:32
  • In such case presence of lot of small files maybe it's the reason. – Romeo Ninov Mar 5 at 13:46
  • That's what I'm afraid of. A far as the load goes it has 48 cores with 251G of RAM . rsync is the only thing running on it and it's at about 96% idle. – Loom Mar 5 at 14:00
2

Two points:

  • First, by default, rsync works over SSH. It's slow. Check the output from top or htop and you'll probably see something like that:
    top - 18:04:39 up 113 days,  3:47,  3 users,  load average: 0,50, 0,59, 0,62
    Tâches: 489 total,   4 en cours, 485 en veille,   0 arrêté,   0 zombie
    %Cpu(s): 40,7 ut, 14,5 sy,  0,0 ni, 36,3 id,  3,4 wa,  0,0 hi,  5,1 si,  0,0 st
    MiB Mem :   7976,3 total,    212,8 libr,   2717,9 util,   5045,7 tamp/cache
    MiB Éch :   8583,0 total,   8381,2 libr,    201,8 util.   4598,0 dispo Mem 

      PID UTIL.     PR  NI    VIRT    RES    SHR S  %CPU  %MEM    TEMPS+ COM.                                                                                                             
    27262 emmanuel  20   0   33956   7924   4204 R  58,3   0,1   0:21.51 ssh                                                                                                              
    31185 emmanuel  20   0   52164   3208   2140 S  35,1   0,0   0:05.03 rsync                                                                                                            
    27249 emmanuel  20   0 1340140 158896  45432 S   8,9   1,9   4:40.63 python2                                                                                                          
       52 root      20   0       0      0      0 R   6,3   0,0   9:51.41 kswapd0                                                                                                          
    25149 root      20   0  324716 126192  63120 S   2,0   1,5  25:26.24 Xorg                                                                                                             
    25679 emmanuel  20   0 2555068 774108 100220 S   1,3   9,5   9:28.86 WebExtensions                                                                                                    

Notice how rsync+ssh max out one CPU almost entirely?

  • Second, we don't know the type and speed of your destination array; its normal write speed may be terrible, for instance if it's a hardware RAID controller with write-caching disabled.

How to get better performance :

  • For initial copy don't use rsync. Seriously. rsync is great to, you know, synchronise data. But for a copy towards and empty target, it sucks. It's much, much slower than good old cp. So my advice is : use cp over NFS and you'll max out your hardware (whatever part is the slowest, destination RAID or network).

  • On the target server, edit /etc/exports :

    /mnt/raid *(rw,async,no_root_squash,no_subtree_check)

Start NFS: systemctl restart nfs-kernel-server

  • On the source machine, mount the export:

mount <server IP>:/mnt/raid /mnt/target

Then copy everything:

cp -av /mnt/source /mnt/target

Preferably use screen or tmux to run your copy and avoid accidents (lost ssh connection, etc).

  • Alternate solution: if NFS isn't available, or some other file sharing protocol (CIFS/SMB, Fuse-FTP, WebDav...) then your best bet is to use netcat in conjunction with tar. The important part is not to encrypt traffic:

On the target machine, run a netcat server:

cd /mnt/target ; nc -l -p 45724 | tar x

On the source side, run this:

cd /mnt/source; tar cf - * | nc <target IP> 45724
  • Thanks for your advice. Unfortunately this server does not have nfs installed and it is currently locked down. – Loom Mar 5 at 19:43
  • @Loom you could try SMB/CIFS if available, then. Or FTP, WebDav, whatever. Any other protocol than SSH will perform better, because you're basically CPU-bound by encryption, and one-CPU bound with that. – wazoox Mar 6 at 14:47
  • As a last (but fast) resort, you can use tar with netcat. I'll edit my answer. – wazoox Mar 6 at 14:50
  • I updated my original post with my ssh + tar attempt (wasnt sure where to post it). It's fast until I start to throttle up. – Loom Mar 6 at 15:25
  • @Loom from what you say it very much looks like it's fast as long as it's reading from the page cache. Please use iostat -mx 5 for a few minutes on both the source and target machines to see if (and how much) the disk subsystems work... – wazoox Mar 6 at 15:27
0

You've lots of cores and plenty of network bandwidth, so I'd suggest you parallelise the requirement. Several rsync processes, each handling a different part of the file set.

  • 1
    I attempted to run 6 processes using tar + ssh each with different parts of the data.Within an hour the processes were taking anywhere from 40min to over an hour to copy ~35G. I had to stop them all. It was faster with a single process. – Loom Mar 7 at 20:24
  • Thanks for the feedback. That surprises me - I've had success with this approach myself. – roaima Mar 7 at 21:03
0

Concluding that many small files is the reason for slow transfer speeds over rsync.

A streaming approach would be more efficient in this case such as using ssh + tar

Update: Actually, in my case this is incorrect (well doesn't solve the issue). I was running these tests on directories I was using as test. It was pointed out that these may be in the page cache so I made sure to test again on a new directory and the speed dropped dramatically.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.