3

I'm trying to backup some files from my PC to an external hard-drive using rsync. The copy works fine but takes a lot of time even for the standard USB 2.0. For an USB 2.0 the real transfer speed should be round 30MB/s, in my test the speed reach sometimes 2MB/s. The pen-drive I'm using is a good 32GB USB3.0 with FAT32 fs.

Note: before each test I erased the whole content of the pen-drive and I also tried on different USB port.

Part of the script I'm using is as follow:

mkdir /media/eusbd0
mount -t vfat -o shortname=mixed,iocharset=utf8 /dev/sdb1 /media/eusbd0 

# copy only data between date1 to date2
for dir in $(find /home/records -type d -newermt "2016-08-04" ! -newermt "2016-08-18" 2>&1); do
    if [ "/home/records" != "$dir" ]; then # skip parent dir
        rsync -ravP --inplace --modify-window=2 --log-file=/var/log/download_records "$dir" /media/eusbd0 > /var/log/rsync_report.txt
    fi     
done
#sync   
umount -l /media/eusbd0

Here is part of the output of rsync:

    sending incremental file list 2016-08-11/ 2016-08-11/gps_00000

     44,179 100%   10.88MB/s    0:00:00 (xfr#1, to-chk=34/36) 2016-08-11/log

    106,792 100%  101.84MB/s    0:00:00 (xfr#2, to-chk=33/36) 2016-08-11/log_cam0

      5,532 100%    5.28MB/s    0:00:00 (xfr#3, to-chk=32/36) 2016-08-11/rec_2016-08-11.13_57_22.mov

  3,513,782 100%  108.10MB/s    0:00:00 (xfr#4, to-chk=31/36) 2016-08-11/rec_2016-08-11.13_59_00.mov

  4,281,177 100%   63.79MB/s    0:00:00 (xfr#5, to-chk=30/36) 2016-08-11/rec_2016-08-11.14_02_09.mov

  3,359,207 100%   36.82MB/s    0:00:00 (xfr#6, to-chk=29/36) 2016-08-11/rec_2016-08-11.14_04_16.mov

  1,877,863 100%   17.91MB/s    0:00:00 (xfr#7, to-chk=28/36) 2016-08-11/rec_2016-08-11.14_05_42.mov


........

 50,482,791 100%  114.90MB/s    0:00:00 (xfr#20, to-chk=15/36) 2016-08-11/rec_2016-08-11.15_14_53.mov

 19,291,527 100%   34.52MB/s    0:00:00 (xfr#21, to-chk=14/36) 2016-08-11/rec_2016-08-11.15_18_42.mov

 50,700,461 100%   58.68MB/s    0:00:00 (xfr#22, to-chk=13/36) 2016-08-11/rec_2016-08-11.15_20_20.mov

.......

sent 802,030,914 bytes  received 685 bytes  94,356,658.71 bytes/sec

total size is 801,832,718  speedup is 1.00 tail:

/var/log/rsync_report.txt: file truncated sending incremental file

list 2016-08-16/ 2016-08-16/log

     41,966 100%    8.77MB/s    0:00:00 (xfr#1, to-chk=16/18) 2016-08-16/obd_00000

     46,798 100%   44.63MB/s    0:00:00 (xfr#2, to-chk=15/18) 2016-08-16/rec_2016-08-16.16_24_12.mov

 50,649,317 100%   18.64MB/s    0:00:02 (xfr#3, to-chk=14/18) 2016-08-16/rec_2016-08-16.16_25_50.mov

 25,602,242 100%    7.78MB/s    0:00:03 (xfr#4, to-chk=13/18) 2016-08-16/rec_2016-08-16.16_57_42.mov

 50,496,580 100%   15.18MB/s    0:00:03 (xfr#5, to-chk=12/18) 2016-08-16/rec_2016-08-16.16_59_20.mov

 50,617,906 100%   13.29MB/s    0:00:03 (xfr#6, to-chk=11/18) 2016-08-16/rec_2016-08-16.17_00_58.mov

 50,759,115 100%   10.16MB/s    0:00:04 (xfr#7, to-chk=10/18) 2016-08-16/rec_2016-08-16.17_02_36.mov

 50,883,325 100%    6.81MB/s    0:00:07 (xfr#8, to-chk=9/18) 2016-08-16/rec_2016-08-16.17_04_14.mov

 47,995,074 100%    4.41MB/s    0:00:10 (xfr#9, to-chk=8/18) 2016-08-16/rec_2016-08-16.17_09_44.mov

 50,813,636 100%    2.35MB/s    0:00:20 (xfr#10, to-chk=7/18) 2016-08-16/rec_2016-08-16.17_11_22.mov

 50,953,015 100%    2.52MB/s    0:00:19 (xfr#11, to-chk=6/18) 2016-08-16/rec_2016-08-16.17_13_00.mov

 50,221,069 100%    2.93MB/s    0:00:16 (xfr#12, to-chk=5/18) 2016-08-16/rec_2016-08-16.17_14_38.mov

 50,445,757 100%    2.87MB/s    0:00:16 (xfr#13, to-chk=4/18) 2016-08-16/rec_2016-08-16.17_16_16.mov

 51,036,959 100%    3.16MB/s    0:00:15 (xfr#14, to-chk=3/18) 2016-08-16/rec_2016-08-16.17_17_54.mov

 51,056,196 100%    3.19MB/s    0:00:15 (xfr#15, to-chk=2/18) 2016-08-16/rec_2016-08-16.17_19_32.mov

 50,434,922 100%    3.12MB/s    0:00:15 (xfr#16, to-chk=1/18) 2016-08-16/rec_2016-08-16.17_21_10.mov

 19,456,000 100%    2.71MB/s    0:00:06 (xfr#17, to-chk=0/18)

sent 701,682,372 bytes  received 343 bytes  4,512,429.04 bytes/sec

total size is 701,509,877  speedup is 1.00

What can be noted in the output is that at the beginning the speed is quiet high whereas at the end it gets slower and slower. Why is this happening?

The file I'm copying are pretty much the same (except the size that may vary a bit) and the amount of them is quiet low, less than 100 files. I also tried with other pen-drives but I got the same result.

Thanks!

NEW EXPERIMENT:

I tried cpio instead of rsync to copy the same files. The modified script:

mkdir /media/eusbd0
mount -t vfat -o shortname=mixed,iocharset=utf8 /dev/sdb1 /media/eusbd0 

# copy only data between date1 to date2
for dir in $(find /home/records -type d -newermt "2016-08-04" ! -newermt "2016-08-18" 2>&1); do
    if [ "/home/records" != "$dir" ]; then # skip parent dir
        find "$dir" -print | cpio -pdm /media/eusbd0 > /var/log/rsync_report.txt
    fi     
done
#sync   
umount -l /media/eusbd0

The amount of data to copy is around 3GB and the time required with cpio is 900 seconds which translates to an average transfer speed of 3-4 MB/s. It seems that the problem concerns the hardware or the drives since also rsync speed is close to that value.

UPDATE I tried with other 3 pen-drive formatted in FAT32 but I got the same behavior. Then I tried the NTFS fs and I got a good result of >30MB/s but unfortunately only on one of the 4 USB sticks in my possession. Other experiment with external HDD and SSD have shown also some reasonably good results >20MB/s.

To sum up, my motherboard (or drivers) probably do not like USB sticks nor FAT32 fs. External hard drives (powered or not) work as expected at least in NTFS. And, I have no clues why this happen.

  • You might try running iostat 5 in another window to see whether the transfer rate to the USB stick is always slow. iostat is part of the sysstat package. The high speed you see at first may just be due to rsync and cpio writing into your system's cache. I have several brand-name USB sticks that only have a write speed of 4MB/sec. – Mark Plotnick Aug 17 '16 at 19:39
  • if I run iostat 5 I can only see the transfer rate on the main HD sda. This last shows the same behavior seen with the script. The transfer speed is very high at the beginning and then it drops toward 3-4MB/s. – lcit Aug 18 '16 at 13:53
  • I don't know offhand why iostat wouldn't display stats for your flash drive /dev/sdb. But the output you see is understandable; after the write cache fills up, writes will be throttled to match the speed of your flash drive, and the read speeds will be the same. – Mark Plotnick Aug 18 '16 at 14:00
  • try zipping the file first then copy – editinit Dec 17 '18 at 8:25
2

rsync is not the best solution for copy. It is the best solution to synchronize files. Speed for most data write/read depends on a buffer. In the case of writing files, buffers get filled and you see a higher speed for that. As soon the buffer will be written to the device (buffer flush) the speed is going down. Write speed to the device depends not only on specification (USB 2.0 or 3.0 ...), also on protocol (implementation), driver, block sizes. Sometimes it is better to copy the whole file instead of letting sync programs first scan for changes. This brings overhead and is lowering the speed in case of local attached devices. If you doing this via network the story may be different.

  • So are you suggesting use cp or cpio instead? I'll try one of them – lcit Aug 17 '16 at 12:01
  • if you copy one large file you could also use "dd if=source-file of=/usb-drive-mount/filename bs=2MB" where bs is the buffer-size for read and write buffering. some people name this block-size, what is the wrong definition. if you would avoide file system buffering you can write an image with "dd" direct to a device. you do this most time with Operating-System images/iso files or for serial byte-stream max. speed I/O test. – 0x0C4 Aug 17 '16 at 12:06
  • Buffers, yes, that's surely the main effect here. But rsync vs cp doesn't make an appreciable difference. Rsync checks if the file exists, very quickly sees that the file is blank, and then copies the files. – Gilles Aug 18 '16 at 0:55
-1

30 MB/s is a pretty fast reading speed for a flash drive (at the time of writing this answer — years later speeds have increased). Writing is a lot slower. (Here are some example benchmarks.) The limiting factor is not the USB connection but the drive physics. And writing separate files is slower than the raw write speed you see in benchmarks: it takes time to write the metadata as well.

The fast speeds you're seeing at the beginning are writing to a write buffer somewhere. At some point the buffer gets full, and then the real disk write speed matters.

You might also see other slowdown effects, although since you're filling only 1/4 of a freshly formatting drive I don't think you're reaching the point where they start. If the filesystem gets close to full and has had many deleted files, it can start getting fragmented, so it takes longer to find a little room here and there for the files (this is mostly an issue with rotating media where it helps a lot to be able to store files in large consecutive chunks). On flash media, when a significant fraction is used and has been deleted, the data is spread between cells; each cell needs to be erased as a whole, so if many cells are partially affected by a write, that's a lot of cells to erase and partially rewrite. Also, on rotating media, data access speeds depend how far the data is from the center, but on a flash drive they're the same everywhere.

  • 30 MB/s is a pretty fast reading speed for a USB3 flash drive. No, it really isn't. – Hurrdurrfurr Aug 17 '18 at 21:53
  • I have reached 1GB transfer on usb3 – Ciasto piekarz Dec 17 '18 at 7:05
  • @Ciastopiekarz On USB3, sure. But not from a cheap flash drive. The limiting speed is the drive, not the transfer protocol. – Gilles Dec 17 '18 at 7:45
  • @Ciastopiekarz, nonsense, USB3.0 Gen1 (in Dec 2017 there were no Gen2) transfer rate is no more than 450 MBytes/s, and 500 MBytes/s is the theoretical maximum: 5000 Mbps /10 (due to 8b/10b encoding) = 500. Period. – Ale..chenski Dec 28 '18 at 3:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.