Cfrtjryk

I'm trying to backup some files from my PC to an external hard-drive using rsync. The copy works fine but takes a lot of time even for the standard USB 2.0. For an USB 2.0 the real transfer speed should be round 30MB/s, in my test the speed reach sometimes 2MB/s. The pen-drive I'm using is a good 32GB USB3.0 with FAT32 fs.

Note: before each test I erased the whole content of the pen-drive and I also tried on different USB port.

Part of the script I'm using is as follow:

mkdir /media/eusbd0

mount -t vfat -o shortname=mixed,iocharset=utf8 /dev/sdb1 /media/eusbd0 



# copy only data between date1 to date2

for dir in $(find /home/records -type d -newermt "2016-08-04" ! -newermt "2016-08-18" 2>&1); do

    if [ "/home/records" != "$dir" ]; then # skip parent dir

        rsync -ravP --inplace --modify-window=2 --log-file=/var/log/download_records "$dir" /media/eusbd0 > /var/log/rsync_report.txt

    fi     

done

#sync   

umount -l /media/eusbd0

Here is part of the output of rsync:

    sending incremental file list 2016-08-11/ 2016-08-11/gps_00000



     44,179 100%   10.88MB/s    0:00:00 (xfr#1, to-chk=34/36) 2016-08-11/log



    106,792 100%  101.84MB/s    0:00:00 (xfr#2, to-chk=33/36) 2016-08-11/log_cam0



      5,532 100%    5.28MB/s    0:00:00 (xfr#3, to-chk=32/36) 2016-08-11/rec_2016-08-11.13_57_22.mov



  3,513,782 100%  108.10MB/s    0:00:00 (xfr#4, to-chk=31/36) 2016-08-11/rec_2016-08-11.13_59_00.mov



  4,281,177 100%   63.79MB/s    0:00:00 (xfr#5, to-chk=30/36) 2016-08-11/rec_2016-08-11.14_02_09.mov



  3,359,207 100%   36.82MB/s    0:00:00 (xfr#6, to-chk=29/36) 2016-08-11/rec_2016-08-11.14_04_16.mov



  1,877,863 100%   17.91MB/s    0:00:00 (xfr#7, to-chk=28/36) 2016-08-11/rec_2016-08-11.14_05_42.mov





........



 50,482,791 100%  114.90MB/s    0:00:00 (xfr#20, to-chk=15/36) 2016-08-11/rec_2016-08-11.15_14_53.mov



 19,291,527 100%   34.52MB/s    0:00:00 (xfr#21, to-chk=14/36) 2016-08-11/rec_2016-08-11.15_18_42.mov



 50,700,461 100%   58.68MB/s    0:00:00 (xfr#22, to-chk=13/36) 2016-08-11/rec_2016-08-11.15_20_20.mov



.......



sent 802,030,914 bytes  received 685 bytes  94,356,658.71 bytes/sec



total size is 801,832,718  speedup is 1.00 tail:



/var/log/rsync_report.txt: file truncated sending incremental file



list 2016-08-16/ 2016-08-16/log



     41,966 100%    8.77MB/s    0:00:00 (xfr#1, to-chk=16/18) 2016-08-16/obd_00000



     46,798 100%   44.63MB/s    0:00:00 (xfr#2, to-chk=15/18) 2016-08-16/rec_2016-08-16.16_24_12.mov



 50,649,317 100%   18.64MB/s    0:00:02 (xfr#3, to-chk=14/18) 2016-08-16/rec_2016-08-16.16_25_50.mov



 25,602,242 100%    7.78MB/s    0:00:03 (xfr#4, to-chk=13/18) 2016-08-16/rec_2016-08-16.16_57_42.mov



 50,496,580 100%   15.18MB/s    0:00:03 (xfr#5, to-chk=12/18) 2016-08-16/rec_2016-08-16.16_59_20.mov



 50,617,906 100%   13.29MB/s    0:00:03 (xfr#6, to-chk=11/18) 2016-08-16/rec_2016-08-16.17_00_58.mov



 50,759,115 100%   10.16MB/s    0:00:04 (xfr#7, to-chk=10/18) 2016-08-16/rec_2016-08-16.17_02_36.mov



 50,883,325 100%    6.81MB/s    0:00:07 (xfr#8, to-chk=9/18) 2016-08-16/rec_2016-08-16.17_04_14.mov



 47,995,074 100%    4.41MB/s    0:00:10 (xfr#9, to-chk=8/18) 2016-08-16/rec_2016-08-16.17_09_44.mov



 50,813,636 100%    2.35MB/s    0:00:20 (xfr#10, to-chk=7/18) 2016-08-16/rec_2016-08-16.17_11_22.mov



 50,953,015 100%    2.52MB/s    0:00:19 (xfr#11, to-chk=6/18) 2016-08-16/rec_2016-08-16.17_13_00.mov



 50,221,069 100%    2.93MB/s    0:00:16 (xfr#12, to-chk=5/18) 2016-08-16/rec_2016-08-16.17_14_38.mov



 50,445,757 100%    2.87MB/s    0:00:16 (xfr#13, to-chk=4/18) 2016-08-16/rec_2016-08-16.17_16_16.mov



 51,036,959 100%    3.16MB/s    0:00:15 (xfr#14, to-chk=3/18) 2016-08-16/rec_2016-08-16.17_17_54.mov



 51,056,196 100%    3.19MB/s    0:00:15 (xfr#15, to-chk=2/18) 2016-08-16/rec_2016-08-16.17_19_32.mov



 50,434,922 100%    3.12MB/s    0:00:15 (xfr#16, to-chk=1/18) 2016-08-16/rec_2016-08-16.17_21_10.mov



 19,456,000 100%    2.71MB/s    0:00:06 (xfr#17, to-chk=0/18)



sent 701,682,372 bytes  received 343 bytes  4,512,429.04 bytes/sec



total size is 701,509,877  speedup is 1.00

What can be noted in the output is that at the beginning the speed is quiet high whereas at the end it gets slower and slower. Why is this happening?

The file I'm copying are pretty much the same (except the size that may vary a bit) and the amount of them is quiet low, less than 100 files. I also tried with other pen-drives but I got the same result.

Thanks!

NEW EXPERIMENT:

I tried cpio instead of rsync to copy the same files. The modified script:

mkdir /media/eusbd0

mount -t vfat -o shortname=mixed,iocharset=utf8 /dev/sdb1 /media/eusbd0 



# copy only data between date1 to date2

for dir in $(find /home/records -type d -newermt "2016-08-04" ! -newermt "2016-08-18" 2>&1); do

    if [ "/home/records" != "$dir" ]; then # skip parent dir

        find "$dir" -print | cpio -pdm /media/eusbd0 > /var/log/rsync_report.txt

    fi     

done

#sync   

umount -l /media/eusbd0

The amount of data to copy is around 3GB and the time required with cpio is 900 seconds which translates to an average transfer speed of 3-4 MB/s. It seems that the problem concerns the hardware or the drives since also rsync speed is close to that value.

UPDATE
I tried with other 3 pen-drive formatted in FAT32 but I got the same behavior. Then I tried the NTFS fs and I got a good result of >30MB/s but unfortunately only on one of the 4 USB sticks in my possession. Other experiment with external HDD and SSD have shown also some reasonably good results >20MB/s.

To sum up, my motherboard (or drivers) probably do not like USB sticks nor FAT32 fs. External hard drives (powered or not) work as expected at least in NTFS. And, I have no clues why this happen.

rsync is not the best solution for copy. It is the best solution to synchronize files. Speed for most data write/read depends on a buffer. In the case of writing files, buffers get filled and you see a higher speed for that. As soon the buffer will be written to the device (buffer flush) the speed is going down. Write speed to the device depends not only on specification (USB 2.0 or 3.0 ...), also on protocol (implementation), driver, block sizes. Sometimes it is better to copy the whole file instead of letting sync programs first scan for changes. This brings overhead and is lowering the speed in case of local attached devices. If you doing this via network the story may be different.

30 MB/s is a pretty fast reading speed for a flash drive (at the time of writing this answer — years later speeds have increased). Writing is a lot slower. (Here are some example benchmarks.) The limiting factor is not the USB connection but the drive physics. And writing separate files is slower than the raw write speed you see in benchmarks: it takes time to write the metadata as well.

The fast speeds you're seeing at the beginning are writing to a write buffer somewhere. At some point the buffer gets full, and then the real disk write speed matters.

You might also see other slowdown effects, although since you're filling only 1/4 of a freshly formatting drive I don't think you're reaching the point where they start. If the filesystem gets close to full and has had many deleted files, it can start getting fragmented, so it takes longer to find a little room here and there for the files (this is mostly an issue with rotating media where it helps a lot to be able to store files in large consecutive chunks). On flash media, when a significant fraction is used and has been deleted, the data is spread between cells; each cell needs to be erased as a whole, so if many cells are partially affected by a write, that's a lot of cells to erase and partially rewrite. Also, on rotating media, data access speeds depend how far the data is from the center, but on a flash drive they're the same everywhere.