1

For a long time I've been using rsync and a usb cable to get the photos from my phone's SD card onto the PC. This is the current command:

rsync -azvu --ignore-existing --progress SD-Karte/DCIM/Camera/  /home/cardamom/Pictures/PhonePictures/

It took me a while to get it right by trial and error and reading the docs. There are probably many thousand of photos in each directory but one would not think that it would take a computer that long to compare two lists a thousand long and if there is anything in the source which is not in the destination, bring it accross.

Yet, if I take a single extra photo with the phone, or even no extra photo, it will sit there for what feels like two or three minutes thinking and then transfer the one photo in about a second.

Am not sure if the bottleneck is in the rsync command which is doing something I don't need like comparing hashes of file, or if it is in the SD card of the old smartphone, which somehow has some big inefficiency.

Is there anything inefficient in the rsync command, or otherwise some kind of measure to make it go quicker inside the phone?

Improve this question
10
  • 1
    I would probably drop the -z, (along with -v and --progress ) but each sync will need to fetch info of 1000s on computer and 1000s on phone, that take time. – Archemar Jun 12 '20 at 14:37
  • Can see that -z means 'compress' which sounds like a performance thing more than a functionality thing, will try without that now and see what happens. – cardamom Jun 12 '20 at 14:42
  • compress might be usefull if you have a slow network connection. – Archemar Jun 12 '20 at 14:46
  • Okay, with only the -z gone it is taking just as long. I think that both -v and --progress do much the same thing, ie tell you what it is doing. Since the slowness happens when almost nothing except 'sendng incremental file list' is printed, and the actual transferring, which is printed on the screen happens quickly, can't see how removing them would help. I think it's probably the SD card on the phone and there is not much you can do to speed this up but empty the photos folder out. – cardamom Jun 12 '20 at 14:52
  • 1
    You can try to do just a find /SD-Karte/DCIM/Camera -ls and find out how long it takes. It is more or less what rsync has to do. If the find takes a long time, then the issue is your connection to the phone and it might be improved with other mount options, if you tell us how it is mounted now. – Eduardo Trápani Jun 12 '20 at 23:07
3

If it takes too long, it might be because rsync cannot apply its quick check that only relies on size and time.

Not different filesystems store the time in different ways that means that what rsync stores is not what it later retrieves.

When that happens, rsync cannot apply the quick check and turns to a checksum check, that forces it to read the whole file just to be sure it is the same one, thus slowing the whole transfer.

ls -l --time-style=full-iso original_file
ls -l --time-style=full-iso already_synced_file

Tells you if that is the case. If it is, you have at least two options, depending on how far off they are:

  • Add --size-only option to just check the size and no the time.
  • Use the --modify-window option to take that into account.
Improve this answer
6
  • That is interesting, that modify-window thing and size-only, had never heard of them. I just did this: rsync -azvu --size-only --dry-run --modify-window=120 from/here/ /to/here/ Forgot to time it but it still took forever (ie two or three minutes) Will have a play with your ls commands and see what happens – cardamom Jun 15 '20 at 18:26
  • But you forgot the --omit-dir-timesand --no-perms. I also suggest build your flags the other way around, instead of starting with -a and removing things, I would start with just size-only and v and build from there. -a is meant mainly for the same (or similar) type of filesystem. – Eduardo Trápani Jun 15 '20 at 18:30
  • Ok, will try it, thanks. Did not know that order mattered.. – cardamom Jun 15 '20 at 18:34
  • Am playing with it, time rsync --dry-run --omit-dir-times --no-perms --size-only --modify-window=120 from/here/ /to/here/ seems not to be enough - it runs almost instantly telling me skipping directory . but it fails to transfer the one new photo. Will leave --dry-run in there till something is working, want to test it first before maybe wrecking any folders. – cardamom Jun 15 '20 at 19:35
  • From the docs: -a, --archive is is equivalent to -rlptgoD. – cardamom Jun 15 '20 at 19:41
1

So the following seems to make my photo syncing run really quickly - three seconds instead of three minutes. I got rid of the abbreviations so it is clear which is which:

time rsync --dry-run --verbose --compress --update --ignore-existing --recursive --group --times --progress --omit-dir-times --no-perms  --modify-window=120   from/here/  /to/here/

Get rid of time when no longer ned to time it, get rid of dry run when it seems to be doing what is meant to and is not harming anything.

Noticed as was building it there were quite a few accelerator and brake at the same time situations in the command and rsync never complained. For example, -a being -rlptgoD , the p is for --perms and for a while I also had --no-perms in there.

Will keep observing the dry runs for a while and make sure it continues to behave.

Improve this answer
2
  • If your SD card is FAT you must use modify-window=2 as per the man page – roaima Jun 15 '20 at 20:27
  • Thanks, just checked it with df -Th and it's not, it's ext4. There's a few others in there like tempfs and squashfs but no FAT. – cardamom Jun 15 '20 at 20:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.