40

I'm trying to copy the contents of a failing USB thumb drive. If I read the data too fast, the drive's controller chip overheats and the drive vanishes from the system. When that happens, I need to unplug the drive, wait a minute or so for it to cool, plug it back in, and re-start the copy.

I've got an old backup of the contents of the drive, so the obvious way to get the rest of the data is to use rsync to bring the backup up to date, but this runs into the whole "read too fast, the drive vanishes, and I need to start over" issue. Is there a way to tell rsync to only read X megabytes of data per minute? Alternatively, is it possible to tell it to suspend operations when the drive vanishes, and resume when it gets plugged back in?

4
  • 24
    Have you thought about cooling the drive?
    – AEonAX
    Dec 28, 2016 at 4:46
  • 3
    How full is the drive? If the drive is more than 50% full it is likely better to just mirror the failing drive first and then only start to deal with more complicated stuff on the mirror.
    – kasperd
    Dec 28, 2016 at 10:19
  • 1
    @AEonAX: While a cool (sic!) out-of-the-box idea, I wouldn't recommend this: you're putting additional thermal stress on the device, and it might fail completely. Dec 29, 2016 at 15:32
  • This feels like the geek's equivalent of a first world problem. I have it too, my USB dock gets overloaded by the huge backup job. Mar 5, 2021 at 23:36

4 Answers 4

48

Unlike DopeGhoti's experience, the --bwlimit flag does limit data transfer, with my rsync (v3.1.2).

test:

$ dd if=/dev/urandom bs=1M count=10 of=data
10+0 records in
10+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.0871822 s, 120 MB/s

$ du -h data
10M     data

$ time rsync -q data fast
0.065 seconds

$ time rsync -q --bwlimit=1M data slow
10.004 seconds

(note: my time output looks different to most time invocations (zsh feature), those times weren't edited by me)

Else, perhaps something along the lines of a double -exec in find. I believe that rsync -R should create & copy the parent folders, but if it doesn't, then cp --parents should.

$ find /failing/usb -exec rsync -R {} /somewhere/safe/ \; -exec sleep 1 \;

Note: also check out ddrescue, it might be right what you're looking for :)

2
  • 1
    Yup. DG may have gotten confused by the word "bandwidth" - but that applies to any data-carrying channel, not only network. Man page specifically says: --bwlimit=KBPS limit I/O bandwidth; KBytes per second, no mention of network. Casually browsing the code, this limits the read speed, regardless of destination. Plus, anecdote: I've been using this option for years now to limit local-to-local write speed. linux.die.net/man/1/rsync Dec 29, 2016 at 15:29
  • 2
    Okay, on second look, it actually counts the number of bytes written and sleeps for a calculated time. io.c, lines 832-3: if (bwlimit_writemax) sleep_for_bwlimit(n); In any case, this would probably work as intended, reading in short bursts and sleeping in-between. I would recommend starting out with a low value and carefully increasing, or just let the slow transfer run overnight. Dec 29, 2016 at 15:39
27

A bit of a MacGyver solution but I have had good success with it in the past:
Put an old USB 1 hub between the stick and the computer. No way in hell it is going to copy fast that way :-)

Other possibility: If you've got another stick available put it on another USB port connected to the same root-hub (so it shares bandwidth with your problem stick).
Now start a big write operation to the extra stick so both sticks are competing for bandwidth. For further control you can use ionice to lower the priority on the problem stick even further.

4
  • 2
    I'm sure there should be a way to make the ports work as USB 1.1 via a software solution, possibly by giving options to the USB kernel module. Dec 28, 2016 at 18:00
  • @AndréBorie: There is - bbs.archlinux.org/viewtopic.php?pid=1342593#p1342593 , but since there are effective throttles in SW, this is a bit of a brute-force solution. Dec 29, 2016 at 15:27
  • @Piskvor it negotiates a lower transfer rate with the device, just like using an USB 1 hub would do. I don't see anything "brute force" about it. Dec 29, 2016 at 18:13
  • Okay, bad choice of word, I admit. But IMHO reconfiguring devices would require special privileges (root or at least %adm), whereas controlling per-process read speed would be simpler. (Plus, there are devices that fail to work outside USB 2.0 at all - in violation of the spec) Dec 29, 2016 at 18:17
15

You can use rsync --bwlimit=RATE to throttle the file transfer speed which, based on comments below, actually specifically seems to work by throttling the read speed of the data, which is precisely what you're looking for.

5
  • 7
    ionice ? linux.die.net/man/1/ionice
    – steve
    Dec 27, 2016 at 23:29
  • 3
    ionice just gives priority, if two things are reading then, by default they get 50/50. ionice can adjust that, but if there is only one thing reading, then it will get 100% .
    – coteyr
    Dec 29, 2016 at 3:30
  • @coteyr You're right and that is why you need to combine it with a 2nd bandwidth eating process on the same USB root-hub. See my answer.
    – Tonny
    Dec 29, 2016 at 9:41
  • 3
    Fortunately, your answer is incorrect: --bwlimit sets I/O limit, not necessarily a network limit. Tested many times in various scenarios: this option makes rsync limit itself on read, regardless of the destination. (Found this out in a similar scenario, where fast writes to a slow local device made them fail) Dec 29, 2016 at 15:24
  • 1
    @piskvor - thanks for that! That's what I love about this place; I keep learning new things (:
    – DopeGhoti
    Dec 29, 2016 at 15:28
12

More solutions, in addition to the rsync --bwlimit=100, for 100KiB/s.

  1. Use ddrescue with the --max-read-rate option for a full disk copy. This will also allow you to restart a full disk copy where it left off after an error.

    ddrescue --max-read-rate=100K /failing/usb output.img output.logfile
    

ddrescue also has lots of other options relevant to data recovery, see the manual here.

  1. Use pv ("pipe viewer") with the --rate-limit option, or the throttle tool to rate limit pipes. pv can also show progress. You can make this work for any tool that uses a pipe or can use a pipe:

    tar -cf - /failing/usb | throttle -K 100 | tar -C /your/outputfolder/ -xvf -
    

    or

    tar -cf - /failing/usb | pv --rate-limit 100k | tar -C /your/outputfolder/  -xvf -
    

    Take a look at the pv manpage for more settings and examples. For throttle, you might want to also try out -w option, which determines how the speed is calculated, see the throttle manpage.

2
  • 1
    Now that's a useful tool, didn't know that, thanks! pv manpage has even more examples of various usages. Dec 29, 2016 at 18:21
  • I've reached for pv over and over again ever since discovering back in my tape jocking days. Back when tar wrote to tape drives. :) Jan 3, 2017 at 19:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.