I have two partial disk images from a failing hard drive. File B contains the bulk of the disk's contents, with gaps where sector reads failed. File A is the result of telling ddrescue to retry all the failed sectors, so it is almost entirely gaps, but contains a few places where rereads succeeded. I now need to merge the interesting contents of File A back into File B. The algorithm is simple:

while not eof(A):
   read 512 bytes from A
   if any of them are nonzero:
       seek to corresponding offset in B
       write bytes into B

and I could sit down and write this myself, but I would first like to know if someone else has already written and debugged it.

(To complicate matters, due to limited space, File B and File A are on two different computers -- this is why I didn't just tell ddrescue to attempt to fill in the gaps in B in the first place -- but A can be transferred over the network relatively easily, being sparse.)

  • I don't really get why you used two computers, merging afterwards won't make it use any less space, ...? – frostschutz Nov 26 '13 at 18:48
  • @frostschutz One computer has enough disk space to accommodate a complete (500GB) disk image, but (for reasons too tedious to get into here) cannot be persuaded to communicate with USB-to-SATA adapters. The other computer can talk to USB-to-SATA adapters just fine, but only has enough space for half the disk image at a time. (In retrospect, we probably could've come up with a better plan for this operation, but having sunk nearly a month of I/O time into it, I'm not bloody starting over.) – zwol Nov 26 '13 at 19:18

Your algorithm is implemented in GNU dd.

dd bs=512 if=A of=B conv=sparse,notrunc

Please verify this beforehand with some test files of your choice. You don't want to inadvertently damage your File B. A better algorithm would be to check whether B also has zeroes at that position, alas that's something dd does not do.

As for two different computers, you have several options. Use a network filesystem that supports seeks on writes (not all do); transfer the file beforehand; or pipe through SSH like so:

dd if=A | ssh -C B-host dd of=B conv=sparse,notrunc
# or the other way around
ssh -C A-host dd if=A | dd of=B conv=sparse,notrunc

The ssh -C option enables compression, you'd be transferring gigabytes of zeroes over the network otherwise.

| improve this answer | |
  • I'd suggest netcat over ssh. I'm guessing that a machine that can't talk to a USB drive probably also can't run AES or even blowfish at line rate. – derobert Nov 26 '13 at 20:43
  • @derobert I don't think network was the main issue, but there are countless ways... File B could be exported as a network block device (nbd) as well. If you're fine with network traffic being unencrypted. – frostschutz Nov 26 '13 at 21:46
  • Well, this is the kind of thing I'd do over a direct gigabit cross-connect, so I'd be OK with that being unencrypted. But yeah, plenty of ways to do it. But you probably want to mention one of them, especially since you mention gzip/vs. ssh -C – derobert Nov 26 '13 at 21:48
  • @derobert, actually I tested it just now out of curiosity, and ssh -C works fine for the purpose, so I removed the gzip reference to avoid confusion. The ssh method is the one I mentioned as I believe it's the most readily available; you're free to pick any other. – frostschutz Nov 26 '13 at 22:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.