This might be a edge case, but here goes...

I'm recovering data from a 2TB harddrive. It's mountable only if I use mount -r, and I'm using rsync --files-from to copy important directories.

Every once in a while it fails and needs to be restarted (powercycled). I wish there was a way to cache the entire directory tree, and use that cache spanning multiple mounts: in other words, when I need to remount the disk, I would provide the cache to be used for file lookup instead of doing more seeking on the disk.

Take as a given that the disk cannot be imaged in its entirety (with dd or ddrescue) because of insufficient backup space.

  • You don't even wanna tell us the filesystem...? – Hauke Laging Mar 20 '18 at 22:16
  • it happens to be NTFS. but next time it might be FAT32. or something elese. – afuna Mar 21 '18 at 7:17
  • 1. If the data really is that important, go get another disk (borrow one if necessary). If it's larger than 2TB you could create a 2TB image file and dd_rescue to that. 2. If you're selecting files and directories to copy, presumably once you've copied that directory you don't need to reference it again. So why would you need a cache? – roaima Mar 23 '18 at 9:11
  • Thinking aloud, your rsync should already be building a cache. When rsync has copied a file once, subsequent runs will not read from that file (rsync is actually a big cheater and uses file timestamps - I think it could be thrown off if you have file timestamps in the future and did not use -a to copy the exact timestamps). So the only thing left is to keep a persistent record of the directory traversals somehow, to benefit if you have a relatively large number of files. That's probably possible without hacking the kernel, but I think it will have to be coded from scratch – sourcejedi Mar 23 '18 at 9:13
  • I'm not aware of other cases where something like this was needed and implemented. (This assumes the code can use subdirectories as resume points. If the problem is too many files in a single directory to ever finish scanning that directory before a reboot failure, then it's a really nasty situation and I think it would require kernel hacking) – sourcejedi Mar 23 '18 at 9:14

The (meanwhile) only way I see to do that is independent of the used filesystem because it acts on the block layer.

What one would want is a kind of lvmcache: Seit it up for the device, then read the filesystem metadata (find /path/to/mountpoint -perm 700 -printf "") so that it gets copied to the cache device and then freeze the cache device.

Unfortunately it seems that lvmcache does not have such a freeze function.

But you can do something similar: You can set up a snapshot. You have to do that manually because LVM does support it only within a volume group. Create a snapshot in a VG and have a look at the details with dmsetup ls and dmsetup table for the involved devices. dmsetup does not care whether the involved devices belong to a volume group (or are LVM devices at all).

Usually you create a snapshot and continue using the main device. The previous state of the main device can be restored by reverting the snapshot because the original data is written to the snapshot before being changed on the original device. This is not what you want. You want to minimize accesses to the origin device. Thus you do not mount the (new) origin device but the snapshot device.

Then you modify all sectors which contain metadata so that they are written to the snapshot device. Even if you write twice (because you change the data and then change it back to its original state) so that the data itself has not changed at all the sector keeps part of the snapshot device. This means that it is always read from the snapshot device. In case of a crash you just have to restore the dmsetup configuration and the origin device will not be accessed for metadata any more.

You may need a rather large snapshot with a small cluster size, though. But you do not have to cache the whole filesystem metadata at once. You can set up a snapshot, cache the metadata for a directory tree, copy all the files from there, delete the snapshot, create a new one and cache the metadata for the next subtree. You can see (IIRC with dmsetup status) how much of the snapshot space is in use.

For writing the inodes you can change the time stamp (touch /path/to/file), especially if you do not need the original value. Or you (maybe after writing the original value somewhere) execute e.g. chmod o=rwx /path/to/file ; chmod o= /path/to/file. For writing all the sectors which belong to directory entries you may rename all the files (to some unused name and back):

mv "$file" "${file}${unique_suffix}"
mv "${file}${unique_suffix}" "$file"

I am not sure whether the page cache is clever enough to detect that there has been no change so you may check that the snapshot is really written to.

These operations on all files should be done (perhaps for a subdir only) with

find /path/to/mountpoint -type f -exec ... + # for chmod and mv
find /path/to/mountpoint -type d -exec ... + # for mv
  • I'm tempted to mark this as the correct answer, but will hold out a bit longer to see if there's a read-only solution. – afuna Mar 28 '18 at 22:01
  • @afuna From the disk's perspective this is read-only. – Hauke Laging Mar 28 '18 at 22:16
  • Oh. I hadn't fully understood the mechanism before, but now you've made me think it through and I see how it works. When I mount the [empty] snapshot, it passes any reads to an 'unchanged' sector through to the physical disk. When I write that sector back, it will be saved in the snapshot and then subsequent reads to that sector won't be passed through. Is that correct? – afuna Mar 28 '18 at 22:28
  • @afuna Yes.But I just realize that my suggestions how to "write" to the relevant parts would work well with ext4 but I do not know enough about NTFS and FAT32 to assess how to do that with them. – Hauke Laging Mar 28 '18 at 22:42
  • @afuna New idea: It might be possible to use blktrace to get all the sector numbers which get read. Read-ahead should be disables for the device when doing this. Then all these sectors could be written (with the same data). This would lead to all these sectors to be read twice, though. – Hauke Laging Mar 28 '18 at 22:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.