2

I have a filesystem with a series of date stamped archival backups of another filesystem. The backup uses hardlinks to copy only the delta between consecutive archival instances. The drive has had some data corruption and I am working to replace it, however some files are corrupted and not copied successfully. I have other copies of the damaged files to restore with, but I do not know a good way to replace the damaged file within the structure of hardlinks.

01/01       02/01       03/01        
 -file1  >>  -file1  x               Added in 01/01, deleted by 03/01
 -file2  >>  -file2  >>  -file2      Added in 01/01, never deleted
             -file3  >>  -file3      Added in 02/01, never deleted

In the above case there is one data store for file2, with two (or three depending how you count) hardlinks. If the base file data is corrupted, how can I use my backup file to restore file2 and retain its hardlinks?

Further Information:

  1. The original data lives on one physical device in a folder structure.
  2. The backups/archive copies are of the complete folder structure held on the original data device (1.). They are consecutive in time, deduplicated and hardlinked between themselves on the backup device.
  3. The copy I will be restoring from is a third device used to store images of the backup device for long-term cold storage.
  4. The errors have occurred on the backup device listed in (2.). I would like to have the corrupted files on this backup device restored from either the original location (1.) or the cold-storage device (3.), within the structure of the backups/archive.
  5. The backup logic:

    5.1. Find last date/time stamped backup folder <last backup folder> on device (2.)

    5.2. Make new empty backup folder <new folder> with current date/time stamp.

    5.3. Make hardlinked copy of files in last backup folder:

    cp -al <last backup folder> <new folder>
    

    5.4. Make copy of data from <source data> from (1.) to new backup folder, over hardlinked backup folder made in (5.3.):

    rsync -azH --delete <source data> <new folder>
    

Update: 14/03/2017

Having attempted to use the advice from one answer, the corrupt file at the destination cannot be inplace replaced. Clearly the destination at that location has a hard fault of some kind and the replacement data needs to go to a new physical location on disk.

  • "I have other copies of the damaged files to restore": how did you made these copies? cp loses hardlink information, various other tools don't. – kubanczyk Mar 7 '17 at 12:47
  • The files exist from their original location, and this backup/archive hierarchy has previously been copied to a long-term storage drive. Each are not on the same physical device so will have been copied without using a link – J Collins Mar 7 '17 at 13:42
  • 3
    If they are hard links, all names point to the same data, so if one of them is corrupted, all of them are. (That's what you basically get with any sort of deduplication.) But if they are on separate physical devices, they probably aren't hard links. So I can't tell what it really is. Without the relevant part of the logic that does the backup, or something like the ls -li output of the actual files it's really hard to say anything. – ilkkachu Mar 7 '17 at 14:15
  • @ilkkachu have clarified the structure within which I'm working so you can understand where the files are, are coming from and going to. – J Collins Mar 7 '17 at 16:00
  • Why the ZFS tag? If you used zfs, I'd expect you use its own snapshotting features rather than hard links. Also, ZFS does data checksumming so would detect corruptions introduced by the underlying device (resulting in I/O errors if there's no redundancy). – Stéphane Chazelas Mar 7 '17 at 16:17
2

Use rsync -azH --inplace

There are multitude of warning regarding this option in the man page:

--inplace

This option changes how rsync transfers a file when the file's data needs to be updated: instead of the default method of creating a new copy of the file and moving it into place when it is complete, rsync instead writes the updated data directly to the destination file. This has several effects:

(1) in-use binaries cannot be updated (either the OS will prevent this from happening, or binaries that attempt to swap-in their data will misbehave or crash),

(2) the file's data will be in an inconsistent state during the transfer,

(3) a file's data may be left in an inconsistent state after the transfer if the transfer is interrupted or if an update fails,

(4) a file that does not have write permissions can not be updated, and

(5) the efficiency of rsync's delta-transfer algorithm may be reduced if some data in the destination file is overwritten before it can be copied to a position later in the file (one exception to this is if you combine this option with --backup, since rsync is smart enough to use the backup file as the basis file for the transfer).

WARNING: you should not use this option to update files that are being accessed by others, so be careful when choosing to use this for a copy. This option is useful for transfer of large files with block-based changes or appended data, and also on systems that are disk bound, not network bound.

The option implies --partial (since an interrupted transfer does not delete the file), but conflicts with --partial-dir and --delay-updates. Prior to rsync 2.6.4 --inplace was also incompatible with --compare-dest and --link-dest.

  • Unfortunately because the backup file is current, rsync will not copy. When I include -I to force the update, the base file being corrupt can't be overwritten. I seem to need to make a new copy of the file and update all inode links to the new file, then delete the old. rsync is giving (code 11) at receiver.c – J Collins Mar 14 '17 at 12:01
  • I implicitly assumed that first of all you are on a good hardware already and on a corrected filesystem. On a fully functioning disk/filesystem, you will be able to overwrite a corrupted file. Migrate using just about any backup tool. Even tar is hard-link-aware. – kubanczyk Mar 16 '17 at 8:21
  • The problem is I can't actually copy the corrupted files off the bad hardware until I can fix them on the bad hardware, it's a bit precarious! Well I can get the file backup but not the whole deduplicated structure it is in.! – J Collins Mar 16 '17 at 9:59
  • Firstly, you need to tar the entire corrupted "deduplicated structure" and restore it to a good hardware with a sound filesystem. Then you can proceed with the tools to make the "deduplicated structure" correct&useful again. – kubanczyk Mar 16 '17 at 10:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.