3

This evening, I had to hard shut down my computer after some kind of kernel panic.

When I rebooted, I noticed my ~/.ssh/id_rsa had been replaced with an empty file.

Rebooting to a USB and running fsck on my home partition reported that the filesystem was in good shape.

This alone is not a problem. I access to the original key. However, I am concerned that other files may have been similarly truncated.

My last backup, using deja-dup, was three days ago, so I could just do a full roll-back, but I would rather just ask deja-dup what files have changed since then and look for "suspicious" files.

This seems to be exactly the purpose of duplicity verify, so after some man page skimming, I tried:

duplicity verify --verbosity 4 --no-encryption file:///path/to/backup/ /home/${USER}

which ran to completion without reporting changes. At a minimum, I expected my ~/.ssh/id_rsa to be detected, but I have added, removed, and changed other files.

My next try was then the same, but with the --compare-data flag:

duplicity verify --verbosity 4 --no-encryption file:///path/to/backup/ /home/${USER}

Which seems to report that every file in my home folder is new, starting like:

Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Fri Dec 15 11:43:22 2017
Difference found: File . has permissions 1000:1001 700, expected 0:0 555
Difference found: New file .AndroidStudio2.3
Difference found: New file .AndroidStudio2.3/config
Difference found: New file .AndroidStudio2.3/config/inspection
Difference found: New file .AndroidStudio2.3/config/inspection/Default.xml

I have had Android Studio installed for months, so it was most certainly in my backup from three days ago, and ls reports that Default.xml still exists and is 108 bytes long.

As a final effort, I changed the target directory to /, since that seemed to be the root when using duplicity list-current-files, which required adding some regular expressions to limit duplicity to only consider my home folder:

duplicity verify --verbosity 4 --compare-data --no-encryption --include-regexp ".*home/${USER}/\.ssh.*" --exclude-regexp ".*" file:///path/to/backup/ /

Which had the interesting effect of reporting that my home folder doesn't exist:

Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Fri Dec 15 11:43:22 2017
Difference found: File home is missing
Difference found: File home/${USER} is missing
Difference found: File home/${USER}/.AndroidStudio2.3 is missing
Difference found: File home/${USER}/.AndroidStudio2.3/config is missing
Difference found: File home/${USER}/.AndroidStudio2.3/config/inspection is missing
Difference found: File home/${USER}/.AndroidStudio2.3/config/inspection/Default.xml is missing

At this point, I am certainly just misunderstanding how I should use duplicity. How can I verify a backup generated by deja-dup?

duplicity list-current-files has output starting:

Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Fri Dec 15 11:43:22 2017
Tue Feb  6 19:36:56 2018 .
Wed Aug  2 17:32:09 2017 home
Tue Feb  6 00:38:20 2018 home/${USER}
Sat May 13 18:49:24 2017 home/${USER}/.AndroidStudio2.3
Thu Jun 22 19:42:14 2017 home/${USER}/.AndroidStudio2.3/config
Sat May 13 18:57:45 2017 home/${USER}/.AndroidStudio2.3/config/inspection
Sat May 13 18:57:45 2017 home/${USER}/.AndroidStudio2.3/config/inspection/Default.xml
4
  • To be clear, the various outputs where I have listed ${USER} actually had my user account name, I have just replaced them for privacy
    – Sompom
    Feb 16, 2018 at 15:27
  • Sompom, can you try duplicity verify --compare-data --no-encryption --file-to-restore home file:///path/to/backup/ / ?
    – ede-duply.net
    Feb 17, 2018 at 19:11
  • @ede Thanks for the reply! I have actually been running that command for an hour, and it appears to be working. If you would like to post an answer I will accept it (once it runs to completion and I verify it does what I need), otherwise I will self-answer
    – Sompom
    Feb 17, 2018 at 19:52
  • @ede with --file-to-restore and --compare-data duplicity seems to have done something useful, in that it correctly noticed lots of files had been modified/added/deleted since the backup was taken, but it still has not detected ~/.ssh/id_rsa is the wrong size. At this point, I am not sure that --compare-data does what the man page seems to say it does!
    – Sompom
    Feb 17, 2018 at 20:45

1 Answer 1

Reset to default
4

@ede and I found the same solution at the same time, in my case on the duplicity mailing list

duplicity verify needs the --compare-data flag in order to verify the on-disk files, and it needs the --file-to-restore flag in order to look in the proper directory, so the final command that solved my problem is:

duplicity verify --verbosity 4 --compare-data --file-to-restore=/home/${USER} --no-encryption file:///path/to/backup/ /home/${USER}/

Unfortunately, this still does not detect that ~/.ssh/id_rsa was damaged. At the same time, trying to restore from the backup restored a 0-byte file... It's quite possible something happened to my file weeks ago.

3
  • hey Sompom, you correctly found out that duplicity has a quirk where in/exclude only works while looking at the local storage. for restore/verify you will need --file-to-restore=path to limit the incrementing over the backup contents. --compare-data actually is needed if you want to actually coompare against the local data, w/o it the backup will just be tested for successful restorability. ..ede/duply.net
    – ede-duply.net
    Feb 18, 2018 at 11:26
  • When only testing for successful restorability (the backup files still match a checksum stored alongside the backup at backup time), why does duplicity still need the local directory to be referenced (twice)? I can understand that --file-to-restore could be to restrict the test to only the feasibility of a partial restoration - but why must we also reference the actual original location of the data we backed up if we are not actually comparing against it? Or have I misunderstood? (I made the same assumption as @Sompom based on this syntax).
    – Phil
    Apr 10, 2020 at 22:53
  • Also when selecting a specific sub-directory to compare I use a different syntax to above, --file-to-restore should only contain the path from within the backup not the full path - eg: duplicity verify -v4 --compare-data --no-encryption --file-to-restore=bkup-subdir sftp://my-server://archives/duplicity/my-bkup ~/data/bkup-subdir. So my local data directory is backed up to my-bkup on my remote server. Here we compare a sub-directory under the backup parent dir called bkup-subdir with the equivalent on local disk. Note file-to-restore doesn't contain the full path.
    – Phil
    Apr 10, 2020 at 23:56

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .