I'm using duplicity on Ubuntu Xenial for creating and encrypting backups and storing them on an S3 bucket (Scaleway).

This is the backup script I'm using:

source ./.scw-configrc

DATE=`date +%Y-%m-%d`
TODAY=$(date +%d%m%Y)

currently_backuping=$(ps -ef | grep duplicity  | grep python | wc -l)

if [ $currently_backuping -eq 0 ]; then
# Clear the recent log file
cat /dev/null > ${LOGFILE_RECENT}

log ">>> removing old backups"
duplicity remove-older-than ${KEEP_BACKUP_TIME} ${SCW_BUCKET} >> ${LOGFILE_RECENT} 2>&1

log ">>> creating and uploading backup"
duplicity \
    incr --full-if-older-than ${FULL_BACKUP_TIME} \
    --asynchronous-upload \
    --exclude-globbing-filelist exclude-list.txt \
    --encrypt-key=${GPG_FINGERPRINT} \
    --sign-key=${GPG_FINGERPRINT} \


I exclude these files from the backup (exclude-list.txt):

  • /tmp
  • /proc
  • /sys
  • /boot
  • /dev
  • /mnt
  • /lost+found
  • /cdrom
  • /swapfile
  • /vmlinuz
  • /vmlinuz.old
  • /initrd.img
  • /initrd.img.old

I want a full backup, so I'm setting / as the source in .scw-configrc: export SOURCE="/"

I have no problems with GPG keys or connecting to the bucket. The issue I'm having that even though I'm running the backup script with sudo, it seems to not back up certain files, and so when I do a full restore on a new server instance, some services are missing.

When I tried to do a full copy from the source server to a new instance with rsync, those same services copied and ran successfully. One example of this is postgres, jenkins and postfix. What I can tell is that all three of the services are running under different users rather than root. And both postgresql and postfix are located under /usr/lib which seems to be missing a lot of files, while jenkins runs from /usr/bin/, which is also missing the files.

This is the restore script (although I'm pretty sure the files I'm talking about don't back up in the first place):

source ./.scw-configrc

if [ $# -lt 2 ]; then
        echo -e "Usage $0 <time or delta> [file to restore] <restore to>
    \t$ $0 2018-7-21 recovery/  ## recovers * from closest backup to date
\t$ $0 0D secret data/  ## recovers most recent file nammed 'secret'";
exit; fi

if [ $# -eq 2 ]; then
    duplicity -vd --force \
        --rsync-options="--super --del --recursive --force --ignore-errors" \
        --time $1 \
        ${SCW_BUCKET} $2

if [ $# -eq 3 ]; then
    duplicity -vd --force \
        --rsync-options="--super --del --recursive --force --ignore-errors" \
        --time $1 \
        --file-to-restore $2 \
        ${SCW_BUCKET} $3

Are there issues with file ownerships? Or am I doing something wrong? The services that don't seem to run all seem to be running from subdirectories of /usr/

  • 1
    You should probably do a duplicity verify after every backup to increase confidence on what is happening. And try the same backup to some local machine or filesystem, as this doesnt need much effort to setup, and then test for similar problems. Does the remote bucket have size limits, or data throughput limits? – meuh May 27 at 16:51
  • it doesn't have limits and it's weird that only some specific things are missing. Thanks for letting me know about duplicity verify, will definitely utilize it. – n3onis Jun 3 at 9:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.