how to do version-specific backup/restore of Debian packages?

Question

summary

Note version-specific in the question title! This question is not, how to

1. On one Debian host, capture a list of all installed packages.
2. On another Debian host, install the latest versions of the listed packages.

This question is rather, how to

1. On a Debian host at some point in time,
   1. capture a package map containing
      • the name of each installed package
      • the version of each package currently installed
   2. any additional data required to ...
2. On that same host later in time, restore the package map, such that subsequently
   1. only the packages in the package map are installed
   2. for each packages in the map, only the appropriate version is installed

details

motivation

Recently I did a "big update" (i.e., involving lots of packages, or major packages like kernels, or both) to one of my Debian boxes, which did not update "cleanly" (i.e., with no significant problems). In future, when I apply a bad update to a Debian box, I'd like to be able to rollback (a) the package set on that same box (b) to the previous versions. (Without previously doing an image backup, which is slow.) Unfortunately I don't know how to do that. I do know how to

1. On one Debian box, capture a list of every package name in its installed package set.
2. On another box, install the latest version of every package in the list.

thanks to numerous webpages discussing how to do this, including several SEs. Most (including this, this, and this) do more or less sophisticated variations on

1. backup: dpkg --get-selections > ${PACKAGE_LIST_FILEPATH}
2. restore: dpkg --set-selections < ${PACKAGE_LIST_FILEPATH}

Probably the most comprehensive (and among the most scripted, which I prefer) SE on this topic is this one: it's one of few to recommend also backing up repository lists (with cp -R /etc/apt/sources.list*) and repo keys (with apt-key exportall), but it also uses dpkg as above. This SE recommends using aptitude (which, FWIW, I prefer, and suspect does a better job than dpkg) for both backup and install, and this SE recommends deborphan for the backup.

But none of the above save package versions, and so none work for my usecase. (IIUC--am I missing something?)

hypothesis

Obviously one must backup (and be able to restore as needed) arbitrary files from one's devices. Similarly, one should be able to backup/restore ones packages, and integrate that into one's periodically-run backup tool. Fortunately, backup/restore of a package list is well-understood. One can backup with, e.g., bash code like

## setup

PACKAGE_LIST_FILENAME='package.list'         # see use below
REPO_KEYS_FILENAME='repo.keys'               # ditto
DATE_FORMAT="%Y%m%d_%H%M%S"
BACKUP_TIMESTAMP="$(date +${DATE_FORMAT})"   # get current timestamp, to the second

# Parent directory of that to which you will backup.
BACKUP_ROOT="/media/$(whoami)/my_backup_drive"
BACKUP_DIR="${BACKUP_ROOT}/backup_${BACKUP_TIMESTAMP}"
echo -e "Backing up to ${BACKUP_DIR}" # debugging
# TODO: check BACKUP_DIR is writable, has sufficient freespace, etc

# ASSERT: all following are subdirs of BACKUP_DIR
ETC_BACKUP_DIR="${BACKUP_DIR}/etc"           # backup /etc , typically not large
PKG_BACKUP_DIR="${BACKUP_DIR}/packages"
REPO_BACKUP_DIR="${BACKUP_DIR}/repos"   # duplicates some of /etc, but disk is cheap ...
for DIR in \
  "${ETC_BACKUP_DIR}" \
  "${PKG_BACKUP_DIR}" \
  "${REPO_BACKUP_DIR}" \
; do
  # make the backup dirs
  mkdir -p "${DIR}"  # TODO: test retval/errorcode, exit on failure
done

PACKAGE_LIST_FILEPATH="${PKG_BACKUP_DIR}/${PACKAGE_LIST_FILENAME}"
touch "${PACKAGE_LIST_FILEPATH}"             # TODO: check that it's writable, exit if not

REPO_KEYS_FILEPATH="${REPO_BACKUP_DIR}/${REPO_KEYS_FILENAME}"
touch "${REPO_KEYS_FILEPATH}"                # TODO: check that it's writable, exit if not

## backup
## TODO: for all following: test retval/errorcode, exit on failure
## TODO: give user some progress indication (e.g., echo commands)

deborphan -a --no-show-section > "${PACKAGE_LIST_FILEPATH}" # or other op you prefer
sudo cp -R /etc/apt/sources.list* "${REPO_BACKUP_DIR}/"
sudo apt-key exportall > "${REPO_KEYS_FILEPATH}"
rsync --progress /etc "${ETC_BACKUP_DIR}"

And one can restore with, e.g., bash code like

## setup
## (remember to transfer constants used above)

RESTORE_DIR="set this equal to the BACKUP_DIR from which you are restoring!"
ETC_RESTORE_DIR="${RESTORE_DIR}/etc"
PKG_RESTORE_DIR="${RESTORE_DIR}/packages"
REPO_RESTORE_DIR="${PKG_RESTORE_DIR}/repos"
PACKAGE_LIST_FILEPATH="${PKG_RESTORE_DIR}/${PACKAGE_LIST_FILENAME}"
REPO_KEYS_FILEPATH="${REPO_RESTORE_DIR}/${REPO_KEYS_FILENAME}"

## restore
## TODO: test that all following are readable, exit if not
## TODO: for all following: test retval/errorcode, exit on failure
## TODO: give user some progress indication (e.g., echo commands)

rsync --progress "${ETC_BACKUP_DIR}" /etc
# following will overwrite some of /etc restored above
sudo apt-key add "${REPO_KEYS_FILEPATH}"
sudo cp -R "${REPO_RESTORE_DIR}/sources.list*" /etc/apt/
# TODO: CHECK THIS SYNTAX!
# see my question @ https://serverfault.com/questions/56848/install-the-same-debian-packages-on-another-system/61472#comment920243_61472
sudo xargs aptitude --schedule-only install < "${PACKAGE_LIST_FILEPATH}" ; aptitude install

So IIUC, we must extend the above to handle package versions. This seems doable using a strategy like the following:

backup

1. Write a package map file, instead of a package list file. Thanks to RobertL and cas, I now know that I can get a package map file for a Debian host in a useful format (one package per line, each line containing {package name (and architecture, but only if more than one), TAB, package version string}. One can then access
   • package names with awk '{print $1}' < "${PACKAGE_LIST_FILEPATH}"
   • package versions with awk '{print $2}' < "${PACKAGE_LIST_FILEPATH}"
2. (extra credit) Archive the current versions of the raw/.deb packages (as indicated by cas) by feeding the package names to dpkg-repack.
3. (extra credit) Convert the archived packages into a local APT repository. This is hopefully described in operational detail by this Debian wikipage, but I'm still trying to figure that out.

restore

glossary: * current refers to the state of the host before the user attempts to restore. * current PMF refers to a package map file generated for the current host, containing package tuples (key=package name, value=package version) currently installed. * backup PMF refers to a package map file generated for the host at some prior time, containing package tuples to be restored.

1. Create a current PMF.
2. Restore repositories and keys by "the usual means" (see above).
3. Read the backup PMF.
4. For each package tuple in the current PMF but not the backup PMF: uninstall the package (e.g., sudo aptitude remove [package_name]).
5. For each package tuple in the backup PMF: install the given version of the package (e.g., sudo aptitude install [package_name]=[package_version]). Thanks to Rui F Ribeiro and bigbenaugust for noting that aptitude and apt-get support install package=version.
   • (extra credit) If the restored repositories do not contain the desired version of a package, install it from the backup-local APT repository.

test

TODO! I'll start by doing a manual restore for a box for which I don't yet have a backup like the above (since, of course, I don't yet have a backup like the above for any box :-) but for which I have console spew.

Answer 1

Some useful notes to help you achieve your goal:

1. You can get a list of installed packages and their versions by running:

   dpkg-query -W
   

2. You can create an archive of .deb packages of all currently installed packages by installing dpkg-repack and running something like this:

   dpkg-query -W | awk '{print $1}' | xargs dpkg-repack
   

   This will dpkg-repack all currently installed packages. This archive is the crucial missing part to your restore - without it, you may not be able to restore the exact same package set (especially if you use testing or unstable).

   WARNING: dpkg-repack repacks the current, possibly modified, contents of all conffiles. If you want the pristine, original packages, you'll have to get the .deb files from /var/cache/apt/archives or from a Debian mirror.

   The .deb archive can be turned into an apt-gettable repository by following the instructions at https://wiki.debian.org/HowToSetupADebianRepository or you can just install them all with dpkg -iBE *.deb (or dpkg -iRBE /path/to/deb/files/ if there are too many to fit on one command line).

3. You'll still need to use -get-selections and --set-selections in order to retain details like de-installed packages.

Answer 2

After struggling with this for a bit I came up with the below which is a combination of few different approaches

Snapshot all installed packages with exact debian name

dpkg --list | awk '/^ii/ { print $2,$3,$4 ;}'| tr ' ' '_'| tr '\t' '_'| sed 's/$/.deb/g'| sed 's|:amd64||g'| sort -u > dpkg_query_pkgs.txt

Clean your apt cache and download all installed packages again fresh so their isn't multiple versions.

apt-get clean; dpkg -l | grep "^ii"| awk ' {print $2} ' | xargs sudo apt-get -y install --reinstall --download-only

Or to explicitly pass the currently installed version in

apt-get clean; dpkg --list | awk '/^ii/ { print $2"="$3 ;}'| xargs sudo apt-get -y install --reinstall --download-only ;

Snapshot installed debs from archives

ls -1 /var/cache/apt/archives| grep deb|sed 's|\%3a|:|g' | sort -u > apt_archives_pkgs.txt

Then you can confirm installed debs vs archives

diff -y apt_archives_pkgs.txt dpkg_query_pkgs.txt

This allows you to test that you can export just the exact names from dpkg --list and it will match what you have in apt cache archives.

Answer 3

You can use apt-clone:

1. Install it with sudo apt install apt-clone

2. Run apt-clone clone filename to create a small archive filename.apt-clone.tar.gz that includes only references to the installed package names and versions. Plus the APT sources.list files, the APT keyrings and other information to later restore the same package configuration in the same versions.

3. On your target system (which may be the same system if it's for backup purposes), run sudo apt-clone restore filename.apt-clone.tar.gz to restore the system state.

Additional hints:

• Including packages. When creating the clone with apt-clone clone --with-dpkg-repack filename, the clone archive will also include .deb files of manually installed packages that are not found in the package archives, and of packages that are still available in the package archives but not in the installed version. Those .deb files are repacked from the installed files, so the original .deb file does not have to be available.

• "Version mismatch". Sometimes, apt-clone clone will complain about version mismatches for some packages. These packages are still included in the clone and will be restored, just not in the exact same version. Not sure if and how this could be fixed.

• No long-term backups. This tool is most useful when the time difference between cloning and restoring from a clone is small (hours to weeks, maybe). It's not suitable for longer-term system backups if you want to restore the system in its exact state. This is because apt-clone works with references to package names and versions, and some package versions may vanish from package archives, and even whole PPA package archives may vanish.

• Docs. apt-clone --help provides accurate information about other available options, while man apt-clone is slightly outdated.