16

I am trying to set up a staging environment in a VM, in order to test updates before applying them to my main system.

In order to do so, I have done a basic installation of Debian Wheezy (same as on the main system) in the VM, then ran as root from within the VM:

# dpkg --clear-selections
# dpkg --add-architecture i386
# apt-get update
# ssh me@main-system 'dpkg --get-selections | grep -v deinstall' | \
  dpkg --set-selections

The i386 architecture is unfortunately needed in my case; the system is amd64 native.

The problem is with dpkg --set-selections run in the VM. I do have some packages that require special handling (those are actually the main reason why I want a staging environment in the first place) but when I run the last command above, I get about a gazillion lines of output like:

dpkg: warning: package not in database at line NNN: package-name

for packages that really should be available in the base system. Examples include xterm, yelp and zip.

Now for my question:

What is the specific process for transferring the package selection list from one Debian system to another (assuming same Debian release level, in Wheezy) and then subsequently applying those changes? The goal is that both have the same list of installed packages, ideally such that doing a diff between the outputs of dpkg --get-selections or dpkg --list on the two comes back showing no differences.

The grep -v deinstall part is borrowed from Prevent packages from being removed after doing dpkg --set-selections over on Ask Ubuntu.

I have changed the source in the VM to be the same as on the main system, also installing apt-transport-https:

deb https://ftp-stud.hs-esslingen.de/debian/ wheezy main non-free
deb-src https://ftp-stud.hs-esslingen.de/debian/ wheezy main non-free
deb https://ftp-stud.hs-esslingen.de/debian/ wheezy-updates main non-free
deb-src https://ftp-stud.hs-esslingen.de/debian/ wheezy-updates main non-free
deb [arch=amd64] http://archive.zfsonlinux.org/debian wheezy main

Looking at the --set-selections output, I'm seeing:

dpkg: warning: package not in database at line 1: a2ps
dpkg: warning: package not in database at line 1: abiword
dpkg: warning: package not in database at line 1: abiword-common
dpkg: warning: package not in database at line 1: abiword-plugin-grammar
dpkg: warning: package not in database at line 1: abiword-plugin-mathview
dpkg: warning: package not in database at line 1: accountsservice
dpkg: warning: package not in database at line 1: acl
dpkg: warning: package not in database at line 4: aglfn
dpkg: warning: package not in database at line 4: aisleriot
dpkg: warning: package not in database at line 4: alacarte
dpkg: warning: package not in database at line 4: alien
...

The line numbers looked odd, and the corresponding portion of the output of --get-selections is:

a2ps install
abiword install
abiword-common install
abiword-plugin-grammar install
abiword-plugin-mathview install
accountsservice install
acl install
acpi-support-base install
acpid install
adduser install
aglfn install
aisleriot install
alacarte install
alien install

Notice that in between acl and aglfn are acpi-support-base, acpid and adduser for which no errors are being reported. It seems that the packages for which errors are being reported are either un according to dpkg -l, or dpkg -l doesn't have any idea at all about them (dpkg-query: no packages found matching ...). I know there are some locally installed packages, but not many. i386 doesn't figure until gcc-4.7-base:i386 install much farther down the list (line 342 in the --get-selections output).

  • dpkg --get-selections … | … dpkg --set-selections is the way to replicate package selections. If it's failing for you, you're going to need to provide more information as to why. Post your full set of sources on both systems, the output from dpkg --get-selections, and the full transcript from dpkg --get-selections (maybe we can discern a pattern by seeing what works and what doesn't). If the main system amd64 or i386? Does it work if you don't add the i386 architecture? – Gilles Mar 21 '15 at 19:18
  • @Gilles Does it really need to be the full output from dpkg --get-selections (and I assume you meant the dpkg --set-selections transcript?)? Even just the former is over 3,600 lines and right around 100 KB. – a CVn Mar 21 '15 at 19:27
  • You need to post enough for us to discern the pattern of which packages trigger errors and which ones don't (unless you can figure out by yourself). A copy of /var/lib/dpkg/Packages (which is even longer) could be useful, too — this could be a symptom of a broken Packages file. – Gilles Mar 21 '15 at 19:29
  • @Gilles Interesting point. I hadn't really considered the possibility, but I do use different mirrors on these. Let me see if it still happens if I point the VM at the same Debian mirror. Stand by. – a CVn Mar 21 '15 at 19:31
  • @Gilles That didn't really make any discernable difference, but I did find something else. See the updated question. – a CVn Mar 21 '15 at 19:52
14

To clone a Debian installation, use the apt-clone utility. It's available (as a separate package, not part of the default installation) in Debian since wheezy and in Ubuntu since 12.04. On the existing machine, run

apt-clone clone foo

This creates a file foo.apt-clone.tar.gz. Copy it to the destination machine, and run

apt-get install apt-clone
apt-clone restore foo.apt-clone.tar.gz

If you're working with an old system where apt-clone isn't available, or if you just want to replicate the list of installed packages but not any configuration file, here are the manual steps.

  • On the source machine:

    cat /etc/apt/sources.list /etc/apt/sources.list.d >sources.list
    dpkg --get-selections >selections.list
    apt-mark showauto >auto.list
    
  • On the target machine:

    cp sources.list /etc/apt/
    apt-get update
    /usr/lib/dpkg/methods/apt/update /var/lib/dpkg/
    dpkg --set-selections <selections.list
    apt-get dselect-upgrade
    xargs apt-mark auto <auto.list
    

I believe that you're affected by an incompatible change in dpkg that first made it into wheezy. See bug #703092 for background.

The short story is that dpkg --set-selections now only accepts package names that are present in the file /var/lib/dpkg/status or /var/lib/dpkg/available. If you only use APT to manage packages, like most people, then /var/lib/dpkg/available is not kept up-to-date.

After running apt-get update and before running dpkg --set-selections and apt-get -u dselect-upgrade, run the following command:

apt-cache dumpavail >/tmp/apt.avail
dpkg --merge-avail /tmp/apt.avail

From jessie onwards, you can simplify this to

apt-cache dumpavail | dpkg --merge-avail

Alternatively, run

/usr/lib/dpkg/methods/apt/update /var/lib/dpkg/

or even simpler

apt-get install dctrl-tools
sync-available

Another simple method that doesn't require installing an additional package but will download the package lists again is

dselect update

See the dpkg FAQ for more information. (This is mentioned in the dpkg man page, but more in a way that would remind you of the issue if you were already aware, not in a way that explains how to solve the problem!)

Note that cloning a package installation with dpkg --set-selections doesn't restore the automatic/manual mark in APT. See Restoring all data and dependencies from dpkg --set-selections '*' for more details. You can save the marks on the source system with

apt-mark showauto >auto.list

and restore them on the target system with

xargs apt-mark auto <auto.list
  • 2
    This problem ought to be much more visible than it is. I can't believe I've not heard of it before! – Faheem Mitha Mar 21 '15 at 20:24
  • Interesting. Going to give this a shot. Thanks! – a CVn Mar 21 '15 at 20:24
  • @FaheemMitha Yeah, I'm surprised that I never heard of it before today. I can't believe I haven't used dpkg --set-selections since pre-wheezy! – Gilles Mar 21 '15 at 20:28
  • Okay, that gave a very much shorter list of warnings from --set-selections. Now how do I apply the changes? Even apt-get -u dist-upgrade only mentions a handful of packages (all ZFS-related) where there should be hundreds if not thousands of packages affected. – a CVn Mar 21 '15 at 20:38
  • @MichaelKjörling That's apt-get -u dselect-upgrade – Gilles Mar 21 '15 at 20:46
0

How I did it

  1. On source host:

    apt-get update && apt-get dist-upgrade
    dpkg --get-selection >/tmp/source.sel
    cat /etc/apt/sources.list
    

    care about /etc/apt/sources.list.d if used...

    Then copy source.sel file to destination host.

  2. On destination host:

    vi /etc/apt/sources.list 
    

    adapt sources.list even with different mirrors, but same component list.

    apt-get update
    dselect upgrade
    dpkg --set-selection </tmp/source.sel
    dselect install
    

    And now, my destination host is currently installing a lot of things, then I could take some time for posting this.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.