How do I resolve package dependencies for a different target system?

Question

(TL;DR: how do I determine the minimum set of packages to install, given an initial state and desired package list?)

I'm creating an Ubuntu installer ISO for systems without access to distribution mirrors. These are embedded devices which need only the base packages and a handful of locally-built packages plus all their dependencies. For instance, we're using Qt/Xcb, so the various libqt5* packages need to be pulled in.

At the moment, I'm using a brute-force approach to determine what to put in the pool/extras directory of the installer. First I seed with the packages that are mentioned in d-i pkgsel/include answer to debconf, then hunt down the recursive dependencies with the following bash command:

# 1. Extract the dependencies from packages/*, removing version
#    conditions and changing separator to newline.
# 2. Find the recursive dependencies of all these packages using
#    package database, extracting Depends and PreDepends.
# 3. For each package we require:
#    a. if we find it in our cache, just copy it into the ISO (using
#       hard link for speed and space)
#    b. else, if we find it already in the ISO, do nothing
#    c. else, pass it through to step 4.
# 4. Search for matching packages in repositories that are not virtual
#    or foreign-architecture.
# 5. Download matching package files to the ISO.
(for i in "$TOP/packages"/*.deb; do dpkg-deb -f $i depends; done) | sed 's/([^()]*)//g;s/[|,]/\n/g'                                     \
  | xargs -r apt-cache --recurse --important depends | sed -re 's/^  (Pre)?Depends://;/[<|:]/d;s/^  *//;' | sort -u                     \
  | ( while read p;                                                                                                                     \
    do test $(find "$TOP/extras" -name "${p}_*.deb" -exec mv '{}' "$REMASTER_HOME"/remaster-iso/pool/extras \; -printf "true" -quit)    \
      || test $(find "$REMASTER_HOME"/remaster-iso/pool/{main,extras} -name "${p}_*.deb" -printf "true" -quit)                          \
      || echo $p;                                                                                                                       \
    done; )                                                                                                                             \
  | uniq | sed -r 's/[^ ]+/!~rforeign!~v^&$/' | xargs -r aptitude -q2 -F '%p' search                                                    \
  | (cd "$REMASTER_HOME"/remaster-iso/pool/extras; xargs -r apt-get download)

Here, $TOP/packages is the seed-package directory, $TOP/extras is a cache of previously-downloaded packages, and $REMASTER_HOME"/remaster-iso is the root of the unpacked installer ISO.

Whilst the above works, it tends to fetch much more than we actually require, because apt-cache depends reports all the alternative dependencies of each package. For example, grub-pc depends on debconf; cdebconf provides debconf, so both debconf and cdebconf are included, along with their dependencies. Also, xserver-xorg depends on xserver-xorg-video-all | xorg-driver-video, so my script pulls all the video drivers - but I've already declared a dependency on xserver-xorg-video-fbdev, so I would like that to satisfy the requirements on its own.

What I would like is the equivalent of running aptitude --download-only install ${MY_PACKAGES} on the target system. But of course, running that on the build host will miss most of the packages because they are already installed there.

I'm currently looking at the python-apt library, but I can't find any examples of initializing it for a target system rather than the host where it's running. Has anybody done something similar, from which I can copy?

Alternatively, are there other approaches that might work? I'm considering creating a pristine, minimal chroot and running the aptitude command in that, but I'm concerned that will be a maintenance burden and hard for others to set up on their own machines.

I'm hoping the solution can be used (with a new initial state) to find the necessary dependencies when any packages are to be updated, as I'll need to write a consistent set to removable media for the target system (still with no access to distribution mirrors).

Answer 1

debootstrap a clean new system in a directory, do what you need within that directory using chroot, and afterwards take all that you find within chrootdir/var/cache/apt/archives/.

Edit after the second answer: I just checked the manual and it appears that one can also use debootstrap to do most if not all of what you want: it can be run under fakeroot, and it has --download-only and --print-debs options as well as --include to add additional packages into the mix and resolve their dependencies, too.

Wasn't sure about using two different sets of sources before I read the source - it seems to support that, too :)

For example:

% fakeroot /usr/sbin/debootstrap --variant=minbase --download-only --include=dnsutils wheezy /tmp/foo "http://ftp.us.debian.org/debian file:///local_packages_dir/"
[...]
I: Resolving dependencies of required packages...
I: Resolving dependencies of base packages...
I: Found additional required dependencies: [...]
I: Found additional base dependencies: [...]
[...]

There is a bit of overhead:

% du -sh /tmp/foo/var/cache/apt/archives/
40M     /tmp/foo/var/cache/apt/archives/
% du -sh /tmp/foo/
68M     /tmp/foo/

The actual download can be optimized using something like apt-mirror, so identical files don't get downloaded from a remote location each time.

Answer 2

It's a bash script rather than python-apt, but I have found dpkg-offline which contains similar logic. I've looked at how it works and have managed to create a script of my own to do the necessary downloading - reproducibly and as an ordinary non-root user:

#!/bin/bash
set -e

die() {
    exec >&2
    printf '%s\n' "$@"
    exit 1
}

test "$UID" != "0" || die \
    "Do not run '$0' as root, because it will break your packages."

test -n "$1" || die \
    "Usage: $0 <package-dir> [package] ..." \
    "  package may be followed by _ - + &m etc; like 'aptitude install'" \
    "  local packages must be listed in package-dir/Packages.gz" \
    "  downloaded packages are saved to package-dir/ubuntu"

PACKAGE_DIR="$1"; shift

BASE_DIR=$(mktemp -d)
trap 'rm -r "$BASE_DIR"' EXIT

(
    cd $BASE_DIR

    # Skeleton directory structure
    mkdir -p \
        etc/apt/preferences.d \
        etc/apt/sources.list.d \
        var/cache/apt/archives \
        var/lib/apt/lists \
        var/lib/aptitude \
        var/lib/dpkg \
        var/log

    # Use GPG keys from host, and create empty dpkg status
    cp /etc/apt/trusted.gpg etc/apt
    touch var/lib/dpkg/status

    cat >etc/apt/sources.list <<-EOF
    # Locally-built packages
    deb file://$PACKAGE_DIR /

    # Upstream distribution to pillage
    deb http://gb.archive.ubuntu.com/ubuntu/ trusty main restricted universe
    EOF

    # Don't let user's ~/.aptitude/* influence anything
    export HOME="$BASE_DIR"
    APT_OPTIONS=(-q2 \
        -o "Apt::Architecture=amd64" \
        -o "Apt::Install-Recommends=0" \
        -o "Dir::Etc=$BASE_DIR/etc/apt" \
        -o "Dir::State=$BASE_DIR/var/cache/apt" \
        -o "Dir::Cache=$BASE_DIR/var/cache/apt" \
        -o "Dir::Cache::Archives=$BASE_DIR/var/cache/apt/archives" \
        -o "Dir::Log=$BASE_DIR/var/log" \
        -o "Dir::State::Lists=$BASE_DIR/var/lib/apt/lists" \
        -o "Dir::State::status=$BASE_DIR/var/lib/dpkg/status" \
        -o "DPkg::Options::--admindir=$BASE_DIR/var/lib/dpkg" \
        )

    APTITUDE="aptitude ${APT_OPTIONS[@]}"

    $APTITUDE update
    $APTITUDE --download-only -y --allow-untrusted install "$@"
)

# We now have all required package files download into cache; put them
# into output directory.
ln -t "$PACKAGE_DIR/ubuntu" -vf var/cache/apt/archives/*.deb

# Create updated package index
( cd "$PACKAGE_DIR" && dpkg-scanpackages -m . | gzip -c >Packages.gz )

I could probably use this to create a Python program to do the same, but the shell script is likely to be adequate for my purposes.