What is the motivation or cause behind inconsistent configuration file modification screens?

Question

Through updating debian OS on various hosts (apt-get dist-upgrade), I've seen many often small variations of the same screen; when a configuration file has been modified, it will tell you a new version is available, and ask you to fix the problem: you've modified the file and apt/dpkg cannot reliably integrate your changes with the new version.

Now this options menu is different between various files. Sometimes even within the same package.

Some files will have a "package configuration" console UI, others will use plain console output. The options available in the package configuration UI are not always the same.

The name of the modified/new version file is not always the same. The location where the new file is stored is not always the place where it should be (sometimes it's in 'tmp'.). The naming scheme for the new file isn't consistent either (some packages use a random name, some use -new, others -dpkg-new, and so on.)

Some packages refuse to tell you where the dpkg default is. Some packages don't provide you with the previous version with extension dpkg-old or -old or -dist or -dpkg-dist or -old-nameofpackage, but others do. (So in some cases you can manually 3-way merge, in other cases you can't). Some packages add a handy 'do a 3 way merge between versions' (which is always the first thing I try as it can often automatically solve the problem: most changes in the config files are usually just typo fixes in comments.

The program or way side-by-side differences are shown can vary between packages/files too.

So I thought the entire purpose of a package management system is to create a consistent way of installing and uninstalling programs. Simple to manage for the user.

What went wrong here; why is this so horrible in its UI/UX? Is there anything a user can do to change/modify apt so at least

• 'do a 3-way merge' is available
• The new files are always renamed/placed in a consistent way?

Having to re-read and figure out exactly what to do with each and every config file manually because of 1,000 tiny variations of doing the same thing when it boils down to the same rote procedure is painful.

To illustrate what I mean, here's two from an upgrade;

Configuration file '/etc/ssh/ssh_config'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
 
 A new version (/tmp/tmp.3RoEfdEm3M) of configuration file /etc/ssh/sshd_config is available, but the version installed currently has been locally modified.  
                                                                                                                                                              
 What do you want to do about modified configuration file sshd_config?                                                                                        
                                                                                                                                                              
                                                     install the package maintainer's version                                                                 
                                                     keep the local version currently installed                                                               
                                                     show the differences between the versions                                                                
                                                     show a side-by-side difference between the versions                                                      
                                                     show a 3-way difference between available versions                                                       
                                                     do a 3-way merge between available versions                                                              
                                                     start a new shell to examine the situation 

Those options in the second variant? They're not always there in that order.

It feels almost like this is just made to trip you up.

Answer 1

I’ll address the “user-facing” concerns in your question first:

'do a 3-way merge' is availble

This can’t be controlled by users, it depends on how the package manages its configuration files (see below).

The new files are always renamed/placed in a consistent way?

No, but it shouldn’t matter in practice. In particular, the temporary files shown for example in your second example (the sshd_config one) really are temporary files: they are a variant of the configuration file made available so that the comparison tools etc. can find the data they need to compare. (The user experience could be improved here by not showing the temporary file name by default, and only displaying it if the user starts a shell “to examine the situation”.)

As for why this happens, in Debian specifically, it’s because configuration files can be handled in a variety of ways.

The package manager, dpkg, provides its own management of configuration files, as you expect. This is what produces the first variant in your example (the ssh_config upgrade). It only has two pieces of information to work with: the configuration file as it exists currently on disk, and the new configuration file shipped in the package it’s trying to install. As a result, it can’t offer three-way merges. For package maintainers it’s very simple to use (in fact in most cases it requires no work at all). dpkg’s configuration-file handling produces consistent file names after the fact: files ending in .dpkg-dist and .dpkg-old depending on whether the user kept the existing file (in which case, the new file is kept for later reference, with the .dpkg-dist extension) or installed the new one (in which case, the old file is kept with the .dpkg-old extension).

To improve this situation, the ucf tool was developed. It stores the original version of configuration files, which is why it can provide three-way comparisons and merges. However it requires some work from package maintainers, which is why many packages don’t use it. Further, support for three-way merges is optional. You can see openssh-server’s ucf integration here.

On top of all that, there’s debconf, the Debian configuration management system. This is what packages which require interaction during installation are supposed to use, but again, this requires work from package maintainers, so while it is now very common, there are still some exceptions (including very old packages which haven’t been updated to use debconf).

Of course, some maintainers don’t like any of the above, or developed their own solution before some of the above were available, so a few packages do their own thing.

Joey Hess wrote about his involvement in the VA Linux box set of Debian from 1999, which gives some context around the creation of debconf and the revamping of package configuration.

Answer 2

Before doing any update, first install distro-info, as otherwise looking up previous release names from the CLI is tricky without access to a web browser.

The commands:

diff3 -m config.file config.file.dpkg-old config.file.dpkg-new > config.file.merged
diff config.file config.file.merged

Then either editing, or, if the merge worked:

mv config.file.merged config.file

are a way to do the same thing debconf/ucf does manually.

This just leaves the packages that don't inform the users where they left their old/new configuration files. To download a missing configuration file from an older version of the package, while halfway through an upgrade process, you could install a whole separate system and install the package in that. Or, you could manually download it through a browser, ftp it to the linux machine. But, if it has to be done on the command line of the upgrading machine, things get a little complicated.

This is what I've got so far. There's still one unimplemented thing: GETPACKAGERELEASEVER --> what version of a package is part of 'buster'? How do I print this? Also, how to get whether it is 'main' or 'contrib'?

(Related question for this part; How do I check package version for a given distribution version from the cli? )

# Todo: Look this up in some way using apt. (main, non-free, or contrib).
pkgType=main
previousReleaseName = $(distro-info -o)
# If not possible to use distro-info: 
# This is rather hacky, and will fail when debian updates this webpage.
# release=$(cat /etc/issue | sed -E 's/[^0-9]*([0-9]+).*/\1/')
# previousRelease=$(($release-1))
# wget -O- https://www.debian.org/releases/  > releases.html
# previousReleaseName = $(cat releases.html | grep "Debian $previousRelease" | sed -E 's/.*Debian [0-9]+ \(<q>([a-z]+).*/\1/'))

firstLetter="$(echo $packagename | head -c 1)"
packageVersion=GETPACKAGERELEASEVER($packagename,$previousReleaseName)
mkdir /tmp/$packagename/
cd /tmp/$packagename
rm -r /tmp/$packagename/*
wget -O package.deb "https://deb.debian.org/pool/$pkgType/${firstLetter}/${packageName}/${packagename}_${packageVersion}_amd64.deb" 
ar x package.deb
tar -xvf data.tar.xz

diff3 -m /etc/$packagename/$configfile /tmp/$packagename/etc/$configfile /etc/$packagename/$newconfigfile > /etc/$packagename/$configfile.merged`