What introduce package/binary incompatibility?

Question

Following up on Is Ubuntu LTS binary compatible with Debian?

I know Ubuntu and Debian binary packages are incompatible more often than not. I know mixing packages from different sources is generally a bad idea, and people get warned against doing that all the time. So let's keep the discussion pure technical --

What exactly will introduce incompatibility for packages from different sources, when of course the dependencies are not the problem?

------ Separation, more details below ------

Like the saying:

about binary compatibility (https://wiki.ubuntu.com/MarkShuttleworth#What_about_binary_compatibility_between_distributions.3F): Debian packages are likely built with different toolchain versions, so you may incur in troubles

Why different toolchain versions will give problem? Like

• I know how to pull the minimum set of packages from Debian sid into my Debian stable, and had been doing that all the time,
• I used to carry packages from older version of Ubuntu / Debian to their newer versions, or even
• copying a single executable from my Ubuntu / Debian to another distro, be it RedHat or even FreeBSD,

and never had problem before. So what exactly is causing the problem that people are saying?

Is it gcc or kernel version? Unlikely to me, as they get upgraded all the time throughout the lifespan of me using that release.

So it is the version of glibc? But it'll be backward compatible normally and most probably, right?

Quoting from the answer from my first link:

There really is no guarantee or even implication of cross-compatibility. Don't expect either the Debian or Ubuntu communities to give you much sympathy if things go wrong. In that event you're mostly on your own. As long as you're okay with that then feel free to give it a try.

So basically I see warning against the practice everywhere, but nobody give further technical explanation. Can anyone list the risks of doing so, those potential technical problems please?

That answer will help me, if I want/need to mixing packages from different sources, say Debian or Ubuntu, or within the same distro but different releases, (if the dependencies are not the problem), to choose the safest approach, to pull a PPA that I know for sure will never end up in Debian, into the Debian Bullseye that I'm currently using.

Answer 1

Binary and package incompatibilities are different and are worth explaining separately.

Binary incompatibility

This is usually what is referred to when people talk about toolchain differences etc. Toolchain incompatibilities themselves are unusual, because the toolchain is, along with the kernel, one of the areas were developers are most careful about preserving backwards compatibility. As a result, a binary built in the past should continue running, as long as its binary dependencies continue to be available; that boils down to keeping the libraries it needs.

Where things break is with forwards compatibility: a binary built “in the future” can’t be guaranteed to run. This commonly shows up as missing symbols in the C library (which are detected precisely because the C library developers take great care to maintain compatibility). One might think that the C library doesn’t change much, so building a program with different C libraries shouldn’t change the symbols it needs, and it should remain compatible. That isn’t the case, and functions do change regularly in backwards-incompatible ways; the C library preserves backwards compatibility by continuing to provide versions of functions compatible with previous interfaces, with an appropriate version symbol. For example, version 2.33 of the GNU C library has incompatible changes to such common functions as the stat family (fstat/lstat/stat etc.); a program built with a default setup of 2.33, using those functions, will require version 2.33 of the C library to run.

Toolchain-related libraries, and the C library, are maintained in such a way that such incompatibilities show up as library symbol changes, or soname changes, and thus end up encoded in package dependencies (for packaged sofware) or are caught by the dynamic linker (for separate binaries).

In libraries which aren’t as carefully maintained as the C library, such incompatibilities don’t show up as immediately, they only show up if the faulty combination is tested (and even then, perhaps only in certain circumstances). Distribution developers usually only test packages in the context of the release being developed, so they won’t know that the package they built for Ubuntu 20.04 installs correctly on Debian 10 but kills your pet squirrel if you use it between 1am and 2am CEST on June 21.

This leads nicely to ...

Package incompatibility

Whether this is done consciously or not, packages are rarely built in a vacuum, they are part of a distribution release. This starts with the package source (and more generally, project source) itself: projects and packages are built on their developers’ systems, and unless great effort is put into it, might not accurately encode their dependencies.

This trickles down to binary package dependencies, and dependencies described in documentation. A project maintainer might not realise that their project in its current configuration only works because, say, systemd version 239 started setting the system up in a certain way. Package maintainers might not realise that either, if the release of the distribution they’re working on happens to already have version 239 of systemd (bear in mind that distribution maintainers usually live in the future, i.e. they develop in what will be the next release).

All this could be caught by testing, but once you start mixing and matching binaries from different distributions and releases, you’re likely to be the first person ever to test your exact combination of package and binary versions. And that is why this isn’t recommended: most users want to use their systems, not test them.

Of course, and this fits in with your experience, in many cases everything just works. This is also contributes to the bias you perceive: people tend not to write posts (or questions, in a Stack Exchange context) explaining how they installed package X from distro Y on their Z system, and it just worked. So most of the content you see in this domain is scenarios where things didn’t work, or ended up being complicated to set up, or broke something else; and people are understandably reluctant to spend time helping someone fix a probably one-off problem that they brought on themselves by doing something which was explicitly recommended against.