Updating the Linux kernel, while leaving rest of system as is

Question

I'm an OpenBSD user. In the OpenBSD FAQ it says:

OpenBSD is a complete system, intended to be kept in sync. It is not a kernel plus utilities that can be upgraded separately from each other.

When you upgrade a system, you do so in one go; the kernel and the base system is replaced. Then you go and update your 3rd party packages. If compiling from source, you recompile the kernel and boot it. Then you rebuild the base system, and then the packages that you've got installed. If more than a couple of weeks/months have past since you last rebuilt everything, you first install a snapshot and rebuild from there (if you're following the most current CVS branch).

Having an out of sync kernel, base system and/or 3rd party packages is a potential source of issues and more or less disqualifies you from getting any serious help from the official mailing lists.

I'm quite okay with this. In fact, this is one of the reasons I use OpenBSD. It makes the system a consistent unit and it makes it easy for me to form a mental overview of it.

What's it like on Linux? Most Linuxes that I'm aware of don't have a "base system" in the same sense as the BSDs, but rather a collection of packages assembled by the distribution provider. Further software is then added to this by a local administrator in such a way that the boundary between what was there from the start and what was added later is, at best, blurry.

Does Linux (in general) not have a strong kernel to userspace coupling? The kernel is updated, as far as I know, like any other software package, and it confuses me slightly that this is at all possible. Add to this the fact that some even compile custom kernels (which is discouraged on OpenBSD), and have a multitude of various kernel versions listed in their boot menus.

Who or what guarantees that the various subsystems of a Linux system are able to cooperate with each other even though they are updated independently from each other?

The reason I'm asking is because another user on this site asked me whether replacing the kernel in his Linux system with a newer version "would be doable". Coming from the OpenBSD side of things, I couldn't say that yes, this would be guaranteed to not break the system.

---

I use "Linux" above as a shorthand for "Linux distribution", kernel + utilities.

Answer 1

Linus Torvalds has a very strong opinion against kernel changes resulting in userspace regressions (see the question "The Linux kernel: breaking user space" for details).

Interface between userspace and kernel is provided by system calls. Newer kernels can have more system calls, and changes in exiting ones when those changes do not break existing applications. When a system call interface has a flag parameter, new kernels often expose the new functionality with a new bit flag. This way kernel maintains backwards compatibility to old applications.

When it has not been possible to alter existing interface without breaking userspace, additional system calls have been added that provide the extended functionality. This is why there are three versions of dup and two versions of umount system call.

The policy of having a stable userspace is the reason why kernel updates rarely cause issues in userspace applications and you do not generally expect issues after upgrading the kernel.

However same API stability is not guaranteed for kernel interfaces and other implementation details. Sysfs (on /sys) and procsfs (on /proc/) expose kernel implementation details on runtime configuration, hardware, network, processes etc. which are used by low-level applications. It is possible for those interfaces to change in an incompatible way between kernel versions if there is a good reason to. Changes still try to minimize incompatibilities if possible and there are rules for how applications can use the interfaces in a way least likely to cause issues. The impact is also limited because non low-level applications shouldn't be using these interfaces.

@PeterCordes pointed out if a change in procfs or sysfs breaks an application used by your distributions init scripts, you could have a a problem.

This depends somewhat on how your distribution updates kernel (long term support or mainline) and even then the issues are relatively rare as distributions usually ship the updated tools at the same time.

@StephenKitt added that upgraded userspace might require a newer version of the kernel, in which case the system might not be able to boot with the old kernel and that distribution release notes mention this when appropriate.

Answer 2

The Linux kernel and the user space of a Linux distribution, which historically was dominated by user tools developed by the GNU project, are loosely coupled. In part this is by design, and in part it is by necessity.

Unlike the BSDs, where the kernel as well as the base user space are designed and maintained together, the Linux kernel and its user space were developed and are maintained by different people. So keeping them tightly coupled together would be difficult, even if the community desired it, which I don't think it does.

And @sebasth also makes the excellent point that a major policy of the Linux kernel project is that it must not break user space. Which of course is another factor enforcing loose coupling.

However, there is still some degree of coupling. A sufficiently old Linux kernel will not work with modern distributions.

Answer 3

My understanding is that Linux is the kernel, and everything else is ancillary. You definitely can upgrade the kernel independently of (many) installed packages, as they are generally tied to libraries rather than the kernel itself. You will generally only see such coupling between kernel versions and hardware drivers (e. g. GPU drivers). Some software is only compatible with certain versions of the kernel, but that should be specified in those programs' individual documentation.

It's rather common to have two kernel package suites installed on a system - the currently used package, and the previously installed package. When upgrading, after ensuring the new version works properly, the oldest can be removed and you still have a known-safe fallback. Red Hat (and its cousins) for example, has package-cleanup --oldkernels --count 2 to do this automagically.

Answer 4

Besides all the good arguments here, I may add a few points that will help.

We already know kernel team policy, and as Linux distributions, we try to keep the kernel source code as pure as possible. This means, whenever a vulnerability arises, or something that needs a patch, we inform the kernel team and try to help with patches, but at the end, the final decision is from kernel team.

Some distributions add extra patches more than others, but the idea is to keep it as it comes from upstream developers. Obviously, there are some programs that need special kernel configuration, especially virtualization and third-party drivers and usually, both of them are used to work with a specific kernel version or at least a not too old version... the reason is they try to work with the newest kernel features and because of that, if you try to run them with an old kernel, they may not run properly .

One extra element to keep in mind is that all the Linux distributions have a maintainers team, people that ensure that all the software is compatible with the system. That's why almost every distribution has a stable and an unstable version. Developers work with "unstable" software and when all is tested they mark it as stable, only after that a normal user can upgrade the package, so if the person that asked you is a normal user is 90% secure that the software is well tested.

So the who is, the community, and then what should be the common approach, we need software working together, and not break the system, that's why we try to follow some standards like Filesystem Hierarchy Standard.