I'm an OpenBSD user. In the OpenBSD FAQ it says:

OpenBSD is a complete system, intended to be kept in sync. It is not a kernel plus utilities that can be upgraded separately from each other.

When you upgrade a system, you do so in one go; the kernel and the base system is replaced. Then you go and update your 3rd party packages. If compiling from source, you recompile the kernel and boot it. Then you rebuild the base system, and then the packages that you've got installed. If more than a couple of weeks/months have past since you last rebuilt everything, you first install a snapshot and rebuild from there (if you're following the most current CVS branch).

Having an out of sync kernel, base system and/or 3rd party packages is a potential source of issues and more or less disqualifies you from getting any serious help from the official mailing lists.

I'm quite okay with this. In fact, this is one of the reasons I use OpenBSD. It makes the system a consistent unit and it makes it easy for me to form a mental overview of it.

What's it like on Linux? Most Linuxes that I'm aware of don't have a "base system" in the same sense as the BSDs, but rather a collection of packages assembled by the distribution provider. Further software is then added to this by a local administrator in such a way that the boundary between what was there from the start and what was added later is, at best, blurry.

Does Linux (in general) not have a strong kernel to userspace coupling? The kernel is updated, as far as I know, like any other software package, and it confuses me slightly that this is at all possible. Add to this the fact that some even compile custom kernels (which is discouraged on OpenBSD), and have a multitude of various kernel versions listed in their boot menus.

Who or what guarantees that the various subsystems of a Linux system are able to cooperate with each other even though they are updated independently from each other?

The reason I'm asking is because another user on this site asked me whether replacing the kernel in his Linux system with a newer version "would be doable". Coming from the OpenBSD side of things, I couldn't say that yes, this would be guaranteed to not break the system.

I use "Linux" above as a shorthand for "Linux distribution", kernel + utilities.

  • It is another way of saying that OpenBSD only has a single distribution. The multiple grub menu entries on a regular Linux system are for falling back to an earlier kernel in case that the (usually) newest cannot boot for some reason. Sep 16, 2017 at 21:51

4 Answers 4


Linus Torvalds has a very strong opinion against kernel changes resulting in userspace regressions (see the question "The Linux kernel: breaking user space" for details).

Interface between userspace and kernel is provided by system calls. Newer kernels can have more system calls, and changes in exiting ones when those changes do not break existing applications. When a system call interface has a flag parameter, new kernels often expose the new functionality with a new bit flag. This way kernel maintains backwards compatibility to old applications.

When it has not been possible to alter existing interface without breaking userspace, additional system calls have been added that provide the extended functionality. This is why there are three versions of dup and two versions of umount system call.

The policy of having a stable userspace is the reason why kernel updates rarely cause issues in userspace applications and you do not generally expect issues after upgrading the kernel.

However same API stability is not guaranteed for kernel interfaces and other implementation details. Sysfs (on /sys) and procsfs (on /proc/) expose kernel implementation details on runtime configuration, hardware, network, processes etc. which are used by low-level applications. It is possible for those interfaces to change in an incompatible way between kernel versions if there is a good reason to. Changes still try to minimize incompatibilities if possible and there are rules for how applications can use the interfaces in a way least likely to cause issues. The impact is also limited because non low-level applications shouldn't be using these interfaces.

@PeterCordes pointed out if a change in procfs or sysfs breaks an application used by your distributions init scripts, you could have a a problem.

This depends somewhat on how your distribution updates kernel (long term support or mainline) and even then the issues are relatively rare as distributions usually ship the updated tools at the same time.

@StephenKitt added that upgraded userspace might require a newer version of the kernel, in which case the system might not be able to boot with the old kernel and that distribution release notes mention this when appropriate.

  • 1
    It would be useful in the long term to expand this explanation with a summary of the kernel-user interface (aka system calls) since it explains the mechanism by which differently configured kernels don't break applications.
    – wallyk
    Sep 16, 2017 at 1:06
  • 3
    Even procfs (/proc) and sysfs (/sys) are kept as stable as possible, following the same "don't break user-space" policy/philosophy. Also, ioctl() codes on device files en.wikipedia.org/wiki/Ioctl. It goes far beyond simple system-call ABI compatibility, but as you say when there's a good reason, things do change in /proc and /sys. It won't break most programs directly, but if it does break a low-level user-space program used by your distro's init scripts, you could have a problem. Fortunately this is rare. Sep 16, 2017 at 10:32
  • Actually some files like IIRC rfkill switch have changed locations across some kernel upgrades. So /proc and /sys are much less stable than syscall interface. Fortunately, distributions usually don't include such kernel upgrades unless it's a major distribution version upgrade.
    – Ruslan
    Sep 16, 2017 at 11:46
  • 3
    There are two aspects to consider here: upgrading the kernel, and upgrading userspace. Thanks to the kernel’s ABI stability, upgrading the kernel is quite safe (even with /proc and /sys changes usually — removals take years). However, upgrading userspace can require a new kernel, and you can end up with an unbootable system if you don’t have a new enough kernel. Distro release notes mention this when appropriate (so don’t upgrade distros blindly, read the release notes). Sep 16, 2017 at 12:18
  • 1
    There are guidelines for /proc and /sys files and how userspace should read them in order to allow for adding more fields in newer kernels. /proc/stat for example, or /proc/meminfo. User space is expected to ignore added fields.
    – Zan Lynx
    Sep 16, 2017 at 21:11

The Linux kernel and the user space of a Linux distribution, which historically was dominated by user tools developed by the GNU project, are loosely coupled. In part this is by design, and in part it is by necessity.

Unlike the BSDs, where the kernel as well as the base user space are designed and maintained together, the Linux kernel and its user space were developed and are maintained by different people. So keeping them tightly coupled together would be difficult, even if the community desired it, which I don't think it does.

And @sebasth also makes the excellent point that a major policy of the Linux kernel project is that it must not break user space. Which of course is another factor enforcing loose coupling.

However, there is still some degree of coupling. A sufficiently old Linux kernel will not work with modern distributions.

  • 2
    @Abigail this is nit-picking, but forwards compatibility can be provided, if userspace implements appropriate fallbacks (even degraded fallbacks) for missing kernel features. It’s often not desirable or even worth it, admittedly, but some software does this (glibc for example). (But yes, this isn’t a kernel promise, it’s a userspace promise.) Sep 16, 2017 at 12:23

My understanding is that Linux is the kernel, and everything else is ancillary. You definitely can upgrade the kernel independently of (many) installed packages, as they are generally tied to libraries rather than the kernel itself. You will generally only see such coupling between kernel versions and hardware drivers (e. g. GPU drivers). Some software is only compatible with certain versions of the kernel, but that should be specified in those programs' individual documentation.

It's rather common to have two kernel package suites installed on a system - the currently used package, and the previously installed package. When upgrading, after ensuring the new version works properly, the oldest can be removed and you still have a known-safe fallback. Red Hat (and its cousins) for example, has package-cleanup --oldkernels --count 2 to do this automagically.

  • Even something such as kmod, which you would think would need to be tied to the kernel version, has a bit of leeway in which versions it will work with. Sep 16, 2017 at 1:42

Besides all the good arguments here, I may add a few points that will help.

We already know kernel team policy, and as Linux distributions, we try to keep the kernel source code as pure as possible. This means, whenever a vulnerability arises, or something that needs a patch, we inform the kernel team and try to help with patches, but at the end, the final decision is from kernel team.

Some distributions add extra patches more than others, but the idea is to keep it as it comes from upstream developers. Obviously, there are some programs that need special kernel configuration, especially virtualization and third-party drivers and usually, both of them are used to work with a specific kernel version or at least a not too old version... the reason is they try to work with the newest kernel features and because of that, if you try to run them with an old kernel, they may not run properly .

One extra element to keep in mind is that all the Linux distributions have a maintainers team, people that ensure that all the software is compatible with the system. That's why almost every distribution has a stable and an unstable version. Developers work with "unstable" software and when all is tested they mark it as stable, only after that a normal user can upgrade the package, so if the person that asked you is a normal user is 90% secure that the software is well tested.

So the who is, the community, and then what should be the common approach, we need software working together, and not break the system, that's why we try to follow some standards like Filesystem Hierarchy Standard.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .