3

Over the past few years, various live kernel patching techniques have become popular among sysadmins who strive to ensure the highest possible uptimes for their systems.

For that process to be possible, a human being prepares custom patches, which are then typically distributed to paying customers, and - sometimes - free of charge to home users.

Why is it not possible to automatically create these patches using the difference between the source codes of the running kernel version and the latest available? As I understand it, server kernels, which could profit the most from this, typically only undergo major changes once every a couple of years, and otherwise only receive major bugfix and security updates, seemingly making this even easier. Likewise, if stability was the concern, it would seem quite simple to set up a system where volunteers running machines of relatively low importance would build their patches first, and automatically report back on how well they work.

Yet, none of this happens. What am I missing there that makes this the case?

2
  • 1
    Possibly lack of practical interest? Mission critical systems are always redundant, so you can upgrade & reboot one of the systems while the others are kept running. Any serious IT organization has test machines on which it can test such patches before going to production, so its not a matter of (not) trusting such patches. – xenoid Dec 13 '19 at 21:42
  • Well, as much as the supply is able to suggest, there exists practical interest for the patches at a charge, and if it's not the most mission-critical systems exhibiting it, I'd argue that only the larger interest for ones virtually free of charge should be hiding somewhere in there... :) – S. Exchange Considered Harmful Dec 13 '19 at 21:54
7

We like to think of running programs like the static source code that creates them. But they are really continually changing. Likewise the kernel in memory is NOT the same as the kernel on disk.

To quote Dijkstra in his letter "goto considered harmful"...

My first remark is that, although the programmer's activity ends when he has constructed a correct program, the process taking place under control of his program is the true subject matter of his activity, for it is this process that has to accomplish the desired effect; it is this process that in its dynamic behavior has to satisfy the desired specifications. Yet, once the program has been made, the "making' of the corresponding process is delegated to the machine.

My second remark is that our intellectual powers are rather geared to master static relations and that our powers to visualize processes evolving in time are relatively poorly developed. For that reason we should do (as wise programmers aware of our limitations) our utmost to shorten the conceptual gap between the static program and the dynamic process, to make the correspondence between the program (spread out in text space) and the process (spread out in time) as trivial as possible

From this I would infer that it is a bad idea to have a program or kernel in memory that isn't the result of loading the kernel from disk. If nothing else you want to know that you can reboot and end up with the same kernel as you are running now.

As a sys-admin you want to know you got a bonafide regular kernel at the end not some Frankenstein's monster because your live kernel had subtle differences to the one they patched.


And live patching is very hard indeed. It is technically impossible to automatically generate live patches

It's important to understand that program code effectively rewrites itself.

int X = 10;
void run(){
    X=5;
}

In this code example X=10 is never executed as code. The number 10 is placed in the location "X" by the compiler. When the 3rd line is executed at run time it replaces the value at location "X". It literally overwrites the value, meaning the number 10 disappears from the running program code entirely.

Now we try to live patch this with:

int X = 20;
void run(){
    X=15;
}

What should X be patched to 20 or 15? Should we patch it at all or just leave it? We are not just changing code here we are changing dynamically generated values. You might think that because they are dynamically generated you might not need to change them, but if we don't change them do we know 5 or 10 is still a valid value in the new code? This cannot be done automatically!

In short there are techniques with associated tools that can create live patches, but using them and testing the result requires experts. Releasing these tools and expecting home users to understand how to use them is a good way for a lot of home users to screw up their system.

2

Any mechanism that can be used for good to introduce patches to the running kernel could also be used for evil, to introduce a rootkit to the running kernel.

If the evil code is introduced through a suitable other network-level vulnerability, there might be no local modified files at all to act as evidence for the intrusion.

Also, things like kernel address-space layout randomization (KASLR), intended to make attacks against the kernel harder, will also complicate live patching, as the information it tries to protect (the addresses of various kernel routines in memory) is also the information needed for successful live patching. It might be possible to cleverly manage the information so that it will only be available for legitimate purposes, but I'd expect that to be a tricky problem.

The modern "cloud" application design principles emphasise virtualization, parallelism and multiple smaller servers instead of a single large one, so perhaps the uptime of individual servers is becoming less important than before? If the application architecture includes a load-balanced set of multiple servers and the size of the set is variable, you could just fire up a new already-patched virtual server and tear down an old unpatched one, then repeat that until all the servers have been replaced with updated ones. If you can do this at all levels of your application stack, you won't need live patching at all, as the problem is solved in another way.

7
  • +1 . though on the point of your last paragraph. This is not ubiquitous. There are many that say you should always run some services on bare metal. DBMS is an example. Often a single DBMS on a single server is at the heart of many application suits meaning downtime is more necessary and less easy to achieve. – Philip Couling Dec 15 '19 at 17:56
  • @PhilipCouling True... "legacy" systems deployed just before the cloud model was really established will be with use for a quite a while yet. And if an application suite is relying on a single, non-clustered, bare-metal DBMS instance, then that is a clear design-level single-point-of-failure. If the application has no hard uptime needs, that might even be an acceptable design tradeoff for simplicity... but planning for updates is something that must be done at the application design level, one way or other. There may still be a niche for live patching, although it seems to be getting smaller. – telcoM Dec 15 '19 at 20:24
  • That's not entirely true. Single central server does not mean there is no replication or failover. But it does mean there is only one "master of the truth". This is completely normal for applications and by no means legacy. Not yet. – Philip Couling Dec 15 '19 at 20:49
  • But if the "source of truth" can failover, then you can patch its inactive host platform, make a failover, and then patch the other one - and in my opinion, you probably should do it that way, to ensure that you can trust your failover mechanism to save you when something actually fails. A failover system that does not get periodically exercised tends to suffer from neglect-caused bit-rot. You need live-patching only when you can't have planned downtime and can't failover - and that combination of restrictions is a bad place to be in in other ways also. – telcoM Dec 16 '19 at 8:09
  • That's a lovely ideal that I wouldn't argue with. Good luck getting a sysadmin to agree to it. – Philip Couling Dec 16 '19 at 9:48
-1

Kernel is the heart of the *nix based operating system. It controls all the behaviour of the operating system. Therefore, patching it live would impede the current ongoing operations. This is the reason it cannot be patched live.

1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.