45

I can't find any good information on the rt and lowlatency Linux kernels.

I am wondering why anybody would not want to use a lowlatency kernel.

Also, if anyone can tell what the specific differences are, that would be great too.

  • 18
    As Stephen Kitt points points out in his answer. There's no free luch, lower latency means lower throughput. Most people would chose throughput unless they specifically need the low latency. Think internet connections; computer gamers and stock market traders want low latency so they can respond to events first; almost everyone else prefer faster download speeds. – Philip Couling Nov 25 '19 at 10:26
  • 3
    @Kusalananda - I don't agree - the low-latency kernel obviously has various engineering trade-offs vs the "standard" kernel and these are factual (though maybe both qualitative and quantitative). True, this question might attract opinions but the OP is clearly asking for facts. Which should be available. – davidbak Nov 25 '19 at 20:39
  • @davidbak The question was edited after my previous comment to remove the request for material. It still lacks specificity, but Stephen gave a good answer none the less. – Kusalananda Nov 25 '19 at 20:52
  • @Kusalananda - ok! (sorry, I know post edits make comments appear flawed ...) – davidbak Nov 25 '19 at 20:54
  • 1
    The reason I don't use them is because they tend to cause kernel panics for me. – Darius Jahandarie Nov 26 '19 at 9:28
60

The different configurations, “generic”, “lowlatency” (as configured in Ubuntu), and RT, are all about balancing throughput versus latency. Generic kernels favour throughput over latency, the others favour latency over throughput. Thus users who need throughput more than they need low latency wouldn’t choose a low latency kernel.

Compared to the generic configuration, the low-latency kernel changes the following settings:

  • IRQs are threaded by default, meaning that more IRQs (still not all IRQs) can be pre-empted, and they can also be prioritised and have their CPU affinity controlled;
  • pre-emption is enabled throughout the kernel (CONFIG_PREEMPT instead of CONFIG_PREEMPT_VOLUNTARY);
  • the latency debugging tools are enabled, so that the user can determine what kernel operations are blocking progress;
  • the timer frequency is set to 1000 Hz instead of 250 Hz.

RT kernels add a number of patches to the mainline kernel, and a few more configuration tweaks. The purpose of most of those patches is to allow more opportunities for pre-emption, by removing or splitting up locks, and to reduce the amount of time the kernel spends handling uninterruptible tasks (notably, by improving the logging mechanisms and using them less). The goal of all this is to allow the kernel to meet deadlines, i.e. ensure that, when it is required to handle something, it isn’t busy doing something else; this isn’t the same as high throughput or low latency, but fixing latency issues helps.

The generic kernels, as configured by default in most distributions, are designed to be a “sensible” compromise: they try to ensure that no single task can monopolise the system for too long, and that tasks can switch reasonably frequently, but without compromising throughput — because the more time the kernel spends considering whether to switch tasks (inside or outside the kernel), or handling interrupts, the less time the system as a whole can spend “working”. That compromise isn’t good enough for latency-sensitive workloads such as real-time audio or video processing: for those, low-latency kernels provide lower latencies at the expense of some throughput. And for real-time requirements, the real-time kernels remove as many low-latency-blockers as possible, at the expense of more throughput.

Main-stream distributions of Linux are mostly installed on servers, where traditionally latency hasn’t been considered all that important (although if you do percentile performance analysis, and care about top percentile performance, you might disagree), so the default kernels are quite conservative. Desktop users should probably use the low-latency kernels, as suggested by the kernel’s own documentation. In fact, the more low-latency kernels are used, the more feedback there will be on their relevance, which helps get generally-applicable improvements into the default kernel configurations; the same goes for the RT kernels (many of the RT patches are intended, at some point, for the mainstream kernel).

This presentation on the topic provides quite a lot of background.

|improve this answer|||||
  • 1
    RT means real time? – Minix Nov 26 '19 at 10:17
  • 1
    That’s right, RT means real-time (in this context). – Stephen Kitt Nov 26 '19 at 10:31
  • RE: timer frequency, didn't the kernel go tickless years ago? Or was that reverted at some point? – genpfault Nov 27 '19 at 17:36
  • 1
    @genpfault most kernels use idle dynticks, so the timer interrupt is enabled when the system is busy. – Stephen Kitt Nov 27 '19 at 17:42
22

Stephen Kitt explained the configurations and balancy and all nice in technical parameters. I would like to offer just a small intuitive distinction:

  • You are on safari, riding through the terrain in a jeep. Your prey is running. When the prey is in crosshairs, you pull the trigger and the rifle shoots - the computation is simple - prey in crosshairs=hit, prey not in crosshairs=miss - you desperately need low latency - you then recover, reload rifle, find another prey - no need for extra speed, no need for regularity. Latency is all.

  • You are converting a video from that safari. It is long, it takes hours. You do not care, when particular frame is processed and if some frames take more time to process than others. You need finish the process as fast as possible - better throughput means less hours, nothing else matters

  • You are getting telegraph - just shorts, longs and spaces - Morse is easy to decipher and you do not need exactly when each pulse started or ended, but you need a guarantee, that you do not miss any one of them - you need realtime (it may be slow, telegraph is not so fast, but it must be regular

In this three examples you clearly select just one of latency, throughput or regularity, sacrificing other two - for obvious reasons. And only in one you really want low latency, if you cannot have all three at the same time.

|improve this answer|||||
  • 3
    System runs many tasks "at the same time", but from closer look it is just skiping from one to another really fast. To skip (switch content) it have to save all registers, program counter, statistics etc, etc of one task and load those of the next. It take time. Also it must decide, what is the next (based on priorites, already consumed sources etc, etc) - another time spend. You need read keyboard, mouse, ethernet, wifi, receive data from discs, save cached data to disc (if possible), swap unused memory, maybe you have desctop, where time&date is shown as well as system stats - lots of tasks – gilhad Nov 26 '19 at 6:52
  • 5
    if you need throughput, you leave each task running as long as possible, to spend only fraction of time in management of the switching. If you need really low latency, you may (in extreme) spend more time skipping between tasks, then doing those tasks. Just now my system does nearly nothing and I have 149 tasks running "concurently" - each with its own set of rights, its mapped memory pages etc etc. Imagine you doing your work and waiting for paper email - if you check it every day (get up from work, walk to mail box, look, go back) = high latency. Low latency = you check it every 5 minutes – gilhad Nov 26 '19 at 7:02
  • 4
    The Golden rule of real-time: Performance doesn't mean anything, all that matters is keeping your deadlines – rasmus91 Nov 26 '19 at 7:33
  • 3
    @rasmus91 I'd reword that as "Performance doesn't mean anything if you miss the deadline". You still prefer getting the answer sooner, it just shouldn't cause your rocket to crash because another module didn't get to run soon enough :) – Luaan Nov 26 '19 at 11:04
  • 4
    @OlegV.Volkov if OP asks, what is the differention between them, and why would somebody even wanted something else, then latency, then I suppose, the words does not have for him as simple and intuitive meaning, as for us two. And if OP says, that it hepls clarify things for him, then I suppose, my explanation was much intuitive for him, than the simple words instead. – gilhad Nov 26 '19 at 11:10
3

Because there is a trade-off. Switching processes or entering/exiting interrupts takes time. For example running the scheduler at 1000Hz instead of 250Hz means you’ll have a timer interrupt and potentially switch processes four times as often. This can allow a process to react faster because it’s more regularly allowed to execute. However, as a human you’ll probably not notice any difference (250Hz means every 4ms which is already much faster than any human reaction time).

The total amount of processing power or I/O throughput is limited and calling the scheduler more often only means you’ll waste a part of it.

|improve this answer|||||
0

In layman's terms:

  • If you are not regularly giving your PC work to be done then the low latency kernel is for you.
  • If you are constantly giving your PC CPU intensive tasks to be done then the generic kernel is for you.
  • If you are just casual using your PC for browsing, playing videos, music...then low latency is for you.
  • If you are gaming the theoretical difference between low latency and generic would be low latency would win in all scenarios except one; level loading, but this is more dependent on whether you have an SSD or not, and the quality of the SSD, because your basic SSD performs at 1X speed, whereas your highest PCI-Express SSD is at 6X speed. A traditional hard drive is at 0.1X speed. So yes an SSD can be 60X faster than a traditional hard drive.

Basically, the generic kernel gets the most work done in the shortest time whereas the low latency kernel makes the PC the most responsive; assuming no bugs are encountered in the kernel or system.

For most people, the low latency kernel makes the most sense. But if you are always giving your PC a lot of work to be done then the generic kernel is for you.

Another way to put it if 90% of the time your CPU load across all cores is < 60% then the low latency kernel is for you. If on the other hand your cpu load across all cores is greater than 80%, 90% or more of the time, then the generic kernel is for you.

Quiz: Which kernel would score better in a benchmark? As long as latency is ignored as benchmark parameter, generic always wins the race, but low latency is like having an octopus by your side ready to delegate.

|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.