Why can kernel features that are reimplemented in user-space be faster than in kernel-space?

Question

I was reading a blog post and noticed the following sentence:

Then he said something really surprising: that in the Seastar HTTP framework, they wrote their own TCP stack, and it made everything several times times faster. What?!

I'm trying to understand why kernel functionalities would be re-implemented in user-space for performance reasons. I would assume that features present in a kernel are in the kernel exactly because they execute (many) privileged instructions, because otherwise the feature could simply be implemented as a user-space program. So, if one were to re-implement kernel features or functionality in user-space, such as a network stack (this is what gVisor does with its netstack for example), would you not end up having to execute many system calls back into the kernel anyway, causing a lot of overhead?

Are such user-space re-implementations of features that are traditionally part of the kernel somehow able to avoid making many system calls? If so, how does that work for e.g. a network stack, since you would probably have to e.g. send() or recv() often, I can imagine.

I do understand that two potential advantages of re-implementing features in user-space are that:

• you are not dependent on what is added to the kernel (which seems to be an arduous process)
• if an exploit is found within a traditionally kernel feature re-implemented in user-space, it is 'only' an unprivileged user-space process anyway

But I am more interested in the performance aspect in this question.

Answer 1

Some of it is avoiding some of the trips across the system call boundary.

That's true, but another aspect is that the Linux system call interface is simultaneously very general (i.e. has to deal with many different kinds of applications and systems) and very narrow (the system call parameters deal very specifically with the current request only). The kernel typically has next to no idea what your code is going to do next.

Let's take find as an example. It spends a lot of its time in system calls like getdents and opendir. You can do a lot of things with find but here's a typical command line:

find . -name 'report_201[89].txt' -print -quit

The find program is going to open a lot of directories and read a lot of filenames. It's going to feed those file names to the user-space function fnmatch to find out if they are report_2018.txt or report_2019.txt.

But, let's suppose that . is in some modern filesystem. The directories are really B-trees or hash tables. If only the kernel knew which filename we were looking for, we could save a lot of processing.

Suppose instead we look at git status. If you trace its system calls, it issues a ton of lstat calls. But what it's really trying to figure out is, did the user change the file system? The kernel basically knows the answer, but there is no way for git to tell the kernel that's what it wants to know. So it has to examine everything itself (though it does so in quite a smart way).

The general theme here is that things could be much more efficient if the kernel API were application-specific. But design-wise that's crazy, because there are so many different applications. Maintaining a much wider kernel interface probably has super-linear complexity. But this is why there are efficiencies to be gained by solving more of the problem (for some problems) in user-space.

Answer 2

Short answer, the kernel has to deal with many different situations/application/hardware. Reimplementation of the stack is done when you know your hardware and/or your application communication needs. You can then code just for that universe.

Say you have a small sensing device sending data periodically over UDP. You could create UDP/IP packets with most of the values fixed so that it reaches the server (you know your IP, your port, the destination port and address, the length of the message, the flags ... you just need to change the reading).

Running a full IP kernel stack just for that would be overkill, slower and maybe not even feasable (running inside a bare bones Arduino for example).

But the broader answer is more complicated, so I suggest the article: Why we use the Linux kernel's TCP stack linked at the bottom of the article you cited.