In any inter-dependent systems there are basically two choices. Abstraction and integration. (I am purposely not using technical terms). With Abstraction, you're saying that when you make a call to an API that, while the code behind the API may change, the result will always be the same. For example when we call fs.open() we don't care whether it's a network drive, a SSD or a hard drive, we will always get an open file descriptor that we can do stuff with. With "integration" the goal is to provide the "best" way to do a thing, even if the way changes. For example, opening a file may be different for a network share than for a file on disk. Both ways are used pretty extensively in the modern Linux desktop.
From a developers point of view it's a question of "works with any version" or "works with a specific version". A great example of this is OpenGL. Most games are set to work with a specific version of OpenGL. It doesn't matter if you're compiling from source. If the game was written to use OpenGL 1.1 and you're trying to get it to run on 3.x, you're not going to have a good time. On the other end of the spectrum, some calls, are expected to work no matter what. For example, I want to call fs.open() I don't want to care what kernel version I am on. I just want a file descriptor.
There are benefits to each way. Integration provides "newer" features at the cost of backwards compatibility. While abstraction provides stability over "newer" calls. Though it's important to note it's a matter of priority, not possibility.
From a communal stand point, without a really really good reason, abstraction is always better in a complex system. For example, imagine if fs.open() worked differently depending on kernel version. Then a simple file system interaction library would need maintain several hundred different "open file" methods (or blocks probably). When a new kernel version came out, you wouldn't be able to "upgrade", you would have to test every single piece of software you used. Kernel 6.2.2 (fake) may just break your text editor.
For some real world examples OSX tends to not care about breaking User Space. They aim for "integration" over "abstraction" more frequently. And at every major OS update, things break. That's not to say one way is better then the other. It's a choice and design decision.
Most importantly, the Linux eco-system is filled with awesome opensource projects, where people or groups work on the project in their free time, or because the tool is useful. With that in mind, the second it stops being fun and starts being a PIA, those developers will go somewhere else.
For example, I submitted a patch to BuildNotify.py. Not because I am altruistic, but because I use the tool, and I wanted a feature. It was easy, so here, have a patch. If it were complicated, or cumbersome, I would not use BuildNotify.py and I would find something else. If every time a kernel update came out my text editor broke, I would just use a different OS. My contributions to the community (however small) would not continue or exist, and so on.
So, the design decision was made to abstract system calls, so that when I do fs.open() it just works. That means maintaining fs.open long after fs.open2() gained popularity.
Historically, this is the goal of POSIX systems in general. "Here are a set of calls and expected return values, you figure out the middle." Again for portability reasons. Why Linus chooses to use that methodology is internal to his brain, and you would have to ask him to know exactly why. If it were me however, I would choose abstraction over integration on a complex system.
