Why do Unix-like systems execute a new process when calling a new function?

Question

Why do Unix-like systems execute a new process when calling a function rather than a dynamic library? Creating a new process is costly in terms of performance when compared to calling a dynamic library.

Answer 1

Unix-like systems don't "call functions by executing new processes". They (now) have shared libraries like pretty much all other relatively modern operating systems.

Shells on the other hand do execute other processes to do certain tasks. But not all. They have build-in functions,implemented directly in the shell (or via shared libraries) for the most common and simple tasks (echo for instance is implemented as a built-in by a lot of shells).
(The windows cmd shell is no different from Unix shells in this respect, BTW.)

Creating a process in modern Unix-like systems is certainly more expensive than doing an in-process function call, but not by such a huge margin. Kernels are optimized for fast forking, using techniques like copy on write for address space management to speed up "cloning" of processes, and sharing the text (code) pages of dynamic libraries.

If every executable on your machine that could be called from a shell script was implemented as a shared library, either:

• starting your shell would take a lot of time (and memory) just to load all that stuff up front (even with caching, the dynamic linker has non-trivial work to do, and libraries have data sections, not only text sections - we're talking hundreds if not thousands of libraries here)
• you would have to load each necessary library on-demand – possibly a bit faster than starting a process, but the advantage here is really thin. And the data part of your shared libraries becomes really hard to manage (the global state of your shell now depends on the state of a lot of unrelated code and data loaded in its address space).

So you probably would not gain much for typical usage, and stability/complexity becomes more of an issue.

Another thing is that the separate process model isolates each task very effectively (assuming virtual memory management & protection). In the "everything is a library" model, a bug in any utility library could pollute (i.e. corrupt) the entire shell. A bug in some random utility could kill your shell process completely.
This is not the case for the multi-process model, the shell is shielded from that type of bug in the programs it runs.

Something else: lower coupling. When I look at what's in my /usr/bin directory right now, I have:

• ELF 64bit executables,
• ELF 32bit executables,
• Perl scripts,
• Shell scripts (some of those run Java programs),
• Ruby scripts and
• Python scripts

... and I probably don't have the most fancy system out there. You simply can't mix the first two types in the same process. Having an interpreter in-process for all the other ones simply isn't practical.
Even if you look only at your "native binary" file format, having the interface between the "utilities" being simple streams and exit codes makes things simpler.
The only requirements on the utilities is to implement the operating system's ABI and system calls. You get (nearly) no dependency between the different utilities. That's either extremely hard, or plain impossible, for an in-process interface, unless you impose things like "everything must be compiled with version X of compiler Y, with such and such flags/settings.

There are things for which in-process calls do make a lot of sense performance wise, and those are already, very often, done as built-ins by the shells. For the rest, the separate processes model works very effectively, and its flexibility is a great advantage.