Why are POSIX mandatory utilities not built into the shell?

Question

The purpose of this question is to answer a curiosity, not to solve a particular computing problem. The question is: Why are POSIX mandatory utilities not commonly built into shell implementations?

For example, I have a script that basically reads a few small text files and checks that they are properly formatted, but it takes 27 seconds to run, on my machine, due to a significant amount of string manipulation. This string manipulation makes thousands of new processes by calling various utilities, hence the slowness. I am pretty confident that if some of the utilities were built in, namely grep, sed, cut, tr, and expr, then the script would run in a second or less (based on my experience in C).

It seems there would be a lot of situations where building these utilities in would make the difference between whether or not a solution in shell script has acceptable performance.

Obviously, there is a reason it was chosen not to make these utilities built in. Maybe having one version of a utility at a system level avoids having multiple unequal versions of that utility being used by various shells. I really can't think of many other reasons to keep the overhead of creating so many new processes, and POSIX defines enough about the utilities that it does not seem like much of a problem to have different implementations, so long as they are each POSIX compliant. At least not as big a problem as the inefficiency of having so many processes.

Answer 1

Why are POSIX mandatory utilities not built into shell?

Because to be POSIX compliant, a system is required¹ to provide most utilities as standalone commands.

Having them builtin would imply they have to exist in two different locations, inside the shell and outside it. Of course, it would be possible to implement the external version by using a shell script wrapper to the builtin, but that would disadvantage non shell applications calling the utilities.

Note that BusyBox took the path you suggested by implementing many commands internally, and providing the standalone variant using links to itself. One issue is while the command set can be quite large, the implementations are often a subset of the standard so aren't compliant.

Note also that at least ksh93, bash and zsh go further by providing custom methods for the running shell to dynamically load builtins from shared libraries. Technically, nothing then prevents all POSIX utilities to be implemented and made available as builtins.

Finally, spawning new processes has become quite a fast operation with modern OSes. If you are really hit by a performance issue, there might be some improvements to make your scripts run faster.

¹ POSIX.1-2008

_{However, all of the standard utilities, including the regular built-ins in the table, but not the special built-ins described in Special Built-In Utilities, shall be implemented in a manner so that they can be accessed via the exec family of functions as defined in the System Interfaces volume of POSIX.1-2008 and can be invoked directly by those standard utilities that require it (env, find, nice, nohup, time, xargs).}

Answer 2

Shell scripts are not expected to run with that type of speed. If you want to improve the speed of your script, try it in perl. If that is still too slow, then you'll have to move to a statically typed language such as java or c, or write a C module for perl that runs the parts which are too slow.

Shell is the first level of prototyping, if you can prove the concept with shell, then move to a better scripting language which can do more bounds checking which would take acres of shell.

A Unix OS is expected to include many small programs which do well defined tasks which make up a larger picture. This is a good thing as it compartmentalises bigger programs. Take a look at qmail, for example and compare that with sendmail. qmail is made of many programs:

http://www.nrg4u.com/qmail/the-big-qmail-picture-103-p1.gif

Exploiting the network daemon would not help you exploit the queue manager.

Answer 3

From the BASH reference manual,

Builtin commands are necessary to implement functionality impossible or inconvenient to obtain with separate utilities.

As I'm sure you've heard, the UNIX philosophy relies heavily on multiple applications that all have limited functionality. Each built-in has a very good reason why it is built in. Everything else is not. I think a more interesting class of questions is along the lines of, "why exactly is pwd built-in?"

Answer 4

The guys at AT&T asked themselves the same thing

If you look at the history of the AT&T Software Toolkit (currently lying dormant on github since the core team left), this is exactly what they did with the AT&T Korn shell, a.k.a. ksh93.

Performance was always part of the motivation for the ksh93 maintainers, and when building ksh you can choose to build many common POSIX utilities as dynamically loaded libraries. By binding these commands to a directory name like /opt/ast/bin, you could control which version of the command would be used, based on the position of that directory name in $PATH.

Examples:

cat chmod chown cksum cmp cp cut date expr fmt head join ln
mkdir mkfifo mktemp mv nl od paste rm tail tr uniq uuencode wc

The full list can be found in the github ast repository.

Note that most of the ast tools have their own provenance and would differ strongly from the more common gnu implementations. The AT&T Research team abided by official standards, which was the way to achieve interoperability when you could not share code.

Answer 5

So we didn't marshal resources into optimizing the original tool, to meet every specific desire. I guess what we need to explain is how much this specific desire would have cost to implement.

POSIX defines enough about the utilities that it does not seem like much of a problem to have different implementations.

this is a bad assumption :-P.

Post-POSIX systems continue to become more powerful and convenient for good reasons; as an after-the-fact standard it never actually catches up.

Ubuntu started an effort to switch to a stripped-down POSIX shell for scripts, to optimize the old System V init boot process. I'm not saying it failed, but it did trigger many bugs that had to be cleaned up: "bashisms", scripts which ran under /bin/sh while assuming that bash features were available.

POSIX sh is not a good general-purpose programming language. Its primary purpose is to work well as an interactive shell. As soon as you start to save your commands to a script, be aware that you approach a Turing tarpit. E.g. it's not possible to detect failures in the middle of a normal pipeline. bash added set -o pipefail for this, but this is not in POSIX.

Similar useful but unstandardized features are provided by almost every utility more complex than true.

For the class of task you outline, you can draw a rough line to Awk, Perl, and nowadays Python. Different tools were created, and evolved independently. Would you expect e.g. GNU Awk to be subsumed into a libutilposixextended?

I'm not saying we now have one universally better approach I can point you to. I have a soft spot for Python. Awk is surprisingly powerful, although I've been frustrated by some features being specific to GNU Awk. But the point is that processing large numbers of strings individually (presumably from lines of the files) was not a design goal of the POSIX shell.

Answer 6

There is also the question of: Which shell would you build it into?

Most Unix/Linux systems have multiple different shells which are developed independently(sh/bash/korn/???). If you build the tools into the shell, you would end up with a different implementation of these tools for each shell. This would cause overhead, and you might end up with different features/bugs in for example grep, depending on which shell you used to invoke it.

Answer 7

Many have answered well. I intend only to compliment those answers. I think the UNIX philosophy is that a tool should do one thing and do it well. If one tries to make an all encompassing tool, that's lot's more places for failure. Limiting functionality in this way makes a tool set that's reliable.

Also, consider, if functionality like sed or grep were built into the shell, would it be as easy to invoke from the command line when you'd like it?

In closing, consider, some of the functionality you're desiring to be in BASH, is in BASH. For example, the ability for RE matching in BASH is implemented using the =~ binary operator (see Shell Grammar in the Manual Page for more, specifically, reference the discussion of the [[ ]] construct for if). As a very quick example, say I'm searching a file for 2 hex digits:

while read line; do
    if [[ $line =~ 0x[[:xdigit:]]{2} ]]; then
        # do something important with it
    fi
done < input_file.txt

As for sed-like functionality, look under Parameter Expansion in the Expansion heading of the same man page. You'll see a wealth of things you can do that are reminiscent of sed. I most often use sed to make some substitution type change to text. Building off of the above:

# this does not take into account the saving of the substituted text
# it shows only how to do it
while read line; do
    ${line/pattern/substitution}
done < input_file.txt

In the end though, is the above "better" than?

grep -E "[[:xdigit:]]{3}" input_file.txt
sed -e 's/pattern/substitution/' input_file.txt

Answer 8

This is, I guess, a historical accident.

When UNIX was created in the late 1960s and early 1970s, computers did not have nearly as much memory as they do today. It would have been possible, at the time, to implement all this functionality as shell builtins, but due to memory limitations, they would have had to limit the amount of functionality that they could implement, or risk out of memory and/or swap trashing problems.

On the other hand, by implementing the given functionality as separate programs, and by making the two required system calls for starting a new process as light as possible, they could create a scripting environment that does not have those problems and that still runs at reasonable speed.

Of course, once those things are implemented as separate processes, people will start them from programs that are not shells, and then they have to remain like that, or suddenly all this software starts breaking.

That's not to say you can't implement some functionality twice, however, and indeed some shells implement some functionality that's supposed to be an external program as a shell builtin; e.g., bash implements the echo command as a builtin, but there's also a /usr/bin/echo