How stable are Unix shell "stdin/stdout APIs"?

Question

grepping, awking, sedding, and piping are day-to-day routine of a user of any Unix-like operating system, may it be on the command line or inside a shell script (collectively called filters from now on).

At their essence, when working with "standard" Unix CLI programs and shell builtins (collectively called commands from now on), filters need a precise expected format for stdin, stdout, and stderr in each filter step in order to work correctly. I call this precise expected format of some command an API of this command in the following.

As someone with web development background, I compare this kind of data collecting and data processing technically with web scraping - a technique which is very instable whenever there is the slightest change in data presentation.

My question now relates to the stability of Unix command APIs.

1. Do commands in a Unix-like operating systems adhere to a formal standardization with respect to their input and output?
2. Have there been instances in history where updates to some important command caused to break the functionality of some filter that was built using an older version of said command?
3. Have Unix commands matured over time that it is absolutely impossible to change in such a way that some filter could break?
4. In case filters may break from time to time due to changing command APIs, how can I as a developer protect my filters against this problem?

Answer 1

The POSIX 2008 standard has a section describing "Shell and Utilities". Generally, if you stick to that your scripts should be fairly future-proof, except possibly for deprecations, but those hardly happen overnight so you should have plenty of time to update your scripts.

In some cases where output format for a single utility varies widely across platforms and versions, the POSIX standard may include an option typically called -p or -P which specifies a guaranteed and predictable output format. An example of this is the time utility, which has widely varying implementations. If you need a stable API/output format, you would use time -p.

If you need to use a filter utility that is not covered by the POSIX standard, then you are pretty much at the mercy of the distribution packagers / upstream developers, just as you are at the mercy of the remote web developers when doing web scraping.

Answer 2

I'll try to answer from my experience.

1. Commands don't really adhere to a formal specification, but they do adhere to a requirement to consume and generate line-oriented text.

2. Yes, of course. Before the GNU utilities became a de facto standard, a lot of vendors would have quirky output, especially with respect to ps and ls. This caused a lot of pain. Today, only HP delivers super-quirky commands. Historically, the Berkeley Software Distribution (BSD) utilities were a major break with the past. The POSIX specification was a break with the past, but now it's widely accepted.

3. Unix commands have indeed matured over time. It's still not impossible to break some script written for an older version. Think about the recent trend towards UTF-8 as a text file encoding. This change necessitated changing basic utilities like tr. In the past, simple text was almost always ASCII (or something close), so uppercase letters formed a numerical range, as did lowercase letters. That's no longer true with UTF-8, so tr gets to accept different command line options to specify things like "uppercase" or "alphanumeric".

4. One of the best ways to "ruggedize" your filters is to not depend on particular text layout. For example, don't do cut -c10-24, which depends on positions of a line. Use cut -f2 instead, which would chop out the 2nd, tab-separated field. awk breaks any input line into $1, $2, $3... which are white-space separated by default. Depend on higher-level concepts like "fields" rather than lower-level concepts like column position. Also, use regular expressions: sed and awk can both do things with regular expressions that don't care about some variance in input. Another trick is to process the input into something whose format your filter can be picky about. Use tr -cs '[a-zA-z0-9]' '[\n]' to break text into a single word per line, without punctuation. You just don't care what the input text is like in that case.

Answer 3

First, very brief answers to your questions:

1. Formal standardization of input/output conventions: no
2. Breakage in the past due to changing output: yes
3. Absolutely impossible to break future filters: no
4. How can I protect myself against changes: be conservative

When you say "API", you're using a term that (for good or ill) implies too much formality around filter input/output conventions. Very (and I do mean "very") broadly, the primary conventions for data that is ameable to easy filtering are

• each input line is a complete record
• within each record, fields are separated by a known delimiter character

A classic example would be the format of /etc/passwd. But, these default conventions are probably violated to some degree more often than they're followed to the letter.

• There are lots of filters (often written in awk or perl) that parse multiline input formats.
• There are lots of input patterns (eg, /var/log/messages) where there is no well defined field structure, and more general regular expression-based techniques must be used.

Your fourth question, how to protect yourself against variations in output structure, is really the only one that you can do anything about.

• As @jw013 said, look at what the posix standards say. Of course, posix doesn't specify all commands you'll want to use as input sources.
• If you want your scripts to be portable, try to avoid the idiosyncracies of the whatever version of the some command you happen to have isntalled. For example, many GNU versions of standard unix commands have non-standard extensions. These may be useful, but you should avoid them if you want maximum portability.
• Try to learn what subsets of commands arguments and output formats tend to be stable across platforms. Unfortunately, this requires access to multiple platforms along with time, because these differences won't be written down anywhere, even informally.

In the end, you can't protect yourself fully from the problems you're worried about, and there's no single place to look to for a "definitive" statement of what a certain command should do. For many shell scripts, especially those written for personal or small-scale use, this simply isn't a problem

Answer 4

Only covering 1) of your question.

Naturally APIs can always change at the will of their creators, and thusly break dependent software, in any language. That said, the great idea of the Unix tools' I/O "APIs" is that there is practically none (maybe 0x0a as line end). A good script filters data with the Unix tools instead of creating it. That means that your script may break because the input or output spec changed, but not because the I/O format (again, there isn't really one) of the individual tools used in the script changed (because something that does not really exist can't really change).

Going through a list of basic tools there are few that I would also attribute producer, as opposed to only filter:

• wc - print number of bytes, words, lines -- very simple format, thusly absolutely unlikely to change, and furthermore not very likely to be used in a script.
• diff - there have evolved different output formats but I haven't heard of any problems. Also not normally used without supervision.
• date - Now here we really have to take care what we produce, especially regarding the system locale. But otherwise the output format is RFC'ed given you don't exactly specify it yourself.
• cal - let's not talk about it, I know that the output format does differ very much across systems.
• ls, who, w, last - I can't help if you want to parse ls, it just wasn't meant to be. Also, who, w, last, are more interactive listers; If you use them in a script you have to take care what you do.
• time was pointed out in another post. But yeah, it's the same as with ls. More for interactive/local use. And the bash builtin is very different from GNU version, and GNU version has had unfixed bugs for many years. Just don't rely on it.

Here are tools that expect a particular input format more specific than being a byte stream:

• bc, dc -- calculators. Already on the more hackish side of things (really, I don't use them in scripts), and presumably very stable I/O formats.

There's another area with a much higher risk of breakage, namely command-line interface. Most tools have differing features both across systems and across the timeline. Examples are

• All tools using regex -- regex can change meaning based on system locale (for example LC_COLLATE) and there are many subtleties and pecularities across regex implementations.
• Simply don't use fancy switches. You can easily use man 1p find for example, to read the POSIX find manpage instead of the system manpage. On my system, I need manpages-posix installed.

And even when using such switches, normally there errors won't be subtly introduced and poison your data. Most programs will simply refuse to work with an unknown switch.

To conclude, I would say that shell has actually the potential of being one of the most portable languages (it's portable when you script portably). Compare to your favorite scripting languages where subtle errors occur, or your favorite compiled program which will cede to compile.

Additionally, at the rare places where breakage may occur because of incompatibilities, it probably would not because of time induced, but because of diversity across the different systems (meaning if it works for you, it did so 20 years before and will in 20 years, too). That is a corollary of the tools' simplicity.

Answer 5

There are only de facto IO standards — whitespace and null separated output.

As for compatibility, we usually revert to checking version numbers of individual filters. Not that they change much, but when you want to use a brand new feature and still want the script to run on older versions, you have to "ifdef" it out somehow. There is practically no capability reporting mechanism, save for manually writing test cases.

Answer 6

Scripts do break, some more often than others. The old and famous software tends to stay relatively the same, and often has compatibility flags when it changes anyway.

Scripts written on one system tend to carry on working, but often break another.