73

POSIX defines standard error as for writing diagnostic output This doesn't limit its use to error messages only. I would consider progress information as diagnostic output, so it belongs on standard error.


68

OS X currently comes with a FreeBSD sed from 2005. Most of the differences below also apply to other BSD sed versions. OS X's sed uses -E for ERE and GNU sed uses -r. -E is an alias for -r in GNU sed (added in 4.2, not documented until 4.3). Newer versions of FreeBSD and NetBSD sed support both -E and -r. OpenBSD sed only supports -E. -i '' works with OS X'...


60

Most important things POSIX 7 defines C API Greatly extends ANSI C with things like: more file operations: mkdir, dirname, symlink, readlink, link (hardlinks), poll(), stat, sync, nftw() process and threads: fork, execl, wait, pipe, semaphors sem_*, shared memory (shm_*), kill, scheduling parameters (nice, sched_*), sleep, mkfifo, setpgid() networking: ...


46

Unfortunately there is no guarantee of anything being available. However, most systems will have GNU coreutils. That alone provides about 105 commands. You can probably rely on those unless it's an embedded system, which might use BusyBox instead. You can probably also rely on bash, cron, GNU findutils, GNU grep, gzip, iproute2, iputils, man-db, module-...


46

There is no standard indentation in shell scripts that matters. Slightly less flippant answer: Pick a standard in your team that you can all work to, to simplify things. Use something your editor makes easy so you don't have to fight to stick to the standard.


29

Debian Live Standard is Debian without the Graphical User Interface. Debian Live Gnome is Debian Standard with Gnome.


28

Posix defines the standard streams thus: At program start-up, three streams shall be predefined and need not be opened explicitly: standard input (for reading conventional input), standard output (for writing conventional output), and standard error (for writing diagnostic output). When opened, the standard error stream is not fully buffered; the standard ...


26

I've never encountered shell specified style guide but for bash programming this is the most popular one: Bash Style Guide and Coding Standard.pdf | lug.fh-swf.de The indentation of program constructions has to agree with the logic nesting depth. The indentation of one step usually is in line with the tabulator steps of the editor selected. In most cases ...


26

No, that would be a bad idea. cat hugeregularfile.txt > /dev/null and touch -a hugeregularfile.txt are not the same. cat will read the whole file, even if you redirect the output to /dev/null. And reading the whole file might be exactly what you want. For example in order to cache it so that later reads will be significantly faster. The shell can't know ...


20

No, since /dev/null is just a name, which could be used for any other device or for a file other than what "normally" is a data sink. So a shell (or any other program) has no idea, based on the name, whether the file it is writing to is doing something "for real" with the data. There are AFAIK also no system calls the shell program can make, to determine ...


18

Google says 2 spaces: Shell Style Guide - Indentation | google.github.io Agree with others that it is an arbitrary choice. Note: If link above does not work in future, Google's complete style project is hosted here: Style guides for Google-originated open-source projects | github.com


18

I'd use .sh (for files in the POSIX sh language, .bash for non-sh-compatible bash files, that is the extension identifies the language the script is written in) for files intended to be sourced (or more generally not intended to be executed), and no extension for files that are meant to be executed. You can also add a: #! /bin/echo Please-source she-bang, ...


18

Adapted from the POSIX standard's "Utility Argument Syntax" section: utility_name [-a] [-b] [-c option_argument] [-d|-e] [-f[option_argument]] [operand...] The utility in the example is named utility_name. It is followed by options, option-arguments, and operands. The arguments that consist of - characters and single letters or digits, such ...


17

The POSIX 2008 standard has a section describing "Shell and Utilities". Generally, if you stick to that your scripts should be fairly future-proof, except possibly for deprecations, but those hardly happen overnight so you should have plenty of time to update your scripts. In some cases where output format for a single utility varies widely across ...


17

There are at least 4 relevant pieces of text in the POSIX.2018 specification of awk: Emphasis (bold text) is mine in all the quoted text below: Input files to the awk program from any of the following sources shall be text files That means that if the input contains NUL characters (which would make it non-text as per the POSIX definition of text), then ...


16

As written in the manual page, the /etc/networks file is to describe symbolic names for networks. With network, it is meant the network address with tailing .0 at the end. Only simple Class A, B or C networks are supported. In your example the google-dns entry is wrong. It's not a A,B or C network. It's an ip-address-hostname-relationship therefore it ...


14

It will not optimise out running commands (and you've already received a number of fine answers telling you why it should not), but it may optimise out forks, pipe/socketpairs, reads in some cases. The kind of optimisations it may do: With most modern shells, the last command in a script will generally be executed in the process of the shell unless some ...


13

Yes and no. In a POSIX environment, the utilities must behave as described by the specification. In practice, this means that conforming versions of the utilities must be present in $PATH. However, when running your program on a POSIX-compliant system, you may be running it in a non-conforming environment. In practice, what often happens is that the OS has a ...


12

TLDR; see Ivan's answer below https://unix.stackexchange.com/a/203328/25985 People coming from monolithic mainstream OS's like windows or OSX are often confused by the heterogeneous nature of GNU/Linux. By "heterogeneous" vs. "monolithic" I mean that while windows and OSX are both essentially gigantic, singular pieces of integrated software, linux is a ...


12

I'll try to answer from my experience. Commands don't really adhere to a formal specification, but they do adhere to a requirement to consume and generate line-oriented text. Yes, of course. Before the GNU utilities became a de facto standard, a lot of vendors would have quirky output, especially with respect to ps and ls. This caused a lot of pain. Today,...


12

It seems that they are standardized in the POSIX spec, POSIX.1-2017 by proxy of unistd.h The header shall define the following symbolic constants for file streams: STDERR_FILENO File number of stderr; 2. STDIN_FILENO File number of stdin; 0. STDOUT_FILENO File number of stdout; 1. But also the POSIX docs on "stderr, stdin, stdout - ...


11

The files that are opened are not files on disk. They are the streams (pseudo files), stdin (0), stdout (1), and stderr (2). Here is the relevant excerpt from the POSIX standard: A file with associated buffering is called a stream and is declared to be a pointer to a defined type FILE. The fopen() function shall create certain descriptive data for a ...


10

POSIX is slightly more concrete about "diagnostic information" in Shell and Utilities, 1.4: Utility Description Defaults (emphasis mine): STDERR The STDERR section describes the standard error output of the utility. Only those messages that are purposely sent by the utility are described. Use of a terminal for standard error may cause any of the ...


9

First, very brief answers to your questions: Formal standardization of input/output conventions: no Breakage in the past due to changing output: yes Absolutely impossible to break future filters: no How can I protect myself against changes: be conservative When you say "API", you're using a term that (for good or ill) implies too much formality around ...


9

By the principle of exclusion, it can only go to stderr. Yes, I know you asked about an official specification, which I cannot present you beyond the link to the POSIX specification, given by Stephen Kitt, which states that stderr is for diagnostic purposes. The more important point is that stdin and stdout have a function that disallows printing progress ...


7

On non-embedded Linux systems, you can generally count on most GNU utilities: coreutils bash findutils (at least find and xargs, not necessarily locate and updatedb) grep gzip sed tar plus the util-linux suite and the procps suite. Note that /bin/sh is not always bash, it can be a shell with less features such as one of the multiple forks of ash. The ...


7

When seeing cat hugeregularfile.txt > /dev/null, the shell is not allowed to believe that the action is useless — cat is not part of the shell and could do anything at all in theory, and also in practice. For example, the user may have renamed the executable rm to cat, and suddenly the line performs externally observable behavior, i.e., removing ...


7

If my goal is to write a script for POSIX compatible shells, scripts that I would begin with #!/bin/sh, how permissible is this [[ syntax. Not at all. Many modern Linux-based systems use dash as /bin/sh by default, which is fully POSIX-compliant and no more, and so [[ will not work. This includes every recent Debian and Ubuntu release, and their derivatives,...


6

The best way I have found to have the same script work on both Linux and Mac is to: sed -i.bak -e 's/foo/bar/' -- "${TARGET}" && rm -- "${TARGET}.bak"


6

There is a set of commands most if not all Linux distributions, and for that matter, also Unix distributions will provide. These are the mandatory commands specified by the POSIX standard. Most of the commands you cite (cd, mkdir, ls, echo, grep, sed, awk, etc.) are of it. The exception being ping as WhiteWinterWolf rightly commented.


Only top voted, non community-wiki answers of a minimum length are eligible