27

If a shell is asked to perform a probably useless (or partially useless) command known to terminate, such as cat hugeregularfile.txt > /dev/null, can it skip that command's execution (or execute a cheaper equivalent, say, touch -a hugeregularfile.txt)?

More generally, is the shell similar to C compilers in that it may perform any transformation on the source code, so long as the externally observable behaviour is as-if the abstract machine evaluated it?

EDIT

Nota Bene: My question as originally posed had a title that asked whether the shell is permitted to do these optimizations, not whether it should or even whether implementations that can do them exist. I'm interested in the theory more than the practice, although both are welcome.

Improve this question
8
  • No, the shell isn't as smart as modern compilers. In fact, it's rather dumb. It wouldn't optimize any useless code.
    – devnull
    Mar 10, 2014 at 6:09
  • 12
    Guessing what the user's intention is is not something the shell should do. The user could be trying to do almost anything with that command, optimising it out would be the wrong thing to do, even if it was possible.
    – Chris Down
    Mar 10, 2014 at 6:11
  • 1
    No matter saying that if the file was a device then catting it makes a big difference. The shell can get to know that the file is a device, but it need not be reliable.
    – yo'
    Mar 10, 2014 at 16:29
  • 3
    @StephaneChazelas C compilers don't need to "ask permission from someone" to optimize their compiled programs; There is an as-if rule in the C standard which permits them to do so. The POSIX standard appears to have standardized at least one shell (pubs.opengroup.org/onlinepubs/009695399/utilities/…), as well as numerous other utilities (pubs.opengroup.org/onlinepubs/009604499/utilities/wc.html for wc, for instance). But to the best of my knowledge POSIX doesn't take a position on shell optimization; Or does it?
    – Iwillnotexist Idonotexist
    Mar 10, 2014 at 21:04
  • 2
    Optimisation is improving performance with shortcuts without affecting functionality. As long as the functionality is guaranteed, I can't see POSIX objecting. Your proposed optimisation would break the cat spec though though. There are specific wordings in the POSIX spec that are there to accommodate the type of optimisation done by ksh. Like they don't say separate process but subshell environment to allow fork-saving optimisations.
    – Stéphane Chazelas
    Mar 10, 2014 at 21:12

4 Answers 4

Reset to default
26

No, that would be a bad idea.

cat hugeregularfile.txt > /dev/null and touch -a hugeregularfile.txt are not the same. cat will read the whole file, even if you redirect the output to /dev/null. And reading the whole file might be exactly what you want. For example in order to cache it so that later reads will be significantly faster. The shell can't know your intention.

Similarly, a C compiler will never optimize out reading a file, even if you don't look at the stuff you read.

Improve this answer
6
  • 2
    @Iwillnotexist: Every useful command (except arguably true and false) has potential side effects, and the side effects are almost always the point of invoking the command. The shell couldn't know those side effects in advance (for external programs like cat) without solving the Halting Problem. So it rightly doesn't try, and assumes you meant what you said.
    – cHao
    Mar 10, 2014 at 14:00
  • 5
    @IwillnotexistIdonotexist No, the shell can't see all that is about to happen. It has no idea about cat. In fact, cat could do anything from formatting your hard drive to downloading the Internet.
    – scai
    Mar 10, 2014 at 14:36
  • 5
    "Unix was not designed to stop its users from doing stupid things, as that would also stop them from doing clever things." – Doug Gwyn
    – Agi Hammerthief
    Mar 10, 2014 at 16:44
  • 7
    @cHao And even true and false set $?.
    – Kyle Strand
    Mar 10, 2014 at 19:17
  • 3
    As @scai pointed out above, executables are not like language keywords: cat and /dev/null have typical meanings but they aren't guaranteed to behave that way. To perform optimizations while guaranteeing no change to expected behaviour, the optimization could only be allowed involve constructs implemented within the shell itself and not things found in the execution environment... no matter how intuitive their names might seem.
    – andybuckley
    Mar 11, 2014 at 19:59
20

No, since /dev/null is just a name, which could be used for any other device or for a file other than what "normally" is a data sink.

So a shell (or any other program) has no idea, based on the name, whether the file it is writing to is doing something "for real" with the data. There are AFAIK also no system calls the shell program can make, to determine that e.g. the file descriptor is actually not doing anything.

Your comparison with optimising away code in a C program does not work, as a shell does not have the total overview that a C compiler has over a piece of source code. A shell doesn't know enough about /dev/null to optimize your example away, more like a C compiler doesn't know enough about code in a function call it dynamically links to, to not make the call.

Improve this answer
4
  • 4
    As it turns out, ksh93 will treat /dev/null specially, sometimes. A builtin that has its stdout directed to /dev/null, e.g echo foo >/dev/null, will not result in any writes being done to /dev/null. It doesn't do anything special if it's invoking a non-builtin command (such as cat file >/dev/null).
    – Mark Plotnick
    Mar 10, 2014 at 19:05
  • Matter of fact, cat could also be something else. Anything else in fact.
    – orion
    Mar 10, 2014 at 19:19
  • 3
    Actually /dev/null is one of the very few standardized paths, along with /dev/tty, /dev/console, /tmp, /dev/ and /.
    – Gilles 'SO- stop being evil'
    Mar 10, 2014 at 22:08
  • 2
    @MarkPlotnick Actually cat is a ksh93 builtin (not enabled unless you put /opt/ast/bin before /bin (or wherever any cat is available) in $PATH). And yes, though cat file > /dev/null with that builtin does read the content of file, it does not write it to /dev/null (though it opens and fstats it).
    – Stéphane Chazelas
    Mar 11, 2014 at 17:35
15

It will not optimise out running commands (and you've already received a number of fine answers telling you why it should not), but it may optimise out forks, pipe/socketpairs, reads in some cases. The kind of optimisations it may do:

  • With some modern shells, the last command in a script can be executed in the process of the shell unless some traps have been set. For instance in sh -c ls, most sh implementations (bash, mksh, ksh, zsh, yash, some versions of ash) won't fork a process to run ls.
  • in ksh93, command substitution will not create a pipe or fork a process until an external command is called ($(echo foo) for instance will expand to foo without a pipe/socketpair or fork).
  • the read built-in of some shells (bash, AT&T ksh) will not do single-byte reads if they detect stdin is seekable (in which case they will do large reads and seek back to the end of what they are meant to read).
Improve this answer
7
  • I like this answer, but it's unclear whether this is original reason, or whether the info is taken from some reference (which I'd like to delve into)
    – yoniLavi
    Mar 10, 2014 at 14:40
  • 3
    @yoniYalovitsky, that's original reason. ksh93 is the shell leading the way on the optimisation front as its aim is/was to be seen as being on par with programming languages like perl. So you can have a look at ksh documentation, code (good luck) and mailing list for further info.
    – Stéphane Chazelas
    Mar 10, 2014 at 20:38
  • 1
    @HenkLangeveld, yes, you can verify that with sh -c 'ps -p "$$"' which will give you ps and not sh with those sh implementations, or with strace/truss/tusc...
    – Stéphane Chazelas
    Mar 14, 2014 at 9:37
  • 1
    The difference between ksh -c 'ps; ps' and bash -c 'ps;ps' is interesting. Ksh93 goes further in its optimisation.
    – Henk Langeveld
    Mar 14, 2014 at 9:52
  • 1
    @HenkLangeveld, depends what implementation of ksh we're talking about here. mksh behaves like bash. That behaviour is mostly meant to optimize things like system("some command"). Note that there's a side effect of that optimisation when it comes to the exit status of processes terminated by a signal (in some shells). ksh93 used to have a bug in that it was doing the optimisation even when traps were set.
    – Stéphane Chazelas
    Mar 14, 2014 at 10:04
7

When seeing cat hugeregularfile.txt > /dev/null, the shell is not allowed to believe that the action is useless — cat is not part of the shell and could do anything at all in theory, and also in practice.

For example, the user may have renamed the executable rm to cat, and suddenly the line performs externally observable behavior, i.e., removing the file.

The user may have compiled a version of cat that goes into an infinite loop, thus the shell cannot assume that it is 'known to terminate' as you suggest.

Someone may have installed a version of cat that works as intended, but with an extra side effect of installing a rootkit if it's ever run with adequate privileges — again, the shell should duly execute it.

Improve this answer
3
  • 2
    Actually, mksh does in fact optimize V=$(cat file) by making it a builtin. So the shell may optimize it out, but not transform it to just a touch -a.
    – Steve Schnepp
    Mar 11, 2014 at 12:41
  • 1
    @SteveSchnepp, cat is a builtin in mksh, but that builtin resorts to the system's cat if passed any option, which is why with GNU cat, mksh -c 'cat /dev/null --help' doesn't yield the same result as bash -c 'cat /dev/null --help', but mksh -c 'cat --help /dev/null' does give you the same as bash -c 'cat --help /dev/null' (as mksh cat builtin parses options the POSIX way, while GNU cat parses them the GNU way).
    – Stéphane Chazelas
    Mar 12, 2014 at 16:02
  • In bash and ksh93, the V=$(cat file) can be optimised with V=$(< file). This speeds up things even without a builtin cat.
    – Henk Langeveld
    Mar 14, 2014 at 11:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.