What are the shell's control and redirection operators?

Question

I often see tutorials online that connect various commands with different symbols. For example:

command1 |  command2
command1 &  command2
command1 || command2    
command1 && command2

Others seem to be connecting commands to files:

command1  > file1
command1  >> file1

What are these things? What are they called? What do they do? Are there more of them?

---

Meta thread about this question..

Answer 1

These are called shell operators and yes, there are more of them. I will give a brief overview of the most common among the two major classes, control operators and redirection operators, and how they work with respect to the bash shell.

A. Control operators

POSIX definition

In the shell command language, a token that performs a control function.
It is one of the following symbols:

&   &&   (   )   ;   ;;   <newline>   |   ||

And |& in bash.

A ! is not a control operator but a Reserved Word. It becomes a logical NOT [negation operator] inside Arithmetic Expressions and inside test constructs (while still requiring an space delimiter).

A.1 List terminators

• ; : Will run one command after another has finished, irrespective of the outcome of the first.

    command1 ; command2
  

First command1 is run, in the foreground, and once it has finished, command2 will be run.

A newline that isn't in a string literal or after certain keywords is not equivalent to the semicolon operator. A list of ; delimited simple commands is still a list - as in the shell's parser must still continue to read in the simple commands that follow a ; delimited simple command before executing, whereas a newline can delimit an entire command list - or list of lists. The difference is subtle, but complicated: given the shell has no previous imperative for reading in data following a newline, the newline marks a point where the shell can begin to evaluate the simple commands it has already read in, whereas a ; semi-colon does not.

• & : This will run a command in the background, allowing you to continue working in the same shell.

     command1 & command2
  

Here, command1 is launched in the background and command2 starts running in the foreground immediately, without waiting for command1 to exit.

A newline after command1 is optional.

A.2 Logical operators

• && : Used to build AND lists, it allows you to run one command only if another exited successfully.

     command1 && command2
  

Here, command2 will run after command1 has finished and only if command1 was successful (if its exit code was 0). Both commands are run in the foreground.

This command can also be written

    if command1
    then command2
    else false
    fi

or simply if command1; then command2; fi if the return status is ignored.

• || : Used to build OR lists, it allows you to run one command only if another exited unsuccessfully.

     command1 || command2
  

Here, command2 will only run if command1 failed (if it returned an exit status other than 0). Both commands are run in the foreground.

This command can also be written

    if command1
    then true
    else command2
    fi

or in a shorter way if ! command1; then command2; fi.

Note that && and || are left-associative; see Precedence of the shell logical operators &&, || for more information.

• !: This is a reserved word which acts as the “not” operator (but must have a delimiter), used to negate the return status of a command — return 0 if the command returns a nonzero status, return 1 if it returns the status 0. Also a logical NOT for the test utility.

    ! command1
  
    [ ! a = a ]
  

And a true NOT operator inside Arithmetic Expressions:

    $ echo $((!0)) $((!23))
    1 0

A.3 Pipe operator

• | : The pipe operator, it passes the output of one command as input to another. A command built from the pipe operator is called a pipeline.

     command1 | command2
  

  Any output printed by command1 is passed as input to command2.

• |& : This is a shorthand for 2>&1 | in bash and zsh. It passes both standard output and standard error of one command as input to another.

    command1 |& command2
  

A.4 Other list punctuation

;; is used solely to mark the end of a case statement. Ksh, bash and zsh also support ;& to fall through to the next case and ;;& (not in ATT ksh) to go on and test subsequent cases.

( and ) are used to group commands and launch them in a subshell. { and } also group commands, but do not launch them in a subshell. See this answer for a discussion of the various types of parentheses, brackets and braces in shell syntax.

B. Redirection Operators

POSIX definition of Redirection Operator

In the shell command language, a token that performs a redirection function. It is one of the following symbols:

<     >     >|     <<     >>     <&     >&     <<-     <>

These allow you to control the input and output of your commands. They can appear anywhere within a simple command or may follow a command. Redirections are processed in the order they appear, from left to right.

• < : Gives input to a command.

    command < file.txt
  

The above will execute command on the contents of file.txt.

• <> : same as above, but the file is open in read+write mode instead of read-only:

    command <> file.txt
  

If the file doesn't exist, it will be created.

That operator is rarely used because commands generally only read from their stdin, though it can come handy in a number of specific situations.

• > : Directs the output of a command into a file.

    command > out.txt
  

The above will save the output of command as out.txt. If the file exists, its contents will be overwritten and if it does not exist it will be created.

This operator is also often used to choose whether something should be printed to standard error or standard output:

    command >out.txt 2>error.txt
 

In the example above, > will redirect standard output and 2> redirects standard error. Output can also be redirected using 1> but, since this is the default, the 1 is usually omitted and it's written simply as >.

So, to run command on file.txt and save its output in out.txt and any error messages in error.txt you would run:

    command < file.txt > out.txt 2> error.txt
  

• >| : Does the same as >, but will overwrite the target, even if the shell has been configured to refuse overwriting (with set -C or set -o noclobber).

    command >| out.txt
  

If out.txt exists, the output of command will replace its content. If it does not exist it will be created.

• >> : Does the same as >, except that if the target file exists, the new data are appended.

    command >> out.txt
  

If out.txt exists, the output of command will be appended to it, after whatever is already in it. If it does not exist it will be created.

• >& : (per POSIX spec) when surrounded by digits (1>&2) or - on the right side (1>&-) either redirects only one file descriptor or closes it (>&-).

A >& followed by a file descriptor number is a portable way to redirect a file descriptor, and >&- is a portable way to close a file descriptor.

If the right side of this redirection is a file please read the next entry.

• >&, &>, >>& and &>> : (read above also) Redirect both standard error and standard output, replacing or appending, respectively.

    command &> out.txt
  

Both standard error and standard output of command will be saved in out.txt, overwriting its contents or creating it if it doesn't exist.

    command &>> out.txt
 

As above, except that if out.txt exists, the output and error of command will be appended to it.

The &> variant originates in bash, while the >& variant comes from csh (decades earlier). They both conflict with other POSIX shell operators and should not be used in portable sh scripts.

• << : A here document. It is often used to print multi-line strings.

     command << WORD
         Text
     WORD
  

  Here, command will take everything until it finds the next occurrence of WORD, Text in the example above, as input . While WORD is often EoF or variations thereof, it can be any alphanumeric (and not only) string you like. When any part of WORD is quoted or escaped, the text in the here document is treated literally and no expansions are performed (on variables for example). If it is unquoted, variables will be expanded. For more details, see the bash manual.

  If you want to pipe the output of command << WORD ... WORD directly into another command or commands, you have to put the pipe on the same line as << WORD, you can't put it after the terminating WORD or on the line following. For example:

     command << WORD | command2 | command3...
         Text
     WORD
  

• <<< : Here strings, similar to here documents, but intended for a single line. These exist only in the Unix port or rc (where it originated), zsh, some implementations of ksh, yash and bash.

    command <<< WORD
  

Whatever is given as WORD is expanded and its value is passed as input to command. This is often used to pass the content of variables as input to a command. For example:

     $ foo="bar"
     $ sed 's/a/A/' <<< "$foo"
     bAr
     # as a short-cut for the standard:
     $ printf '%s\n' "$foo" | sed 's/a/A/'
     bAr
     # or
     sed 's/a/A/' << EOF
     $foo
     EOF

A few other operators (>&-, x>&y x<&y) can be used to close or duplicate file descriptors. For details on them, please see the relevant section of your shell's manual (here for instance for bash).

That only covers the most common operators of Bourne-like shells. Some shells have a few additional redirection operators of their own.

Ksh, bash and zsh also have constructs <(…), >(…) and =(…) (that latter one in zsh only). These are not redirections, but process substitution.

Answer 2

Warning regarding ‘>’

Unix beginners who have just learned about I/O redirection (< and >) often try things like

command … input_file > the_same_file

or

command … < file     > the_same_file

or, almost equivalently,

cat file | command … > the_same_file

(grep, sed, cut, sort, and spell are examples of commands that people are tempted to use in constructs like these.)  Users are surprised to discover that these scenarios result in the file becoming empty.

A nuance that doesn’t seem to be mentioned in the other answer can be found lurking in the first sentence of the Redirection section of bash(1):

Before a command is executed, its input and output may be redirected using a special notation interpreted by the shell.

The first five words should be bold, italic, underlined, enlarged, blinking, colored red, and marked with a icon, to emphasize the fact that the shell performs the requested redirection(s) before the command is executed.  And remember also

Redirection of output causes the file … to be opened for writing ….  If the file does not exist it is created; if it does exist it is truncated to zero size.

1. So, in this example:

   sort roster > roster
   

   the shell opens the roster file for writing, truncating it (i.e., discarding all its contents), before the sort program starts running.  Naturally, nothing can be done to recover the data.

2. One might naïvely expect that

   tr "[:upper:]" "[:lower:]" < poem > poem
   

   might be better.  Because the shell handles redirections from left to right, it opens poem for reading (for tr’s standard input) before it opens it for writing (for standard output).  But it doesn’t help.  Even though this sequence of operations yields two file handles, they both point to the same file.  When the shell opens the file for reading, the contents are still there, but they still get clobbered before the program is executed. 

So, what to do about it?

Solutions include:

• Check whether the program you’re running has its own, internal, capability to specify where the output goes.  This is often indicated by a -o (or --output=) token.  In particular,

  sort -o roster roster
  

  is roughly equivalent to

  sort roster > roster
  

  except, in the first case, the sort program opens the output file.  And it’s smart enough not to open the output file until after it has read all of the input file(s).

  Similarly, at least some versions of sed have a -i (edit in place) option that can be used to write the output back out to the input file (again, after all the input have been read).  Editors like ed/ex, emacs, pico, and vi/vim allow the user to edit a text file and save the edited text in the original file.  Note that ed (at least) can be used non-interactively.

  • vi has a related feature.  If you type :%!commandEnter, it will write the contents of the edit buffer out to command, read the output, and insert it into the buffer (replacing the original contents).

• Simple but effective:

  command … input_file > temp_file  &&  mv temp_file input_file

  This has the drawback that, if input_file is a link, it will (probably) be replaced by a separate file.  Also, the new file will be owned by you, with default protections.  In particular, this carries the risk that the file will be end up being world-readable, even if the original input_file wasn’t.

  Variations:

  • command … input_file > temp_file && cp temp_file input_file && rm temp_file
    which will still (potentially) leave the temp_file world-readable.  Even better:
  • cp input_file temp_file && command … temp_file > input_file && rm temp_file
    These preserve the link status, owner, and mode (protection) of the file, potentially at the cost of twice as much I/O.  (You may need to use an option like -a or -p on cp to tell it to preserve attributes.)
  • command … input_file > temp_file &&
cp --attributes-only --preserve=all input_file temp_file &&
mv temp_file input_file
    (broken into separate lines only for readability)  This preserves the mode of the file (and, if you’re root, the owner), but makes it owned by you (if you’re not root), and makes it a new, separate file.

• This blog (“In-place” editing of files) suggests and explains

  { rm input_file  &&  command … > input_file; } < input_file

  This requires that the command be able to process standard input (but almost all filters can).  The blog itself calls this a risky kludge and discourages its use.  And this will also create a new, separate file (not linked to anything), owned by you and with default permissions.

• The moreutils package has a command called sponge:

  command … input_file | sponge the_same_file

  See this answer for more information.

Here’s something that came as a complete surprise to me: syntaxerror says:

[Most of these solutions] will fail on a read-only file system, where “read-only” means that your $HOME will be writable, but /tmp will be read-only (by default).  For instance, if you have Ubuntu, and you’ve booted into the Recovery Console, this is commonly the case.  Also, the here-document operator <<< will not work there either, as it requires /tmp to be read/write because it will write a temporary file into there as well.
(cf. this question includes an strace’d output)

The following may work in that case:

• For advanced users only: If your command is guaranteed to produce the same amount of output data as there is input (e.g., sort, or tr without the -d or -s option), you can try

  command … input_file | dd of=the_same_file conv=notrunc

  See this answer and this answer for more information, including an explanation of the above, and alternatives that work if your command is guaranteed to produce the same amount of output data as there is input or less (e.g., grep, or cut).  These answers have the advantage that they do not require any free space (or they require very little).  The answers above of the form command … input_file > temp_file && … clearly require that there be enough free space for the system to be able to hold the entire input (old) file and output (new) file simultaneously; this is non-obviously true for most of the other solutions (e.g., sed -i and sponge) as well.  Exception: sort … | dd … will probably require lots of free space, because sort needs to read all of its input before it can write any output, and it probably buffers most if not all of that data in a temporary file.
• For advanced users only:

  command … input_file 1<> the_same_file

  may be equivalent to the dd answer, above.  The n<> file syntax opens the named file on file descriptor n for both input and output, without truncating it – sort of a combination of n< and n>.  Note: Some programs (e.g., cat and grep) may refuse to run in this scenario because they can detect that the input and the output are the same file.  See this answer for a discussion of the above, and a script that makes this answer work if your command is guaranteed to produce the same amount of output data as there is input or less.
  Warning: I haven’t tested Peter’s script, so I don’t vouch for it.

So, what was the question?

This has been a popular topic on U&L; it is addressed in the following questions:

• Is there a way to modify a file in-place?
• How can I make iconv replace the input file with the converted output?
• Why does the command shuf file > file leave an empty file?
• Can I read and write to the same file in Linux without overwriting it?
• Redirect to the same file as the source file processed by the command
• Why does this sort command give me an empty file?
• Redirecting tr stdout to a file
• grep: input file 'X' is also the output
• Do redirection operators open file descriptors in parallel?
• Redirection not overwriting file but just producing a blank one

… and that’s not counting Super User or Ask Ubuntu.  I have incorporated a lot of the information from the answers to the above questions here in this answer, but not all.  (I.e., for more information, read the above-listed questions and their answers.)

^{P.S. I have no affiliation with the blog that I cited, above.}

Answer 3

More observations on ;, &, ( and )

• Note that some of the commands in terdon’s answer may be null.  For example, you can say

  command1 ;
  

  (with no command2).  This is equivalent to

  command1
  

  (i.e., it simply runs command1 in the foreground and waits for it to complete.  Comparably,

  command1 &
  

  (with no command2) will launch command1 in the background and then issue another shell prompt immediately.

• By contrast, command1 &&, command1 ||, and command1 | don’t make any sense.  If you type one of these, the shell will (probably) assume that the command is continued onto another line.  It will display the secondary (continuation) shell prompt, which is normally set to >, and keep on reading.  In a shell script, it will just read the next line and append it to what it has already read.  (Beware: this might not be what you want to happen.)

  Note: some versions of some shells may treat such incomplete commands as errors.  In such cases (or, in fact, in any case where you have a long command), you can put a backslash (\) at the end of a line to tell the shell to continue reading the command on another line:

  command1  &&  \
  command2
  

  or

  find starting-directory -mindepth 3 -maxdepth 5 -iname "*.some_extension" -type f \
                          -newer some_existing_file -user fred -readable -print
  

• As terdon says, ( and ) can be used to group commands.  The statement that they are “not really relevant” to that discussion is debatable.  Some of the commands in terdon’s answer may be command groups.  For example,

  ( command1 ; command2 )  &&  ( command3; command4 )
  

  does this:

  • Run command1 and wait for it to finish.
  • Then, regardless of the result of running that first command, run command2 and wait for it to finish.

  • Then, if command2 succeeded,

    • Run command3 and wait for it to finish.
    • Then, regardless of the result of running that command, run command4 and wait for it to finish.

    If command2 failed, stop processing the command line.

• Outside parentheses, | binds very tightly, so

  command1 | command2 || command3
  

  is equivalent to

  ( command1 | command2 )  ||  command3
  

  and && and || bind tighter than ;, so

  command1 && command2 ; command3
  

  is equivalent to

  ( command1 && command2 ) ;  command3
  

  i.e., command3 will be executed regardless of the exit status of command1 and/or command2.