Conditional pipeline

Question

Say I've got the following pipeline:

cmd1 < input.txt |\
cmd2 |\
cmd4 |\
cmd5 |\
cmd6 |\
(...) |\
cmdN > result.txt

Under certain conditions I would like to add a cmd3 between cmd2 and cmd4. Is there a way to create a kind of conditional pipeline without saving the result of cmd2 into a temporary file? I would think of something like:

cmd1 < input.txt |\
cmd2 |\
(${DEFINED}? cmd3 : cat ) |\
cmd4 |\
cmd5 |\
cmd6 |\
(...) |\
cmdN > result.txt

Answer 1

Just the usual && and || operators:

cmd1 < input.txt |
cmd2 |
( [ -n "$DEFINED" ] && cmd3 || cat ) |
cmd4 |
cmd5 |
cmd6 |
(...) |
cmdN > result.txt

Although, as specified by this answer, you would generally prefer if ... else, if you're after the if-else syntax:

  ...
  cmd2 |
  if [ -n "$DEFINED" ]; then cmd3; else cat; fi |
  cmd4 |
  ...

(Note that no trailing backslash is needed when the line ends with pipe.)

Update according to Jonas' observation.
If cmd3 may terminate with non-zero exit code and you not want cat to process the remaining input, reverse the logic:

cmd1 < input.txt |
cmd2 |
( [ -z "$DEFINED" ] && cat || cmd3 ) |
cmd4 |
cmd5 |
cmd6 |
(...) |
cmdN > result.txt

Answer 2

if/else/fi works. Assuming any Bourne-like shell:

cmd1 < input.txt |
cmd2 |
if [ -n "$DEFINED" ]; then cmd3; else cat; fi |
cmd4 |
cmd5 |
cmd6 |
(...) |
cmdN > result.txt

Answer 3

All the answers given so far replace cmd3 with cat. You can also avoid running any command with:

if [ -n "$DEFINE" ]; then
  alias maybe_cmd3='cmd3 |'
else
  alias maybe_cmd3=''
fi
cmd1 |
cmd2 |
maybe_cmd3
cmd4 |
... |
cmdN > result.txt

That's POSIX, but note that if in a bash script where bash is not in sh-mode (like with a script starting with #! /path/to/bash -), you'll need to enable alias expansion with shopt -s expand_aliases (or set -o posix).

Another approach that still doesn't run any unnecessary command is to use eval:

if [ -n "$DEFINE" ]; then
  maybe_cmd3='cmd3 |'
else
  maybe_cmd3=''
fi
eval "
  cmd1 |
  cmd2 |
  $maybe_cmd3
  cmd4 |
  ... |
  cmdN > result.txt"
  

Or:

eval "
  cmd1 |
  cmd2 |
  ${DEFINE:+cmd3 |}
  cmd4 |
  ... |
  cmdN > result.txt"

On Linux (at least), instead of cat, you could use pv -q which uses splice() instead of read() + write() to pass the data across between the two pipes which avoids having the data moved twice between kernel and user space.

Another option could be to define a function that runs a give command piped to or from cmd3 or not depending on $DEFINE.

if [ -n "$DEFINE" ]; then
  maybe_preprocess_with_cmd3() { cmd3 | "$@"; }
  maybe_postprocess_with_cmd3() { "$@" | cmd3; }
else
  maybe_preprocess_with_cmd3() { "$@"; }
  maybe_postprocess_with_cmd3() { "$@"; }
fi

And then:

cmd1 |
cmd2 |
maybe_preprocess_with_cmd3 cmd4 |
... |
cmdN > result.txt

Or:

cmd1 |
maybe_postprocess_with_cmd3 cmd2 |
cmd4 |
... |
cmdN > result.txt

Answer 4

As an addendum to manatwork's accepted answer: be aware of the and-false-or gotcha and its interaction with streams. For instance,

true && false || echo foo

outputs foo. Not surprisingly,

true && (echo foo | grep bar) || echo baz

and

echo foo | (true && grep bar || echo baz)

both output baz. (Note that echo foo | grep bar is false and has no output). However,

echo foo | (true && grep bar || sed -e abaz)

outputs nothing. This may or may not be what you want.

Answer 5

I had a similar situation that I solved with bash functions:

if ...; then
  my_cmd3() { cmd3; }
else
  my_cmd3() { cat; }
if

cmd1 < input.txt |
cmd2 |
my_cmd3 |
cmd4 |
cmd5 |
cmd6 |
(...) |
cmdN > result.txt

Answer 6

Here is an alternative possible solution:
concatenating named pipes in order to create a pseudo-pipeline:

dir="$(mktemp -d)"
addfifo() {
  stdin="$dir/$fifo_n"
  [ "$1" ] || mkfifo "$dir/$((fifo_n=fifo_n+1))"
  stdout="$dir/$fifo_n"
}

addfifo; echo "The slow pink fox crawl under the brilliant dog" > "$stdout" &

# Restoring the famous line: "The quick brown fox jumps over the lazy dog"
# just replace 'true' with 'false' in any of the 5 following lines and the pseudo-pipeline will still work
true && { addfifo; sed 's/brilliant/lazy/' < "$stdin" > "$stdout" & }
true && { addfifo; sed 's/under/over/'     < "$stdin" > "$stdout" & }
true && { addfifo; sed 's/crawl/jumps/'    < "$stdin" > "$stdout" & }
true && { addfifo; sed 's/pink/brown/'     < "$stdin" > "$stdout" & }
true && { addfifo; sed 's/slow/quick/'     < "$stdin" > "$stdout" & }

addfifo -last; cat < "$stdin"

wait; rm -r "$dir"

Answer 7

For future reference, if you are coming here looking for conditional pipes in fish, this is how you do it:

set -l flags "yes"
# ... 
echo "conditionalpipeline" | \
  if contains -- "yes" $flags 
    sed "s/lp/l p/"
  else
    cat
  end | tr '[:lower:]' '[:upper:]'