39

Say I've got the following pipeline:

cmd1 < input.txt |\
cmd2 |\
cmd4 |\
cmd5 |\
cmd6 |\
(...) |\
cmdN > result.txt

Under certain conditions I would like to add a cmd3 between cmd2 and cmd4. Is there a way to create a kind conditional pipeline without saving the result of cmd2 into a temporary file ? I would think of something like:

cmd1 < input.txt |\
cmd2 |\
(${DEFINED}? cmd3 : cat ) |\
cmd4 |\
cmd5 |\
cmd6 |\
(...) |\
cmdN > result.txt
34

Just the usual && and || operators:

cmd1 < input.txt |
cmd2 |
( [[ "${DEFINED}" ]] && cmd3 || cat ) |
cmd4 |
cmd5 |
cmd6 |
(...) |
cmdN > result.txt

(Note that no trailing backslash is needed when the line ends with pipe.)

Update according to Jonas' observation.
If cmd3 may terminate with non-zero exit code and you not want cat to process the remaining input, reverse the logic:

cmd1 < input.txt |
cmd2 |
( [[ ! "${DEFINED}" ]] && cat || cmd3 ) |
cmd4 |
cmd5 |
cmd6 |
(...) |
cmdN > result.txt
  • Handy tip Noted! – invert May 10 '12 at 11:08
  • 5
    Be aware of the gotchas: unix.stackexchange.com/a/39043/19055 – Jonas Kölker May 20 '12 at 11:09
  • 2
    Reversing the logic doesn't help. If cat returns with a non-zero exit status (common when getting a SIGPIPE), then cmd3 will be run. Better use if/then/else as in Thor's answer or other solutions that avoid running cat. – Stéphane Chazelas May 9 '14 at 21:54
  • so cat can be used to connect things, kind of like a "through" stream – Alexander Mills Jul 31 '17 at 19:20
18

if/else/fi works. Assuming any Bourne-like shell:

cmd1 < input.txt |
cmd2 |
if [ -n "$DEFINED" ]; then cmd3; else cat; fi |
cmd4 |
cmd5 |
cmd6 |
(...) |
cmdN > result.txt
10

All the answers given so far replace cmd3 with cat. You can also avoid running any command with:

if [ -n "$DEFINE" ]; then
  alias maybe_cmd3='cmd3 |'
else
  alias maybe_cmd3=''
fi
cmd1 |
cmd2 |
maybe_cmd3
cmd4 |
... |
cmdN > result.txt

That's POSIX, but note that if in a bash script where bash is not in sh-mode (like with a script starting with #! /path/to/bash), you'll need to enable alias expansion with shopt -s expand_aliases (or set -o posix).

Another approach that still doesn't run any unnecessary command is to use eval:

if [ -n "$DEFINE" ]; then
  maybe_cmd3='cmd3 |'
else
  maybe_cmd3=''
fi
eval "
  cmd1 |
  cmd2 |
  $maybe_cmd3
  cmd4 |
  ... |
  cmdN > result.txt"

Or:

eval "
  cmd1 |
  cmd2 |
  ${DEFINE:+cmd3 |}
  cmd4 |
  ... |
  cmdN > result.txt"

On Linux (at least), instead of cat, you could use pv -q which uses splice() instead of read() + write() to pass the data across between the two pipes which avoids having the data moved twice between kernel and user space.

8

As an addendum to manatwork's accepted answer: be aware of the and-false-or gotcha and its interaction with streams. For instance,

true && false || echo foo

outputs foo. Not surprisingly,

true && (echo foo | grep bar) || echo baz

and

echo foo | (true && grep bar || echo baz)

both output baz. (Note that echo foo | grep bar is false and has no output). However,

echo foo | (true && grep bar || sed -e abaz)

outputs nothing. This may or may not be what you want.

  • I can't see how is this relevant here. The else (or ||) should act as a null operation keeping the input unchanged. So replacing cat with echo changes everything making your code irrelevant. Regarding the “and-false-or gotcha”, I see no interference with the pipeline: pastebin.com/u6Jq0bZe – manatwork May 20 '12 at 11:29
  • 4
    @manatwork: One thing Jonas is pointing out is that, in [[ "${DEFINED}" ]] && cmd3 || cat, cmd3 and cat are not the mutually exclusive then and else branches of the same if, but that if cmd3 fails, then cat will also be executed. (Of course, if cmd3 always succeeds, or if it consumes all the input so there's nothing left in stdin for cat to process, then the it may not matter.) If you need both then and else branches, it's better to use an explicit if command, not && and ||. – musiphil Dec 2 '12 at 21:17
  • 1
    Thank you for the explanation, @musiphil. Now I understand Jonas' argument. – manatwork Dec 3 '12 at 7:10
4

I had a similar situation that I solved with bash functions:

if ...; then
  my_cmd3() { cmd3; }
else
  my_cmd3() { cat; }
if

cmd1 < input.txt |
cmd2 |
my_cmd3 |
cmd4 |
cmd5 |
cmd6 |
(...) |
cmdN > result.txt
2

Here is an alternative possible solution:
concatenating named pipes in order to create a pseudo-pipeline:

dir="$(mktemp -d)"
fifo_n='0'
function addfifo {
 stdin="$dir/$fifo_n"
((fifo_n++))
[ "$1" ] || mkfifo "$dir/$fifo_n"
stdout="$dir/$fifo_n"; }

addfifo; echo "The slow pink fox crawl under the brilliant dog" >"$stdout" &

# Restoring the famous line: "The quick brown fox jumps over the lazy dog"
# just replace 'true' with 'false' in any of the 5 following lines and the pseudo-pipeline will still work
true && { addfifo; sed 's/brilliant/lazy/' <"$stdin" >"$stdout" & }
true && { addfifo; sed 's/under/over/'     <"$stdin" >"$stdout" & }
true && { addfifo; sed 's/crawl/jumps/'    <"$stdin" >"$stdout" & }
true && { addfifo; sed 's/pink/brown/'     <"$stdin" >"$stdout" & }
true && { addfifo; sed 's/slow/quick/'     <"$stdin" >"$stdout" & }

addfifo -last; cat <"$stdin"

wait; rm -r "$dir"; exit 0
2

For future reference, if you are coming here looking for conditional pipes in fish, this is how you do it:

set -l flags "yes"
# ... 
echo "conditionalpipeline" | \
  if contains -- "yes" $flags 
    sed "s/lp/l p/"
  else
    cat
  end | tr '[:lower:]' '[:upper:]'

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.