Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It only takes a minute to sign up.
Sign up to join this community
Say I've got the following pipeline:
cmd1 < input.txt |\
cmd2 |\
cmd4 |\
cmd5 |\
cmd6 |\
(...) |\
cmdN > result.txt
Under certain conditions I would like to add a cmd3 between cmd2 and cmd4. Is there a way to create a kind of conditional pipeline without saving the result of cmd2 into a temporary file? I would think of something like:
cmd1 < input.txt |\
cmd2 |\
(${DEFINED}? cmd3 : cat ) |\
cmd4 |\
cmd5 |\
cmd6 |\
(...) |\
cmdN > result.txt
Just the usual && and || operators:
cmd1 < input.txt |
cmd2 |
( [ -n "$DEFINED" ] && cmd3 || cat ) |
cmd4 |
cmd5 |
cmd6 |
(...) |
cmdN > result.txt
Although, as specified by this answer, you would generally prefer if ... else, if you're after the if-else syntax:
...
cmd2 |
if [ -n "$DEFINED" ]; then cmd3; else cat; fi |
cmd4 |
...
(Note that no trailing backslash is needed when the line ends with pipe.)
Update according to Jonas' observation.
If cmd3 may terminate with non-zero exit code and you not want cat to process the remaining input, reverse the logic:
cmd1 < input.txt |
cmd2 |
( [ -z "$DEFINED" ] && cat || cmd3 ) |
cmd4 |
cmd5 |
cmd6 |
(...) |
cmdN > result.txt
cat returns with a non-zero exit status (common when getting a SIGPIPE), then cmd3 will be run. Better use if/then/else as in Thor's answer or other solutions that avoid running cat.
May 9, 2014 at 21:54
if/else/fi works. Assuming any Bourne-like shell:
cmd1 < input.txt |
cmd2 |
if [ -n "$DEFINED" ]; then cmd3; else cat; fi |
cmd4 |
cmd5 |
cmd6 |
(...) |
cmdN > result.txt
All the answers given so far replace cmd3 with cat. You can also avoid running any command with:
if [ -n "$DEFINE" ]; then
alias maybe_cmd3='cmd3 |'
else
alias maybe_cmd3=''
fi
cmd1 |
cmd2 |
maybe_cmd3
cmd4 |
... |
cmdN > result.txt
That's POSIX, but note that if in a bash script where bash is not in sh-mode (like with a script starting with #! /path/to/bash), you'll need to enable alias expansion with shopt -s expand_aliases (or set -o posix).
Another approach that still doesn't run any unnecessary command is to use eval:
if [ -n "$DEFINE" ]; then
maybe_cmd3='cmd3 |'
else
maybe_cmd3=''
fi
eval "
cmd1 |
cmd2 |
$maybe_cmd3
cmd4 |
... |
cmdN > result.txt"
Or:
eval "
cmd1 |
cmd2 |
${DEFINE:+cmd3 |}
cmd4 |
... |
cmdN > result.txt"
On Linux (at least), instead of cat, you could use pv -q which uses splice() instead of read() + write() to pass the data across between the two pipes which avoids having the data moved twice between kernel and user space.
As an addendum to manatwork's accepted answer: be aware of the and-false-or gotcha and its interaction with streams. For instance,
true && false || echo foo
outputs foo. Not surprisingly,
true && (echo foo | grep bar) || echo baz
and
echo foo | (true && grep bar || echo baz)
both output baz. (Note that echo foo | grep bar is false and has no output). However,
echo foo | (true && grep bar || sed -e abaz)
outputs nothing. This may or may not be what you want.
else (or ||) should act as a null operation keeping the input unchanged. So replacing cat with echo changes everything making your code irrelevant. Regarding the “and-false-or gotcha”, I see no interference with the pipeline: pastebin.com/u6Jq0bZe
May 20, 2012 at 11:29
[[ "${DEFINED}" ]] && cmd3 || cat, cmd3 and cat are not the mutually exclusive then and else branches of the same if, but that if cmd3 fails, then cat will also be executed. (Of course, if cmd3 always succeeds, or if it consumes all the input so there's nothing left in stdin for cat to process, then the it may not matter.) If you need both then and else branches, it's better to use an explicit if command, not && and ||.
condition && command1 || command2 is not the same as if condition; then command1; else command2; fi
Aug 15, 2020 at 11:43
I had a similar situation that I solved with bash functions:
if ...; then
my_cmd3() { cmd3; }
else
my_cmd3() { cat; }
if
cmd1 < input.txt |
cmd2 |
my_cmd3 |
cmd4 |
cmd5 |
cmd6 |
(...) |
cmdN > result.txt
Here is an alternative possible solution:
concatenating named pipes in order to create a pseudo-pipeline:
dir="$(mktemp -d)"
addfifo() {
stdin="$dir/$fifo_n"
[ "$1" ] || mkfifo "$dir/$((fifo_n=fifo_n+1))"
stdout="$dir/$fifo_n"
}
addfifo; echo "The slow pink fox crawl under the brilliant dog" > "$stdout" &
# Restoring the famous line: "The quick brown fox jumps over the lazy dog"
# just replace 'true' with 'false' in any of the 5 following lines and the pseudo-pipeline will still work
true && { addfifo; sed 's/brilliant/lazy/' < "$stdin" > "$stdout" & }
true && { addfifo; sed 's/under/over/' < "$stdin" > "$stdout" & }
true && { addfifo; sed 's/crawl/jumps/' < "$stdin" > "$stdout" & }
true && { addfifo; sed 's/pink/brown/' < "$stdin" > "$stdout" & }
true && { addfifo; sed 's/slow/quick/' < "$stdin" > "$stdout" & }
addfifo -last; cat < "$stdin"
wait; rm -r "$dir"
For future reference, if you are coming here looking for conditional pipes in fish, this is how you do it:
set -l flags "yes"
# ...
echo "conditionalpipeline" | \
if contains -- "yes" $flags
sed "s/lp/l p/"
else
cat
end | tr '[:lower:]' '[:upper:]'
