2

Without and with the dry run, the script below executes as expected for A and B, respectively. The problem arises with the dry run for A: the output is inhibited, as if 1>/dev/null persisted after the dry run. Could someone explain and resolve the problem?

function baz() {
    local file="$1"; shift

    # dry run
    local err=$(source "$file" "$@" 2>&1 1>/dev/null)
    [[ -z $err ]] || { echo "Exiting"; return 1; }

    # live run
    local output=$(source "$file" "$@")
    [[ -z $output ]] || echo "$output"
}

baz <(cat <<EOF
echo "I was given $# argument(s):"  # A
# eecho "I was given $# argument(s):" # B
printf "%s " "$@"
EOF
) 'foo' 'bar'

Other:

$ uname  -a
6.7.3-arch1-2 #1 SMP PREEMPT_DYNAMIC Fri, 02 Feb 2024 17:03:55 +0000 x86_64 GNU/Linux
$ bash --version
GNU bash, version 5.2.26(1)-release (x86_64-pc-linux-gnu)
3
  • What happens if you invert the positions of the # dry run and the # live run sections of the code?
    – Samveen
    Mar 7 at 4:14
  • @Samveen that would defeat the purpose so please make clear what you're leading at.
    – Erwann
    Mar 7 at 4:19
  • You're expecting a socket descriptor to work the same way as a file name. That isn't going to happen, as the semantics are different.
    – Samveen
    Mar 7 at 4:26

3 Answers 3

5

There are two separate problems in your code:

  1. You forgot quotes around EOF so the $#/$@ will be expanded in the here document at the time that here document is parsed.
  2. You're trying to read twice until the end-of-file from a named pipe (aka fifo)

For 1, compare:

$ bash --norc -s {1..42}
bash-5.3$ cat << EOF
> echo "$#"
> EOF
echo "42"
bash-5.3$ cat << 'EOF'
> echo "$#"
> EOF
echo "$#"

Quoting the delimiter for the heredoc (here with 'EOF' but you could also quote only some part of it like in \EOF same as 'E'OF to the same effect) causes expansions inside not to be performed.

For 2, a simpler reproducer would be:

myfunction() {
  local file="$1"
  stat -Lc '$file (%n) is a %F.' -- "$file"
  echo First round:
  cat -- "$file"
  echo Second round:
  cat -- "$file"
  echo Done.
}
myfunction <(echo some text)

(here assuming a GNU or GNU-like implementation of stat).

Which gives:

bash-5.3$ myfunction <(echo some text)
$file (/dev/fd/63) is a fifo.
First round:
some text
Second round:
Done.

<(cmd) expands to a named pipe or something that behaves like a named pipe on systems that have /dev/fd. And you can only read once the contents of a pipe.

In cmd1 <(cmd2), cmd1 and cmd2 run concurrently like in cmd2 | cmd1, the difference being that in cmd2 | cmd1, cmd2's output is readily available on cmd1's fd 0 while in cmd1 <(cmd2), cmd1 needs to open its first argument (the /dev/fd/63 that <(cmd2) expanded to above) to get a fd where to read the output from.

In either case, once cmd2's output has been read, it can't be read again.

With older versions of bash (5.0 or below), you could have gotten away with:

bash-5.0$ myfunction /dev/fd/3 3<< 'EOF'
> echo "$#"
> EOF
$file (/dev/fd/3) is a regular file.
First round:
echo "$#"
Second round:
echo "$#"
Done.

As back then, here-documents were still implemented as deleted regular files as in the original Bourne shell implementation, or as they still are in other shells such as zsh. In newer versions, bash has switched to using pipes when the here-doc is small enough that pipes can be used without deadlocks:

bash-5.3$ myfunction /dev/fd/3 3<< 'EOF'
> echo "$#"
> EOF
$file (/dev/fd/3) is a fifo.
First round:
echo "$#"
Second round:
Done.
bash-5.3$ myfunction /dev/fd/3 3<< EOF | grep -e 9999 -e '[[:alpha:]]'
> $(seq 40000)
> EOF
$file (/dev/fd/3) is a regular file.
First round:
9999
19999
29999
39999
Second round:
9999
19999
29999
39999
Done.

To be able to read the file more than once, you need it to be a regular file, so the options are either to

  • switch to zsh for instance and use that /dev/fd/3 3<< 'EOF' approach

  • switch to zsh and use its =(...) form of process substitution that uses temporary files (and takes care of cleanup itself) instead of fifos (and where cmd2 and cmd1 run sequentially in cmd1 =(cmd2)):

    zsh% myfunction =(<<'EOF'
    cmdsubst¹ heredoc> echo "$#"
    cmdsubst¹ heredoc> EOF
    cmdsubst¹> )
    $file (/tmp/zsh7xSwLQ) is a regular file.
    First round:
    echo "$#"
    Second round:
    echo "$#"
    Done.
    
  • or if you have to use bash, do the temp file handling by hand using mktemp for instance on systems that have one (most these days). That can be done with it deleted in advance like in original here-documents so you don't have to worry about clean-up as much:

    file=$(mktemp) || exit
    <<'EOF' cat > "$file"
    echo "$#"
    EOF
    {
      rm -f -- "$file" && myfunction /dev/fd/3
    } 3< "$file"
    

    Though that approach only works on Linux or Cygwin where /dev/fd/n are magic symlinks to the original files. On other systems, opening /dev/fd/n acts like dup(n) so it does not move back to the beginning of the file each time you open it.

Or in your case, you could use eval instead of source and pass the code to run in memory rather than files/fifos:

myfunction() {
  local code="$1"; shift
  echo First round:
  eval -- "$code"
  echo Second round:
  eval -- "$code"
}

myfunction "$(cat <<'EOF'
echo "I got $# argument${2+s}${1+: $@}"
EOF
)" more args

Note the $(...) command substitution (which needs to be quoted to prevent split+glob (split only in zsh)) instead of <(...)/=(...) process substitution.

Which gives:

First round:
I got 2 arguments: more args
Second round:
I got 2 arguments: more args

¹ note that despite what that cmdsubst may suggest, that's a process substitution here, not command substitution.

1

The section below creates a socket pipe stream descriptor, not a file descriptor:

cat <<EOF
...
EOF

The first time this stream is read, (in the # dry run section), the read pointer of the descriptor reaches the End-of-File.

The # live run section only sees the EOF (i.e. the End-of-File, not the EOF string), and so there is nothing actually run in the # live run section.

A solution I usually use in cases like this is to use mktemp as below:

TS="$(mktemp /tmp/subscript-XXXXXX.sh)"

cat >$TS <__EOF
...
__EOF

bash -x $TS fee fi fo fum

rm -f $TS

Edit: As pointed out by the OP, the usage of mktemp is very flexible, and need not follow my exact example as above. The example was picked from one of my use cases, and not tailored to the OP's scenario.

6
  • 1
    And the resolution: copying the content of the descriptor to a temporary file?
    – Erwann
    Mar 7 at 4:24
  • Yes. That is the resolution.
    – Samveen
    Mar 7 at 4:27
  • The solution you're proposing seems to be at the calling point. Here is one within the function itself: temp=$(mktemp); cat "$file" > "$temp", and replace file by temp in the dry and live runs. This works as expected for A and B. OTOH < $file > $temp does not work, presumably for the reason you pointed out.
    – Erwann
    Mar 7 at 4:42
  • @Erwann Updated to include your point about my example.
    – Samveen
    Mar 7 at 4:52
  • it's a pipe, not a socket, as shown in Stéphane Chazelas's answer.
    – Barmar
    Mar 7 at 15:36
0

The function expects a file, but in the example, it was passed <(cat << EOF ...), in other words a here-doc inside a process substitution. As noted about the latter, "The first time this stream is read, the read pointer of the descriptor reaches the End-of-File." Accordingly, to accommodate such file objects, the resolution is to copy the content thereof, to a temporary file, and use it thereafter:

script.sh:

function baz() {
    local file="$1"; shift

    temp=$(mktemp)
    trap '{ rm -f "$temp"; }' EXIT ERR
    cat "$file" > "$temp"

    # dry run
    local err=$(source "$temp" "$@" 2>&1 1>/dev/null)
    [[ -z $err ]] || { echo "Exiting"; return 1; }

    # live run
    local output=$(source "$temp" "$@")
    [[ -z $output ]] || echo "$output"
}

baz <(cat << 'EOF'
# echo "I was given $# argument(s):"  # A
eecho "I was given $# argument(s):" # B
printf "%s " "$@"
EOF
) 'foo' 'bar'

Under A:

$ source script.sh 'foo' 'bar'
I was given 2 argument(s):
foo bar 

Under B:

$ source script.sh 'foo' 'bar´
Exiting

Extra:

  • As noted, "without quotes around [the lead] EOF the positional parameters are expanded in the here document at the time that here document is parsed." Corrected accordingly.
1
  • 3
    Not related to the problem you're having, but you should note that since you haven't escaped $# or $@ in the heredoc and as the heredoc marker isn't quoted either, $# and $@ are substituted in the heredoc itself (if you do cat "$temp" you'll see it already has 2 and foo bar in place of $# and $@). Passing 'foo' 'bar' to baz or using "$@ in source ... "$@" isn't doing anything useful given the code in the question or here.
    – muru
    Mar 7 at 5:19

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .