0

I have two arrays

$ arr1=( 1 2 2 3)
$ arr2=( 2 3 3 4)

Why does this generate strange output when I follow the good practice of double quoting an array expansion

$ tsort << EOF
> "${arr1[@]}" "${arr2[@]}"
> EOF
"1
"2
2
3
3"
4"

whereas this one generates correct results, when I do not follow the good practice? What shall I do? Thanks.

$ tsort << EOF
${arr1[@]} ${arr2[@]} 
EOF

1
2
3
4

I think word splitting is still performed in here document:

$ wc -l << EOF
a    
b  
1
EOF
3
  • 1
    Where is word splitting in your example? wc receives a single stream and counts newlines in it. – Kamil Maciorowski Nov 18 '18 at 2:52
4

From the bash manual:

[...] all lines of the here-document are subjected to parameter expansion, command substitution, and arithmetic expansion, the character sequence \newline is ignored, and ‘\’ must be used to quote the characters ‘\’, ‘$’, and ‘`’.

It doesn't say anything about removing quotes, or field splitting or filename expansion, so the quotes in a heredoc are taken literally, and don't perform their usual functions.

  • Thanks. "It doesn't say anything about removing quotes, or field splitting or filename expansion, so... don't perform their usual functions." I think word splitting is still performed. Otherwise, how the command receives multiple words? – Tim Nov 18 '18 at 2:45
  • 1
    Who said the command receives multiple words? How do you know the command didn't decide to parse the input into words? – Olorin Nov 18 '18 at 2:46
  • See my update. If word splitting is unmentioned but still performed, why can't quote removal? – Tim Nov 18 '18 at 2:48
  • How does your update prove word splitting is performed? – Olorin Nov 18 '18 at 2:51
2

The other answer is basically "because the manual says so", but I think there is some rationale behind this behavior.

This type of redirection instructs the shell to read input from the current source until a line containing only word (with no trailing blanks) is seen. All of the lines read up to that point are then used as the standard input […] for a command.

[…]

If any part of word is quoted, the delimiter is the result of quote removal on word, and the lines in the here-document are not expanded. If word is unquoted, all lines of the here-document are subjected to parameter expansion, command substitution, and arithmetic expansion […].

(emphasis mine).

Consider main reasons to quote something in a command line:

  1. to tell the shell how to split the line and create a list of arguments (two arguments a b vs. one argument "a b");
  2. to tell the shell if expansions should be performed ('' vs. "" or unquoted).

The first reason doesn't apply while creating a standard input because standard input is a stream and there is no concept of number of arguments in it. The second reason could apply but the design choice to control expansions solely by quoting or not quoting word ("EOF" vs. EOF) invalidates it.

What shall I do?

You shall remember with here document you create a stream. The reasons to use quotes that wouldn't appear in it have been invalidated. In effect the premise is every quote you use should appear in the stream.

If this worked more as you expected then unescaped quotes would disappear without making any difference. I guess here document would then support \" and \' to denote quotes that should survive. But these would be the only quotes you would really need to use. The current situation simplifies things.

  • 1
    If the implementations had elected to perform all expansions and quote removal, somebody would have made a rationalisation for that too. – Olorin Nov 18 '18 at 2:40
1

Let's look at what actually happens here by replacing tsort with cat:

$ arr1=( 1 2 2 3 )
$ arr2=( 2 3 3 4 )
$ cat <<END
> "${arr1[@]}" "${arr2[@]}"
> END
"1 2 2 3" "2 3 3 4"

As you can see, the here-document is nothing but a text string with the values of the arrays expanded inside it. The double quotes come from the double quotes in the document itself (the shell only cares about the ${...} bit and won't touch the quote characters).

The output when deleting the double quotes would be the same, but without the double quotes,

1 2 2 3 2 3 3 4

This would be interpreted by tsort as the pairs

1 2   <-- first two numbers from arr1
2 3   <-- last two numbers from arr1
2 3   <-- first two numbers from arr2
3 4   <-- last two numbers from arr2

It's unfortunate that you chose this particular example, as this happens to be exactly the same as

1 2   <-- first number from arr1 and arr2
2 3   <-- second number from arr1 and arr2
2 3   <-- etc.
3 4

i.e., the entries of each array running down in two columns (one column per array).

To generate this second list (correctly), you can't really use a here-document. Instead you could use a shell loop:

for (( i=0; i<${#arr1[@]}; ++i)); do
    printf '%d %d\n' "${arr1[i]}" "${arr2[i]}"
done | tsort

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.