20

In this question someone reports a problem using a here document with a quoted delimiter word inside $(...) command substitution, where a backslash \ at the end of a line inside the document triggers newline-joining line continuation, while the same here document outside command substitution works as expected.

Here is a simplified example document:

cat <<'EOT'
abc ` def
ghi \
jkl
EOT

This includes one backtick and one backslash at the end of a line. The delimiter is quoted, so no expansions occur inside the body. In all Bourne-alikes I can find this outputs the contents verbatim. If I put the same document inside a command substitution as follows:

x=$(cat <<'EOT'
abc ` def
ghi \
jkl
EOT
)
echo "$x"

then they no longer behave identically:

  • dash, ash, zsh, ksh93, BusyBox ash, mksh, and SunOS 5.10 POSIX sh all give the verbatim contents of the document, as before.
  • Bash 3.2 gives a syntax error for an unmatched backtick. With matched backticks, it attempts to run the contents as a command.
  • Bash 4.3 collapses "ghi" and "jkl" onto a single line, but has no error. The --posix option does not affect this. Kusalananda tells me (thanks!) that pdksh behaves the same way.

In the original question, I said this was a bug in Bash's parser. Is it? [Update: yes] The relevant text from POSIX (all from the Shell Command Language definition) that I can find is:

  • §2.6.3 Command Substitution:

    With the $(command) form, all characters following the open parenthesis to the matching closing parenthesis constitute the command. Any valid shell script can be used for command, except a script consisting solely of redirections which produces unspecified results.

  • §2.7.4 Here-Document:

    If any part of word is quoted, the delimiter shall be formed by performing quote removal on word, and the here-document lines shall not be expanded.

  • §2.2.1 Escape Character (Backslash):

    If a <newline> follows the <backslash>, the shell shall interpret this as line continuation. The <backslash> and <newline> shall be removed before splitting the input into tokens.

  • §2.3 Token Recognition:

    When an io_here token has been recognized by the grammar (see Shell Grammar), one or more of the subsequent lines immediately following the next NEWLINE token form the body of one or more here-documents and shall be parsed according to the rules of Here-Document.

    When it is not processing an io_here, the shell shall break its input into tokens by applying the first applicable rule below to the next character in its input. ...

    ...

    1. If the current character is <backslash>, single-quote, or double-quote and it is not quoted, it shall affect quoting for subsequent characters up to the end of the quoted text. The rules for quoting are as described in Quoting . During token recognition no substitutions shall be actually performed, and the result token shall contain exactly the characters that appear in the input (except for <newline> joining), unmodified, including any embedded or enclosing quotes or substitution operators, between the and the end of the quoted text.

My interpretation of this is that all characters after $( until the terminating ) comprise the shell script, verbatim; a here document appears, so here-document processing occurs instead of ordinary tokenisation; the here document then has a quoted delimiter, meaning that its contents is processed verbatim; and the escape character never comes into it. I can see an argument, however, that this case is simply not addressed, and both behaviours are permissible. It's possible that I've skipped over some relevant text somewhere, too.


  • Is this situation made clearer elsewhere?
  • What should a portable script be able to rely on (in theory)?
  • Is the specific treatment given by of any of these shells (Bash 3.2/Bash 4.3/everyone else) mandated by the standard? Forbidden? Permitted?
  • Can you show us how you produce your output in the second case? – Julie Pelletier Jan 29 '17 at 6:07
  • @JuliePelletier echo "$x", but any way of inspecting the variable works. I've edited that line into the bottom. – Michael Homer Jan 29 '17 at 6:13
  • 2
    Looks like it's an easy fix. This patch seem to work at least: ignore_quoted_newline_in_quoted_heredoc.patch – geirha Jan 31 '17 at 8:04
  • 1
    I think you are interpreting this correctly and imo the standard is pretty clear since "The shell shall expand the command substitution by executing command in a subshell environment [...] and replacing the command substitution [...] with the standard output of the command [...]" So it runs the command in a subshell and replaces $(...) with whatever that output is... Now, when running the command in your example in a subshell (in bash) it does output the expected result. It's only when turning it into command substitution that it collapses "ghi" and "jkl". So this is a bug imo – don_crissti Feb 3 '17 at 0:30
  • 2
    @geirha I reported a Bash bug; I'm not going to bother about pdksh since it doesn't seem to have even a shadow of current maintenance. – Michael Homer Feb 11 '17 at 6:23
5

This was asked on Bash's mailing list, and the maintainer confirmed it was a bug

They also mentioned that the text in POSIX "is not necessarily ambiguous, but it does require close reading.", so I asked for a clarification on that. Their answer including a description of the issue and interpretation of the standard was as follows:

The command substitution is a red herring; it's relevant only in that it pointed out where the bug was.

The delimiter to the here-document is quoted, so the lines are not expanded. In this case, the shell reads lines from the input as if they were quoted. If a backslash appears in a context where it is quoted, it does not act as an escape character (see below), and the special handling of backslash-newline does not take place. In fact, if any part of the delimiter is quoted, the here-document lines are read as if single-quoted.

The text in Posix 2.2.1 is written awkwardly, but means that the backslash is only treated specially when it's not quoted. You can quote a backslash and inhibit all all expansion only with single quotes or another backslash.

The close reading part is the "not expanded" text implying the single quotes. The standard says in 2.2 that here documents are "another form of quoting," but the only form of quoting in which words are not expanded at all is single quotes. So it's a form of quoting that is just about exactly like single quotes, but not single quotes.

  • @Scott (1) I believe this answers all questions and nothing is superfluous. My comment that starts off the answer is regarding a deletion made by a moderator who misunderstood the situation. (2) I do not have enough reputation. (3) I would have appreciated similar behavior by those deleting my answers, but I will certainly keep that in mind in the future. Thanks for the thoughts. – Kevin Feb 25 '17 at 2:03
  • My point was that most of your first paragraph is a conversation with Michael Mrozek and not an answer to the question.  I realize that you don’t have enough reputation to comment on any post, but I believe that you have enough for meta and chat. – Scott Feb 25 '17 at 2:13
  • 1
    @Scott I understand and appreciate that you're trying to streamline the answer, but I posted that exactly streamlined answer previously (just the quote and a link to it), and it was deleted by said moderator (without any discussion!) and I see no links in the deleted post to chat and dispute that decision. I hoped that by answering his unfounded criticism, it would survive deletion, be accepted by the asker and then I would modify the answer to remove the preamble. – Kevin Feb 25 '17 at 2:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.