5

Edit to clarify my question:

POSIX says:

If a <newline> follows the (unquoted) <backslash>, the shell shall interpret this as line continuation. The <backslash> and <newline> shall be removed before splitting the input into tokens.

However, dash or other implementations, tokenize input at first. As a result, \<newline> is not recognized but # this is a comment \ is discarded. Is this behavior POSIX compliant? Again, POSIX says that line continuation shall be removed before tokenizing.

Isn't the following procedure really POSIX compliant?

  1. read the whole input: "echo hello ... \<newline> ... bye"
  2. search for unquoted \<newline> and remove them: "echo hello ... bye"
  3. tokenize: "echo"(discard ' ')"hello"(discard ' ')(discard "# ... bye")

On Ubuntu with dash-0.5.10.2-6 sh (dash) we get the following

$ cat /var/tmp/test.sh
echo hello # this is a comment \
echo bye

$ sh /var/tmp/test.sh
hello
bye

This is because everything after # is treated as a comment, and everything up to \ is discarded, so line continuation of \<newline> does not work.

However, POSIX "Escape Character (Backslash)" section states

The <backslash> and <newline> shall be removed before splitting the input into tokens.

And since comment processing of # is done in tokenization,

echo hello # this is a comment \
echo bye

should be equivalent to

echo hello # this is a comment echo bye

Does this mean that sh is not POSIX compliant? Or is there some rationale for comment taking precedence over line continuation in this situation?

Improve this question
2
  • The spec says "The <backslash> and <newline> shall be removed before splitting the input into tokens. Since the escaped <newline> is removed entirely from the input and is not replaced by any white space, it cannot serve as a token separator." and I wonder if the second sentence is the clue here, i.e. that the idea is that the removed newline doesn't split a token. (i.e. echo foo<backslash><newline>bar prints foobar, not foo bar)
    – ilkkachu
    May 20, 2022 at 8:30
  • "Does this mean that sh is not POSIX compliant?" -- which sh? It works like that in every shell I tried, Bash, Dash, Busybox, zsh, kshs, yash, posh. If all shells do A, and the standard says B, then it's the standard that's wrong... If you're looking for rationale, be prepared to accept that the rationale might be "it was done like this the first time, 40 years ago".
    – ilkkachu
    May 20, 2022 at 8:31

3 Answers 3

Reset to default
7

The shell's input is scanned character by character to divide it into tokens, as described in the section on Token Recognition.

[...] the shell shall break its input into tokens by applying the first applicable rule below to the next character in its input.

Quoting is handled as part of the token recognition process, but given the example in the question, the shell will encounter the # before the quoted newline.

When the shell arrives at an unquoted comment character during its scanning of the input line, the rest of the line, including the final backslash, is discarded as a comment:

If the current character is a #, it and all subsequent characters up to, but excluding, the next <newline> shall be discarded as a comment. The <newline> that ends the line is not considered part of the comment.

The part of the standard that you quote, the Quoting section, says that when encountering a newline preceded by a backslash...

A <backslash> that is not quoted shall preserve the literal value of the following character, with the exception of a <newline>. If a <newline> follows the <backslash>, the shell shall interpret this as line continuation. The <backslash> and <newline> shall be removed before splitting the input into tokens. [...]

Note that this does not come into effect until the scanner actually encounters an unquoted backslash, which is handled by the token recognition process:

If the current character is <backslash>, single-quote, or double-quote and it is not quoted, it shall affect quoting for subsequent characters up to the end of the quoted text. The rules for quoting are as described in "Quoting".

As already mentioned in this answer, the scanner will encounter the comment character first, before seeing the backslash, which will trigger the token recognition rule that handles the rest of the line, including any quoting characters, as a comment. Therefore, the quoting of the newline at the end of the line will never come into effect.

Improve this answer
2
  • I edited my question. Please take a look.
    – quasar
    May 20, 2022 at 7:32
  • @quasar See added bit.
    – Kusalananda ♦
    May 20, 2022 at 7:56
6

Interesting thought. It seems you think of backslash escaping as a step applied to the whole input prior to any token recognition, which is not the case.

Input is still evaluated strictly from left to right, which is neccessary in different situations like quoting:

echo 'foo\
bar'

is another case where the cited rule does not apply because of the context. In this case the context is a quoted string, in your case it was a commentary according to

If the current character is a '#', it and all subsequent characters up to, but excluding, the next shall be discarded as a comment.

The reason for the cited sentence is the fact that you can do

ec\
ho foo

So you go from left to right, token by token. If you are still in the normal context, the backslash + newline is removed prior to token splitting, evaluation is simple echo.

The bahaviour is not only dash, but any POSIX shell.

Improve this answer
3
  • I edited my question. Please take a look.
    – quasar
    May 20, 2022 at 7:32
  • I see, but your update is already covered by my answer: Escaping is not performed for the whole input before any token splitting, but only for the next token before splitting.
    – Philippos
    May 20, 2022 at 7:58
  • Your answer is also helpful, sorry not to accept your answer.
    – quasar
    May 20, 2022 at 10:08
-1

Thanks for the answers.

I have understood that we know whether \ is quoted or not in tokenizing, so we can't remove unquoted \<newline> before tokenizing.

The <backslash> and <newline> shall be removed before splitting the input into tokens.

This statement simply says that, unlike other quotings, \<newline> is immediately discarded. I have to think that way.

Improve this answer
3
  • You need to let go of this notion that there's some objectively-best-in-all-circumstances boundary between tokenization and readin. The shell applies quoting rules in its tokenization, and escapes are treated as ordinary quotes that only affect the immediately-following character when encountered. But it also applies comment recognition in its tokenization, and quoting does not apply in comments. So backslashes have no special meaning in comments.
    – jthill
    May 21, 2022 at 0:10
  • If "the <backslash> and <newline> shall be removed before splitting the input into tokens" means that unquoted \<newline> must be removed before tokenizing the input, we need to know whether any <backslash> is quoted or not before tokenization. But this is not the case; quoting recognition is a part of tokenization. The statement cited previously means that unquoted \<newline> must be removed immediately, unlike other quotings which keeps both quoting and quoted characters unmodified. Am I wrong?
    – quasar
    May 21, 2022 at 3:17
  • Yes. Quote markers alter the tokenization of affected characters. So does a comment marker. Inside any of the four (three different kinds of quoting, one comment) tokenization changes and the other three kinds have their own context-dependent rules. Inside single quotes and betwen a comment marker and the following newline nothing else is treated specially.
    – jthill
    May 21, 2022 at 4:20

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .