What are the ideas behind the absence of quote removal and escaping in variable expansion?

Question

Commands can be stored in variables and executed (though not a good practice) in shell like:

command='ls -l A* "B\" type"'
$command

It lists files beginning with A, "B\" and type". Argument separating and globbing are performed, but no quote removal and escaping. This behavior makes passing arbitrary arguments using one variable very difficult in shells not supporting arrays, and impossible to combine find and other commands with for safely (which is often discussed). It even limits the usage of globbing as a lot of characters are uncontrolled in unquoted variable expansion (you cannot store globbing sequences containing `'"*?\n literals and reuse them properly).

The situation would be very different if quotes and escape sequences in variables can be processed. But why most shells don't do that in fact? Is it specially designed with some obscure considerations I didn't notice, or just simply passed down to keep compatibility? I know there are similar questions like Why does bash variable expansion retain quotes? and Quoting / escaping / expansion issue in "command in a variable" discussing the behavior, but the answers there didn't talk about reasons.

Answer 1

This behavior makes passing arbitrary arguments using one variable very difficult [...]

Perhaps. But having the results of expansions go through all the usual command line processing would make it impossible to pass even one arbitrary argument intact.

Consider e.g. a script that gets a filename from somewhere, and tries to pass it to a command. Let's say we get the filename with read:

echo -n "please enter filename: "
read -r filename
some command "$filename"

Now, if the user enters a filename like don't stop me now.txt, running some command will crash with a syntax error due to the single quote.

Similarly if the script is run e.g. as myscript don*.txt and gets the filename from a command line argument:

filename=$1
some command "$filename"

Again, $filename (or $1 already) would contain that single quote.

Worse, the filename or user input string could contain a command substitution, making it possible for merely using the variable to run arbitrary commands. The script writer would have to go through hoops to add escapes to every string read from outside the script, and some ways of doing that might already trigger the expansion processing. Plus people just wouldn't do that, and the shell would be even more unsafe a tool to use.

(For what you want it wouldn't be necessary to process expansions, just quotes and backslashes, but the issues with unpaired quotes would still be there.)

Of course, you could also say that read should just add the necessary escapes, but would they need to be added for all other sorts of input too? How would string operations work, would they need to process the quotes too? Even something as simple as ${#var} for the length of a variable would turn much more expensive to implement. And what would the length even mean for a variable that contained multiple distinct quoted strings?

In the end, it's best to consider the code of the script distinct from the data the script processes, and to organize it so that they don't get mixed up, so that the data only gets processed in ways explicitly set in the code. That's what the shell pretty much does, if you remember to quote the variable expansions.

Using data in variables as-is is what every other programming language also does. E.g. in this C snippet, the string that gets printed is "foo bar", with the quotes, they're not parsed by the runtime environment:

char *s = "\"foo bar\"";
printf("%s\n", s);

Similarly, if it was s = "foo()" instead, that printf() call would not call the function foo(), but would just print the string foo(). (If you want to object about interpreted vs. compiled languages, we could change the example to Perl or Python.)

---

Now, that's just an argument as to why what you suggest doesn't seem a good idea, to me, in 2022. But really, you asked about the "why" and the design rationale. Those didn't happen in 2022, but in the 1970's and 1980's or so. Wikipedia mentions the initial release of the Bourne Shell as having happened in 1979. That's a long time ago, and the existing history of computing was a lot shorter back then than it is now. We now have the some benefit of hindsight, which probably has helped in the creation of other tools, like those shell arrays. Faster computers and more memory too.

I wouldn't reject the idea that the actual explanation behind the design might be something like "that's what they came up with when they were first figuring all this out, and for some reason it stuck". Backward compatibility goes two ways. At least now you have those shells with arrays, and completely different shells too.

Answer 2

Because quote removal works only on quotes that are in the shell syntax, it is, in princple a compilable feature. So that is to say, there doesn't actually have to be any run-time quote removal going on in the shell which takes place after substitutions. It's an abstraction which is explained that way, but in fact it is possible for the quotes to be removed when the syntax is parsed.

A command line component like "foo $bar" can be turned into a quoted unit. The shell's parser remembers that this was quoted, but doesn't remember the actual quotes; only the foo $bar interior. When that item is processed at run time, then $bar is interpolated verbatim and that's it. Whereas a command line item like abc$bar can become an unquoted unit, whose run-time semantics is to interpolate $bar, then split into fields, and perform pathname expansions.

Under this model, if the processing of quotes were done on the contents of variables, it means that the shell has to do lexical scanning and parsing activities at run time too.

It's basically the same reason why a syntactic keyword cannot come from a variable, like:

thenvar=then
fivar=fi

# nonsense
if command; $thenvar
  echo command succeeded
$fivar

Why can't the shell recognize the content of $thenvar as the then keyword?

For exactly the same reason why it won't recognize a quote stored in a variable as a syntactic quote.

Now the shell does "mix levels" between run time and parse time to some extent. If an expansion is unquoted, then it is split into fields. Moreover, if the expansion contains globbing characters, then those are active.

Arguably, those features are also syntax, yet are allowed to dynamically come from the data, too.

Alas, those very features and their level mixing cause confusion and bugs in shell programming. Forget to quote and the *, ? or spaces that were supposed to be a verbatim part of the data get mangled.

Processing quotes that come from the data would cause more confusion and errors. For instance, if a variable contained an unbalanced quote, there would be a syntax error, right? Wait, what? Syntax error caused by run-time data? Or, would you allow a quote contained in a variable to balance one in the syntax? Would this be valid:

quote='"'
echo "foo bar$quote   # does $quote close the open quote?

You see it can get absurd pretty quickly.