bash quote escaping change in 4.4

Question

I am updating bash on our embedded platform from 4.1.9 to the latest (4.4.12), and I am seeing a behaviour change in this simple scenario of passing escaped arguments into a script.

Script /tmp/printarg:

#! /bin/sh
echo "ARG |$*|"

And I invoke the script like this:

bash -c "/tmp/printarg \\"abc\\""

I've tried this on several platforms (native x86_64 Linux) running bash 4.3.42, as well as several embedded platforms (ARM and PPC) running bash 4.1.9 and 4.2.37, and all of these platforms report what I would expect:

38$ bash -c "/tmp/printarg \\"abc\\""
ARG |abc|

But, when I run this using bash 4.4.12 (native X86 or embedded platforms), I get this:

$ bash -c "/tmp/printarg \\"abc\\""
ARG |abc\|            <<< trailing backslash

And if I add a space in the command line between the second escaped quote and the ending quote, then I no longer see the extra backslash:

$ bash -c "/tmp/printarg \\"abc\\" "
ARG |abc |            <<< trailing space, but backslash is gone

This feels like a regression. Any thoughts? I also did try enabling the various compat options (compat40, compat41, compat42, compat43) with change.

Answer 1

bash -c "/tmp/printargs \\"abc\\""

Does not escape what you think it does. A backslash-backslash is an escaped backslash, handled by the calling shell — so that is the same as running:

/tmp/printargs \abc\

because the double-quotes are not escaped. You could have just written:

bash -c '/tmp/printargs \abc\'

I'm guessing you actually wanted:

bash -c "/tmp/printargs \"abc\""

which escapes the double quotes, passing a quoted "abc" to the bash -c.

(I'm guessing the different behavior you're seeing is different versions of bash handling the escaped nothing at end of input differently.)

Perl version of printargs (slightly improved behavior):

#!/usr/bin/perl
use feature qw(say);

for (my $i = 0; $i < @ARGV; ++$i) {
        say "$i: |$ARGV[$i]|";
}

Answer 2

bash -c "/tmp/printargs \\"abc\\""

Are you sure this is what you want to do? If you run that with set -x in effect, you'll see that the command that runs, is

bash -c '/tmp/printargs \abc\'

i.e. you're passing the shell a string that ends in a backslash. Your first quoted string contains an escaped backslash, then you have an unquoted abc, an escaped backslash, and then an empty quoted string. (Note how the syntax highlighting done by Stackexchange shows the abc is not quoted.)

The unquoted backslash at the end of input doesn't make much sense. A backslash either escapes the following character, or starts a continuation line, where it's deleted along with the following newline, like here:

$ bash -c $'echo "foo\\\nbar"'                                                                                                              
foobar 

This case has neither. You're possibly trying to do either of these:

bash -c "/tmp/printargs \"abc\""
bash -c '/tmp/printargs "abc"'

Both of which produce the output ARG |abc|.

---

We can see the difference between shells with a bit simpler test:

$ bash -c 'echo $BASH_VERSION; echo abc\'
4.4.12(1)-release
abc\

$ ./bash -c 'echo $BASH_VERSION; echo abc\'
4.3.30(1)-release
abc

$ dpkg -l dash |grep ^i
ii  dash           0.5.8-2.4    amd64        POSIX-compliant shell
$ dash -c 'echo abc\'
abc\

$ dpkg -l zsh |grep ^i
ii  zsh            5.3.1-4+b2   amd64        shell with lots of features
$ zsh -c 'echo abc\'
abc

---

If I had to guess, I'd start looking for the source of the change in this change:

This document details the changes between this version, bash-4.4-alpha, and
the previous version, bash-4.3-release.

1.  Changes to Bash

cccc. Fixed a bug that resulted in short-circuited evaluation when reading
      commands from a string ending in an unquoted backslash, or when sourcing
      a file that ends with an unquoted backslash.

Answer 3

I'll explain this one:

$ bash -c "/tmp/quotefail \\"abc\\" "
ARG |abc |            <<< trailing space, but backquote is gone

As @ilkkachu explained, with set -x we see how this is interpreted:

+ bash -c '/tmp/quotefail \abc\ '

Of course, "\a" is just "a", and "\ " is just " ", so the parameter /tmp/quotefail receives is "abc ", and the result:

ARG |abc |

In the first test, the backslash wasn't followed by anything, so it remained a backslash.