2

I need to pass an array of filenames to a command, preserving proper quoting. So far, so good. Unfortunately the command is actually a sub-command that is, in turn, invoked by another command. Concretely, the command is:

git filter-branch --index-filter \
    'git rm -rf --cached ‹file1› ‹file2›…' \
    HEAD

For simplicity, I’m going to replace this in the following by a simpler command that exhibits the same problem:

printf '%s\n' 'cmd file1 file2…'

Now I’ve got an array files=('a b' c). My desired result is that the above command prints in a single line, and individually quotes every token after cmd as necessary (e.g. when there’s a space).

It works if I manually expand and quote the file names:

$ printf '%s\n' 'cmd '\''a b'\'' c'
→ cmd 'a b' c

(Alternatively I could mix single and double quotes to achieve the same result.)

But it no longer works if I am trying to pass an array:

  1. $ (set -x; printf '%s\n' "cmd '${files[@]}'")
    + printf '%s\n' 'cmd '\''a b' 'c'\'''
    → cmd 'a b
    c'
    
  2. $ (set -x; printf '%s\n' 'cmd '\'"${files[@]}"\')
    + printf '%s\n' 'cmd '\''a b' 'c'\'''
    → cmd 'a b
    c'
    
  3. $ (set -x; printf '%s\n' 'cmd '"${files[@]}")
    + printf '%s\n' 'cmd a b' c
    → cmd a b
    c
    

I’m not surprised (3) doesn’t work (and it’s only included for completeness). Based on the output of set -x, the shell correctly quotes the individual array elements in (1) and (2) and it even puts escaped quotes around the whole thing. But then it breaks apart the individually quoted items. Is there a way to prevent this?


Incidentally, Shellcheck (SC2145) suggests replacing the [@] part by [*] in the above. This obviously breaks for filenames with spaces.

  • Defining gitCmd=( git filter-branch --index-filter 'git rm -rf --cached [MAGIC]' HEAD ) and doing an array expansion "${gitCmd[@]}" doesn't work? It preserves the quoted expressions as defined. You can see the part within the quotes is not broken and preserved – Inian May 9 at 16:30
3
  1. Instead of an array, use set -- file1 file2 ... to fill the parameter list, then use bash parameter transformation with the Quote operator:

    set -- 'a "b' c "d 'e" "f 'g "'"h' ; (set -x; printf 'cmd %s\n' "${*@Q}")
    

    Output:

    + printf 'cmd %s\n' ''\''a "b'\'' '\''c'\'' '\''d '\''\'\'''\''e'\'' '\''f '\''\'\'''\''g "h'\'''
    cmd 'a "b' 'c' 'd '\''e' 'f '\''g "h'
    

    Or, if we remove the set -x; part, the output becomes:

    cmd 'a "b' 'c' 'd '\''e' 'f '\''g "h'
    
  2. A comment from LL3 suggests a better way that doesn't require set -- ...:

    export x; n=(a "b 'c"); x="${n[@]@Q}"
    ( n=($x); printf 'cmd %s\n' "${n[*]}"; )
    

    Output:

    cmd 'a' 'b '\''c'
    
  3. Yet another method is to use bash parameter transformation with the Assignment operator, (which also needs an eval):

    export x;n=(a b 'c d');x="${n[@]@A}"; (eval "$x";printf '%s\n' "${n[@]}")
    

    Output showing what printf sees:

    a
    b
    c d
    
  • 1
    Uh that's cool! I didn't know of that parameter expansion. I think you could then use it from the array itself and expand it directly on the command ? Something like cmd "${files[*]@Q}" (thus without passing through set -- etc.) – LL3 May 10 at 0:53
  • @LL3 That's a good idea. See revised answer. – agc May 10 at 1:13
  • My real use case is a parameter array rather than a regular array anyway, so I wouldn’t need set -- either way, but having these solutions is brilliant. – Konrad Rudolph May 10 at 8:22
  • 1
    @KonradRudolph, As a diagnostic please try: set -- a b 'c d' ; (echo ${@@Q}); it should print 'a' 'b' 'c d'. – agc May 10 at 17:12
  • 1
    "Instead of an array, (which can't be passed to a subshell)" -- Of course you can pass an array to a subshell, something like a=('a b' c); (printf "cmd "; printf "%s " "${a[@]@Q}"; printf "\n") works just fine and prints cmd 'a b' 'c' , which seems to be what is required. You can't pass an array through the environment to another process, though. But I'm not sure you need to do that here, whatever the argument to --index-filter is going to be, it can be made a single string in the main shell. – ilkkachu May 10 at 21:41
3

git filter-branch runs /bin/sh /usr/lib/git-core/git-filter-branch and that script evaluates the argument of --index-filter using eval.

So that argument is evaluated as /bin/sh code.

On most systems, /bin/sh will be more or less an interpreter of the POSIX sh language, though in a few like Solaris 10 and older, it could still be the ancient Bourne sh language instead.

When it comes to quoting syntax, it makes little difference though.

In any case, none of the ksh/bash/zsh extended quoting operators like $'...' can be used. What that means is that you can't use GNU/bash/zsh/ksh printf %q or mksh/bash ${var@Q} operator, or the xtrace tracing to generate the quoting as those resort to $'...' in some cases. They also use some forms of quoting that are not localisation-safe (like \).

One builtin quoting operator you could use is zsh's qq parameter expansion flag as it uses single quotes:

files=(foo 'a b c' $'a\nb\nc' --foo-- "a'b")
git filter-branch --index-filter "git rm -rf --cached -- ${${(@qq)files}}" HEAD

To see how zsh quotes those:

$ printf '<%s>\n' "${${(@qq)files}}"
<'foo' 'a b c' 'a
b
c' '--foo--' 'a'\''b'>

With bash/ksh/yash/zsh, you could do that same quoting using a function like:

shquote() {
  LC_ALL=C awk -v q=\' '
    BEGIN{
      for (i=1; i<ARGC; i++) {
        gsub(q, q "\\" q q, ARGV[i])
        printf "%s ", q ARGV[i] q
      }
      print ""
    }' "$@"
}

And then:

git filter-branch --index-filter "git rm -rf --cached -- $(shquote "${files[@]}")" HEAD
  • Couldn't the same result be obtained with a LC_ALL=C printf -v var "'%s' " "${files[@]//\'/\'\\\'\'}" ? It would then be used as git filter-branch --index-filter "git rm -rf --cached -- ${var}" HEAD. It seems to be doing the same quoting as your Awk script. Or what am I missing ? – LL3 Jul 7 at 1:58
0
$ foo=(1 2 '3 4' 4 5)
$ printf "'%s'\n" "${foo[@]}"
'1'
'2'
'3 4'
'4'
'5'
$ subcommand() { printf "'%s'\n" "$@"; }
$ subcommand "${foo[@]}"
'1'
'2'
'3 4'
'4'
'5'

So let's adapt this to your specific use-case:

git filter-branch --index-filter \
    'git rm -rf --cached file1 file2 […]' \
    HEAD

In your case we need to be a little more creative though, and break things up into smaller pieces.

git filter-branch --index-filter  \
    'git rm -rf --cached [MAGIC]' \
    HEAD

The file list we're creating is where you need the "magic" to happen. The rest is all static, yes? And since you're scripting this, you don't need it to be on three lines, which simplifies things:

git filter-branch --index-filter 'git rm -rf --cached [MAGIC]' HEAD

So:

prefix="git filter-branch --index-filter 'git rm -rf --cached "
postfix="' HEAD"
magic="$(printf '"%s" ' "${file[@]}"'

And then if we execute:

${prefix}${magic}${postfix}

Thereby, we have assembled your command - albeit using "s to enclose your filenames rather than 's, since the filter-branch command is in 's already.

  • Unfortunately this doesn’t help me … In the printf code, I need all items in one line, properly quoted. Consider my actual use-case: your command would break the git filter-branch call. – Konrad Rudolph May 9 at 16:05
  • So don't use \n, that was just for demonstrative purposes. e. g. printf "'%s' " "$@". – DopeGhoti May 9 at 16:05
  • Again, consider my actual use-case. I’m using \n in the printf example because this emulates what the git filter-branch command is doing. The issue isn’t the newline, it’s that shell quoting doesn’t work inside a string. – Konrad Rudolph May 9 at 16:06
  • So considered; see expanded discussion. – DopeGhoti May 9 at 16:18
  • This is a nice enough hack and I think it does work with spaces — but unlike normal shell quoting it’s not generally valid. For instance, it fails if a filename contains a double-quote (crazy, but valid). I don’t think this issue can be worked around by using printf. – Konrad Rudolph May 9 at 16:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.