0

Edit:

Important note before reading further: since Kusalananda's anwser, my question is now quite useless, since he pointed out a bug in another script I used. Now this bug is fixed, the below errors I described no longer occur!

There are two issues in your code which causes the value in $extension to be expanded
The second issue is the function itself


TL;DR

I was stuck when I started writing the question, but found a solution that works (using shell noglob), BUT I still have a question of "would it be possible to write".

I'm sure this has already been answered, but I get lost in hundreds of topics about bash expansion.

Here is my case:

  • I want to run a script (lets say ffmpeg with option -pattern_type glob -i mode that accepts a regexp like *.JPG)
  • taking an argument (lets say *.JPG)
  • argument coming from a variable (lets say $extension)
  • variable coming from script parameters (lets say $1)
  • but I do not want shell to expand $extension as by default for a pattern like *.JPG

Note that all this actions take place in a script script.sh.

Thus, the user could run:

script.sh 'JPG' ; # where $1='JPG'

Note: in script.sh, I concatenate *. and $1 to get a more final regexp like *.JPG

I read (from here https://stackoverflow.com/a/11456496/912046) that shell expansion occurs before ffmpeg receives arguments (ffmpeg even does not know Shell expansion took place)

I tried to play with quotes, double quotes, or backslashes around ${extension} in different ways, unsuccessfully.
Here are some tries:

ffmpeg ... "${extension}"

leads to: ffmpeg 1.jpg 2.jpg [...] (expansion occurs)

ffmpeg ... ${extension}

same

ffmpeg ... '${extension}'

leads to no file match because of searching pattern ${extension} (variable name is used literally)

extension="\'${extension}\'" ; # wrapping the single quotes (preventing expansion) directly in variable
ffmpeg ... ${extension}

leads to no file match because of searching pattern '*.jpg' (including single quotes)

ffmpeg ... \"${extension}\"

same but searching for "*.jpg" (including double quotes)

I finally succeeded using Shell noglob option.
Here is the command for curious:

set -f ; # disable glob (prevent expansion on '*.jpg')
ffmpeg -pattern_type glob -i ${extension} movie.mp4 ;
set +f ; # restore, but what if glob was already disabled? I should not restore it then?

But for my curiosity, is there another way of preventing expansion on a script argument coming from a variable we WANT to be resolved? (thus, the way I know of preventing expansion (using single quotes) makes variable no longer resolved/replaced).

Something like : (but I read string concatenation should be avoided for safety, injection)

ffmpeg '${extension}'
       ||           | 3
       || 2
       | 1

where :

  • 1: would open single quote for final result being ffmpeg '*.jpg'
  • 2: would inject and resolve our variable extension to .jpg
  • 3: would close single quote for ffmpeg to receive argument as if I manually had written '*.jpg'

Edit: after comment about how extension variable is assigned:

local extension="${2}" ;
echo $extension ;
extension="${extension:=jpg}" ; # Default on "jpg" extension
# wrapp extension type in single quotes + prefix with "*."
extension="*.${extension}" ;

Then, using it with:

runprintcommand ffmpeg -loglevel verbose -pattern_type glob -i ${extension} "$movieName" ;

Where the runprintcommand function/script being:

function runprintcommand() {
    echo "Command to run: (Note: '\\' escape character may not be printed)" ;
    echo "$*" ;
    $* ;    
}
  • 2
    "$extension" (or "${extension}") would expand the variable, but would under no circumstances perform filename globbing on the result, unless you used eval on the result of the expansion. You are doing something very strange if it did that. Or, ffmpeg might possibly do its own globbing, which it would do regardless of set -f in that case. Show us how you assign to extension to create it from $1. – Kusalananda Mar 6 at 21:42
  • indeed, I delegate the call of ffmpeg to a custom homemade, possibly broken runprintcommand function, which may interfer with globbing. – el-teedee Mar 6 at 21:51
  • 2
    Then you have a bug in that function, or in the creation of the extension variable's value. – Kusalananda Mar 6 at 21:52
4
  • An unquoted variable would undergo variable expansion, word-splitting (on spaces, tabs and newlines by default), and each generated word would in turn undergo filename generation (globbing).
  • A variable within double quotes would have its value expanded, but the shell would not do word-splitting or globbing on the expanded value.
  • A variable within single quotes would not be expanded at all.

Example:

$ ls
script.sh
$ var='* *'
$ echo $var
script.sh script.sh
$ echo "$var"
* *
$ echo '$var'
$var

There are two issues in your code which causes the value in $extension to be expanded to the glob pattern that it contains and also causes the glob pattern to be matched against filenames prematurely (you want to hand it as-is to ffmpeg -i which does its own glob expansion internally).

The first is your call to your function:

runprintcommand ffmpeg -loglevel verbose -pattern_type glob -i ${extension} "$movieName" ;

Here, ${extension} is unquoted, so you would definitely get (word-splitting and) filename globbing happening on its value.

The second issue is the function itself:

function runprintcommand() {
    echo "Command to run: (Note: '\\' escape character may not be printed)" ;
    echo "$*" ;
    $* ;    
}

Here, you use $* unquoted, which again would (split the value up into words and then) expand the glob, even if you double-quoted ${extension} in the call to the function.

Instead of a bare $*, use "$@" (with the double quotes). This would expand to the individually quoted positional parameters of the function.

This is the difference between "$@" and "$*":

  • "$*" is a single double-quoted string. This can generally not be used to execute a command with arguments.
  • "$@" is a number of double quoted words. This could be used to execute a command with arguments.
  • Originally, the unquoted ${extension} was a (single quoted, but hardcoded) '*.jpg' that I wanted to turn into an argument, for script to have such an option. (I want to say, its a remaining bug after my today tests, it has to be quoted I agree.). Thank you for the hint in runprintcommand ("$@") and for your explanations and your time. – el-teedee Mar 6 at 22:07
  • should I accept it as best answer (as it does not directly answer to my original question about preventing expansion syntax, but explains a bug in a third script that I used?) Should I delete my question? or accept ikkachu's anwer ? – el-teedee Mar 6 at 22:12
  • 1
    @el-teedee I was just about to add something about that. In the end, you decide what answer is the best or most helpful one (which may in fact be an answer not yet written, don't be in a hurry). – Kusalananda Mar 6 at 22:14
  • 1
    For information, (you were right about the bug on other function), I replaced it with "$@" and the call by runprintcommand ffmpeg -loglevel verbose -pattern_type glob -i "${extension}" "$movieName" and it works perfectly now, variable is expanded, and word splitting and globbing NO longer occur. ffmpeg receives the expected *.JPG argument, and make its own globbing as @ikkachu does point out. Thanx again – el-teedee Mar 6 at 22:27
  • Now that you completed your answer with quoting, expansion and "a single double-quoted string", "a number of double quoted words" explanations, no scrupl chosing it as accepted. – el-teedee Mar 6 at 22:34
1

The way to stop the shell from globbing is to quote the variable expansion. (It also stops word splitting, which is usually also what you want). That doesn't stop the command itself from processing glob-like patterns, of course. ffmpeg does that for -i, and so does e.g. find -name. Using the latter as an example:

$ touch a.foo b.foo c.bar
$ extension=foo
$ set -x
$ find . -name "*.$extension"
+ find . -name '*.foo'
./b.foo
./a.foo

(The line that starts with a + comes from set -x, and shows the command the shell actually ran.)

While you can stop globbing with set -f, too, it doesn't help with word splitting. If the variable contained whitespace, the unquoted expansion would break, even with set -f.

  • thank you for explanation, I fixed the bug that occured in fact in another script where I missed to double quote the right thing. Now that it is double quoted well at all the place it needs, no word splitting nor globbing will occur as expected. – el-teedee Mar 6 at 22:30
0

As I found an alternate syntax of doing this, I write it down here.

By using a flag to disable Shell globbing (http://tldp.org/LDP/abs/html/globbingref.html), it does prevent expansion:

set -f ; # disable glob (prevent expansion on '*.jpg' to turn in "1.jpg 2.jpg 3.jpg...")
ffmpeg -pattern_type glob -i ${extension} movie.mp4 ;
set +f ; # restore

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.