3

Is there any special bash reserved characters in bash regex expression?

  • ex:

    • eg: if [[ $url =~ ^https:\/\/www\.youtube\.com\/playlist\?(.+&)?list= ]]; then echo "URL matches the regex"; fi
      This is just a normal regex test.
  • problem:

    • Do I need to be careful about special case where:
      I have things like eg: %VAR%, $VAR, &VAR, (or any potential special bash reserved characters), inside my regex expression.
      Do I need to escape those?
    • eg: if [[ $url =~ ^https:\/\/%VAR%/$VAR/&VAR/!@#$%^&*()-=_+/wte;/ ]]; then echo "URL matches the regex"; fi
      // assume I intentionally need to match such regex
  • ex:

  • again:

    • I am not asking about escaping the regex (I know how to use regex, be it PCRE or ERE doesnt matter.)
    • I am asking about escaping the bash reserved special characters, when inside a regex expression, if there is any.
  • In another words, is the regex expression in bash

    • more like /^http:\/\/www/g in js literal regex, where the inside will always be literal regex;
    • or more like "^http:\\/\\/www" in js RegExp, where you need to escape the reserved characters ' " ` \ etc due to being inside a string

Update:

  • For "Is there any special bash reserved characters in bash regex expression?"
    The simple answer is yes.

  • As for an example:

    url="https://www.youtube.com/playlist?list=PLmHVyfmcRKyxvxehq3fiGjKDsEyy6d4Tz"
    regex='^https:\/\/www\.youtube\.com\/playlist\?(.+&)?list='
    
    idx=0
    (( idx++ )) # normal
    if [[ $url =~ ^https:\/\/www\.youtube\.com\/playlist\?(.+&)?list= ]]; then echo "URL matches the regex - $idx"; fi
    (( idx++ )) # normal
    if [[ $url =~ $regex ]]; then echo "URL matches the regex - $idx"; fi
    (( idx++ ))
    yt='www\.youtube\.com\/'
    if [[ $url =~ ^https:\/\/${yt}playlist\?(.+&)?list= ]]; then echo "URL matches the regex - $idx"; fi
    (( idx++ ))
    if [[ $url =~ ^https:\/\/'${yt}'playlist\?(.+&)?list= ]]; then echo "URL matches the regex - $idx"; fi
    (( idx++ ))
    if [[ $url =~ '^https:\/\/'${yt}'playlist\?'(.+&)?list= ]]; then echo "URL matches the regex - $idx"; fi
    

Update:

For what literal regex that I want means:

text='The pizza is 2'\'' and 100$price' # The pizza is 2' and 100$price

# literal regex I want: [0-9]+' and [0-9]+\$price
regex="[0-9]+' and [0-9]+\\\$price"
if [[ "$text" =~ $regex ]]; then echo "this is the normal way of doing things, do you see I need to escape the \ & $ ?"; fi
# if [[ "$text" =~ [0-9]+' and [0-9]+\$price ]]; then echo "this is what I prefer to have -- literal regex, like / /g in js. But this wont even compile"; fi

js anology:

const str = "The pizza is 2' and 100$price";

if (/[0-9]+' and [0-9]+\$price/.test(str)) {
  console.log('literal regex');
}
if (new RegExp("[0-9]+' and [0-9]+\\$price").test(str)) {
  console.log('non literal regex');
}

enter image description here

  • conclusion from comments & discussions:
    • use ' single quote & a variable outside the if statement.
    • here, ' in bash is actually analogous to / in js
      • -- so, the literal regex I mean (should have learnt more before asking.)
        (you may read comments & discussions below)
    • though, escaping single quote ' in bash -- '\'', is not as pretty as escaping / in js -- \/.)
0

2 Answers 2

4

Short answer: yes, of course you need to quote Bash special characters in a regex.

Longer answer, as well as an idea to overcome the problem of quoting Bash special characters, is contained in section 3.2.5.2 Conditional Constructs of the GNU Bash Reference Manual:

You can quote any part of the pattern to force the quoted portion to be matched literally instead of as a regular expression (see above). If the pattern is stored in a shell variable, quoting the variable expansion forces the entire pattern to be matched literally.

and

If you want to match ‘initial string’ at the start of a line, the following will work:

[[ $line =~ ^"initial string" ]]

but this will not:

[[ $line =~ "^initial string" ]]

because in the second example the ‘^’ is quoted and doesn’t have its usual special meaning.

It is sometimes difficult to specify a regular expression properly without using quotes, or to keep track of the quoting used by regular expressions while paying attention to shell quoting and the shell’s quote removal. Storing the regular expression in a shell variable is often a useful way to avoid problems with quoting characters that are special to the shell. For example, the following is equivalent to the pattern used above:

pattern='[[:space:]]*(a)?b'
[[ $line =~ $pattern ]]

The manual then continues detailing the specifics and giving the examples for backslashes, bracket expressions, and so on.

1
  • It seems the quotes are like \Q\E in PCRE. \ And it isnt for making literal regex, but literal string. \ It also mixs up the ability of making both special regex reserved character & special bash reserved characters into literal strings. \ Quite messy. \ (Also thanks for the link to the docs.) \ And seems there is no good way to write literal regex.
    – Nor.Z
    Commented Jun 16 at 21:17
2

It's safe to put any character except ^ or \ inside a bracket expression to make it literal (i.e. change x to [x]) but never put a \ in front of any character that isn't a metacharacter (e.g. the /s in the regexps in your question) because doing so can turn it into a metacharacter for any given tool since a backslash before an ordinary character is undefined behavior per POSIX. ^ and \ need to have \ put in front of them to make them literal as they take on a different meaning inside [].

I'm not sure but I think you may be asking how to both protect against bash globbing, word splitting, etc. constructs and ERE metachars so you can create a regexp that includes parts to be treated literally. If so:

For example:

$ cat tst.sh
#!/usr/bin/env bash

goodUrl='https://www.youtube.com/playlist?list=PLmHVyfmcRKyxvxehq3fiGjKDsEyy6d4Tz'
badUrl='https://wwwxyoutubexcom/playlist?list=PLmHVyfmcRKyxvxehq3fiGjKDsEyy6d4Tz'

echo "goodUrl=$goodUrl"
echo "badUrl=$badUrl"

echo "######################"

orig_yt='www.youtube.com'

regex='^https://'"$orig_yt"'/playlist\?(.+&)?list='

if [[ $goodUrl =~ $regex ]]; then
    echo "goodUrl matched orig_yt: $regex"
else
    echo "goodUrl did not match orig_yt: $regex"
fi

if [[ $badUrl =~ $regex ]]; then
    echo "badUrl matched orig_yt: $regex"
else
    echo "badUrl did not match orig_yt: $regex"
fi

echo "######################"

sanitized_yt="${orig_yt//[^^\\]/[&]}"
sanitized_yt="${sanitized_yt//[\\^]/\\&/}"

regex='^https://'"$sanitized_yt"'/playlist\?(.+&)?list='

if [[ $goodUrl =~ $regex ]]; then
    echo "goodUrl matched sanitized_yt: $regex"
else
    echo "goodUrl did not match sanitized_yt: $regex"
fi

if [[ $badUrl =~ $regex ]]; then
    echo "badUrl matched sanitized_yt: $regex"
else
    echo "badUrl did not match sanitized_yt: $regex"
fi

$ ./tst.sh
goodUrl=https://www.youtube.com/playlist?list=PLmHVyfmcRKyxvxehq3fiGjKDsEyy6d4Tz
badUrl=https://wwwxyoutubexcom/playlist?list=PLmHVyfmcRKyxvxehq3fiGjKDsEyy6d4Tz
######################
goodUrl matched orig_yt: ^https://www.youtube.com/playlist\?(.+&)?list=
badUrl matched orig_yt: ^https://www.youtube.com/playlist\?(.+&)?list=
######################
goodUrl matched sanitized_yt: ^https://[w][w][w][.][y][o][u][t][u][b][e][.][c][o][m]/playlist\?(.+&)?list=
badUrl did not match sanitized_yt: ^https://[w][w][w][.][y][o][u][t][u][b][e][.][c][o][m]/playlist\?(.+&)?list=

Regarding the update to your question that includes this:

text='The pizza is 2'\'' and 100$price' # The pizza is 2' and 100$price
...
# if [[ "$text" =~ [0-9]+' and [0-9]+\$price ]]; then echo "this is what I prefer to have -- literal regex, like / /g in js. But this wont even compile"; fi

You simply cannot write such a regexp in-line, you need to store it in a variable to avoid issues with bash metachars. You would also need to further escape any regexp metachars that you want treated literally but you're already escaping the $ manually in your regexp so you can skip that step and just write:

regex='[0-9]+'\'' and [0-9]+\$price'
if [[ "$text" =~ $regex ]]; ...

Note, though, that when you say "do you see I need to escape the \ & $ ?" - to be clear, you need to escape the \ $ and ? in a single-quoted string as I used because they're regexp metachars, not because they're bash metachars. You do not need to escape a & as it's not a regexp metachar (unless bash regexps support backreferences in the regexp, which I doubt).

In the js example in your question if (/[0-9]+' and [0-9]+\$price/... - that regexp is identical to the one you'd store in a variable for bash except that in bash we use ' (or ") as the delimiter and so can't use ' (or ") in the string as-is while js is using / as the delimiter and so, I suspect, you couldn't use / in the string as-is and so the js regexp isn't actually any more literal than the bash regexp, it just uses different delimiters and doesn't need to be stored in a variable.

If you don't want to use a variable in bash, you can always just write a function to do the comparison:

$ cat tst.sh
#!/usr/bin/env bash

cmpr() { [[ "$1" =~ $2 ]]; }

text='The pizza is 2'\'' and 100$price' # The pizza is 2' and 100$price

if cmpr "$text" '[0-9]+'\'' and [0-9]+\$price'; then
    echo "this is what I prefer to have -- literal regex, like / /g in js. But this wont even compile"
fi

$ ./tst.sh
this is what I prefer to have -- literal regex, like / /g in js. But this wont even compile

but to me that seems like overkill and makes the code a little bit harder to understand.

3
  • Note for js, yes, you do need to escape /, eg: if (/[0-9]+' and [0-9]+\$price\/lb/.test(str)) { is valid -- but, to me it feels like you are escaping the / for regex, not for js (just a feeling, cuz pcre escapes that too, I can be wrong).
    – Nor.Z
    Commented Jun 22 at 19:56
  • 1
    You're definitely escaping it for js - it's the same with any tool/language, you can never include a character that delimits a string within the string without doing something to escape it. A / is simply not a regexp metachar in BRE or ERE, I'm not familiar enough with PCRE to comment on that.
    – Ed Morton
    Commented Jun 22 at 19:57
  • 1
    stackoverflow.com/a/21335941/1745001 apparently confirms that / isn't a metachar in a PCRE either since in perl it also only needs escaping if you're using / as the delimiter.
    – Ed Morton
    Commented Jun 22 at 20:03

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .