Match ASCII codes in shell regex

Question

I've trouble about matching ASCII codes in regex in my bash script.

My payload is : payload='#1243669, start 2018-12-17 16:32:50 CET'

My old regex is : regold='(#[^,]+),' My new regex is : regnew='(#[^\x2c]+)\x2c'

When I do the test in shell, it confirms that new regex doesn't work, whereas it works on regex101.com :

root@test# payload='#1243669, start 2018-12-17 16:32:50 CET'
root@test# regold='(#[^,]+),'
root@test# regnew='(#[^\x2c]+)\x2c'
root@test# if [[ $payload =~ $regold ]]; then echo yes; fi
yes
root@test# if [[ $payload =~ $regnew ]]; then echo yes; fi

Edit:

Question partially answered by telling bash doesn't support PCRE expressions. So I move my regex to this one :

root@test# regex=$'(\x23[^\x2c]+)\x2c[[:space:]]start[[:space:]][[:digit:]]{4}\x2d[[:digit:]]{2}\x2d[[:digit:]]{2}[[:space:]]([^[:space:]]+)[[:space:]][^\x2c]+\x2c[^\x2c]+\x2c[[:space:]](captureme)'
root@test# pattern='Message #1243669, start 2018-12-17 16:32:50 CET, duration 20, captureme'

Again, my expression works great on regex101, sample here https://regex101.com/r/g4UsUN/1

Expression works to this point : ... [[:space:]][^\x2c]+ ... but bash had some difficulties when I add the \x2c

What's going on ?

Answer 1

In bash, you may insert the literal character that the hexadecimal codes are the codes for with $'\xHH'. bash will not recognise them in a string otherwise, and the regular expression [^\x2c] would match any character that is not a \, x, 2, or c. The regex101.com site that you tested your expression in may have used a PCRE regular expression engine. This is not what bash uses, it uses POSIX extended regular expressions (which does not seem to be supported by that web site). This also means that bash does not support other regular expression syntax that you would otherwise use in a PCRE, such as \d for digits, \s for space characters, or look-aheads or similar things.

POSIX regular expression (both basic and extended) should be documented on your system in the re_format(7) manual (man 7 re_format), and are also described in detail by the POSIX standard.

Therefore, use

regnew=$'#[^\x2c]+\x2c'

(the parenthesis seemed unused, so I removed it). Since bash would expand the escape-sequences before assigning the string to the variable, and since they may expand to characters that may have special meaning in regular expressions, care must be taken to escaped the escape-sequences to match literal values. One may want to prefix \x2e (a dot character) with \\ (\\\x2e) when a literal dot is to be matched, for example.

From the bash manual:

Words of the form $'string' are treated specially. The word expands to string, with backslash-escaped characters replaced as specified by the ANSI C standard. Backslash escape sequences, if present, are decoded as follows:

[...]

\xHH

the eight-bit character whose value is the hexadecimal value HH (one or two hex digits)

Note that using $'...' would make the shell also replace any other escape sequences that it recognises.