1

I've trouble about matching ASCII codes in regex in my bash script.

My payload is : payload='#1243669, start 2018-12-17 16:32:50 CET'

My old regex is : regold='(#[^,]+),' My new regex is : regnew='(#[^\x2c]+)\x2c'

When I do the test in shell, it confirms that new regex doesn't work, whereas it works on regex101.com :

root@test# payload='#1243669, start 2018-12-17 16:32:50 CET'
root@test# regold='(#[^,]+),'
root@test# regnew='(#[^\x2c]+)\x2c'
root@test# if [[ $payload =~ $regold ]]; then echo yes; fi
yes
root@test# if [[ $payload =~ $regnew ]]; then echo yes; fi

Edit:

Question partially answered by telling bash doesn't support PCRE expressions. So I move my regex to this one :

root@test# regex=$'(\x23[^\x2c]+)\x2c[[:space:]]start[[:space:]][[:digit:]]{4}\x2d[[:digit:]]{2}\x2d[[:digit:]]{2}[[:space:]]([^[:space:]]+)[[:space:]][^\x2c]+\x2c[^\x2c]+\x2c[[:space:]](captureme)'
root@test# pattern='Message #1243669, start 2018-12-17 16:32:50 CET, duration 20, captureme'

Again, my expression works great on regex101, sample here https://regex101.com/r/g4UsUN/1

Expression works to this point : ... [[:space:]][^\x2c]+ ... but bash had some difficulties when I add the \x2c

What's going on ?

  • Dont' be logged in as root, unless you want a broken operating system. – ctrl-alt-delor Dec 18 '18 at 13:10
1

In bash, you may insert the literal character that the hexadecimal codes are the codes for with $'\xHH'. bash will not recognise them in a string otherwise, and the regular expression [^\x2c] would match any character that is not a \, x, 2, or c. The regex101.com site that you tested your expression in may have used a PCRE regular expression engine. This is not what bash uses, it uses POSIX extended regular expressions (which does not seem to be supported by that web site). This also means that bash does not support other regular expression syntax that you would otherwise use in a PCRE, such as \d for digits, \s for space characters, or look-aheads or similar things.

POSIX regular expression (both basic and extended) should be documented on your system in the re_format(7) manual (man 7 re_format), and are also described in detail by the POSIX standard.

Therefore, use

regnew=$'#[^\x2c]+\x2c'

(the parenthesis seemed unused, so I removed it). Since bash would expand the escape-sequences before assigning the string to the variable, and since they may expand to characters that may have special meaning in regular expressions, care must be taken to escaped the escape-sequences to match literal values. One may want to prefix \x2e (a dot character) with \\ (\\\x2e) when a literal dot is to be matched, for example.

From the bash manual:

Words of the form $'string' are treated specially. The word expands to string, with backslash-escaped characters replaced as specified by the ANSI C standard. Backslash escape sequences, if present, are decoded as follows:

[...]

\xHH

the eight-bit character whose value is the hexadecimal value HH (one or two hex digits)

Note that using $'...' would make the shell also replace any other escape sequences that it recognises.

  • Hello, simple like that :) Now it works : root@test# regnew=$'(\x23[^\x2c]+)\x2c\sstart' root@test# if [[ $pattern =~ $regnew ]]; then echo yes; fi yes root@test# echo ${BASH_REMATCH[1]} #1243669 Thank you so much! – CLB Dec 17 '18 at 16:24
  • Might be worth noting that since the escapes are expanded before passing to the regexp engine, a $'\x2e' (. in ASCII) would still be the . operator and $'[x\x5d] would be [x]]. But [[ x = $'.' ]] would not match as that . is quoted to bash (since 3.2) replaces it with \. before calling the regex engine. – Stéphane Chazelas Dec 18 '18 at 9:49
  • @StéphaneChazelas Thanks, I overlooked that. – Kusalananda Dec 18 '18 at 9:50
  • \x2c expands to , not . so is OK here, I was thinking of characters that are regexp operators or special within [...] to watch out for. – Stéphane Chazelas Dec 18 '18 at 10:18
  • Note that \xHH only works for single-byte characters. For multi-byte characters, you can use \xHH\xHH... to encode each byte of the character, or use \uXXXX or \UXXXXXXXX to use the unicode code point of the character. – Stéphane Chazelas Dec 18 '18 at 10:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.