best / any way to count escapes in glibc regex(7) / GNU sed / grep / egrep Regular Expressions?

Question

Given bash environment variable settings :

 $ declare -g bs=$'\\' bsbs=$'\\\\' q="'";

This Regular Expression will correctly match a sequence of single quote-("'")-ed text , where such text may contain escaped single quotes:

 "[${bs}${q}]((([^${bsbs}]?[^${bs}${q}])|(${bsbs}${bs}${q}))+)[${bs}${q}]"

 $ echo "[${bs}${q}]((([^${bsbs}]?[^${bs}${q}])|(${bsbs}${bs}${q}))+)[${bs}${q}]"
 [\']((([^\\]?[^\'])|(\\\'))+)[\']

(the backtick in "[\']" is not strictly required but is included for clarity, and in case one is trying to encode this value in a singly-quoted string).

The problem lies in how best to generalize this for any escaped quoting character, and how to handle runs of multiple escape sequences ; ONLY if the run of input escape characters is of ODD ((n&1)==1) size (number of bytes), then the last escape is ACTIVE, the last character is INACTIVE (part of string), and otherwise (number of escapes is EVEN ((n&1)==0), then the string contains HALF the number of escapes (n>>1) and the last character is ACTIVE (ie. not escaped) .

Also, in sed and grep / egrep this has some issues :

o The matching sub-groups can occupy subsequent '\1+' group numbers, increasing their number - if any subsequent group does not match -

• ideally, I'd like to be able to express that regexp without any subgroups that can possibly affect subsequent sub-group numbers.

o It doesn't handle Number of Escapes at all, and will fail to
recognize that a quote that is proceeded by an EVEN number of escapes is not escaped.

So my question is :

How best to solve these issues using only glibc-supported POSIX REs or grep / sed REs ?

ie. allow arbitrary length sequences of escapes of ODD (effective escape) or EVEN (ineffective escape) length to be recognized inside RegExps ?

I really think POSIX REs could benefit from special syntax to handle such questions, like:

 [\\]{1,}\#&1\?$A\:$B

Where '}#&1' means the test 'x & 1' on the number of elements matched by previous [\]{...} group, and ?x:y means "if last test is true, substitute x, otherwise y in RE".

Then one could actually easily express this and safely handle any number of escapes in RegExp parsed strings . How to do that without some new RE syntax like this ?

Very difficult, if not impossible / infeasible , with RegExp exprs alone.

Or am I wrong ?

Is there now an easy way to do arithmetic on run length of previous group in modern POSIX REs ?

Example 1 :

$ declare -g bs=$'\\' bsbs=$'\\\\' q="'";

$ echo "'a quot\\'d string' 42" | sed -r 's/'"[${bs}${q}]((([^${bsbs}]?[^${bs}${q}])|(${bsbs}${bs}${q}))+)[${bs}${q}]"'[[:space:]]([0-9]+)/\1\t:\t\2/'
'a quot'd string    :   g

Example 2 :

$ echo "'a quot\\'d string' 42" | 
  sed -r 's/'"[${q}]((([^${bsbs}]?[^${q}])|(${bsbs}${q}))+)[${q}]"'[[:space:]]([0-9]+)/\1\t:\t\2/'
a quot\'d string    :   g

note how the ${bs}-es @rowboat mentioned are removed, and still the same result , as would using only $bs, not $bsbs :

$ echo "'a quot\\'d string' 42" | sed -r 's/'"[${q}]((([^${bs}]?[^${q}])|(${bs}${q}))+)[${q}]"'[[:space:]]([0-9]+)/\1\t:\t\2/'
a quot\'d string    :   g

Conclusion :

I am developing non-POSIX extensions to the "regex(7) - POSIX.2 regular expressions" library, provided by glibc , and to PCRE, and to PERL, and to cl-ppcre (SBCL's Common Lisp RE library) , and to Emacs's RE library for :

o defining a meaning for any named POSIX character class when suffixed by '-esc' or 'esc', eg. '[[:spaceesc:]]' or '[^[:space-esc:]]' or '[[:quote-esc:]]' , which means: A character that is ordinarily a member of character class 'X', is not a member of character class "${X}esc" (a synonym of "${X}-esc") IFF it is preceded by an ODD NUMBER of Escape Characters ('\':ASCII "\x5c").

 All character sequences that are subject to an :*esc: character
 class test will have legal '\\' , '\xXX', '\0OOO', or '\Uxxxxxx' or
'\uXXXX' sequences replaced by :
 ASCII:\x5c , ASCII:\xXX (where XX are hex digits), 
 ASCII:\OOO (where OOO are Octal digits) ,
 24-bit unicode value with code point xxxxxx (x: hex digit) , and
 16-bit unicode value with code point xxxx (x: hex digit) ,
 respectively.

 Also '[[:quote:]]' and '[[:quoteesc:]]' classes must be
 supported that select characters (or non-escaped chars)
 with the Unicode 'Quotation Mark' binary attribute, and
 '[[:punct:]]' or '[[:punctesc:]]' would similarly apply
 to all (non-escaped) chars which have the Punctuation attribute.

 Perhaps a similar '*cesc' or '*escc' character class suffixes
 could be provided that support also the C escapes:
  '\n','\r','\t','\v','\b','\l'... etc.

 If the /𝕦 (\U1D566) flag is specified / UNICODE_NAMES flag,
 then Unicode Names can also be specified :
  \U1D566 == \U{MATHEMATICAL DOUBLE-STRUCK SMALL U}
          == \U{MATHEMATICAL_DOUBLE_STRUCK_SMALL_U}
          .

 There is no point in doing such an exercise unless UTF-8 names
 also are supported, IMHO .

 There no point in just handling escaped spaces if full escape
 handling is not also enable-able somehow or comes along with it.

 Actually, the SBCL Pure-Common-Lisp implementation is about
 the speediest and nicest to use amongst ANY RE implementation
 I have used , and already supports escaped classes & Unicode
 Names.

 The LIBC regex and glob implementations are EXTRA-ORDINARILY SLOW!

 This slows down BASH and command-line tools and all tools that
 use the POSIX RE library, such as Flex / Bison / Yacc,
 tremendously .

 Perhaps either :
 A) Techniques used in SBCL PPCRE, libppcre, and PERL RE library
    can be ported to LIBC Regex library, in a new 'Fast Regex'
    replacement that can optionally replace old implementation
    on demand ;
 B) LIBC RE library can be made to transparently replace itself
    with libppcre or to a connection to a running SBCL instance
    with CL-PPCRE loaded, or to PERL with full PERL RE support,
    to support a UNIX CMSG Message API for Compiling, 
    Match Against String, or Match against FD / stream API, 
    and for Retrieval of Match Numbered N or with Name N,
    where N is in a set of Group Names or 
    Numbers sent in advance as identifying parenthesis groups,
    and which can contain multiple dimensions (numbers in square
    brackets) to denote sub-expressions.

 Also I think that supporting a char-class LENGTH test, of the
 form:
   ']{x,}\#<test>\?<A>\:<B>' , meaning:
 "  If number of characters in character-class just closed
    satisfies test <test> , then RE fragment <A> is parsed / takes
    effect, else RE <B> takes effect.
 ", would be very useful - for <test> in:
 {=X,>X,<X,<=X,>=X,&X,|X,^X,&~X,|~X} , where X is a decimal number.

 But first, I am working on the escaped char-classes support.

I can't understand why no-one seems to understand what I was suggesting with this question.

I hope the above makes things clearer.

Answer 1

If the point is to tokenise shell code the same way the shell language interpreter does, regexps will get you nowhere.

The zsh shell exposes its tokeniser with the z parameter expansion flag (or Z which can take options to process comments or change the treatment of newline) which you can combine with the Q parameter expansion to do quote removal.

For instance:

tokens() printf ' - « %s »\n' ${(Z[Cn])1}
tokens_dequoted() printf ' - « %s »\n' "${(@Q)${(Z[Cn])1}}"

Would report all the shell tokens in its first argument, dropping comments; the second one also removing one layer of quoting:

$ tokens '  foo "a b"; "" "$(echo "x y")" <<'"'qwe '\''qwe' #qwe"
 - « foo »
 - « "a b" »
 - « ; »
 - « "" »
 - « "$(echo "x y")" »
 - « << »
 - « 'qwe '\''qwe' »
$ tokens_dequoted '  foo "a b"; "" "$(echo "x y")" <<'"'qwe '\''qwe' #qwe"
 - « foo »
 - « a b »
 - « ; »
 - «  »
 - « $(echo "x y") »
 - « << »
 - « qwe 'qwe »

You can see that in order to do the same, you need to implement a full shell parser.

You can get somewhere with regexp if you reduce the scope: only consider '...', "...", and \ types of quotes (not $'...'), and only whitespace as delimiter and ignore expansions inside double quotes. In bash 4.4+, which contrary to zsh can't handle NUL bytes in its code anyway, and with GNU grep, you can do:

tokens() {
  local tokens
  readarray -td '' tokens < <(printf %s "$1" |
    grep -Ezo '(\\.|[^[:space:]\\"'\'']|'\''[^'\'']*'\''|"(\\.|[^\\"])*")+'
  )
  printf ' - « %s »\n' "${tokens[@]}"
}

Then:

$ tokens ' foo "a b"\c\\\" c\ d '" 'qwe'\''qwe'\"'\"qwe"
 - « foo »
 - « "a b"\c\\\" »
 - « c\ d »
 - « 'qwe'\''qwe'"'"qwe »

To remove one layer of quoting from that, I'd resort to perl (or zsh which can do that out of the box as seen above).

Answer 2

A better answer: use pcre / PERL RegExps :

$ cat a.pcre
/^[']((?|(?:[^\\]?[^'\t\n\r])|(?:[\\]['\t\n\r]))*)[']\t((?|(?:[^\\]?[^\t])|(?:[\\][^\t\n\r]))+)/
'A quot\'d\ tab containing string'  42

$ pcretest < a.pcre 
PCRE version 8.45 2021-06-15

re> data>  0: 'A quot'd\x09tab containing string'\x0942
1: A quot'd\x09tab containing string
2: 42
data>