0

I was writing a ksh script to validate the column is numerical. The regex pattern is defined in a config file like \d+.\d+. But this is not working when I use d pattern. However [0-9]{1,9} is working. Any insights into this?

  • Here is the ksh version I am using:
    $ ksh --version
      version         sh (AT&T Research) 93u+ 2012-08-01
    
  • Code snippet for the pattern comparison. If I provide $col_patt as \d+ it would not work but [0-9]{1,} will work
    val=$(awk -F "$sep" -v n="$col_pos" -v m="$col_patt" 'NR!=1 && $n !~ "^" m "$" {
                             printf "%s:%s:%s\n", FILENAME, FNR, $n > "/dev/stderr"
                             count++
                           }
                           END {print count+0}' "$cp_input" 2>> $script_path/errors_${file_name_patt}.log
                           )
    
  • Here is the pattern used: \d*\.\d+
3
  • 2
    That very much depends on the tool (and implementation/version thereof) or ksh construct that is used to match strings against that pattern and the implementation and version of ksh if a ksh construct. string=00.99 ksh -c '[[ $string = ~(E:\d+\.\d+) ]] && echo yes' should work if using ksh93 for instance. Commented Jun 15, 2021 at 14:10
  • 5
    See also Why does my regular expression work in X but not in Y? (possible duplicate) Commented Jun 15, 2021 at 14:14
  • 1
    IOW, unless you tell us (by editing your question to provide the additional info) what software is trying to make use of that regexp, we can't help you. Commented Jun 15, 2021 at 15:17

1 Answer 1

3

Various utilities, languages, regex/pattern libraries and APIs support different operators/wildcards.

\d is a perl regex operator that matches a decimal digit (generally any of 0123456789, but under some conditions can match other decimal digits (there are hundreds in Unicode such as 0123456789٠١٢٣٤٥٦٧٨٩۰۱۲۳۴۵۶۷۸۹߀߁߂߃߄߅߆߇߈߉०१२३४५६७८९০১২৩৪৫৬৭৮৯੦੧੨੩੪੫੬੭੮੯૦૧૨૩૪૫૬૭૮૯୦୧୨୩୪୫୬୭୮୯௦௧௨௩௪௫௬௭௮௯౦౧౨౩౪౫౬౭౮౯೦೧೨೩೪೫೬೭೮೯൦൧൨൩൪൫൬൭൮൯෦෧෨෩෪෫෬෭෮෯๐๑๒๓๔๕๖๗๘๙໐໑໒໓໔໕໖໗໘໙༠༡༢༣༤༥༦༧༨༩၀၁၂၃၄၅၆၇၈၉႐႑႒႓႔႕႖႗႘႙០១២៣៤៥៦៧៨៩᠐᠑᠒᠓᠔᠕᠖᠗᠘᠙᥆᥇᥈᥉᥊᥋᥌᥍᥎᥏᧐᧑᧒᧓᧔᧕᧖᧗᧘᧙᪀᪁᪂᪃᪄᪅᪆᪇᪈᪉᪐᪑᪒᪓᪔᪕᪖᪗᪘᪙᭐᭑᭒᭓᭔᭕᭖᭗᭘᭙᮰᮱᮲᮳᮴᮵᮶᮷᮸᮹᱀᱁᱂᱃᱄᱅᱆᱇᱈᱉᱐᱑᱒᱓᱔᱕᱖᱗᱘᱙꘠꘡꘢꘣꘤꘥꘦꘧꘨꘩꣐꣑꣒꣓꣔꣕꣖꣗꣘꣙꤀꤁꤂꤃꤄꤅꤆꤇꤈꤉꧐꧑꧒꧓꧔꧕꧖꧗꧘꧙꧰꧱꧲꧳꧴꧵꧶꧷꧸꧹꩐꩑꩒꩓꩔꩕꩖꩗꩘꩙꯰꯱꯲꯳꯴꯵꯶꯷꯸꯹0123456789𐒠𐒡𐒢𐒣𐒤𐒥𐒦𐒧𐒨𐒩𐴰𐴱𐴲𐴳𐴴𐴵𐴶𐴷𐴸𐴹𑁦𑁧𑁨𑁩𑁪𑁫𑁬𑁭𑁮𑁯𑃰𑃱𑃲𑃳𑃴𑃵𑃶𑃷𑃸𑃹𑄶𑄷𑄸𑄹𑄺𑄻𑄼𑄽𑄾𑄿𑇐𑇑𑇒𑇓𑇔𑇕𑇖𑇗𑇘𑇙𑋰𑋱𑋲𑋳𑋴𑋵𑋶𑋷𑋸𑋹𑑐𑑑𑑒𑑓𑑔𑑕𑑖𑑗𑑘𑑙𑓐𑓑𑓒𑓓𑓔𑓕𑓖𑓗𑓘𑓙𑙐𑙑𑙒𑙓𑙔𑙕𑙖𑙗𑙘𑙙𑛀𑛁𑛂𑛃𑛄𑛅𑛆𑛇𑛈𑛉𑜰𑜱𑜲𑜳𑜴𑜵𑜶𑜷𑜸𑜹𑣠𑣡𑣢𑣣𑣤𑣥𑣦𑣧𑣨𑣩𑥐𑥑𑥒𑥓𑥔𑥕𑥖𑥗𑥘𑥙𑱐𑱑𑱒𑱓𑱔𑱕𑱖𑱗𑱘𑱙𑵐𑵑𑵒𑵓𑵔𑵕𑵖𑵗𑵘𑵙𑶠𑶡𑶢𑶣𑶤𑶥𑶦𑶧𑶨𑶩𖩠𖩡𖩢𖩣𖩤𖩥𖩦𖩧𖩨𖩩𖭐𖭑𖭒𖭓𖭔𖭕𖭖𖭗𖭘𖭙𝟎𝟏𝟐𝟑𝟒𝟓𝟔𝟕𝟖𝟗𝟘𝟙𝟚𝟛𝟜𝟝𝟞𝟟𝟠𝟡𝟢𝟣𝟤𝟥𝟦𝟧𝟨𝟩𝟪𝟫𝟬𝟭𝟮𝟯𝟰𝟱𝟲𝟳𝟴𝟵𝟶𝟷𝟸𝟹𝟺𝟻𝟼𝟽𝟾𝟿𞅀𞅁𞅂𞅃𞅄𞅅𞅆𞅇𞅈𞅉𞋰𞋱𞋲𞋳𞋴𞋵𞋶𞋷𞋸𞋹𞥐𞥑𞥒𞥓𞥔𞥕𞥖𞥗𞥘𞥙🯰🯱🯲🯳🯴🯵🯶🯷🯸🯹 )).

Since perl introduced that operator in the 80s, a few regexp engines have followed suit, but far from all. The ones beside perl that support it that I'm aware of are:

  • vim (not inside bracket expressions).
  • PCRE library (as used in many tools and languages)
  • python or ruby regexps or more generally regular expressions that aim to cover most of or extend the PCRE ones.
  • ast-open regexps, as used by ksh93, or the grep / sed builtins of ksh93 when ksh93 has been compiled as part of ast-open and you've run builtin grep / builtin sed or put /opt/ast/bin at the front of $PATH to enable them. In ksh93 globs, \d is only recognised if found inside one of its x(...) extended operators (where x can be @, +, ~...). [[ 1 = @(\d) ]] will match but not [[ 1 = \d ]] (not inside bracket expressions).

The POSIX equivalent in POSIX basic or extended regular expressions and shell filename patterns (as used in shell filename generation or fnmatch(), or find -path/-name patterns for instance) is [[:digit:]]. That's meant to be the same as [0123456789] but you'll find some systems where that matches other digits.

There's also [0-9] that matches characters that come between 0 and 9. Again, that should include [0123456789] but often includes more. POSIX only guarantees [0-9] to be the same as [0123456789] in the C / POSIX locale.

zsh globs also support the <x-y> operator to match strings that represent a decimal number in the specific range. <3-12> for instance would match on 006, 11, 012, and <-> would be the same as perl's \d+ (though only with 0123456789 digits).

Not all regexp engines will recognise + either. That one comes from egrep / awk in the late 70s, and (contrary to \d) could not be added to already existing regexp engines (like those of grep, sed or vi) as that would have broken backward compatibility and scripts that were doing grep + to look for + characters for instance.

All regexp engines (not globs) recognise the . operator though which matches any single character (same as the ? glob operator) as that was in the original implementation from the 60s.

The most portable regexp to match strings that is made of one or more ASCII decimal digits followed by a literal . followed by 1 or more ASCII decimal digits would be:

^[0123456789][0123456789]*[.][0123456789][0123456789]*$

Though note that in perl that would also match on 12.3<newline> as well. You'd need ^\d+\.\d\Z to avoid it.

Since you've now clarified the tool was awk, I don't know of any awk implementations that support \d as a regexp operator. All will support [0123456789] and +. All will support [0-9] though what it matches outside of the C locale depends on implementation, most will support [[:digit:]].

Also, to pass data to awk that may contain backslash character (which happens to be the case for you), you should avoid -v as that does one level of backslash escape expansion¹ (\n turns into a newline character \\ into \ and \d into \d or d or generate an error depending on the implementation).

So here, you'd want:

val=$(
  RE="$col_patt" awk -F "$sep" -v n="$col_pos" '
    BEGIN {m = ENVIRON["RE"]}
    NR!=1 && $n !~ "^" m "$" {
      printf "%s:%s:%s\n", FILENAME, FNR, $n > "/dev/stderr"
      count++
    }
    END {print count+0}
    ' "$cp_input" 2>> "$script_path/errors_${file_name_patt}.log"
)

And make sure $col_patt contains a regexp compatible with the syntax recognised by your awk implementation.

col_patt='[0123456789]+\.[0123456789]'

at least should be OK.


¹ Recent versions of the GNU implementation of awk also has a problem with values starting with @/ and ending in /. See how awk -v var='@/x/' 'BEGIN{print var}' outputs x instead of @/x/ there

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .