7

How can I refer to a regex group in awk regex? For example, if I have a regex group (\w), how can I refer to it later in the same regex like (\w)\1? Does awk support this feature? Below example doesn't work.

# In this example, I want to change aa to aaa and cc to ccc.
echo ab aa cc de mn | gawk '{print gensub(/(\w)\1/, "\\1\\1\\1", "g")}'
# The result is: ab aa cc de mn
# The expected result is: ab aaa ccc de mn
  • 1
    related: unix.stackexchange.com/questions/361427/… – cas Aug 21 at 12:35
  • @Justalearner please edit your question and show us the output you were expecting. Do you only want to make the change when you have a character repeated? – terdon Aug 21 at 12:40
  • @terdon the comment above the echo says “I want to change aa to aaa and cc to ccc” which I assume to mean that “ab”, “de”, and “mn” are supposed to be left unchanged. The question is “how can I refer to it later in the same regex”. – Stephen Kitt Aug 21 at 12:43
  • I edited the question and added the expected result. – Just a learner Aug 21 at 12:47
  • Thank you. But as @StephenKitt pointed out, it was perfectly obvious before, I just completely failed to see it. Sorry! – terdon Aug 21 at 12:48
11
$ echo ab aa cc de mn | perl -pe 's/(\w)\1/\1\1\1/g'
ab aaa ccc de mn

Sometimes you just have to accept that there are some things awk can't do, but perl can.

On the bright side, if you're skilled enough with awk to be using gensub and wanting to do back-references, you should find perl to be a doddle. i.e. if you can write awk, you can write perl.

  • 2
    Maybe I should take a look at perl. Thanks for your answer and suggestion! – Just a learner Aug 21 at 12:45
  • 1
    I highly recommend doing so, especially if you ever find yourself thinking "i wish awk could do this sed thing" or vice-versa. There's a lot more to perl than just this, but very simple pipeline-filter style usage is like a combination of sed,awk,cut,grep,tr,sh and more. On steroids. – cas Aug 21 at 13:28
10

The busybox implementation of awk is the only one that I know that supports back-references. It does happen to also support gawk's gensub() and \w extensions:

Like with sub() and gsub(), you have to use "..." instead of /.../ and use \\1 instead of \1 (in standard awk, "\1" is the character of value 1 (^A), and /\1/ is required to match that character while "\\1" is (well was) un(der)specified in POSIX; also note that POSIX EREs don't have back-references, that's the one feature BREs have but not EREs).

$ echo ab aa cc de mn | busybox awk '{print gensub("(\\w)\\1", "\\1\\1\\1", "g")}'
ab aaa ccc de mn

Beware though that busybox awk is not internationalised, its \w only matches a-zA-Z0-9_ regardless of the locale (same for [[:alnum:]]) and multi-byte characters are not supported:

$ echo ee éé | busybox awk '{print gensub("(\\w)\\1", "\\1\\1\\1", "g")}'
eee éé

With standard utilities, you'd typically use sed for that job:

sed 's/\([[:alnum:]_]\)\1/&\1/g'

sed regular expressions are basic regular expressions that do support back references. Some sed implementations support extended regexps with -r or -E, and POSIX will specify -E in the next major version of the standard, but still not back references (though capture groups for s's replacement will). GNU and busybox sed do support back-references with -E but FreeBSD's sed doesn't.

  • Nice. i wasn't aware of that. I'm not likely to use busybox awk instead of perl, but it's good to know in case I ever need it. – cas Aug 21 at 13:01
  • with perl you have to enable UTF-8 for input and output for unicode text. e.g. echo ee éé | perl -CIO -pe 's/(\w)\1/$1$1$1/g' ---> eee ééé – cas Aug 21 at 13:06
  • 1
    @cas, that only works for UTF-8 locale. You should rather use: perl -Mopen=locale -pe 's/(\w)\1/$1$1$1/g' – Stéphane Chazelas Aug 21 at 13:08
  • cool. for other readers: to get some info on perl's -C options, run man perlrun and search for unicode. for info on perl's open pragma, run perldoc open. – cas Aug 21 at 13:14
4

This may be going beyond the question, but the reason why awk doesn't support backreferences is because awk has always used real regular expressions, ie ones that can be implemented without recursion by a finite-state machine. Such an implementation cannot support backreferences in any form (it can support capture groups, though the implementation is not straight-forward).

The idea with awk as I see it is that you're supposed to use regular expression for straight time- and memory-bounded matches, and its C-like turing-complete language for anything more complex than that.

The "regexps" from perl / pcre / etc on the contrary have grown into a compact syntax to describe recursive matching procedures which can only implemented by a turing machine. This has security implications: any search box, etc. where a untrusted user can enter such a regexp is an invitation to a denial of service attack; nobody can know how much time or memory such a match will take, and only gross measures like hard arbitrary limits on it and banning of persistent hogs are possible.

Here is an old article by Russ Cox, where all this is described more in depth.

  • Right. Also the general approach in awk is to introduce language constructs only for common tasks that aren't easy to do with existing language constructs (i.e. don't introduce constructs just for the sake of brevity) and this fits right in with that. The result is a simple, tiny language that you can do all text manipulation with which is well worth the tradeoff vs other much larger languages that can sometimes accomplish some tasks more briefly than awk but usually are an unintelligible pile of runes! – Ed Morton Aug 25 at 15:28
1

Not regex, but works. Will add exactly 1 extra character, if all of the characters of the word are the same:

echo ab aa cc de mn | \
      awk '{
        for(i=1;i<=NF;i++)
        {
          char=substr($i,1,1)
          for(j=2;j<=length($i);j++)
          {
            if(substr($i,j,1)==char) y=1
          else
            y=0
            char=substr($i,j,1)
          }
          if(y) $i=$i""char
        }
        print $0
        }'

    ab aaa ccc de mn         #output
  • 1
    That changes aba to abaa. That also affects the spacing (like tabs converting to space, leading and trailing whitespace removed) in lines where there have been at least one substitution. – Stéphane Chazelas Aug 23 at 10:37
  • Thanks for checking. Updated it to deal with aba to abaa changes.I didn't deal with the space-tab conversion, or the trailing whitespace issues. – zabidima Aug 23 at 14:53
0

Unfortunately in POSIX or any other non-busybox awk that requires a loop on all unique chars in each line since backreferences are not supported within a regexp:

$ cat tst.awk
{
    old = new = $0
    while (old != "") {
        char = substr(old,1,1)
        gsub(char,"",old)
        if ( char ~ /[[:alnum:]_]/ ) {
            gsub(char char,char char char,new)
        }
    }
    print new
}

$ echo ab aa cc de mn | awk -f tst.awk
ab aaa ccc de mn

The above works when the target characters are not regexp metachars as in this case. If they could be RE metachars then you'd need to escape them before using them in the regexp context of the gsub()s. You could use char"{2}" instead of char char in the gsub() regexp if you prefer.

See @Stephane's answer for how to do this task with busybox awk.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.