How to use regex with AWK for string replacement?

Question

Suppose there is some text from a file:

(bookmarks
("Chapter 1 Introduction 1" "#1"
("1.1 Problem Statement and Basic Definitions 23" "#2")
("Exercises 31" "#30")
("Notes and References 42" "#34"))
)

I want to add 11 to each number followed by a " in each line if there is one, ie

(bookmarks
("Chapter 1 Introduction 12" "#12"
("1.1 Problem Statement and Basic Definitions 34" "#13")
("Exercises 42" "#41")
("Notes and References 53" "#45"))
)

Here is my solution by using GNU AWK and regex:

awk -F'#' 'NF>1{gsub(/"(\d+)\""/, "\1+11\"")}'

i.e., I want to replace (\d+)\" with \1+10\", where \1 is the group representing (\d+). But it doesn't work. How can I make it work?

If gawk is not the best solution, what else can be used?

Answer 1

Try this (gawk is needed).

awk '{a=gensub(/.*#([0-9]+)(\").*/,"\\1","g",$0);if(a~/[0-9]+/) {gsub(/[0-9]+\"/,a+11"\"",$0);}print $0}' YourFile

Test with your example:

kent$  echo '(bookmarks
("Chapter 1 Introduction 1" "#1"
("1.1 Problem Statement and Basic Definitions 2" "#2")
("Exercises 30" "#30")
("Notes and References 34" "#34"))
)
'|awk '{a=gensub(/.*#([0-9]+)(\").*/,"\\1","g",$0);if(a~/[0-9]+/) {gsub(/[0-9]+\"/,a+11"\"",$0);}print $0}'   
(bookmarks
("Chapter 1 Introduction 12" "#12"
("1.1 Problem Statement and Basic Definitions 13" "#13")
("Exercises 41" "#41")
("Notes and References 45" "#45"))
)

Note that this command won't work if the two numbers (e.g. 1" and "#1") are different. or there are more numbers in same line with this pattern (e.g. 23" ...32"..."#123") in one line.

UPDATE

Since @Tim (OP) said the number followed by " in same line could be different, I did some changes on my previous solution, and made it work for your new example.

BTW, from the example I feel that it could be a table of content structure, so I don't see how the two numbers could be different. First would be the printed page number, and 2nd with # would be the page index. Am I right?

Anyway, you know your requirement best. Now the new solution, still with gawk (I break the command into lines to make it easier to read):

awk 'BEGIN{FS=OFS="\" \"#"}{if(NF<2){print;next;}
        a=gensub(/.* ([0-9]+)$/,"\\1","g",$1);
        b=gensub(/([0-9]+)\"/,"\\1","g",$2); 
        gsub(/[0-9]+$/,a+11,$1);
        gsub(/^[0-9]+/,b+11,$2);
        print $1,$2
}' yourFile

test with your new example:

kent$  echo '(bookmarks
("Chapter 1 Introduction 1" "#1"
("1.1 Problem Statement and Basic Definitions 23" "#2")
("Exercises 31" "#30")
("Notes and References 42" "#34"))
)
'|awk 'BEGIN{FS=OFS="\" \"#"}{if(NF<2){print;next;}
        a=gensub(/.* ([0-9]+)$/,"\\1","g",$1);
        b=gensub(/([0-9]+)\"/,"\\1","g",$2); 
        gsub(/[0-9]+$/,a+11,$1);
        gsub(/^[0-9]+/,b+11,$2);
        print $1,$2
}'                        
(bookmarks
("Chapter 1 Introduction 12" "#12"
("1.1 Problem Statement and Basic Definitions 34" "#13")
("Exercises 42" "#41")
("Notes and References 53" "#45"))
)

EDIT2 based on @Tim 's comment

(1) Does FS=OFS="\" \"#" mean the separator of field in both input and output is double quote, space, double quote and #? Why specify double quote twice?

You are right for the separator in both input and output part. It defined separator as:

" "#

There are two double quotes, because it is easier to catch the two numbers you want (based on your example input).

(2) In /.* ([0-9]+)$/, does $ mean the end of the string?

Exactly!

(3) In the third argument of gensub(), what is the difference between "g" and "G"? there is no difference between G and g. Check this out:

gensub(regexp, replacement, how [, target]) #
    Search the target string target for matches of the regular expression regexp. 
    If "how" is a string beginning with ‘g’ or ‘G’ (short for “global”), then 
        replace all matches of regexp with replacement.

This is from http://www.gnu.org/s/gawk/manual/html_node/String-Functions.html. you can read to get detailed usage of gensub.

Answer 2

Unlike just about every tool that provides regexp substitutions, awk does not allow backreferences such as \1 in replacement text. GNU Awk gives access to matched groups if you use the match function, but not with ~ or sub or gsub.

Note also that even if \1 was supported, your snippet would append the string +11, not perform a numerical computation. Also, your regexp isn't quite right, you're matching things like "42"" and not "#42".

Here's an awk solution (warning, untested). It only performs a single replacement per line.

awk '
  match($0, /"#[0-9]+"/) {
    n = substr($0, RSTART+2, RLENGTH-3) + 11;
    $0 = substr($0, 1, RSTART+1) n substr($0, RSTART+RLENGTH-1)
  }
  1 {print}'

It would be simpler in Perl.

perl -pe 's/(?<="#)[0-9]+(?=")/$1+11/e'

Answer 3

awk can do it, but it isn't direct, even using backreferencing.
GNU awk has (partial) backreferecing, in the form of gensub.

Instances of 123" are temporarily wrapped in \x01 and \x02 to mark them as unmodified (for sub(). co

Or you could just step through the loop changing candidates as you go, in which case, the backreferencing and "brackets" aren't needed; but keeping track of the character index is needed.

awk '{$0=gensub(/([0-9]+)\"/, "\x01\\1\"\x02", "g", $0 )
      while ( match($0, /\x01[0-9]+\"\x02/) ) {
        temp=substr( $0, RSTART, RLENGTH )
        numb=substr( temp, 2, RLENGTH-3 ) + 11
        sub( /\x01[0-9]+\"\x02/, numb "\"" ) 
      } print }'

---

Here is another way, using gensub and array split and \x01 as a field delimiter (for split).. \x02 marks an array element as a candidate for the arithmetic addition.

awk 'BEGIN{ ORS="" } {
     $0=gensub(/([0-9]+)\"/, "\x01\x02\\1\x01\"", "g", $0 )
     split( $0, a, "\x01" )
     for (i=0; i<length(a); i++) { 
       if( substr(a[i],1,1)=="\x02" ) { a[i]=substr(a[i],2) + 11 }
       print a[i]
     } print "\n" }'

Answer 4

Since solutions in (g)awk seem to become quite complex, I wanted to add an alternative solution in Perl:

perl -wpe 's/\d+(?=")/$&+11/eg' < in.txt > out.txt

Explanation:

• Option -w enables warnings (which will warn you of possible unwanted effects).
• Option -p implies a loop around the code that works similar to sed or awk, saving each line of input automatically in the default variable, $_.
• Option -e tells perl that the program code is following on the command line, not in a script file.
• The code is a regex substitution (s/.../.../) on $_, where a sequence of digits, if it is followed by a ", will be replaced by the sequence, interpreted as a number in the addition, plus 11.
• The zero-width positive look-ahead assertion (?=pattern) looks for the " without taking it into the match, so we do not have to repeat it in the replacement. The MATCH variable $& in the replacement will then contain only the number.
• The /e modifier to the regex tells perl to "execute" the replacement as code instead of taking it as a string.
• The /g modifier makes the replacement "global", repeating it on every match in the line.

The MATCH variable $& will unfortunately be detrimental to code performance in Perl versions before 5.20. A faster (and not much more complex) solution would use grouping and the backreference $1 instead:

perl -wpe 's/(\d+)?="/$1+11/eg' < in.txt > out.txt

And if the look-ahead assertion looks too confusing, you could also replace the quotation mark explicitely:

perl -wpe 's/(\d+)"/$1+11 . q{"}/eg' < in.txt > out.txt