Using variables in gawk for the repeat interval value of an interval expression

Question

OS Distribution: Ubuntu 22.04.3 LTS
gawk version: GNU Awk 5.1.0, API: 3.0 (GNU MPFR 4.1.0, GNU MP 6.2.1)

I have a text file that every once in a while a line will have a significant amount of white space followed by some random text. I am using gawk to search for these rows and make a modification to the left portion of the line.

Input:

formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text
formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text
formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text
                                                                           random_text
formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text
formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text
                                                                           random_text
formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text

Expected Output:

formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text
formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text
formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text
                WORDWRAP                                                   random_text
formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text
formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text
                WORDWRAP                                                   random_text
formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text

This command works and returns the expected output:

gawk '/^[[:space:]]{75}/ { $0 = substr($0,1,15) " WORDWRAP " substr($0,26) }1' input.txt

What I want to be able to do is assign variables to the repeat interval on the interval expression and to the start, length values on the substr function as these values can change depending on the input file.

I set three environment variables:

export PH1="75"; export PH2="15"; export PH3="26"

and then attempt to run this command:

gawk -v gph1="${PH1}" -v gph2="${PH2}" -v gph3="${PH3}" '/^[[:space:]]{gph1}/ { $0 = substr($0,1,gph2) " WORDWRAP " substr($0,gph3) }1' input.txt

which just returns the input unmodified. If I set the repeat interval to the actual value and leave the variables on the substr start and length values:

gawk -v gph2="${PH2}" -v gph3="${PH3}" '/^[[:space:]]{75}/ { $0 = substr($0,1,gph2) " WORDWRAP " substr($0,gph3) }1' input.txt

It works and returns the expected output.
I've have also tried this without success as it just returns the input unmodified:

gawk '/^[[:space:]]{ENVIRON["PH1"]}/ { $0 = substr($0,1,ENVIRON["PH2"]) " WORDWRAP " substr($0,ENVIRON["PH3"]) }1' input.txt

However, this does work when setting the repeat interval to the actual value:

gawk '/^[[:space:]]{75}/ { $0 = substr($0,1,ENVIRON["PH2"]) " WORDWRAP " substr($0,ENVIRON["PH3"]) }1' input.txt

Is there a way to use variables for the repeat intervals in an interval expression?

(Added 2023-09-30) The answer to this question is yes, but not in a regexp constant. Based on the feedback received, I ended up changing the command to this:

gawk -v gph1="${PH1}" -v gph2="${PH2}" -v gph3="${PH3}" ' $0 ~ "^[[:blank:]]{" gph1 "}" { $0 = substr($0,1,gph2) " WORDWRAP " substr($0,gph3) }1' input.txt

Answer 1

You can build the desired regex in the BEGIN{...} block, eg:

BEGIN { regex = "^[[:space:]]{" gph1 "}" }

Then in the main script you compare the input line ($0) to the regex, eg:

# replace this:

/^[[:space:]]{gph1}/

# with this:

$0 ~ regex

Rolling these changes into your current gawk script:

gawk -v gph1="${PH1}" -v gph2="${PH2}" -v gph3="${PH3}" '
BEGIN      { regex = "^[[:space:]]{" gph1 "}" }
$0 ~ regex { $0 = substr($0,1,gph2) " WORDWRAP " substr($0,gph3) }
1
' input.txt

This generates:

formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text
formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text
formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text
                WORDWRAP                                                   random_text
formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text
formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text
                WORDWRAP                                                   random_text
formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text

Answer 2

Using any POSIX awk:

$ cat tst.sh
PH1='75'; PH2='15'; PH3='26'

awk -v gph1="$PH1" -v gph2="$PH2" -v gph3="$PH3" '
    $0 ~ "^[[:space:]]{"gph1"}" { $0 = substr($0,1,gph2) " WORDWRAP " substr($0,gph3) }1
' input.txt

$ ./tst.sh input.txt
formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text
formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text
formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text
                WORDWRAP                                                   random_text
formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text
formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text
                WORDWRAP                                                   random_text
formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text

The above is using a dynamic (aka "computed") regexp instead of a literal (aka "constant") regexp, see https://www.gnu.org/software/gawk/manual/gawk.html#Computed-Regexps.

You shouldn't really pass in a hard-coded gph3, though, as that position depends on the length of the replacement text so you should instead pass in the replacement text and calculate gph3 from it's length so you can change the replacement text as you like without having to manually recalculate gph3 when you do so:

$ cat tst.sh
PH1='75'; PH2='15'; new=' WORDWRAP '

awk -v gph1="$PH1" -v gph2="$PH2" -v new="$new" '
    BEGIN { gph3 = gph2 + length(new) + 1 }
    $0 ~ "^[[:space:]]{"gph1"}" { $0 = substr($0,1,gph2) new substr($0,gph3) }1
' input.txt

But, back to the original question....

You could alternatively store the dynamic regexp in a variable so you're only doing string concatenation to construct the regexp once at the start of the script executing rather than every time an input line is read:

awk -v gph1="$PH1" -v gph2="$PH2" -v gph3="$PH3" '
    BEGIN { re = "^[[:space:]]{"gph1"}" }
    $0 ~ re { $0 = substr($0,1,gph2) " WORDWRAP " substr($0,gph3) }1
' input.txt

and you could define that re variable on the command line rather than inside the script if you prefer:

awk -v re="^[[:space:]]{$PH1}" -v gph2="$PH2" -v gph3="$PH3" '
    $0 ~ re { $0 = substr($0,1,gph2) " WORDWRAP " substr($0,gph3) }1
' input.txt

With GNU awk you could alternatively define a variable to contain a strongly typed regexp constant and then use that:

$ cat tst.sh
PH1="75"; PH2="15"; PH3="26"

gawk -v re="@/^[[:space:]]{$PH1}/" -v gph2="$PH2" -v gph3="$PH3" '
    $0 ~ re { $0 = substr($0,1,gph2) " WORDWRAP " substr($0,gph3) }1
' input.txt

$ ./tst.sh input.txt
formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text
formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text
formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text
                WORDWRAP                                                   random_text
formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text
formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text
                WORDWRAP                                                   random_text
formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text formatted_text

If you'd prefer to create the strongly typed regexp (re) dynamically inside the script than on the command line you can do it but the syntax looks a bit kludgy IMO as, unlike strings, there is no concatenation operator for strongly typed regexps but, like strings, you can do a *sub() on them and they'll retain their type so either of these would work (x can be whatever char or string you like, it's just a placeholder for the sub() to act on):

gawk -v gph1="$PH1" -v gph2="$PH2" -v gph3="$PH3" '
    BEGIN { re = @/^[[:space:]]{x}/; sub(/x/,gph1,re) }
    $0 ~ re { $0 = substr($0,1,gph2) " WORDWRAP " substr($0,gph3) }1
' input.txt

gawk -v gph1="$PH1" -v gph2="$PH2" -v gph3="$PH3" '
    BEGIN { re = @/x/; sub(/x/,"^[[:space:]]{"gph1"}",re) }
    $0 ~ re { $0 = substr($0,1,gph2) " WORDWRAP " substr($0,gph3) }1
' input.txt

There's no benefit to using a strongly typed regexp over a regular dynamic regexp for the OPs case, I'm just showing them here as they are an option and can be useful in other contexts, see the gawk manual.

The only tiny benefit to a strongly typed regexp in THIS case would be if the OP wanted to use \s instead of [[:space:]] in their regexp then they wouldn't have to remember to add an extra backslash to \s (consumed when a string containing a dynamic regexp is internally converted to a regexp):

$ echo 'foo bar' | gawk -v re='\s' '$0 ~ re'
gawk: warning: escape sequence `\s' treated as plain `s'

$ echo 'foo bar' | gawk -v re='\\s' '$0 ~ re'
foo bar

$ echo 'foo bar' | gawk -v re='@/\s/' '$0 ~ re'
foo bar

Answer 3

You can't use awk var or envvar (or any other nonliteral) for {repeat} in a constant regexp, but you can instead test only the first var columns:

substr($0,1,ph1) ~ /^[[:space:]]+$/
# or equivalent but perhaps confusing
substr($0,1,ph1) !~ /[^[:space:]]/
# ENVIRON["PH1"] if you don't make it awk var

But do you really want to match characters like HT FF VT CR here? From your description of the data, I think you only want to match actual space characters, not the space character class which includes much more. For that

substr($0,1,ph1) == sprintf("%.*s",ph1,"")

or for efficiency

BEGIN{ ph1spaces = sprintf("%.*s",ph1,"") } substr($0,1,ph1)==ph1spaces { do change }

Alternatively you can use a dynamic regexp, but that has to recompile every time

$0 ~ sprintf("^[[:space:]]{%d}", ph1)
# or for actual space only
$0 ~ sprintf("^ {%d}", ph1)

And although you only asked for gawk, these are not gawk-specific and should work in any POSIX awk.

Answer 4

I would use perl instead of gawk which would make it more portable, easier to read and more reliable, and would allow to edit the file in place with -i¹:

perl -lpse 'substr($_, $offset, length($text)) = $text if /^\s{$spaces}/
  ' -- -offset=14 -text=WORDWRAP -spaces=75 your-file

(offsets for perl's substr() start at 0, hence the 14 instead of 15).

---

^{¹ recent versions of gawk come with an inplace.awk extension, but it's hard to use safely}