2

awk is the swiss army knife of text manipulation. If, however, I need to change small bits in a text, I reach out to sed generally. While it might be the best tool for the job, it is worth it to know how to do such simple tasks with another tool. How would I use awk as a stream editor replacement for sed?

In particular, with the following file text:

Comparing apples with oranges.
Comparing rockets with bicycles.

How can achieve the following result with awk:

sed 's/apples/fruit/' text
sed 's/apples\|oranges/fruit/g' text

As a bonus, how can I alter a variable in awk with those functions?

7

A stream editor is a special type of filter. A filter is a program that takes text on standard input, does some magic, and spits it out on standard output. grep, and basically all of the coreutils are filters. A stream editor is a special type of filter: It applies one or more editing commands on the incoming text.

In awk, the following three functions can be used: sub, gsub, and gensub with the following synopsis:

sub(regexp, replacement [, target])
gsub(regexp, replacement [, target])
gensub(regexp, replacement, how [, target])

In all these three functions, if target is omitted, the current line ($0) is assumed.

sub and gsub

We'll first look at sub.

$ awk '{rt = sub(/apple|orange/, "fruit"); print rt, $0}' text 
1 Comparing fruits with oranges.
0 Comparing rockets with bicycles.

Here, the return value of the sub()-function is stored in rt. The regular expression /apple|orange/, meaning to match either apple or orange is applied once. Nothing will happen after the the call to sub, but in the background, the current line has changed, and the return value has a value.

That the return value is 0 when no changes have been made, means that if the sub was applied outside the {action}, can be used to emulate sed.

$ awk 'sub(/apple|orange/, "fruit")' text    
Comparing fruits with oranges.

Now, as only the first line has changed, only the first line has been printed. Remember, the action if none is specified, is to print the line.

To emulate sed 's/apple/fruit/' text, one can write:

$ awk 'sub(/apple|orange/, "fruit") || 1' text
Comparing fruits with oranges.
Comparing rockets with bicycles.

Now, the first function will be tried. If something has been replaced, the return value is non-zero, and the line is printed. If nothing has been replaced, the second test of PATTERN will be tried, which happens to be always non-zero, namely 1. As a result, the (unmodified) line will be printed.

An alternative way of writing the same, and possibly more idiomatic, would be:

$ awk '{sub(/apple|orange/, "fruit")};1' text 
Comparing fruits with oranges.
Comparing rockets with bicycles.

Here, it is attempted to change the current line in the first ACTION-block. The return code of sub will be silently ignored. Nothing will be printed. The second PATTERN{ACTION}-block (1), always matches, and the default action idf top print it, whether it is a modified or an unmodified line.

You have noticed that the second match of the first line, orange does not get replaced. One solution would be to wrap the sub-function in a while loop:

$ awk '{while (sub(/apple|orange/, "fruit")){}};1' text
Comparing fruits with fruits.
Comparing rockets with bicycles.

As long as sub returns a non-zero value, sub will be repeated. As a handy shorthand for this, and as the while-loop does not work in a PATTERN, a function gsub was introduced.

$ awk 'gsub(/apple|orange/, "fruit")' text             
Comparing fruits with fruits.

This means that the famous sed 's/regex/replacement/g' can be emulated in awk like this:

awk '{gsub(/apple|orange/, "fruit")};1' text

gensub: No side effects

WARNING: gensub is not in the POSIX awk standard, and might not be available on your installation. It is available in gawk, busybox awk, but not in mawk and nawk.

These mechanics already show a bit of how working with variables work. The variable changed in place.

$ awk '{a=$0; rt=sub(/apple|orange/, "fruit", a); print rt, a, $0}' text
1 Comparing fruits with oranges. Comparing apples with oranges.
0 Comparing rockets with bicycles. Comparing rockets with bicycles.

This might not be what you want. A sound principle in computing is to not work on the input itself, but to work on a copy of the input. What if you do not want to alter the input, but assign the result of the replacement to a new variable? Enter gensub.

$ awk '{rt=gensub(/apple|orange/, "fruit", "g"); print rt, $0}' text
Comparing fruits with fruits. Comparing apples with oranges.
Comparing rockets with bicycles. Comparing rockets with bicycles.

Here, the return value is not a return value, but the resulting string is assigned to the variable rt. The fourth argument is now the default, $0.

The third argument to gensub is how. The sensible values for this argument is "g" or "G", which stands for global. This will change all occurrences of /regex/ with the replacement string. One can also specify a positive integer i, where the i-th occurrence will be replaced.

$ gawk '{print gensub(/apple|orange/, "fruit", 1)}' text
Comparing fruits with oranges.
Comparing rockets with bicycles.

$ gawk '{print gensub(/apple|orange/, "fruit", 2)}' text
Comparing apples with fruits.
Comparing rockets with bicycles.

$ gawk '{print gensub(/apple|orange/, "fruit", 3)}' text
Comparing apples with oranges.
Comparing rockets with bicycles.

$ gawk '{print gensub(/apple|orange/, "fruit", "g")}' text
Comparing fruits with fruits.
Comparing rockets with bicycles.

If how is not a positive integer, or not a string starting with G or g, gawk will issue a warning.

Note that another idiomatic use of gensub has been made: directly print the result of the substitution. The last form also doubles as a replacement for sed 's/regex/replacement/g' command.

Doing more with the replacement string

So far, we have done some straight forward string replacing. What if you want to modify the matched string?

There are some special variables that capture the matched text. With the POSIX-conform sub and gsub, one can repeat the matched part with a &:

$ awk '{rt=gsub(/apple|orange/, "a basket of &"); print rt, $0}' text
2 Comparing a basket of apples with a basket of oranges.
0 Comparing rockets with bicycles.

The fancy thing known from sed and perl/PCRE with numbered matches is too modern for the sub and gsub variants. gensub can do the same with &, but a bit more when you use grouping in the regex to specify your regex:

$ awk '{rt=gensub(/(appl|orang)(e)/, "a basket of \\1\\2","g"); print rt}' text
Comparing a basket of apples with a basket of oranges.
Comparing rockets with bicycles.

The TL;DR

Use sub and gsub for quick-and-dirty tasks:

  • When you want to change a variable immediately, and do not care about its old value
  • When you wish to base an action on whether a substitution is made, by utilizing the return code

Use gensub in all other cases:

  • It provides for a more elaborate back-references in replacement strings
  • If you want to keep the original variable intact
  • If you want to assign the result to a variable
4
  • 2
    Recent GNU awk versions also have the -iinplace option which acts like sed -i.
    – terdon
    Apr 26 '16 at 9:21
  • @JJoao: Yes. You can use @load "inplace" in your awkscript, or -i inplace as a command line argument.
    – joepd
    Apr 26 '16 at 12:25
  • @JJoao no, the space is optional. Both awk -i inplace and awk -iinplace work. I like the latter since it's shorter but it's a matter of choice. FOr documentation, see gnu.org/software/gawk/manual/html_node/….
    – terdon
    Apr 26 '16 at 12:50
  • @terdon, sorry you are completely right . No more wine to my glass (I was testing in pre-historic old machine without noticing).
    – JJoao
    Apr 26 '16 at 12:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.