8

This awk expression prints inet 34.45 as expected on OpenBSD:

echo "inet 34.45" | awk '/inet [0-9]+\./ { print }'

However, when I replace the + with a bound {1,3}, I am not getting any match:

echo "inet 34.45" | awk '/inet [0-9]{1,3}\./ { print }'

Both expressions work as expected on Linux with gawk. The gawk man page mentions that what it calls interval expressions were not originally supported by awk but later added to POSIX for consistency with egrep. The awk man page on OpenBSD mentions no such thing and just refers to the man page of re_format, which specifies bounds as usual.

Is this a bug or some undocumented limitation of OpenBSD awk?

Improve this question
3
  • 1
    If gawk works, why not use it?
    – ajgringo619
    Jan 12, 2020 at 23:34
  • That is not the point. I can make the OpenBSD awk work just fine for my purpose by simply expanding the regexp to this: [0-9]?[0-9]?[0-9]?. Actually I settled on the more correct expression [1-9][0-9]?[0-9]?. That does not answer however why OpenBSD awk does not work as documented.
    – Fabian Ritzmann
    Jan 12, 2020 at 23:40
  • 1
    I ran into similar problems with AIX utilities and just gave up & installed the GNU versions. It is weird that the BSD docs don't mention the change, though.
    – ajgringo619
    Jan 12, 2020 at 23:41

2 Answers 2

Reset to default
13

That restriction is precisely documented.

From: http://man.openbsd.org/awk.1#STANDARDS

STANDARDS

The awk utility is compliant with the IEEE Std 1003.1-2008 (“POSIX.1”) specification, except awk does not support {n,m} pattern matching.

Improve this answer
1
  • 1
    Oh, great, I had read the man page multiple times but not what seemed like boiler plate at the very end. (－‸ლ)
    – Fabian Ritzmann
    Jan 12, 2020 at 23:54
2

I cannot vouch for the OpenBSD people, but a reason why bounds / interval expressions are not supported in OpenBSD's and most other awk implementations may be because they're a horrible botch, implementation-wise.

I'll start with a testcase, using GNU awk (gawk), which supports them:

time echo | gawk '/a{1,30000}/'
  # still going strong, after 5 minutes with the CPU at 100%
  # and eating up > 4G of memory

Because awk is using real regular expressions (the state-machine / finite automaton kind, NOT the recursive / backtracking space & time unbounded kind from perl), counted repetitions could only be implemented by statically repeating the regex subexpression in code, as many times as necessary.

A regex like a{1,4} is actually turned into something like a(a(aa?)?)? at compile time. You can easily see how nasty this is with anything but very small repetition counts: even a puny /a{1,500}/ will take half a second and many MBs of memory.

Despite being mandated by POSIX, as of April 2020, interval expressions are NOT supported in the default awk from Debian 10 (Buster), OpenBSD 6.6 and FreeBSD 12.1, and the /usr/bin/nawk from Solaris 11. Except for Debian (which is using mawk) all the other are based on the traditional nawk ("new awk").

Besides GNU awk, awk implementation which do support interval expresssions are the busybox awk, and the default awk from NetBSD and MacOS.

The purported descendant of nawk (bwk, "one true awk") has also recently included support for interval expressions, a move which was IMHO absolutely NOT well though-out.

POSIX has a history of mandating broken requirements for awk, that then have to be retired; an example of this is the requirement that 0=="000" should evaluate to 1 (true), which was dropped in the current version of the standard, but which unfortunately has made its way into the /usr/xpg4/bin/awk from Solaris, making it unusable.

Improve this answer
9
  • So if we want to print the string that matches a repetition of 1 to 30000 occurrences of a (assuming a could be some random regexp like (foo.(bar|xy?)z), not necessarily a single character), then what should we write instead of match($0,/a{1,30000}/){print substr($0,RSTART,RLENGTH)}? Do you know if grep and sed behave the same way in this regards?
    – Ed Morton
    Apr 13, 2020 at 0:19
  • 1
    Yes, grep and sed work the same as awk. Try with a simple echo a | grep 'a\{1,2000\}' and see long does it take. If I were the maintainer of some awk implementation, I would let it use alternative regex implementations (as they did with grep -P) instead of trying to retrofit it in awk's regex engine. As to how to do (variable length subexpr){1,1000} without intervals -- I have no idea if that's even possible in general. There may be tricks that work in limited, particular cases.
    – user313992
    Apr 13, 2020 at 0:22
  • I tested with GNU grep and GNU sed and they accepted the interval but had the same performance issue as GNU awk. I then tested with OSX/BSD versions of all 3 and awk kept accepting the values til it failed with a seg fault but their grep and sed both rejected with RE error: invalid repetition count(s). So no tool had any problem up to 255 repetitions, above that YMMV but then you really need to ask yourself what you're doing.
    – Ed Morton
    Apr 13, 2020 at 0:22
  • I'm obviously just a sample of 1 but after nearly 40 years of shell programming, including 25+ using awk, and participating in various shell/awk forums and usenet I've never heard of anyone actually having this problem in their real code. Add to that that only a few awks don't support RE intervals, there's no general workaround, you can always install/use an awk that does support them, they're part of the POSIX standard, the performance problem only occurs for large numbers and it's common to all standard UNIX tools, I think we should just file this under "OK" and keep using awk RE intervals.
    – Ed Morton
    Apr 13, 2020 at 0:33
  • 1
    Opinions like they're a horrible botch and making it unusable and a move which was IMHO absolutely NOT well though-out and I think we should just file this under "OK" are perfectly valid on this site, this isn't a deposition so simply listing facts isn't a requirement. You're welcome.
    – Ed Morton
    Apr 13, 2020 at 0:46

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .