1

I use an API to fetch a billion or so records which I run through awk to filter and pipe to wc -l to count the records. I'm currently doing this successfully:

$apiCall | awk '$1=="xx"' | wc -l > file

But I think combining an AWK command will consume less compute than piping to wc -l, something like this:

$apiCall | awk '$1=="xx" && END{print NR}' > file

But this isn't working.

2 Answers 2

3

You may find that:

LC_ALL=C grep -Ec '^[[:blank:]]*xx([[:blank:]]|$)'

is considerably quicker as it avoids the overhead of awk having to interpret a higher level language.

We're using LC_ALL=C to simplify the text interpretation. That also means that only SPC and TAB are understood as delimiter, but that's the case of many awk implementations as well regardless of the locale.

Some timings on a GNU/Linux amd64 multi-core system from zsh:

$ (repeat 3000 printf '%s\n' {{,xx}{1..1000},xx}" blah blah blah")> a
$ wc a
  6003000  24012000 119412000 a
$ time LC_ALL=C grep -Ec '^[[:blank:]]*xx([[:blank:]]|$)' < a
3000
LC_ALL=C grep -Ec '^[[:blank:]]*xx([[:blank:]]|$)' < a  0.15s user 0.03s system 99% cpu 0.175 total
$ time gawk '$1=="xx"{n++};END{print n}' < a
3000
gawk '$1=="xx"{n++};END{print n}' < a  2.00s user 0.05s system 99% cpu 2.055 total
$ time LC_ALL=C gawk '$1=="xx"{n++};END{print n}' < a
3000
LC_ALL=C gawk '$1=="xx"{n++};END{print n}' < a  1.96s user 0.03s system 99% cpu 1.985 total
$ time mawk '$1=="xx"{n++};END{print n}' < a
3000
mawk '$1=="xx"{n++};END{print n}' < a  1.23s user 0.04s system 99% cpu 1.277 total
$ time gawk '$1=="xx"' < a | wc -l
3000
gawk '$1=="xx"' < a  1.91s user 0.05s system 99% cpu 1.967 total
wc -l  0.00s user 0.00s system 0% cpu 1.967 total
2

The idea is right, but the right syntax for doing that in awk could be do to

$apiCall | awk '$1=="xx"{ count++ }END{ print count }'

Since awk is pattern { action } based, once we match the pattern of xx in the first column, we do the action part of increment a counter on each occurrence and when we finish processing the file, in the END clause we print out the final count. Re-direct the output to a file at the end with > file as needed.

2
  • Beautiful, thank you for both the code and lesson!
    – Steve
    Mar 20, 2018 at 12:57
  • 1
    @Steve If this solves your issue, you should accept the answer.
    – Kusalananda
    Mar 20, 2018 at 13:03

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .