I have the following bash script:

#!/usr/bin/env bash

grep -e '^[a-zA-Z]\{4,8\}$' data/words3.txt | tr '[:upper:]' '[:lower:]' | sort -u > data/passphrase-words.txt

function wordfrequency() {
  awk '{ for (i=1; i<=NF; i++) { word = tolower($i); words[word]++ } } END { for (w in words) printf("%3d %s\n", words[w], w) } ' | sort -rn

function getArticleText() {
  awk '/<text xml:space="preserve">/,/<\/text>/' | sed 's/<.*>//'

function reduceWikiText() {
  tr ' [:punct:]' '[\n*]' | sed '/^$/d' | tr '[:upper:]' '[:lower:]'

bzcat data/enwiki-20161020-pages-articles.xml.bz2 | getArticleText | reduceWikiText | grep -F -f data/passphrase-words.txt | wordfrequency > data/wordFreqs.txt

I'm sure it could be simplified in a number of ways, but it's what I came up with. data/passphrase-words is a list of some ~170k words, one word per line. data/enwiki-* is 12GB of compressed XML (it's a Wikipedia dump). From there, getArticleText grabs the text from each article, reduceWikiText "reduces" that text to one word per line, stripping out all the xml and punctuation, and wordfrequency counts how often each word comes up.

If I'm reading my task manager correctly, the gawk inside wordFrequency() is using massive amounts of memory; 695MB, if I let it run long enough, over 1GB of RAM.

The grep command that's not part of any function limits the number of different words gawk will see, and it takes up a constant 36 MB. I could see gawk needing 50MB or even 100MB, but over 1GB? That seems wrong. And the growth rate implies it will grow unbounded.

I need to figure out why gawk is using this much memory? Due to the size of the BZ2 file, I can't afford to let gawk go too far out of control...

And the reason I'm not using sort | uniq -c | sort -nr is because I really want the word counting aggregation to occur in-memory; I know it can fit, for the number of words I'm dealing with. Less HDD usage = faster, right?

For reference, Linux Subsystem for Windows, and:

$ gawk --version
GNU Awk 4.0.1
Copyright (C) 1989, 1991-2012 Free Software Foundation.

This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program. If not, see http://www.gnu.org/licenses/.

EDIT: Posted what I've got (minus the 12GB .xml.bz2 file) at https://github.com/proegssilb/wikipedia-wordcount. Using mawk, as per the suggestions in the comments, did not seem to do anything, but I stopped the process at 200MB of RAM. Will let the process run overnight without using awk at all, just to see what happens.

EDIT2: After substituting in | sort | uniq -c for the problematic awk, the process completed within the 6-7 hours I was out. I'm going to make some further tweaks to try and strip out HTML usage in articles (getting rid of """ causing so much pollution), and time it again, but at least it runs in a "reasonable" amount of time now.

  • There are a couple of things I would try ... 1) How many distinct words do you have? You could do something like | tr ' \t' '\n' | sort | uniq | wc -l 2) Is the memory usage substantially less for a different awk implementation like mawk? Nov 13, 2016 at 5:57
  • @GregoryNisbet, the OP already told us he had 170k words. I can't yet reproduce this problem using gawk version 4.1.3 on a 64bit linux system. I need about 60MB to process a word list of 99k entries.
    – icarus
    Nov 13, 2016 at 6:07
  • @GregoryNisbet I would need to use | sort | uniq -c in order to get per-word counts, and that would thrash the HDD. Not that I'm getting a lot of HDD traffic that I can measure right now, but still makes me nervous. See also, the question edit. @icarus It may be an issue with how Microsoft did their Linux-on-Windows mess, but I don't have a Linux machine lying around right now.
    – xenrelay
    Nov 13, 2016 at 7:39
  • 1
    Your grep will also accept a word that contains as a SUBSTRING one of the 170k words; add -x for exact only. Or, you could let awk do the filtering as well as the counting, which you can simplify because the words are already one per line and lowercase: awk 'BEGIN{while((getline w <"wantedwords")>0) words[w]=0} $0 in words {++words[$0]} END{for(w in words)if(words[w])printf...}' With gawk 4, you could set PROCINFO["sorted_in"]="@val_num_desc" and skip the separate sort but I don't know if it's any faster. ... Nov 13, 2016 at 8:52
  • ... With Windows, there might be CRs in your data, and if so awk treats fredCR as a different string from fred. Also with XML there could be TAB or C1 controls; to be on the safe side I would use the more robust and simpler tr -sc '[:alpha:]' '\n' to just remove everything except consecutive letters and suppress empty lines so you don't need the sed '/^$/d' . Nov 13, 2016 at 8:58

1 Answer 1


So, there were a couple of things that helped, but the main thing that got this working was using sort | uniq -c instead of gawk, as per Gregory Nisbet.

I also wound up using @dave_thompson_085 's comment about tr -sc '[:alpha:]' '\n'. The flag -s combines repeats, meaning I don't have to delete blank lines, and -c inverts the set of characters to find. A side-effect of -c is you can only use one replacement character, instead of a set. Also thanks to Dave for the catch about grep and exact line matches (-x). If I had the reputation to upvote that comment, I would.

Lastly, I had to use some extra code to remove XML entities ( &quot; ) and strip out html (excess <ref />). In getArticleText, the new sed command is | sed -e 's/&quot;/"/g' -e 's/&lt;/</g' -e 's/&gt;/>/g' -e 's/&amp;/&/g' -e 's/<.*>//g'. Each expression (-e to chain commands) deals with a different HTML entity. I tried some more complete options (like using perl as per StackOverflow), but they didn't work in my case due to machine-specific issues. The final script can be seen on my wordcount repo.

The script completed in 3hrs20min on my machine, but it's also a 6-core AMD from years ago with a HDD. Your mileage may vary, but this was good enough for me.

I'll avoid accepting this answer so that if @Gregory Nisbet or @dave_thompson_085 want to post their own answer, they can.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .