0

I'm writing a script that uses awk to parse JSON data from MediaWiki API pages in order to retrieve information from Wikipedia tables. This is the sample I'm working with, that's being piped into awk.

What I intended was:

  • to substitute the \n text occurences with an actual line-break
  • remove the double square brackets that sorround some of the entries, as well as everything up to the single vertical bar that divides some entries
  • substitute all double vertical bars || with a single one, so as to use it as a Field Separator
  • remove the leading vertical bar at the start of every line
  • print a given field, deleting empty lines and leading whitespace

Now, here's the issue: I've managed to accomplish this, but through piping different awk instances, in this really ugly way. Here's what I've got so far:

curl -s 'https://en.wikipedia.org/w/api.php?action=parse&prop=sections&page=List_of_islands_of_Spain&section=1&prop=wikitext&format=json' |\
    awk 'BEGIN { FS = "|" }\
    gsub (/\\n/, "\n") gsub (/\[\[[^\|]*\||\]\]/, "")\
    gsub (/\|\|/, "|")' |\ # Sub. "\n" for line-break, remove "[[" and "]]", substitute "||" for "|"
    awk 'gsub (/^\|/, "")' |\ # Remove leading "|"
    awk 'BEGIN { FS = "|" } {print $5}' |\ # Print 5th field
    awk '{gsub (/^[ \t]*/, "")} NF' # Remove any leading whitespace and delete empty lines

I'm aware that I could have used sed and cut for the last three instances, but I'm trying to use this script to develop my awk skills.

Now, one thing that I eventually noticed is that the string manipulation done in the first instance, even though it alters the output, doesn't change the NR nor the NF. I guess that this is the root of the issue I'm having, but I'm not sure how to work arount it.

So this is what I'd like to know:

Can you (and how could I) "chain" all of these operations in a single awk instance? Something like "piping" between these operations inside of awk?

Thanks in advance to everyone that replies.

3
  • Not really. You can of course set up a (named) pipe and loop back your output (as in my answer here), but that has very little practical value in your case. I suggest you start by extracting the raw string (with \n replaced with newlines, etc) with curl ... | jq -r '.parse.wikitext["*"]' | .... You could also do many text manipulation with jq itself, which does have the internal "pipes" you're looking for in awk.
    – user414777
    Jan 15 at 20:47
  • Please edit your question to contain a minimal, complete, verifiable example with concise, testable sample input and expected output. No links, no images.
    – Ed Morton
    Jan 15 at 23:31
  • If the answer I posted helped you then please see unix.stackexchange.com/help/someone-answers for what to do next. If there's anything about it you don't understand then please feel free to ask questions as comments under my answer.
    – Ed Morton
    Jan 18 at 15:03
0

I'm not going to address the wisdom of using awk to parse JSON (unless you use gawks JSON library) but I will address how to convert your shell pipeline of awk calls:

awk 'BEGIN { FS = "|" }\
gsub (/\\n/, "\n") gsub (/\[\[[^\|]*\||\]\]/, "")\
gsub (/\|\|/, "|")' |\ # Sub. "\n" for line-break, remove "[[" and "]]", substitute "||" for "|"
awk 'gsub (/^\|/, "")' |\ # Remove leading "|"
awk 'BEGIN { FS = "|" } {print $5}' |\ # Print 5th field
awk '{gsub (/^[ \t]*/, "")} NF' # Remove any leading whitespace and delete empty lines

into a single awk command.

awk is a C-like programming language, it's not shell-like in syntax or semantics. You wouldn't be looking at how to pipe C statements to each other within a C program and similarly you wouldn't do that in an awk program.

Try this:

$ cat tst.awk
BEGIN { FS = "|" }
{
    gsub(/(\[\[[^|]*\|)|(]])/, "")
    gsub(/\|\|/, FS)
    split($0,lines,/\\n/)
    for (i=1; i in lines; i++) {
        $0 = lines[i]
        sub(/^[[:space:]]+/, "", $6)
        if ( $6 !~ /^$/ ) {
            print $6
        }
    }
}

curl -s 'https://en.wikipedia.org/w/api.php?action=parse&prop=sections&page=List_of_islands_of_Spain&section=1&prop=wikitext&format=json' |
awk -f tst.awk

Province
Isla de \u00cdzaro
Garraitz
Santa Clara
Aqueche
Txatxarramendi
Villano
Montehano
Santa Marina o Los Jorganes
Pedrosa
Virgen del Mar
Castril, Am\u00edo o M\u00edo, Las Lastras de Pech\u00f3n
La Pasiega o Solita
La Torre
Ratones o Marnay
Neptuno Ni\u00f1o
Ori\u00f1\u00f3n
Castro
Cuarezo
Llera
\u00c1guila
Suaces
Garfanta
Deva
Pantorgas
Isla Herbosa
Isla del Carmen
Illa de Arousa
Ons
La Toja Peque\u00f1a
Ansar\u00f3n
Guidoiro Areoso
A Creba
Lobeiras
Centoleiras
Beiro
Farall\u00f3ns
Guidoiro Pedregoso
Malveiras
Isla de Santa Cruz
Isla Herbosa
San Clemente
San Vicente
San Ant\u00f3n (Pontevedra)
San Ant\u00f3n (La Coru\u00f1a)
Pancha
Gavoteira
Isla de Santa Catalina
Isla Canela
Isla de Salt\u00e9s
Las Palomas
Trocadero
Sancti Petri
San Andr\u00e9s
Terreros
Isla Negra
Albor\u00e1n
San Sebasti\u00e1n
Piedra del Hombre
Isla Mayor
Rondella
Las Palomas
Isla de Tabarca
Benidorm
Portichol
Descubridor
Medas
Port Lligat
Encalladora
Cabrera
Isla del Rey

It's worth noting that with GNU awk you could set RS to separate the input at \\ns automatically and then you wouldn't need to split on \\ns inside the script:

$ printf 'foo\\\\nbar\n'
foo\\nbar

$ printf 'foo\\\\nbar\n' | awk '{split($0,lines,/\\\\n/); for (i=1; i in lines; i++) print i, lines[i]}'
1 foo
2 bar

$ printf 'foo\\\\nbar\n' | awk 'BEGIN{RS="[\\\\]{2}n|\n"} {print NR, $0}'
1 foo
2 bar

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.