0

From this JSON-HTML webpage I want to filter out only lines with this pattern:

"realname": "SOME_VALUE_COMES_HERE",

I tried this:

curl https://www.mediawiki.org/w/api.php?action=query&meta=siteinfo&siprop=specialpagealiases&formatversion=2 -s |
  grep -oP '"realname": "[A-Za-z0-9]*",' |
  sort -u > special_page_names.txt

A file is created but it's empty. What have I done wrong in the grep -oP command?

Update

I don't use jq, I can install it but I would prefer an "all native" approach.

4
  • 1
    Is parsing the HTML really necessary, or would using the JSON-formatted data itself suffice? Does it need to be with regular expressions, or would a JSON-targetted tool like jq suffice? You seem to be looking for text that does not exist in the source when I retrieve that page; is that something you've examined? You can edit your question. Mar 14, 2022 at 3:44
  • Parsing the HTML isn't necessary as long as I get JSON as-is (as a JSON document per se). I prefer regular expressions over jq (or any non native utility in Bash). I did get HTML source after hitting CTLR+U.
    – Lahor
    Mar 14, 2022 at 9:22
  • & and ? are special characters in the syntax of the shell. That URL should be quoted. Also, you should really get used to putting options (-s) before non-option arguments. Mar 14, 2022 at 9:39
  • I don't understand the "all native" remark. On a Linux system, jq is installed as a package, just like grep is. Both are "native", and jq is additionally a JSON parser which means it is able to work with the structure of any JSON document. grep, on the other hand, is restricted to working with line-based data while JSON documents don't care about newline characters (so two documents could be equivalent even though one is a single line and the other is thousands of lines).
    – Kusalananda
    Apr 15, 2022 at 16:58

3 Answers 3

1

The webiste provides format=json option, so you can use jq:

curl 'https://www.mediawiki.org/w/api.php?action=query&meta=siteinfo&siprop=specialpagealiases&formatversion=2&format=json' \
| jq -r '.query | .specialpagealiases[].realname'
0

First, you need to put the url in quotation marks so the shell doesn't interpret the '&' as the command to put the process into the background.

Next, that curl command does not return JSON. It returns HTML which represents what the JSON looks like when it's pretty printed. But it says at the top that you can use &format=json to get the actual JSON.

Next, when you get the JSON using &format=json it is not pretty printed, so you need to take out the space after the ':'.

curl 'https://www.mediawiki.org/w/api.php?action=query&meta=siteinfo&siprop=specialpagealiases&format=json' -s | grep -oP '"realname":"[A-Za-z0-9]*",' | sort -u > special_page_names.txt 

If you want to remove the "realname": and the quotes around the results, and the commas at the end:

curl 'https://www.mediawiki.org/w/api.php?action=query&meta=siteinfo&siprop=specialpagealiases&format=json' -s | grep -oP '"realname":"[A-Za-z0-9]*",' | sed -E 's/"realname":"([A-Za-z0-9]*)",/\1/' | sort -u > special_page_names.txt 

This should only be used as a quick and dirty method. If you want something robust, use a real JSON parser, or at least jq.

-1

Python is your easiest solution here.

Script

import sys
import requests


url = sys.argv[1]
response = requests.get(url)
if response:
    if not response.ok:
        print('Bad response: {0!r}'.format(response))
        sys.exit(1)
    else:
        data = response.json()
        query = data.get("query", None)
        if query:
            special_page_aliases = query.get("specialpagealiases", None)
            if special_page_aliases:
                for spa in special_page_aliases:
                    real_name = spa.get("realname", None)
                    if real_name:
                        print(spa.get("realname"))
    sys.exit(0)
print('Bad request')
sys.exit(1)

Usage

python script.py "https://www.mediawiki.org/w/api.php?action=query&meta=siteinfo&siprop=specialpagealiases&format=json"

The script will simply dump all the "realname" values under the key "query.specialpagealiases" to the console.

Edit: I know OP is looking for a native solution; I just want to pre-emptively point out that python is native now.

1
  • Downvoted to oblivion with zero comments as to why..
    – Rowshi
    May 26, 2022 at 22:14

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .