18

I have a JSON file members.json as below.

{
   "took": 670,
   "timed_out": false,
   "_shards": {
      "total": 8,
      "successful": 8,
      "failed": 0
   },
   "hits": {
      "total": 74,
      "max_score": 1,
      "hits": [
         {
            "_index": "2000_270_0",
            "_type": "Medical",
            "_id": "02:17447847049147026174478:174159",
            "_score": 1,
            "_source": {
               "memberId": "0x7b93910446f91928e23e1043dfdf5bcf",
               "memberFirstName": "Uri",
               "memberMiddleName": "Prayag",
               "memberLastName": "Dubofsky"
            }
         }, 
         {
            "_index": "2000_270_0",
            "_type": "Medical",
            "_id": "02:17447847049147026174478:174159",
            "_score": 1,
            "_source": {
               "memberId": "0x7b93910446f91928e23e1043dfdf5bcG",
               "memberFirstName": "Uri",
               "memberMiddleName": "Prayag",
               "memberLastName": "Dubofsky"
            }
         }
      ]
   }
}

I want to parse it using bash script get only the list of field memberId.

The expected output is:

memberIds
----------- 
0x7b93910446f91928e23e1043dfdf5bcf
0x7b93910446f91928e23e1043dfdf5bcG

I tried adding following bash+python code to .bashrc:

function getJsonVal() {
   if [ \( $# -ne 1 \) -o \( -t 0 \) ]; then
       echo "Usage: getJsonVal 'key' < /tmp/file";
       echo "   -- or -- ";
       echo " cat /tmp/input | getJsonVal 'key'";
       return;
   fi;
   cat | python -c 'import json,sys;obj=json.load(sys.stdin);print obj["'$1'"]';
}

And then called:

$ cat members.json | getJsonVal "memberId"

But it throws:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
KeyError: 'memberId'

Reference

https://stackoverflow.com/a/21595107/432903

  • 2
    Why do you need to do this in bash? you are clearly using python here so why not just create a python script that does the job? You might not get actual answers on how to do it with bash because when you need to do that much you use another language. – DavidG Feb 21 '14 at 6:36
  • I changed your title from "using bash script" to "using python" since python, and not bash, is what you are using to parse json. E.g., that error is certainly a python error, not a bash error. – goldilocks Feb 21 '14 at 6:38
  • @goldilocks just because his attempt used python, doesn't mean his goal is to use python – jordanm Feb 21 '14 at 6:59
  • @DavidG see my answer. It's not pure shell, it's an external command but it integrates into shell scripts pretty well. – jordanm Feb 21 '14 at 7:00
  • Can I suggest you take out most of the irrelevant fields in the json. It suffices to have 2-3 elements in _source to get the gist of what you try to do. The rest just distracts – Anthon Feb 21 '14 at 8:01
25

If you would use:

 $ cat members.json | \
     python -c 'import json,sys;obj=json.load(sys.stdin);print obj;'

you can inspect the structure of the nested dictonary obj and see that your original line should read:

$ cat members.json | \
    python -c 'import json,sys;obj=json.load(sys.stdin);print obj["hits"]["hits"][0]["_source"]["'$1'"]';

to the to that "memberId" element. This way you can keep the Python as a oneliner.

If there are multiple elements in the nested "hits" element, then you can do something like:

$ cat members.json | \
python -c '
import json, sys
obj=json.load(sys.stdin)
for y in [x["_source"]["'$1'"] for x in obj["hits"]["hits"]]:
    print y
'

Chris Down's solution is better for finding a single value to (unique) keys at any level.

With my second example that prints out multiple values, you are hitting the limits of what you should try with a one liner, at that point I see little reason why to do half of the processing in bash, and would move to a complete Python solution.

7

Another way to do this in bash is using jshon. Here is a solution to your problem using jshon:

$ jshon -e hits -e hits -a -e _source -e memberId -u < foo.json
0x7b93910446f91928e23e1043dfdf5bcf
0x7b93910446f91928e23e1043dfdf5bcG

The -e options extract values from the json. The -a iterates over the array and the -u decodes the final string.

  • Let me install jshon – prayagupd Feb 21 '14 at 7:22
6

Well, your key is quite clearly not at the root of the object. Try something like this:

json_key() {
    python -c '
import json
import sys

data = json.load(sys.stdin)

for key in sys.argv[1:]:
    try:
        data = data[key]
    except TypeError:  # This is a list index
        data = data[int(key)]

print(data)' "$@"
}

This has the benefit of not just simply injecting syntax into Python, which could cause breakage (or worse, arbitrary code execution).

You can then call it like this:

json_key hits hits 0 _source memberId < members.json
  • 1
    Note: This will not loop over each item in "hits". If you want that, you should write specific Python code for that instance. – Chris Down Feb 21 '14 at 6:51
  • But it shows only one memberId. – prayagupd Feb 21 '14 at 7:01
4

Another alternative is jq:

$ cat members.json | jq -r '.hits|.hits|.[]|._source|.memberId'
0x7b93910446f91928e23e1043dfdf5bcf
0x7b93910446f91928e23e1043dfdf5bcG
2

Try this:

$ cat json.txt | python -c 'import sys; import simplejson as json; \
print "\n".join( [i["_source"]["memberId"] for i in json.loads( sys.stdin.read() )["hits"]["hits"]] )'


If you already have pretty printed json, why don't you just grep it?

$ cat json.txt | grep memberId
               "memberId": "0x7b93910446f91928e23e1043dfdf5bcf",
               "memberId": "0x7b93910446f91928e23e1043dfdf5bcG",

You can always get a pretty printed format with simplejson python to grep it.

# cat json_raw.txt
{"hits": {"hits": [{"_score": 1, "_type": "Medical", "_id": "02:17447847049147026174478:174159", "_source": {"memberLastName": "Dubofsky", "memberMiddleName": "Prayag", "memberId": "0x7b93910446f91928e23e1043dfdf5bcf", "memberFirstName": "Uri"}, "_index": "2000_270_0"}, {"_score": 1, "_type": "Medical", "_id": "02:17447847049147026174478:174159", "_source": {"memberLastName": "Dubofsky", "memberMiddleName": "Prayag", "memberId": "0x7b93910446f91928e23e1043dfdf5bcG", "memberFirstName": "Uri"}, "_index": "2000_270_0"}], "total": 74, "max_score": 1}, "_shards": {"successful": 8, "failed": 0, "total": 8}, "took": 670, "timed_out": false}

Use dumps:

# cat json_raw.txt | python -c 'import sys; import simplejson as json; \
print json.dumps( json.loads( sys.stdin.read() ), sort_keys=True, indent=4); '

{
    "_shards": {
        "failed": 0,
        "successful": 8,
        "total": 8
    },
    "hits": {
        "hits": [
            {
                "_id": "02:17447847049147026174478:174159",
                "_index": "2000_270_0",
                "_score": 1,
                "_source": {
                    "memberFirstName": "Uri",
                    "memberId": "0x7b93910446f91928e23e1043dfdf5bcf",
                    "memberLastName": "Dubofsky",
                    "memberMiddleName": "Prayag"
                },
                "_type": "Medical"
            },
            {
                "_id": "02:17447847049147026174478:174159",
                "_index": "2000_270_0",
                "_score": 1,
                "_source": {
                    "memberFirstName": "Uri",
                    "memberId": "0x7b93910446f91928e23e1043dfdf5bcG",
                    "memberLastName": "Dubofsky",
                    "memberMiddleName": "Prayag"
                },
                "_type": "Medical"
            }
        ],
        "max_score": 1,
        "total": 74
    },
    "timed_out": false,
    "took": 670
}

Thereafter, simply grep result with 'memberId' pattern.

To be completely precise:

#!/bin/bash

filename="$1"
cat $filename | python -c 'import sys; import simplejson as json; \
print json.dumps( json.loads( sys.stdin.read() ), sort_keys=True, indent=4)' | \
grep memberId | awk '{print $2}' | sed -e 's/^"//g' | sed -e 's/",$//g'

Usage:

$ bash bash.sh json_raw.txt 
0x7b93910446f91928e23e1043dfdf5bcf
0x7b93910446f91928e23e1043dfdf5bcG
0

Following this thread I'd use json.tool in python:

python -m json.tool members.json | awk -F'"' '/memberId/{print $4}'

0

Using deepdiff you do not need to know the exact keys:

import json
from deepdiff import DeepSearch
DeepSearch(json.load(open("members.json", "r")), 'memberId', verbose_level=2)['matched_paths'].values()
0

Here's a bash solution.

  1. create file find_members.sh
  2. add the following line to file + save

    #!/bin/bash
    
    echo -e "\nmemberIds\n---------"
    cat members.json | grep -E 'memberId'|awk '{print$2}' | cut -d '"' -f2
    
  3. chmod +x find_members.sh

Now run it:

$ ./find_members.sh

memberIds
----------------
0x7b93910446f91928e23e1043dfdf5bcf
0x7b93910446f91928e23e1043dfdf5bcG

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.