How to parse JSON with shell scripting in Linux?

Question

I have a JSON output from which I need to extract a few parameters in Linux.

This is the JSON output:

{
        "OwnerId": "121456789127",
        "ReservationId": "r-48465168",
        "Groups": [],
        "Instances": [
            {
                "Monitoring": {
                    "State": "disabled"
                },
                "PublicDnsName": null,
                "RootDeviceType": "ebs",
                "State": {
                    "Code": 16,
                    "Name": "running"
                },
                "EbsOptimized": false,
                "LaunchTime": "2014-03-19T09:16:56.000Z",
                "PrivateIpAddress": "10.250.171.248",
                "ProductCodes": [
                    {
                        "ProductCodeId": "aacglxeowvn5hy8sznltowyqe",
                        "ProductCodeType": "marketplace"
                    }
                ],
                "VpcId": "vpc-86bab0e4",
                "StateTransitionReason": null,
                "InstanceId": "i-1234576",
                "ImageId": "ami-b7f6c5de",
                "PrivateDnsName": "ip-10-120-134-248.ec2.internal",
                "KeyName": "Test_Virginia",
                "SecurityGroups": [
                    {
                        "GroupName": "Test",
                        "GroupId": "sg-12345b"
                    }
                ],
                "ClientToken": "VYeFw1395220615808",
                "SubnetId": "subnet-12345314",
                "InstanceType": "t1.micro",
                "NetworkInterfaces": [
                    {
                        "Status": "in-use",
                        "SourceDestCheck": true,
                        "VpcId": "vpc-123456e4",
                        "Description": "Primary network interface",
                        "NetworkInterfaceId": "eni-3619f31d",
                        "PrivateIpAddresses": [
                            {
                                "Primary": true,
                                "PrivateIpAddress": "10.120.134.248"
                            }
                        ],
                        "Attachment": {
                            "Status": "attached",
                            "DeviceIndex": 0,
                            "DeleteOnTermination": true,
                            "AttachmentId": "eni-attach-9210dee8",
                            "AttachTime": "2014-03-19T09:16:56.000Z"
                        },
                        "Groups": [
                            {
                                "GroupName": "Test",
                                "GroupId": "sg-123456cb"
                            }
                        ],
                        "SubnetId": "subnet-31236514",
                        "OwnerId": "109030037527",
                        "PrivateIpAddress": "10.120.134.248"
                    }
                ],
                "SourceDestCheck": true,
                "Placement": {
                    "Tenancy": "default",
                    "GroupName": null,
                    "AvailabilityZone": "us-east-1c"
                },
                "Hypervisor": "xen",
                "BlockDeviceMappings": [
                    {
                        "DeviceName": "/dev/sda",
                        "Ebs": {
                            "Status": "attached",
                            "DeleteOnTermination": false,
                            "VolumeId": "vol-37ff097b",
                            "AttachTime": "2014-03-19T09:17:00.000Z"
                        }
                    }
                ],
                "Architecture": "x86_64",
                "KernelId": "aki-88aa75e1",
                "RootDeviceName": "/dev/sda1",
                "VirtualizationType": "paravirtual",
                "Tags": [
                    {
                        "Value": "Server for testing RDS feature in us-east-1c AZ",
                        "Key": "Description"
                    },
                    {
                        "Value": "RDS_Machine (us-east-1c)",
                        "Key": "Name"
                    },
                    {
                        "Value": "1234",
                        "Key": "cost.centre",
                      },
                    {
                        "Value": "Jyoti Bhanot",
                        "Key": "Owner",
                      }
                ],
                "AmiLaunchIndex": 0
            }
        ]
    }

I want to write a file that contains heading like instance id, tag like name, cost center, owner. and below that certain values from the JSON output. The output here given is just an example.

How can I do that using sed and awk?

Expected output :

 Instance id         Name                           cost centre             Owner
    i-1234576          RDS_Machine (us-east-1c)        1234                   Jyoti

Answer 1

The availability of parsers in nearly every programming language is one of the advantages of JSON as a data-interchange format.

Rather than trying to implement a JSON parser, you are likely better off using either a tool built for JSON parsing such as jq or a general purpose script language that has a JSON library.

For example, using jq, you could pull out the ImageID from the first item of the Instances array as follows:

jq '.Instances[0].ImageId' test.json

Alternatively, to get the same information using Ruby's JSON library:

ruby -rjson -e 'j = JSON.parse(File.read("test.json")); puts j["Instances"][0]["ImageId"]'

I won't answer all of your revised questions and comments but the following is hopefully enough to get you started.

Suppose that you had a Ruby script that could read a from STDIN and output the second line in your example output[0]. That script might look something like:

#!/usr/bin/env ruby
require 'json'

data = JSON.parse(ARGF.read)
instance_id = data["Instances"][0]["InstanceId"]
name = data["Instances"][0]["Tags"].find {|t| t["Key"] == "Name" }["Value"]
owner = data["Instances"][0]["Tags"].find {|t| t["Key"] == "Owner" }["Value"]
cost_center = data["Instances"][0]["SubnetId"].split("-")[1][0..3]
puts "#{instance_id}\t#{name}\t#{cost_center}\t#{owner}"

How could you use such a script to accomplish your whole goal? Well, suppose you already had the following:

• a command to list all your instances
• a command to get the json above for any instance on your list and output it to STDOU

One way would be to use your shell to combine these tools:

echo -e "Instance id\tName\tcost centre\tOwner"
for instance in $(list-instances); do
    get-json-for-instance $instance | ./ugly-ruby-scriptrb
done

Now, maybe you have a single command that give you one json blob for all instances with more items in that "Instances" array. Well, if that is the case, you'll just need to modify the script a bit to iterate through the array rather than simply using the first item.

In the end, the way to solve this problem, is the way to solve many problems in Unix. Break it down into easier problems. Find or write tools to solve the easier problem. Combine those tools with your shell or other operating system features.

[0] Note that I have no idea where you get cost-center from, so I just made it up.

Answer 2

You can use following python script to parse that data. Lets assume that you have JSON data from arrays in files like array1.json, array2.json and so on.

import json
import sys
from pprint import pprint

jdata = open(sys.argv[1])

data = json.load(jdata)

print "InstanceId", " - ", "Name", " - ", "Owner"
print data["Instances"][0]["InstanceId"], " - " ,data["Instances"][0]["Tags"][1]["Value"], " - " ,data["Instances"][0]["Tags"][2]["Value"] 

jdata.close()

And then just run:

$ for x in `ls *.json`; do python parse.py $x; done
InstanceId  -  Name  -  Owner
i-1234576  -  RDS_Machine (us-east-1c)  -  Jyoti Bhanot

I haven't seen cost in your data, that's why I didn't include that.

According to discussion in comments, I have updated parse.py script:

import json
import sys
from pprint import pprint

jdata = sys.stdin.read()

data = json.loads(jdata)

print "InstanceId", " - ", "Name", " - ", "Owner"
print data["Instances"][0]["InstanceId"], " - " ,data["Instances"][0]["Tags"][1]["Value"], " - " ,data["Instances"][0]["Tags"][2]["Value"] 

You can try to run following command:

#ec2-describe-instance <instance> | python parse.py

Answer 3

Others have provided general answers for your question which demonstrate good ways of parsing json however I, like you, were looking for a way to extract an aws instance id using a core tool like awk or sed without depending on other packages. To accomplish this you can pass the "--output=text" argument to your aws command which will give you an awk parsable string. With that you can simply get the instance ID using something like the following...

aws ec2 run-instances --output text  | awk -F"\t" '$1=="INSTANCES" {print $8}'

Answer 4

The following jq code:

.Instances[] | (.Tags | map(.value=.Value | .key=.Key) | from_entries) as $tags | "\(.InstanceId) | \($tags.Name) | \($tags["cost.centre"]) | \($tags.Owner)"

used like:

json_producer | jq -r '<jq code...>'

would output:

i-1234576 | RDS_Machine (us-east-1c) | 1234 | Jyoti Bhanot

A few pointers to understand the code:

• from_entries takes an array of objects like {key:a, value:b} and turns it into an object with corresponding key/value pairs ({a: b});
• The Key and Value keys in the Tags array had to be converted to lowercase;
• The last string uses jq's string interpolation feature. You can tweak it as needed.

Fore more details, go see jq's tutorial and manual at https://stedolan.github.io/jq/

Answer 5

If this is limited to the AWS use case provided above, you should use --query and --output flags for your CLI API call

http://docs.aws.amazon.com/cli/latest/userguide/controlling-output.html

Answer 6

Jshon is available in several distributions :

$ echo your_JSON|jshon -e Instances -a -e InstanceId -u -p -e Tags -a -e Key -u -p -e Value -u
i-1234576
Description
Server for testing RDS feature in us-east-1c AZ
Name
RDS_Machine (us-east-1c)
cost.centre
1234
Owner
Jyoti Bhanot

Poor explanation : -e uu will extract object uu, -a will make the array usable (not sure I correctly phrased this one but anyway…), -u will decode the string, -p will go back to previous item (seems that -i N, N being any number, has the same effect).

Depending on your case, the output may require some post-treatment (like yours, as you can see).

Jshon doesn's seem robust against JSON malformation, though (your "Tags" with commas before the closing curly bracket will raise an error).

Someone mentioned jsawk in another thread, but I have not tested it.

Answer 7

Using jq, and assuming that the Tags array is always ordered the way it is in the example data, with the last tree elements referring to Name, cost.centre, and Owner (see end for a more general approach where the ordering of Tags is not important).

jq -r '[ "Instance ID", "Name", "cost centre", "Owner" ],
    (.Instances[] | [.InstanceId, .Tags[-3:][].Value]) | @tsv' file

For each instance (element in the top-level Instances array), this extracts the instance ID and the Value values corresponding to the three last Tags elements, and formats it as a tab-delimited line.

The output is preceded by a header for each column.

The output of this given the data in the question (tab-delimited):

Instance ID     Name    cost centre     Owner
i-1234576       RDS_Machine (us-east-1c)        1234    Jyoti Bhanot

If the order of the Tags values can't be guaranteed, then one may use

jq -r '[ "Instance ID", "Name", "cost centre", "Owner" ],
    (
        .Instances[] |
        .InstanceId as $id |
        [ .Tags[] | with_entries(.key |= ascii_downcase) ] |
        from_entries | [ $id, .Name, ."cost.centre", .Owner ]
    ) | @tsv' file

This converts each {"Value": "somevalue", "Key": "somekey"} element of Tags into {"somekey": "somevalue"} and then extracts the keys we're interested in, in the order that we want them.

Answer 8

Take a look at jtc cli tool:

it lets easily extracting required information from your json (assuming it's in file.json, btw, your JSON needs to be fixed, there's couple extra commas there):

bash $ cat file.json | jtc -x '<InstanceId>l+0[-1]' -y '[InstanceId]' -y "[Key]:<Name>[-1][Value]" -y "[Key]:<cost.centre>[-1][Value]" -y "[Key]:<Owner>[-1][Value]" | sed 's/"/\\"/g' | xargs -L4 echo
"i-1234576" "RDS_Machine (us-east-1c)" "1234" "Jyoti Bhanot"
bash $ 

Answer 9

Warning: This answer uses bashisms.

For this exact purpose (easily access any JSON) I wrote a tool called json2sh which converts any valid JSON into something

• bash can natively understand
• and can be easily be processed line by line (each line is a key/value pair)

The basic idea is to just convert the JSON into environment variables. However with an easy tweak you can process it with some shell function or read it line-by line, too.

Beware: You cannot distinguish between Strings and Numbers, as both are expressed the same way in shells:

So json2sh <<<'"1"' and json2sh <<<1 produce the same output JSON_=1
and json2sh <<<1.1E99 outputs JSON_='1.1E99' (due to the dot it is quoted)

json2sh <<<'{"w":"t", "f":[ 6 ], "one":true, "nothing":null }'

outputs

JSON__0_w=t
JSON__0_f_1_=6
JSON__0_one=$JSON_true_
JSON__0_nothing=$JSON_null_

while

json2sh 'parse ' ' ' <<<'{"w":"t", "f":[ 6 ], "one":true, "nothing":null }'

outputs the same things as

parse _0_w t
parse _0_f_1_ 6
parse _0_one $JSON_true_
parse _0_nothing $JSON_null_

Hence by defining some parse() function like that

parse()
{
  printf 'key=%q\nval=%q\n' "$1" "$2"
}

you can parse it with

. <(json2sh 'parse ' ' ' <<<'{"w":"t", "f":[ 6 ], "one":true, "nothing":null }')

easily. If you are concerned that there might be problems (read: eval is evil or in case your JSON might contain NUL like in "\u0000"), you can work it line by line, as json2sh makes sure each value is on it's own single (perhaps extremely long) line like this:

while read -ru6 k v
do
  {
  # your code here
  } 6<&-
done 6< <(json2sh '' ' ' <<<'{"w":"t", "f":[ 6 ], "one":true, "nothing":null }')

You do not worry about lines with trailing blanks in variable v, as each JSON string which contains a blank is quoted, so the value in v ends on ' in that case.

However v can have following forms:

• value (a simple string made of numbers and letters)
• 'value' (a not-so-simple string containing special characters like SPC or dot)
• $'value' if the value contains very special characters. This is called an ANSI-C quoting.

The only problem here is that v may be quoted for shell usage in the ANSI-C sense and must be dequoted somehow. For simple values there is not much of a problem, though, so you do not need complex dequoting.

json2sh <<<'"\n"' becomes JSON_=$'\n' which is the ANSI-C-String.

Beware: json2sh <<<'"\u0000a\u0000"' gives JSON_=$'\x00a\x00' which creates an empty shell string, as shell strings cannot contain NUL. But such patterns can be used (literally, not as a variable) with printf like in printf '\x00a\x00'.

With json2sh any valid JSON can be converted such that the shell can easily process it, no need to know anything before.

For example your JSON

json2sh < SO121718.json > SO121718.inc

gives:

json2sh:105:24: expected '"' but got '}': Success

Huh? (The Success above means, errno==0, so no IO error or similar.)

sed -n '104,105p' SO121718.json 

gives

                "Key": "cost.centre",
              },

which is not valid JSON. Finally

sed -e '104s/,$//' -e '108s/,$//' SO121718.json | json2sh > SO121718.inc

works and SO121718.inc is

JSON__0_OwnerId=121456789127
JSON__0_ReservationId='r-48465168'
JSON__0_Groups=$JSON_empty_
JSON__0_Instances_1_Monitoring_0_State=disabled
JSON__0_Instances_1_PublicDnsName=$JSON_null_
JSON__0_Instances_1_RootDeviceType=ebs
JSON__0_Instances_1_State_0_Code=16
JSON__0_Instances_1_State_0_Name=running
JSON__0_Instances_1_EbsOptimized=$JSON_false_
JSON__0_Instances_1_LaunchTime='2014-03-19T09:16:56.000Z'
JSON__0_Instances_1_PrivateIpAddress='10.250.171.248'
JSON__0_Instances_1_ProductCodes_1_ProductCodeId=aacglxeowvn5hy8sznltowyqe
JSON__0_Instances_1_ProductCodes_1_ProductCodeType=marketplace
JSON__0_Instances_1_VpcId='vpc-86bab0e4'
JSON__0_Instances_1_StateTransitionReason=$JSON_null_
JSON__0_Instances_1_InstanceId='i-1234576'
JSON__0_Instances_1_ImageId='ami-b7f6c5de'
JSON__0_Instances_1_PrivateDnsName='ip-10-120-134-248.ec2.internal'
JSON__0_Instances_1_KeyName='Test_Virginia'
JSON__0_Instances_1_SecurityGroups_1_GroupName=Test
JSON__0_Instances_1_SecurityGroups_1_GroupId='sg-12345b'
JSON__0_Instances_1_ClientToken=VYeFw1395220615808
JSON__0_Instances_1_SubnetId='subnet-12345314'
JSON__0_Instances_1_InstanceType='t1.micro'
JSON__0_Instances_1_NetworkInterfaces_1_Status='in-use'
JSON__0_Instances_1_NetworkInterfaces_1_SourceDestCheck=$JSON_true_
JSON__0_Instances_1_NetworkInterfaces_1_VpcId='vpc-123456e4'
JSON__0_Instances_1_NetworkInterfaces_1_Description='Primary network interface'
JSON__0_Instances_1_NetworkInterfaces_1_NetworkInterfaceId='eni-3619f31d'
JSON__0_Instances_1_NetworkInterfaces_1_PrivateIpAddresses_1_Primary=$JSON_true_
JSON__0_Instances_1_NetworkInterfaces_1_PrivateIpAddresses_1_PrivateIpAddress='10.120.134.248'
JSON__0_Instances_1_NetworkInterfaces_1_Attachment_0_Status=attached
JSON__0_Instances_1_NetworkInterfaces_1_Attachment_0_DeviceIndex=0
JSON__0_Instances_1_NetworkInterfaces_1_Attachment_0_DeleteOnTermination=$JSON_true_
JSON__0_Instances_1_NetworkInterfaces_1_Attachment_0_AttachmentId='eni-attach-9210dee8'
JSON__0_Instances_1_NetworkInterfaces_1_Attachment_0_AttachTime='2014-03-19T09:16:56.000Z'
JSON__0_Instances_1_NetworkInterfaces_1_Groups_1_GroupName=Test
JSON__0_Instances_1_NetworkInterfaces_1_Groups_1_GroupId='sg-123456cb'
JSON__0_Instances_1_NetworkInterfaces_1_SubnetId='subnet-31236514'
JSON__0_Instances_1_NetworkInterfaces_1_OwnerId=109030037527
JSON__0_Instances_1_NetworkInterfaces_1_PrivateIpAddress='10.120.134.248'
JSON__0_Instances_1_SourceDestCheck=$JSON_true_
JSON__0_Instances_1_Placement_0_Tenancy=default
JSON__0_Instances_1_Placement_0_GroupName=$JSON_null_
JSON__0_Instances_1_Placement_0_AvailabilityZone='us-east-1c'
JSON__0_Instances_1_Hypervisor=xen
JSON__0_Instances_1_BlockDeviceMappings_1_DeviceName='/dev/sda'
JSON__0_Instances_1_BlockDeviceMappings_1_Ebs_0_Status=attached
JSON__0_Instances_1_BlockDeviceMappings_1_Ebs_0_DeleteOnTermination=$JSON_false_
JSON__0_Instances_1_BlockDeviceMappings_1_Ebs_0_VolumeId='vol-37ff097b'
JSON__0_Instances_1_BlockDeviceMappings_1_Ebs_0_AttachTime='2014-03-19T09:17:00.000Z'
JSON__0_Instances_1_Architecture='x86_64'
JSON__0_Instances_1_KernelId='aki-88aa75e1'
JSON__0_Instances_1_RootDeviceName='/dev/sda1'
JSON__0_Instances_1_VirtualizationType=paravirtual
JSON__0_Instances_1_Tags_1_Value='Server for testing RDS feature in us-east-1c AZ'
JSON__0_Instances_1_Tags_1_Key=Description
JSON__0_Instances_1_Tags_2_Value='RDS_Machine (us-east-1c)'
JSON__0_Instances_1_Tags_2_Key=Name
JSON__0_Instances_1_Tags_3_Value=1234
JSON__0_Instances_1_Tags_3_Key='cost.centre'
JSON__0_Instances_1_Tags_4_Value='Jyoti Bhanot'
JSON__0_Instances_1_Tags_4_Key=Owner
JSON__0_Instances_1_AmiLaunchIndex=0

Notes:

• Beware of bugs. This code has not been party reviewed. I tried my best to always create output which does not harm the shell or leads to exploits, but I might fail (I already failed once).- json2sh does not interpret UTF-8. Hence invalid UTF-8 sequences are handed through to the shell as-is.
• I cannot change bad choices of the output format, because too many of my scripts already rely on it
  • If I change the output format, I will add some selector for this, such that it stays backwards compatible by default
• json2sh is written for batch usage
  • That means it only buffers max. 256 byte of the last value and flushes as soon as a line is complete.
  • It only needs as much memory as to keep the biggest string in memory.
  • So it can process documents which are much bigger than the available RAM (as long as each single value fits into the available memory)
  • YMMV if you include the output into the shell ..

Answer 10

A native way to process JSON is to use JavaScript. Use NodeJS to execute a JS file outside of a browser.

// myscript.js
const data = getData();

console.log('Instance id\tName\t\t\t\tcost centre\tOwner');
console.log(data.Instances[0].InstanceId + '\t' + data.Instances[0].Tags[1].Value + '\t' + + data.Instances[0].Tags[2].Value + '\t\t' + data.Instances[0].Tags[3].Value);

function getData() {
    return {
        "OwnerId": "121456789127",
        "ReservationId": "r-48465168",
        "Groups": [],
        "Instances": [] // to fill
    }
}

From bash:

# nacre ./myscript.js works the same way
node ./myscript.js
Instance id    Name                       cost centre      Owner
i-1234576      RDS_Machine (us-east-1c)   1234             Jyoti Bhanot

If you tend to deal often JSON, maybe consider using Nacre.sh which is a JavaScript based shell.

Answer 11

Here is one-liner suggestion:

pr -mt \
 <(grep -o ".*: .*," in.json | grep -iw InstanceId | cut -d: -f2) \
 <(grep -o ".*: .*," in.json | grep -iw Value      | cut -d: -f2) \
 <(grep -o ".*: .*," in.json | grep -iw Key        | cut -d: -f2)

Not perfect, but it'd work if you tweak it a bit.

It basically using pr to print each set result per column. Each result set is returned by process substitution which parse JSON file and return values based on the key.

This works similar as described in: Given key-value content, how do I group values by key and sort by value?

Answer 12

So many suggestions and enough to confuse about choosing one. I just want to keep things simple here with good simple practical examples.

To parse JSON there are various solution. With Shell one best tool is

"jq"

jq tutorial: https://stedolan.github.io/jq/tutorial/

Install jq at centos as:

yum install -y jq

. But I always suggest to know enough about JSON format. It will helps in understanding the way how we go for parsing.

suppose I want to see number of volumes associated with my docker container I will use below command:

To see complete output:

docker inspect <container_name> | jq .

To see all mounts:

docker inspect <container_name> | jq .[0].Mounts

To see 1st Item (0th) of Mount (with screenshot):

docker inspect <container_name> | jq .[0].Mounts[0]

To see specifically 'Source' field of first Mount item:

docker inspect <container_name> | jq .[0].Mounts[1].Source

To see the total counts of volumes (lenght):

docker inspect <container_name> | jq '.[0].Mounts | length'

Answer 13

Pure Bash 3.2+ without dependencies (such as jq, python, grep, etc.):

source <(curl -s -L -o- https://github.com/lirik90/bashJsonParser/raw/master/jsonParser.sh)
read -d '' JSON << EOF
// put your json here
EOF
JSON=$(minifyJson "$JSON")
id=$(parseJson "$JSON" Instances 0 InstanceId)
name=$(parseJson "$JSON" Instances 0 Tags 1 Value)
cost=$(parseJson "$JSON" Instances 0 Tags 2 Value)
owner=$(parseJson "$JSON" Instances 0 Tags 3 Value)
echo -e "Instance id |\t\t Name \t\t\t\t| cost centre | Owner"
echo -e "$id \t| $name  | $cost\t\t  | $owner"

Output:

Instance id |        Name               | cost centre | Owner
i-1234576   | RDS_Machine (us-east-1c)  | 1234        | Jyoti Bhanot

Try it.