Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have a JSON output from which I need to extract a few parameters in Linux.

This is the JSON output:

{
        "OwnerId": "121456789127",
        "ReservationId": "r-48465168",
        "Groups": [],
        "Instances": [
            {
                "Monitoring": {
                    "State": "disabled"
                },
                "PublicDnsName": null,
                "RootDeviceType": "ebs",
                "State": {
                    "Code": 16,
                    "Name": "running"
                },
                "EbsOptimized": false,
                "LaunchTime": "2014-03-19T09:16:56.000Z",
                "PrivateIpAddress": "10.250.171.248",
                "ProductCodes": [
                    {
                        "ProductCodeId": "aacglxeowvn5hy8sznltowyqe",
                        "ProductCodeType": "marketplace"
                    }
                ],
                "VpcId": "vpc-86bab0e4",
                "StateTransitionReason": null,
                "InstanceId": "i-1234576",
                "ImageId": "ami-b7f6c5de",
                "PrivateDnsName": "ip-10-120-134-248.ec2.internal",
                "KeyName": "Test_Virginia",
                "SecurityGroups": [
                    {
                        "GroupName": "Test",
                        "GroupId": "sg-12345b"
                    }
                ],
                "ClientToken": "VYeFw1395220615808",
                "SubnetId": "subnet-12345314",
                "InstanceType": "t1.micro",
                "NetworkInterfaces": [
                    {
                        "Status": "in-use",
                        "SourceDestCheck": true,
                        "VpcId": "vpc-123456e4",
                        "Description": "Primary network interface",
                        "NetworkInterfaceId": "eni-3619f31d",
                        "PrivateIpAddresses": [
                            {
                                "Primary": true,
                                "PrivateIpAddress": "10.120.134.248"
                            }
                        ],
                        "Attachment": {
                            "Status": "attached",
                            "DeviceIndex": 0,
                            "DeleteOnTermination": true,
                            "AttachmentId": "eni-attach-9210dee8",
                            "AttachTime": "2014-03-19T09:16:56.000Z"
                        },
                        "Groups": [
                            {
                                "GroupName": "Test",
                                "GroupId": "sg-123456cb"
                            }
                        ],
                        "SubnetId": "subnet-31236514",
                        "OwnerId": "109030037527",
                        "PrivateIpAddress": "10.120.134.248"
                    }
                ],
                "SourceDestCheck": true,
                "Placement": {
                    "Tenancy": "default",
                    "GroupName": null,
                    "AvailabilityZone": "us-east-1c"
                },
                "Hypervisor": "xen",
                "BlockDeviceMappings": [
                    {
                        "DeviceName": "/dev/sda",
                        "Ebs": {
                            "Status": "attached",
                            "DeleteOnTermination": false,
                            "VolumeId": "vol-37ff097b",
                            "AttachTime": "2014-03-19T09:17:00.000Z"
                        }
                    }
                ],
                "Architecture": "x86_64",
                "KernelId": "aki-88aa75e1",
                "RootDeviceName": "/dev/sda1",
                "VirtualizationType": "paravirtual",
                "Tags": [
                    {
                        "Value": "Server for testing RDS feature in us-east-1c AZ",
                        "Key": "Description"
                    },
                    {
                        "Value": "RDS_Machine (us-east-1c)",
                        "Key": "Name"
                    },
                    {
                        "Value": "1234",
                        "Key": "cost.centre",
                      },
                    {
                        "Value": "Jyoti Bhanot",
                        "Key": "Owner",
                      }
                ],
                "AmiLaunchIndex": 0
            }
        ]
    }

I want to write a file that contains heading like instance id, tag like name, cost center, owner. and below that certain values from the JSON output. The output here given is just an example.

How can I do that using sed and awk?

Expected output :

 Instance id         Name                           cost centre             Owner
    i-1234576          RDS_Machine (us-east-1c)        1234                   Jyoti
share|improve this question
    
Pipe your CLI call into python, suggested because it's native to EC2 instances. Python can easily interpret JSON. See the answer below for an example. Of course, you could use any other SS language too, but they will require installs whereas Python is already there. –  scrowler Oct 31 at 6:24

4 Answers 4

The availability of parsers in nearly every programming language is one of the advantages of JSON as a data-interchange format.

Rather than trying to implement a JSON parser, you are likely better off using either a tool built for JSON parsing such as jq or a general purpose script language that has a JSON library.

For example, using jq, you could pull out the ImageID from the first item of the Instances array as follows:

jq '.Instances[0].ImageId' test.json

Alternatively, to get the same information using Ruby's JSON library:

ruby -rjson -e 'j = JSON.parse(File.read("test.json")); puts j["Instances"][0]["ImageId"]'

I won't answer all of your revised questions and comments but the following is hopefully enough to get you started.

Suppose that you had a Ruby script that could read a from STDIN and output the second line in your example output[0]. That script might look something like:

#!/usr/bin/env ruby
require 'json'

data = JSON.parse(ARGF.read)
instance_id = data["Instances"][0]["InstanceId"]
name = data["Instances"][0]["Tags"].find {|t| t["Key"] == "Name" }["Value"]
owner = data["Instances"][0]["Tags"].find {|t| t["Key"] == "Owner" }["Value"]
cost_center = data["Instances"][0]["SubnetId"].split("-")[1][0..3]
puts "#{instance_id}\t#{name}\t#{cost_center}\t#{owner}"

How could you use such a script to accomplish your whole goal? Well, suppose you already had the following:

  • a command to list all your instances
  • a command to get the json above for any instance on your list and output it to STDOU

One way would be to use your shell to combine these tools:

echo -e "Instance id\tName\tcost centre\tOwner"
for instance in $(list-instances); do
    get-json-for-instance $instance | ./ugly-ruby-scriptrb
done

Now, maybe you have a single command that give you one json blob for all instances with more items in that "Instances" array. Well, if that is the case, you'll just need to modify the script a bit to iterate through the array rather than simply using the first item.

In the end, the way to solve this problem, is the way to solve many problems in Unix. Break it down into easier problems. Find or write tools to solve the easier problem. Combine those tools with your shell or other operating system features.

[0] Note that I have no idea where you get cost-center from, so I just made it up.

share|improve this answer
    
i have installed jq on my machine. but i dont know how to get the information. i am updating the question –  user3086014 Mar 27 at 9:23
    
How to do that. the command ec2-describe instance gives reslut like this. this is data for 1 instance, there are 100 instance. How to do that in a script –  user3086014 Mar 27 at 9:25
    
i have aws cli tools that gives me the output. now how to parse the output and the required tags that i really dont know –  user3086014 Mar 27 at 9:59
1  
@user3086014 I'm sorry, but I won't be putting more work into this answer. Take a look at the Ruby example I have there. It should give you a good place to start on how to get tags out of various parts of the JSON you want. –  Steven D Mar 27 at 10:02

You can use following python script to parse that data. Lets assume that you have JSON data from arrays in files like array1.json, array2.json and so on.

import json
import sys
from pprint import pprint

jdata = open(sys.argv[1])

data = json.load(jdata)

print "InstanceId", " - ", "Name", " - ", "Owner"
print data["Instances"][0]["InstanceId"], " - " ,data["Instances"][0]["Tags"][1]["Value"], " - " ,data["Instances"][0]["Tags"][2]["Value"] 

jdata.close()

And then just run:

$ for x in `ls *.json`; do python parse.py $x; done
InstanceId  -  Name  -  Owner
i-1234576  -  RDS_Machine (us-east-1c)  -  Jyoti Bhanot

I haven't seen cost in your data, that's why I didn't include that.

According to discussion in comments, I have updated parse.py script:

import json
import sys
from pprint import pprint

jdata = sys.stdin.read()

data = json.loads(jdata)

print "InstanceId", " - ", "Name", " - ", "Owner"
print data["Instances"][0]["InstanceId"], " - " ,data["Instances"][0]["Tags"][1]["Value"], " - " ,data["Instances"][0]["Tags"][2]["Value"] 

You can try to run following command:

#ec2-describe-instance <instance> | python parse.py
share|improve this answer
    
but this is just one array there are similar arrays returned by the command. how to do that –  user3086014 Mar 27 at 10:32
    
and this data is generated by ec2-describe instance command at runtime. how to handle that –  user3086014 Mar 27 at 10:34
    
I have modified a bit this python script: import json from pprint import pprint jdata = open('example.json') data = json.load(jdata) print "InstanceId", " - ", "Name", " - ", "Owner" print data["Instances"][0]["InstanceId"], " - " ,data["Instances"][0]["Tags"][1]["Value"], " - " ,data["Instances"][0]["Tags"][2]["Value"] jdata.close() If you have all json data from arrays in files like array1.json, array2.json, ... and so on, you could try to run it like this: # for x in ls *.json; do python parse.py $x; done –  Robert Jonczy Mar 27 at 10:42
    
you can update the answer itself . its not readable –  user3086014 Mar 27 at 10:42
    
also i have arrays.100 of arrays like this –  user3086014 Mar 27 at 10:43

Others have provided general answers for your question which demonstrate good ways of parsing json however I, like you, were looking for a way to extract an aws instance id using a core tool like awk or sed without depending on other packages. To accomplish this you can pass the "--output=text" argument to your aws command which will give you an awk parsable string. With that you can simply get the instance ID using something like the following...

aws ec2 run-instances --output text  | awk -F"\t" '$1=="INSTANCES" {print $8}'
share|improve this answer

I will now quote extremely liberally (read: copy entirely) from this Stack Overflow answer:

You can't parse [X]HTML with regex. Because HTML can't be parsed by regex. Regex is not a tool that can be used to correctly parse HTML. As I have answered in HTML-and-regex questions here so many times before, the use of regex will not allow you to consume HTML. Regular expressions are a tool that is insufficiently sophisticated to understand the constructs employed by HTML. HTML is not a regular language and hence cannot be parsed by regular expressions. Regex queries are not equipped to break down HTML into its meaningful parts. so many times but it is not getting to me. Even enhanced irregular regular expressions as used by Perl are not up to the task of parsing HTML. You will never make me crack. HTML is a language of sufficient complexity that it cannot be parsed by regular expressions. Even Jon Skeet cannot parse HTML using regular expressions. Every time you attempt to parse HTML with regular expressions, the unholy child weeps the blood of virgins, and Russian hackers pwn your webapp. Parsing HTML with regex summons tainted souls into the realm of the living. HTML and regex go together like love, marriage, and ritual infanticide. The <center> cannot hold it is too late. The force of regex and HTML together in the same conceptual space will destroy your mind like so much watery putty. If you parse HTML with regex you are giving in to Them and their blasphemous ways which doom us all to inhuman toil for the One whose Name cannot be expressed in the Basic Multilingual Plane, he comes. HTML-plus-regexp will liquify the n​erves of the sentient whilst you observe, your psyche withering in the onslaught of horror. Rege̿̔̉x-based HTML parsers are the cancer that is killing StackOverflow it is too late it is too late we cannot be saved the trangession of a chi͡ld ensures regex will consume all living tissue (except for HTML which it cannot, as previously prophesied) dear lord help us how can anyone survive this scourge using regex to parse HTML has doomed humanity to an eternity of dread torture and security holes using regex as a tool to process HTML establishes a breach between this world and the dread realm of c͒ͪo͛ͫrrupt entities (like SGML entities, but more corrupt) a mere glimpse of the world of reg​ex parsers for HTML will ins​tantly transport a programmer's consciousness into a world of ceaseless screaming, he comes, the pestilent slithy regex-infection wil​l devour your HT​ML parser, application and existence for all time like Visual Basic only worse he comes he comes do not fi​ght he com̡e̶s, ̕h̵i​s un̨ho͞ly radiańcé destro҉ying all enli̍̈́̂̈́ghtenment, HTML tags lea͠ki̧n͘g fr̶ǫm ̡yo​͟ur eye͢s̸ ̛l̕ik͏e liq​uid pain, the song of re̸gular exp​ression parsing will exti​nguish the voices of mor​tal man from the sp​here I can see it can you see ̲͚̖͔̙î̩́t̲͎̩̱͔́̋̀ it is beautiful t​he final snuffing of the lie​s of Man ALL IS LOŚ͖̩͇̗̪̏̈́T ALL I​S LOST the pon̷y he comes he c̶̮omes he comes the ich​or permeates all MY FACE MY FACE ᵒh god no NO NOO̼O​O NΘ stop the an​*̶͑̾̾​̅ͫ͏̙̤g͇̫͛͆̾ͫ̑͆l͖͉̗̩̳̟̍ͫͥͨe̠̅s ͎a̧͈͖r̽̾̈́͒͑e n​ot rè̑ͧ̌aͨl̘̝̙̃ͤ͂̾̆ ZA̡͊͠͝LGΌ ISͮ̂҉̯͈͕̹̘̱ TO͇̹̺ͅƝ̴ȳ̳ TH̘Ë͖́̉ ͠P̯͍̭O̚​N̐Y̡ H̸̡̪̯ͨ͊̽̅̾̎Ȩ̬̩̾͛ͪ̈́̀́͘ ̶̧̨̱̹̭̯ͧ̾ͬC̷̙̲̝͖ͭ̏ͥͮ͟Oͮ͏̮̪̝͍M̲̖͊̒ͪͩͬ̚̚͜Ȇ̴̟̟͙̞ͩ͌͝S̨̥̫͎̭ͯ̿̔̀ͅ

Bear in mind: XHTML is based on XML. While JSON is not quite as complex to parse as XML, it is still difficult. The answer to your question is: don't use sed or awk. Instead, use a real JSON parser. Please.

Cursory Googling for "bash json" shows that jshon may be something to look into.

share|improve this answer
3  
@user3086014 I think you are missing the entire point of this answer. –  user80551 Mar 27 at 6:01
1  
+1 for jshon: it's perfect for this task... –  jasonwryan Mar 27 at 6:02
8  
To be fair, he wasn't asking to parse it with regular expressions. You could write a fine JSON parser in awk, it would just take a lot more time than anyone would reasonably spend on an answer for stackexchange. :) More seriously, jq has been useful to me: stedolan.github.io/jq –  Steven D Mar 27 at 6:28
4  
@user3086014 You can for example read a bit of documentation, work a bit, and then if you can't manage to find a proper solution ask a new question, with your goal, and the different steps you go through and explain in details what is not working as expected. –  Kiwy Mar 27 at 9:54
1  
If I could downvote you on this site I would. I regularly reference regex questions to this answer on Stack Overflow, but this has got nothing to do with regex. –  scrowler Oct 29 at 5:51

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.