8

How can I parse JSON output on the shell?

For example, Amazon Web Services provides a CLI to retrieve the status of your instances:

$ aws ec2 describe-instances <my_instance_id>

But the command returns a JSON string. The output of that command looks like this:

$ aws ec2 describe-instances x12345
{
    "Reservations" :
     {  
            "OwnerId": "1345345"
            "Groups": [], 
            "SecurityGroups": [
               {
                  "Foo" : "yes"
                  "Bar" : "no
               }
             ]
     }
}

Are there an shells built-ins that could be used to parse JSON output?

For example, I would like to capture in a shell variable FOO, the following output["Reservations"]["SecurityGroups"][0]{"Foo"}.

In case it helps, I am specifically interested in solutions that could work from Zsh.

  • 2
    You would typically use a tool, like jshon. – jasonwryan Oct 7 '14 at 21:01
  • 1
    Use --output text if you want to parse in the shell without using external tools like jshon. – jordanm Oct 7 '14 at 22:49
  • 1
    @jasonwryan - Having only now heard of jshon for the first time, I followed your link. Having read that, I can only say that I left very pleased that I, by chance, heard of and installed jq first. I think you might like to hear of it too if you have not already - it doesn't bother with all of those command-line switches and can do its own regexes - it even allows you to declare functions and variables if you wish. See the answer here about it if you're interested. – mikeserv Oct 8 '14 at 16:20
8

As I understand it you're looking for the value of "Foo". This is really easy to do with the shell command-line tool jq. It is something like sed in that it implements its own kind of parser language. Given your example:

json='
{
    "Reservations" :
     {  
            "OwnerId" : "1345345",
            "Groups" :  [],
            "SecurityGroups" : [
               {
                  "Foo" : "yes",
                  "Bar" : "no"
               }
             ]
     }
}'

jq can get yes as simply as:

printf %s "$json" |
jq '.[].SecurityGroups[0].Foo?'                                                

OUTPUT

"yes"

You can walk through an object hash or dictionary list using the .dot notation, and indexed arrays can be indexed more simply, with, as you have probably guessed, numeric, square-bracketed indices. In the command above I use the empty index form to indicate that I want all of that level's iterable items expanded. That may be easier to understand in this way:

printf %s "$json" | jq '.[][]'

... which breaks out all values for the second level items in the hash and gets me...

"1345345"
[]
[
  {
    "Foo": "yes",
    "Bar": "no"
  }
]

This barely scratches the surface with regards to jq's capabilities. It is an immensely powerful tool for serializing data in the shell, it compiles to a single executable binary in the classic Unix-style, it is very likely available via package-manager for your distribution, and it is very well documented. Please visit its git-page and see for yourself.

By the way, another way to tackle layered-data in json - at least to get an idea of what you're working with - might be to go the other way and use the .dot notation to break out all values at all levels like:

printf %s "$json" | jq '..'

{
  "Reservations": {
    "OwnerId": "1345345",
    "Groups": [],
    "SecurityGroups": [
      {
        "Foo": "yes",
        "Bar": "no"
      }
    ]
  }
}
{
  "OwnerId": "1345345",
  "Groups": [],
  "SecurityGroups": [
    {
      "Foo": "yes",
      "Bar": "no"
    }
  ]
}
"1345345"
[]
[
  {
    "Foo": "yes",
    "Bar": "no"
  }
]
{
  "Foo": "yes",
  "Bar": "no"
}
"yes"
"no"

But far better, probably, would be just to use one of the many discovery or search methods that jq offers for the various types of nodes.

10

This is an answer to your objective, but not your question. Meaning you can accomplish your goal without using a JSON parser.

The AWS cli util has the ability to only output select fields using the --query argument. This is documented here.

For example:

$ aws ec2 describe-instances \
  --query 'Reservations[0].Instances[0].SecurityGroups[0].GroupName' \
  --instance-id i-6b272337 \
  --output text
mongodb

You can even select multiple fields if you'd like:

$ aws ec2 describe-instances \
  --query 'Reservations[0].Instances[0].SecurityGroups[0].[GroupName,GroupId]' \
  --instance-id i-6b272337 \
  --output text
mongodb sg-710ffa14

And you can show multiple matching structs as well:

$ aws ec2 describe-instances \
  --query 'Reservations[0].Instances[0].SecurityGroups[*].[GroupName,GroupId]' \
  --instance-id i-6b272337 \
  --output text
mongodb sg-710ffa14
default sg-a0243bcc
  • 1
    Thanks! That's very helpeful. I am accepting @mikeserv mostly to serve the community with the answer to the question, but your answer is the one I will be using – Amelio Vazquez-Reina Oct 8 '14 at 17:38
  • This is very helpful! Thank you very much!! – MD Sayem Ahmed Sep 27 '18 at 11:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.