I have a json as shown below where I have multiple json objects inside rootLevel. I need to validate my json to make sure each json object should follow this below rules:

  • Value in clientId, procId and machineName in each json object should be unique. If they are not unique then I want to fail it by printing what is not unique.
  • Required fields are: clientId, some_key6, some_key7, some_key8, some_key9, some_key10 and machineName. If they are not present in any json object then I want to fail it as well.

Below is my json. As of now it only has one json object in rootLevel array but in general it has a lot in similar fashion.

    "rootLevel": [
            "clientId": 1234,
            "processId": 1,
            "clientName": "Hello",
            "some_key1": 4,
            "some_key2": 9,
            "some_key3": "rcb.com",
            "some_key4": 7,
            "some_key5": 301,
            "some_key6": 78,
            "some_key7": 2,
            "some_key8": 1,
            "some_key9": "USD",
            "some_key10": "en-US",
            "some_key11": 45,
            "some_key12": {
                "SportsId": [
                "Team": [
            "procId": [
            "machineName": [

Can we do these validations on each json object using shell script or is there anything else suited for this? I have named few keys as some_key# just to make it easier to understand but in general they have different names.

  • Kind of a tall order. The gold standard for processing json data is jq, however I think json is more intuitive/user friendly. Either tool should be able to help you write a validation script that meets your needs.
    – jesse_b
    Nov 16 '20 at 15:40
  • I suggest you to update your question with a minified valid json file. One table, 3 objects, 4 lines each, no scrollbars.
    – thanasisp
    Nov 16 '20 at 18:28
  • I updated the question just now.
    – AndyP
    Nov 16 '20 at 18:31
  • You may want to look into using/writing a JSON schema and using a schema validator. How to do that is (I believe) too big a topic for an answer here. See e.g. code.tutsplus.com/tutorials/…
    – Kusalananda
    Nov 17 '20 at 14:30
  • 1
    It could be possible that some json object might not have some fields but some json object can have those fields. When you say order what does that mean? you mean all fields will they appear in same order in each json object? if yes then no they can be in difference places as well
    – AndyP
    Nov 17 '20 at 17:26

Test if values are unique

If I want to do a fast check for if all objects in the array have unique clientId, I would use jq only to print this value and check in the shell.

< file.json jq '.rootLevel[].clientId' | awk '
    seen[$0]++ {print "FAILURE"; f=1; exit}
    END {if (!f) print "SUCCESS"}'

You could also do it with jq combining length, unique, unique_by or other functions, man jq for more.

Print only duplicates

If you want to print all objects having duplicated .clientId, you could do this:

jq '.rootLevel | group_by(.clientId) | map(select(length>1))' file.json

group_by will create separate arrays per clientId, and we select only them with more than one element, that means those having duplicated clientId. Or you could add the length function to count them (another way to test if values are unique):

jq '.rootLevel | group_by(.clientId) | map(select(length>1)) | length' file.json

Test if field exists for all objects

If I want to check that all objects of this json array have a .processId field, I would do this:

jq '.rootLevel[] | .processId?' file.json | grep -m1 null

and test the exit status of grep. That means this:

if jq '.rootLevel[] | .processId?' file.json | grep -qm1 null; then
    echo "FAILURE"
    echo "SUCCESS"

Combining the above directions, you can do all that you mention into your question.

If you have a lot of work to do with similar stuff, consider using a "JSON Database" (or "Document Database" or "Nosql Database").

  • Sorry I was out for a while. Is there any way to combine all these in one shell script? I am kinda new so having issues in converting that.
    – AndyP
    Nov 23 '20 at 16:14
  • Yes, and basic shell scripting skills are required. If you have a spesific question about how to do that, you can post new questions or find your answers on existing posts.
    – thanasisp
    Nov 23 '20 at 17:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.