3

I have thousands of JSON file that I want to merge in one object. Those all are not in a similar format. Let me explain in detail. Here is the first sample of JSON

[
  {
    "value 1": 1,
    "value 2": 2,
    "value 3": 3,
    "value 4": 4
  }
]

and another types are not similar, something like following having few common fields and other fields

[
  {
    "value 3": 300,
    "value 4": 400,
    "value 5": 500,
    "value 6": 600
  }
]

Such as I have 2 file having first sample format and one file having second example format. I am trying to using jq for merging this.

jq -s '.' *.json > myfile.json

It is returning the following with three different objects

[
  {
    "value 1": 1,
    "value 2": 2,
    "value 3": 3,
    "value 4": 4
  }
],
[
  {
    "value 1": 10,
    "value 2": 20,
    "value 3": 30,
    "value 4": 40
  }
],
[
  {
    "value 3": 300,
    "value 4": 400,
    "value 5": 500,
    "value 6": 600
  }
]

I need to merge this into one object like following and if there any jq option to exclude those files that have specific field. Something like excluding those files which have the field "value 6". So finally the JSON output will be

[
  {
    "value 1": 1,
    "value 2": 2,
    "value 3": 3,
    "value 4": 4
  },
  {
    "value 1": 10,
    "value 2": 20,
    "value 3": 30,
    "value 4": 40
  }
]
5

You can use inputs to apply the filter on the contents of all the JSON files together and apply the select filter. The -n flag is to ensure the output JSON is constructed from scratch from the given input.

jq -n '[ inputs[] | select( has("value 6") | not ) ]' *.json

By doing jq -n 'inputs[]', all the objects in all the constituent JSON files are made available to the select function which discards any object containing key field as "value 6". The [..] surrounding the filter is put the final resultant objects within the array.

Another way would be to use a reduce() function to add the required objects in an iterative way

jq -n 'reduce inputs[] as $data (.; . + [ if $data | has("value 6") | not then $data else empty end ] )'
3
  • That's work like magic....hundred of thousands file merged within 1 hour. Can you help a little bit more? How can I remove duplicate? Something like, if "value 3" is duplicate of any other value then duplicate object will not merge. Thanks – Devil's Dream May 21 '20 at 10:51
  • 2
    @Devil'sDream: Happy to be of help. But your question is completely new. I'm afraid I can't modify my current answer to your new requirement. Can you post a new question? – Inian May 21 '20 at 10:55
  • I have posted a new question here unix.stackexchange.com/q/588098/413732 – Devil's Dream May 21 '20 at 11:07
0

I created a command line tool using Node.js to merge nested json files locally.

https://github.com/divyeshmakwana96/injson

You can use !include('./path-to-json/item1.json') statement to embed contents of another json file. The child json files could also have include statement.

Example:

{
  "pi": 3.141,
  "happy": true,
  "questions:": [
    "!include('./questions/item1.json')",
    "!include('./questions/item2.json')"
  ],
  "answer": "!include('./answers/item1.json')"
}

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.