6

I have thousands of JSON file that I want to merge in one object. Those all are not in a similar format. Let me explain in detail. Here is the first sample of JSON

[
  {
    "value 1": 1,
    "value 2": 2,
    "value 3": 3,
    "value 4": 4
  }
]

and another types are not similar, something like following having few common fields and other fields

[
  {
    "value 3": 300,
    "value 4": 400,
    "value 5": 500,
    "value 6": 600
  }
]

Such as I have 2 file having first sample format and one file having second example format. I am trying to using jq for merging this.

jq -s '.' *.json > myfile.json

It is returning the following with three different objects

[
  {
    "value 1": 1,
    "value 2": 2,
    "value 3": 3,
    "value 4": 4
  }
],
[
  {
    "value 1": 10,
    "value 2": 20,
    "value 3": 30,
    "value 4": 40
  }
],
[
  {
    "value 3": 300,
    "value 4": 400,
    "value 5": 500,
    "value 6": 600
  }
]

I need to merge this into one object like following and if there any jq option to exclude those files that have specific field. Something like excluding those files which have the field "value 6". So finally the JSON output will be

[
  {
    "value 1": 1,
    "value 2": 2,
    "value 3": 3,
    "value 4": 4
  },
  {
    "value 1": 10,
    "value 2": 20,
    "value 3": 30,
    "value 4": 40
  }
]

2 Answers 2

6

You can use inputs to apply the filter on the contents of all the JSON files together and apply the select filter. The -n flag is to ensure the output JSON is constructed from scratch from the given input.

jq -n '[ inputs[] | select( has("value 6") | not ) ]' *.json

By doing jq -n 'inputs[]', all the objects in all the constituent JSON files are made available to the select function which discards any object containing key field as "value 6". The [..] surrounding the filter is put the final resultant objects within the array.

Another way would be to use a reduce() function to add the required objects in an iterative way

jq -n 'reduce inputs[] as $data (.; . + [ if $data | has("value 6") | not then $data else empty end ] )'
3
  • 1
    That's work like magic....hundred of thousands file merged within 1 hour. Can you help a little bit more? How can I remove duplicate? Something like, if "value 3" is duplicate of any other value then duplicate object will not merge. Thanks May 21, 2020 at 10:51
  • 3
    @Devil'sDream: Happy to be of help. But your question is completely new. I'm afraid I can't modify my current answer to your new requirement. Can you post a new question?
    – Inian
    May 21, 2020 at 10:55
  • I have posted a new question here unix.stackexchange.com/q/588098/413732 May 21, 2020 at 11:07
0

I created a command line tool using Node.js to merge nested json files locally.

https://github.com/divyeshmakwana96/injson

You can use !include('./path-to-json/item1.json') statement to embed contents of another json file. The child json files could also have include statement.

Example:

{
  "pi": 3.141,
  "happy": true,
  "questions:": [
    "!include('./questions/item1.json')",
    "!include('./questions/item2.json')"
  ],
  "answer": "!include('./answers/item1.json')"
}

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .