Merging many json files into one by merging it into a common object

Question

I have many json files of such format:

sample file1:

{
  "attributes": [
    {
      "name": "Node",
      "value": "test"
    }
  ]
}

sample file2:

{
  "attributes": [
    {
      "name": "version",
      "value": "11.1"
    }
  ]
}

etc.

I need to merge all of them to one json file, eg.

{
  "attributes": [
    {
      "name": "Node",
      "value": "test"
    },
    {
      "name": "version",
      "value": "11.1"
    }
  ]
}

Could someone please provide a solution with jq?

Answer 1

jq solution:

jq -s '{ attributes: map(.attributes[0]) }' file*.json

• -s (--slurp) - instead of running the filter for each JSON object in the input, read the entire input stream into a large array and run the filter just once.

Sample output:

{
  "attributes": [
    {
      "name": "Node",
      "value": "test"
    },
    {
      "name": "version",
      "value": "11.1"
    }
  ]
}

Answer 2

In contrast to RomanPerekhrest's solution, this will allow you merge the target field without remaking the entire object:

jq -s '.[0].attributes = [.[].attributes | add] | .[0]' file*.json

This returns the entirety of the first json file with the .attributes from all the others concatenated together.

Answer 3

Use jsonmerge

>>> base = {
...         "foo": 1,
...         "bar": [ "one" ],
...      }

>>> head = {
...         "bar": [ "two" ],
...         "baz": "Hello, world!"
...     }

>>> from pprint import pprint

>>> from jsonmerge import merge
>>> result = merge(base, head)

>>> pprint(result, width=40)
{'bar': ['two'],
 'baz': 'Hello, world!',
 'foo': 1}

Answer 4

Another jq variant, that doesn't involve slurp, but with inputs and reduce

jq -n 'reduce inputs as $in ({}; reduce ($in|keys_unsorted)[] as $k (.;
         .[$k] += $in[$k]))'  file1 file2 

Answer 5

Convert the JSON documents to TOML using yq (from https://kislyuk.github.io/yq/), and then back from TOML to JSON using tomlq (from the same yq distribution):

$ yq -t '.' file1 file2 | tomlq .
{
  "attributes": [
    {
      "name": "Node",
      "value": "test"
    },
    {
      "name": "version",
      "value": "11.1"
    }
  ]
}

This works because yq -t creates the concatenation of the two TOML documents

[[attributes]]
name = "Node"
value = "test"

and

[[attributes]]
name = "version"
value = "11.1"

Together, according to the rules of TOML, this happens to be equivalent of the JSON document that you are looking to create (by virtue of attributes being an array). So when tomlq converts it back to JSON, it does the right thing.

---

You can do the equivalent transformations using the yj utility (from https://github.com/sclevine/yj) too:

yj -tj < <( cat <( yj -jt <file1 ) <( yj -jt <file2 ) )

or, without the process substitutions and the cat,

{ yj -jt <file1; yj -jt <file2; } | yj -tj

It turns out you could do the same type of round-trip via Hashicorp's HCL format too, using yj -jc and yj -cj in place of yj -jt and yj -tj respectively.