I am trying to merge 4 JSON files (in the future maybe two more). The only common factor between the newline delimited JSON files is "vulnid".

File 1: 
{"vulnid":"cve1", "product":"product1"}
{"vulnid":"cve2", "product":"product2"}
{"vulnid":"cve3", "product":"product3"}
{"vulnid":"cve4", "product":""}
{"vulnid":"cve5", "product":""}
{"vulnid":"cve6", "product":""}

File 2: 
{"vulnid":"cve1", "version":"version1"}
{"vulnid":"cve2", "version":"version2"}
{"vulnid":"cve3", "version":"version3"}
{"vulnid":"cve4", "version":"version4"}

File 3: 
{"vulnid":"cve1", "patch":"patch1"}
{"vulnid":"cve2", "patch":"patch2"}
{"vulnid":"cve3", "patch":"patch3"}
{"vulnid":"cve4", "patch":""}

File 4: 
{"vulnid":"cve1", "speed":"speed1"}
{"vulnid":"cve2", "power":"power2"}
{"vulnid":"cve3", "amps":"amps3"}
{"vulnid":"cve4", "product":"product4"}
{"vulnid":"cve4", "patch":"patch4"}

Required output:
{"vulnid":"cve1", "product":"product1", "version":"version1", "patch":"patch1", "speed":"speed1"}
{"vulnid":"cve2", "product":"product2", "version":"version2", "patch":"patch2", "power":"power2"}
{"vulnid":"cve3", "product":"product3", "version":"version3", "patch":"patch3", "amps":"amps3"}
{"vulnid":"cve4", "product":"product4", "version":"version4", "patch":"patch4"}
{"vulnid":"cve5", "product":""}
{"vulnid":"cve6", "product":""}

What I have tried so far:

jq -s '.[0] * .[1] * .[3] * .[4]' json1 json2 json3 json4
jq -s '.[0] + .[1] + .[3] + .[4]' json1 json2 json3 json4

... and multiple different combinations of jq, all of them provided incorrect results and proved the point that I am not understand the tool.

Is this even possible to accomplish such merging using the field "vulnid" with jq?


The shortest jq solution:

jq -sc 'group_by(.vulnid)[] | add' file*.json

The output:

  • ah, I just couldn't wrap my head around group_by, but that does make sense.
    – ilkkachu
    Jul 5 '18 at 8:35

Yeah, I think so.

If we change the data so that we have the vulnid strings as keys, with the object itself as the corresponding value (e.g. {"cve1": {"vulnid": "cve1", "product": "product1" }}), then we can use reduce and * to combine the ones with the same keys. The first map() below produces the correct format for the reduce.

So, with your sample input in the file json:

$ < json jq -s 'map({(.vulnid): . }) | reduce .[] as $item ({}; . * $item) '
  "cve1": {
    "vulnid": "cve1",
    "product": "product1",
    "version": "version1",
    "patch": "patch1",
    "speed": "speed1"

Then it's rather simple to drop the fluff around the objects to get the expected output:

$ < json jq -s 'map({(.vulnid): . }) | 
                    reduce .[] as $item ({}; . * $item) | map(.) | .[]'
  "vulnid": "cve1",
  "product": "product1",
  "version": "version1",
  "patch": "patch1",
  "speed": "speed1"

I wouldn't be surprised if there was a better way to do all of this, though.

  • @Anna, uhh, comments don't work too well for formatted text, but if I got that right, I think you changed the input to {"cve1": {...}} manually. The first map(...) in my script was there to do just that, and doing it again doesn't work (since the new objects don't have the vulnid key). I meant to just push the sample data in your question to the command as-is (without the File N headers, of course).
    – ilkkachu
    Jul 4 '18 at 20:24
  • I am sorry, worked! I don't have to parse as you suggested because the jq does that! :) Amazing, thanks!
    – Anna
    Jul 4 '18 at 20:36
  • 1
    I spy a useless cat.
    – steve
    Jul 4 '18 at 22:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.