2

I am trying to merge 4 JSON files (in the future maybe two more). The only common factor between the newline delimited JSON files is "vulnid".

File 1: 
{"vulnid":"cve1", "product":"product1"}
{"vulnid":"cve2", "product":"product2"}
{"vulnid":"cve3", "product":"product3"}
{"vulnid":"cve4", "product":""}
{"vulnid":"cve5", "product":""}
{"vulnid":"cve6", "product":""}

File 2: 
{"vulnid":"cve1", "version":"version1"}
{"vulnid":"cve2", "version":"version2"}
{"vulnid":"cve3", "version":"version3"}
{"vulnid":"cve4", "version":"version4"}

File 3: 
{"vulnid":"cve1", "patch":"patch1"}
{"vulnid":"cve2", "patch":"patch2"}
{"vulnid":"cve3", "patch":"patch3"}
{"vulnid":"cve4", "patch":""}

File 4: 
{"vulnid":"cve1", "speed":"speed1"}
{"vulnid":"cve2", "power":"power2"}
{"vulnid":"cve3", "amps":"amps3"}
{"vulnid":"cve4", "product":"product4"}
{"vulnid":"cve4", "patch":"patch4"}


Required output:
{"vulnid":"cve1", "product":"product1", "version":"version1", "patch":"patch1", "speed":"speed1"}
{"vulnid":"cve2", "product":"product2", "version":"version2", "patch":"patch2", "power":"power2"}
{"vulnid":"cve3", "product":"product3", "version":"version3", "patch":"patch3", "amps":"amps3"}
{"vulnid":"cve4", "product":"product4", "version":"version4", "patch":"patch4"}
{"vulnid":"cve5", "product":""}
{"vulnid":"cve6", "product":""}

What I have tried so far:

jq -s '.[0] * .[1] * .[3] * .[4]' json1 json2 json3 json4
jq -s '.[0] + .[1] + .[3] + .[4]' json1 json2 json3 json4

... and multiple different combinations of jq, all of them provided incorrect results and proved the point that I am not understand the tool.

Is this even possible to accomplish such merging using the field "vulnid" with jq?

2 Answers 2

3

The shortest jq solution:

jq -sc 'group_by(.vulnid)[] | add' file*.json

The output:

{"vulnid":"cve1","product":"product1","version":"version1","patch":"patch1","speed":"speed1"}
{"vulnid":"cve2","product":"product2","version":"version2","patch":"patch2","power":"power2"}
{"vulnid":"cve3","product":"product3","version":"version3","patch":"patch3","amps":"amps3"}
{"vulnid":"cve4","product":"product4","version":"version4","patch":"patch4"}
{"vulnid":"cve5","product":""}
{"vulnid":"cve6","product":""}
1
  • ah, I just couldn't wrap my head around group_by, but that does make sense.
    – ilkkachu
    Jul 5, 2018 at 8:35
1

Yeah, I think so.

If we change the data so that we have the vulnid strings as keys, with the object itself as the corresponding value (e.g. {"cve1": {"vulnid": "cve1", "product": "product1" }}), then we can use reduce and * to combine the ones with the same keys. The first map() below produces the correct format for the reduce.

So, with your sample input in the file json:

$ < json jq -s 'map({(.vulnid): . }) | reduce .[] as $item ({}; . * $item) '
{
  "cve1": {
    "vulnid": "cve1",
    "product": "product1",
    "version": "version1",
    "patch": "patch1",
    "speed": "speed1"
  },
  ...

Then it's rather simple to drop the fluff around the objects to get the expected output:

$ < json jq -s 'map({(.vulnid): . }) | 
                    reduce .[] as $item ({}; . * $item) | map(.) | .[]'
{
  "vulnid": "cve1",
  "product": "product1",
  "version": "version1",
  "patch": "patch1",
  "speed": "speed1"
}
...

I wouldn't be surprised if there was a better way to do all of this, though.

2
  • @Anna, uhh, comments don't work too well for formatted text, but if I got that right, I think you changed the input to {"cve1": {...}} manually. The first map(...) in my script was there to do just that, and doing it again doesn't work (since the new objects don't have the vulnid key). I meant to just push the sample data in your question to the command as-is (without the File N headers, of course).
    – ilkkachu
    Jul 4, 2018 at 20:24
  • I am sorry, worked! I don't have to parse as you suggested because the jq does that! :) Amazing, thanks!
    – Anna
    Jul 4, 2018 at 20:36

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .