Merging multiple JSON files removing duplicate objects by a field [closed]

Question

I am trying to merging hundreds of thousand JSON files that have a single object. with the help of this answer I have successfully added all objects into a single object as an array. Now I want to filter out some objects during merge if a specific field or parameter is duplicate. Something like field 1 of two objects have the same value.

Here is the example

Input File 1:

[
    {
      "field 1": 10,
      "field 2": 25,
      "field 3": 35,
      "field 4": 45
    }
]

Input file 2:

[
    {
      "field 1": 15,
      "field 2": 25,
      "field 3": 35,
      "field 4": 45
    }
]

Input file 3:

[
    {
      "field 1": 10,
      "field 2": 20,
      "field 3": 30,
      "field 4": 40
    }
]

Expected output:

[
    {
      "field 1": 10,
      "field 2": 25,
      "field 3": 35,
      "field 4": 45
    },
    {
      "field 1": 15,
      "field 2": 25,
      "field 3": 35,
      "field 4": 45
    }
]

The file 3 object will not merge due to having field 1 value same

Answer 1

You just use the unique_by() function in jq which expects an array of objects and takes the field name as the input to remove duplicates on

jq -n '[ inputs[] ] | unique_by(."value 1")'

To append on top of my other answer to uniquely select based on one field and remove objects by another field you can do below. The below code groups objects excluding duplicates based on "value 1" and excludes ones containing "value 6"

jq -n '[ [inputs[]] | unique_by(."value 1")[] | select( has("value 6") | not )]  '