4

I am looking to merge two different JSON files into one file -

file 1

[
  {
    "namespace": "lerjr-rb26-sandbox-keycloak",
    "name": "keycloak",
    "url": "REMOVED"
  },
  {
    "namespace": "prometheus-brr1-sandbox",
    "name": "abacus",
    "url": "REMOVED"
  },
  {
    "namespace": "prometheus-brr1-sandbox",
    "name": "keycloak",
    "url": "REMOVED"
  },
  {
    "namespace": "prometheus-nginxredirect-sandbox",
    "name": "riskpoynt-redirect444",
    "url": "REMOVED"
  },
  {
    "namespace": "prometheus-nginxredirect-sandbox",
    "name": "riskpoynt-redirect447",
    "url": "REMOVED"
  },
  {
    "namespace": "prometheus-x105-sandbox",
    "name": "epas",
    "url": "REMOVED"
  },
  {
    "namespace": "prometheus-x105-sandbox",
    "name": "epas-oidc",
    "url": "REMOVED"
  },
  {
    "namespace": "prometheus-x105-sandbox",
    "name": "keycloak",
    "url": "REMOVED"
  }
]

file 2

[
  {
    "namespace": "ler-rb26-sandbox",
    "labels": {
      "pg.customerName": "prometheus",
      "pg.customerCode": "rb26",
      "pg.product": "keycloak",
      "pg.environment": null
    }
  },
  {
    "namespace": "lerjr-rb26-sandbox-keycloak",
    "labels": {
      "pg.customerName": "lerjr",
      "pg.customerCode": "rb26",
      "pg.product": "keycloak",
      "pg.environment": null
    }
  },
  {
    "namespace": "prometheus-brr1-sandbox",
    "labels": {
      "pg.customerName": "prometheus",
      "pg.customerCode": null,
      "pg.product": null,
      "pg.environment": null
    }
  }
]

EXPECTED OUTPUT

[
  {
    "namespace": "lerjr-rb26-sandbox-keycloak",
    "labels": {
      "pg.customerName": "lerjr",
      "pg.customerCode": "rb26",
      "pg.product": "keycloak",
      "pg.environment": null
    },
    "ingresses": [
      {
        "name": "keycloak",
        "url": "REMOVED"
      }
    ]
  },
  {
    "namespace": "prometheus-brr1-sandbox",
    "labels": {
      "pg.customerName": "prometheus",
      "pg.customerCode": null,
      "pg.product": null,
      "pg.environment": null
    },
    "ingresses": [
      {
        "name": "abacus",
        "url": "REMOVED"
      },
      {
        "name": "keycloak",
        "url": "REMOVED"
      }
    ]
  },
  {
    "namespace": "prometheus-nginxredirect-sandbox",
    "labels": {
      "pg.customerName": "null",
      "pg.customerCode": null,
      "pg.product": null,
      "pg.environment": null
    },
    "ingresses": [
      {
        "name": "riskpoynt-redirect444",
        "url": "REMOVED"
      },
      {
        "name": "riskpoynt-redirect447",
        "url": "REMOVED"
      }
    ]
  },
  {
    "namespace": "prometheus-x105-sandbox",
    "labels": {
      "pg.customerName": "null",
      "pg.customerCode": null,
      "pg.product": null,
      "pg.environment": null
    },
    "ingresses": [
      {
        "name": "epas",
        "url": "REMOVED"
      },
      {
        "name": "epas-oidc",
        "url": "REMOVED"
      },
      {
        "name": "keycloak",
        "url": "REMOVED"
      }
    ]
  }
]

NOTE that the final product has labels (null if not present) and if there are no ingresses the namespace entry is removed. the namespace key/value pair is the link between the two files - file 1 is the primary file I suppose, I do not want the namespace entry from file 2 if there is no namespace entry in file 1.

Any and all help would be greatly appreciated I have been struggling with this.

1 Answer 1

5

Passing the "file 2" file first, then the "file 1" file, to the following jq expression, we perform a relational JOIN operation between the two using jq's SQL-style operators.

JOIN(
        INDEX(input[]; .namespace);
        input | group_by(.namespace)[];
        first.namespace;
        {
                namespace: first[0].namespace,
                labels: (
                        last.labels // { 
                                "pg.customerName": null,
                                "pg.customerCode": null,
                                "pg.product": null,
                                "pg.environment": null
                        }
                ),
                ingresses: ( first | map(del(.namespace)) )
        }
)

The JOIN() function expects an index computed for the left join data set, a data stream of objects representing the entries of the data set we're joining with, an expression that picks out the key from the elements we're joining with, and an expression that creates resulting objects from joined pairs.

The INDEX() function, acting on the "file 2" file, will create an index based on the namespace key. Its output will be the following (notice how the namespace values are now used as keys):

{
  "ler-rb26-sandbox": {
    "namespace": "ler-rb26-sandbox",
    "labels": {
      "pg.customerName": "prometheus",
      "pg.customerCode": "rb26",
      "pg.product": "keycloak",
      "pg.environment": null
    }
  },
  "lerjr-rb26-sandbox-keycloak": {
    "namespace": "lerjr-rb26-sandbox-keycloak",
    "labels": {
      "pg.customerName": "lerjr",
      "pg.customerCode": "rb26",
      "pg.product": "keycloak",
      "pg.environment": null
    }
  },
  "prometheus-brr1-sandbox": {
    "namespace": "prometheus-brr1-sandbox",
    "labels": {
      "pg.customerName": "prometheus",
      "pg.customerCode": null,
      "pg.product": null,
      "pg.environment": null
    }
  }
}

We group the incoming data from the other file on the namespace key, which gives us an array with each element being an array of objects with the same namespace value. The namespace value from the first such object serves as the join key.

The last part of the JOIN() function is an expression that combines a pair of objects, one from the first file and one from the second, having been matched by namespace value. The labels are picked from "file 2" if they exist, otherwise a null-value label is inserted.

You would run this using

jq -n -f script.jq file2 file1

... and it would output the following:

{
  "namespace": "lerjr-rb26-sandbox-keycloak",
  "labels": {
    "pg.customerName": "lerjr",
    "pg.customerCode": "rb26",
    "pg.product": "keycloak",
    "pg.environment": null
  },
  "ingresses": [
    {
      "name": "keycloak",
      "url": "REMOVED"
    }
  ]
}
{
  "namespace": "prometheus-brr1-sandbox",
  "labels": {
    "pg.customerName": "prometheus",
    "pg.customerCode": null,
    "pg.product": null,
    "pg.environment": null
  },
  "ingresses": [
    {
      "name": "abacus",
      "url": "REMOVED"
    },
    {
      "name": "keycloak",
      "url": "REMOVED"
    }
  ]
}
{
  "namespace": "prometheus-nginxredirect-sandbox",
  "labels": {
    "pg.customerName": null,
    "pg.customerCode": null,
    "pg.product": null,
    "pg.environment": null
  },
  "ingresses": [
    {
      "name": "riskpoynt-redirect444",
      "url": "REMOVED"
    },
    {
      "name": "riskpoynt-redirect447",
      "url": "REMOVED"
    }
  ]
}
{
  "namespace": "prometheus-x105-sandbox",
  "labels": {
    "pg.customerName": null,
    "pg.customerCode": null,
    "pg.product": null,
    "pg.environment": null
  },
  "ingresses": [
    {
      "name": "epas",
      "url": "REMOVED"
    },
    {
      "name": "epas-oidc",
      "url": "REMOVED"
    },
    {
      "name": "keycloak",
      "url": "REMOVED"
    }
  ]
}

Note that these are loose objects that are not part of an array. If you wish to collect them in an array, either put the whole of that jq expression within [...], or pass the output through jq -s ..

1
  • 1
    This is it!!! Thank you SO MUCH I have been banging my head on this and it's such an awesome learning experience here. It does exactly what I'm looking for. Commented May 30 at 11:50

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .