JSON with jq into bash array / loop through

Question

I have a JSON result which I want to go through and send an alert, if one or several of the values in the JSON string is smashing thresholds.

This bash command:

for sat in `docker exec -i storagenode wget -qO - localhost:14002/api/sno | jq .satellites[].id -r`; do docker exec -i storagenode wget -qO - localhost:14002/api/sno/satellite/$sat | jq .id,.audits; done

Provides the following (excerpt):

"12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo"
{
  "auditScore": 1,
  "suspensionScore": 1,
  "onlineScore": 0.9974358974358974,
  "satelliteName": "us2.storj.io:7777"
}
"1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"
{
  "auditScore": 1,
  "suspensionScore": 1,
  "onlineScore": 0.9989041005632043,
  "satelliteName": "saltlake.tardigrade.io:7777"
}

Now I want to go through the results and e.g. in case, onlineScore drops below 0.9 (or suspensionScore resp. auditScore below 1.0, I want to create a text alert, which also includes the satelliteName. Example: auditScore below threshold: 0.98 for us2.storj.io

I guess I can start with the following (from JSON array to bash variables using jq), but I do not know, how to loop through the results and how to name and validate the fields:

jq -r '.[] | to_entries | .[] | .key + "=" + (.value | @sh)'

UPDATE #1

docker exec -i storagenode wget -qO - localhost:14002/api/sno

provides:

{"nodeID":"1veqEG5xuBNkt...","wallet":"12345","walletFeatures":["zksync"],"satellites":[{"id":"12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo","url":"us2.storj.io:7777","disqualified":null,"suspended":null,"currentStorageUsed":4592556672},{"id":"1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE","url":"saltlake.tardigrade.io:7777","disqualified":null,"suspended":null,"currentStorageUsed":513269323264},{"id":"121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6","url":"ap1.storj.io:7777","disqualified":null,"suspended":null,"currentStorageUsed":70956116864},{"id":"12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S","url":"us1.storj.io:7777","disqualified":null,"suspended":null,"currentStorageUsed":322340591104},{"id":"12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs","url":"eu1.storj.io:7777","disqualified":null,"suspended":null,"currentStorageUsed":148740125312},{"id":"12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB","url":"europe-north-1.tardigrade.io:7777","disqualified":null,"suspended":null,"currentStorageUsed":100681406976}],"diskSpace":{"used":1162257094528,"available":9500000000000,"trash":43196690332,"overused":0},"bandwidth":{"used":31670206976,"available":0},"lastPinged":"2022-01-02T07:55:51.886776586Z","version":"1.45.3","allowedVersion":"1.24.0","upToDate":true,"startedAt":"2021-12-31T00:00:32.209840775Z"}

And this one:

docker exec -i storagenode wget -qO - localhost:14002/api/sno/satellite/1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE

provides:

{"id":"1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE","storageDaily":[{"atRestTotal":11726579066524.266,"intervalStart":"2022-01-01T00:00:00Z"}],"bandwidthDaily":[{"egress":{"repair":390576640,"audit":7424,"usage":2373615872},"ingress":{"repair":2745100032,"usage":26744320},"delete":0,"intervalStart":"2022-01-01T00:00:00Z"},{"egress":{"repair":143168256,"audit":3584,"usage":786989568},"ingress":{"repair":1034401280,"usage":8107264},"delete":0,"intervalStart":"2022-01-02T00:00:00Z"}],"storageSummary":11726579066524.266,"bandwidthSummary":7508714240,"egressSummary":3694361344,"ingressSummary":3814352896,"currentStorageUsed":513269323264,"audits":{"auditScore":1,"suspensionScore":1,"onlineScore":0.9989041005632043,"satelliteName":"saltlake.tardigrade.io:7777"},"auditHistory":{"score":0.9989041005632043,"windows":[{"windowStart":"2021-12-03T00:00:00Z","totalCount":36,"onlineCount":36},{"windowStart":"2021-12-03T12:00:00Z","totalCount":30,"onlineCount":30},{"windowStart":"2021-12-04T00:00:00Z","totalCount":43,"onlineCount":42},{"windowStart":"2021-12-04T12:00:00Z","totalCount":68,"onlineCount":68},{"windowStart":"2021-12-05T00:00:00Z","totalCount":53,"onlineCount":53},{"windowStart":"2021-12-05T12:00:00Z","totalCount":36,"onlineCount":36},{"windowStart":"2021-12-06T00:00:00Z","totalCount":33,"onlineCount":33},{"windowStart":"2021-12-06T12:00:00Z","totalCount":53,"onlineCount":53},{"windowStart":"2021-12-07T00:00:00Z","totalCount":27,"onlineCount":27},{"windowStart":"2021-12-07T12:00:00Z","totalCount":65,"onlineCount":65},{"windowStart":"2021-12-08T00:00:00Z","totalCount":30,"onlineCount":30},{"windowStart":"2021-12-08T12:00:00Z","totalCount":27,"onlineCount":27},{"windowStart":"2021-12-09T00:00:00Z","totalCount":46,"onlineCount":46},{"windowStart":"2021-12-09T12:00:00Z","totalCount":48,"onlineCount":48},{"windowStart":"2021-12-10T00:00:00Z","totalCount":49,"onlineCount":49},{"windowStart":"2021-12-10T12:00:00Z","totalCount":67,"onlineCount":67},{"windowStart":"2021-12-11T00:00:00Z","totalCount":71,"onlineCount":71},{"windowStart":"2021-12-11T12:00:00Z","totalCount":52,"onlineCount":52},{"windowStart":"2021-12-12T00:00:00Z","totalCount":59,"onlineCount":59},{"windowStart":"2021-12-12T12:00:00Z","totalCount":77,"onlineCount":77},{"windowStart":"2021-12-13T00:00:00Z","totalCount":79,"onlineCount":79},{"windowStart":"2021-12-13T12:00:00Z","totalCount":79,"onlineCount":79},{"windowStart":"2021-12-14T00:00:00Z","totalCount":65,"onlineCount":65},{"windowStart":"2021-12-14T12:00:00Z","totalCount":59,"onlineCount":59},{"windowStart":"2021-12-15T00:00:00Z","totalCount":87,"onlineCount":87},{"windowStart":"2021-12-15T12:00:00Z","totalCount":82,"onlineCount":81},{"windowStart":"2021-12-16T00:00:00Z","totalCount":96,"onlineCount":96},{"windowStart":"2021-12-16T12:00:00Z","totalCount":66,"onlineCount":64},{"windowStart":"2021-12-17T00:00:00Z","totalCount":36,"onlineCount":36},{"windowStart":"2021-12-17T12:00:00Z","totalCount":48,"onlineCount":48},{"windowStart":"2021-12-18T00:00:00Z","totalCount":37,"onlineCount":37},{"windowStart":"2021-12-18T12:00:00Z","totalCount":60,"onlineCount":60},{"windowStart":"2021-12-19T00:00:00Z","totalCount":69,"onlineCount":69},{"windowStart":"2021-12-19T12:00:00Z","totalCount":32,"onlineCount":32},{"windowStart":"2021-12-20T00:00:00Z","totalCount":53,"onlineCount":53},{"windowStart":"2021-12-20T12:00:00Z","totalCount":37,"onlineCount":37},{"windowStart":"2021-12-21T00:00:00Z","totalCount":80,"onlineCount":80},{"windowStart":"2021-12-21T12:00:00Z","totalCount":57,"onlineCount":57},{"windowStart":"2021-12-22T00:00:00Z","totalCount":46,"onlineCount":46},{"windowStart":"2021-12-22T12:00:00Z","totalCount":33,"onlineCount":33},{"windowStart":"2021-12-23T00:00:00Z","totalCount":42,"onlineCount":42},{"windowStart":"2021-12-23T12:00:00Z","totalCount":73,"onlineCount":73},{"windowStart":"2021-12-24T00:00:00Z","totalCount":35,"onlineCount":35},{"windowStart":"2021-12-24T12:00:00Z","totalCount":44,"onlineCount":44},{"windowStart":"2021-12-25T00:00:00Z","totalCount":81,"onlineCount":81},{"windowStart":"2021-12-25T12:00:00Z","totalCount":43,"onlineCount":43},{"windowStart":"2021-12-26T00:00:00Z","totalCount":62,"onlineCount":62},{"windowStart":"2021-12-26T12:00:00Z","totalCount":79,"onlineCount":79},{"windowStart":"2021-12-27T00:00:00Z","totalCount":70,"onlineCount":70},{"windowStart":"2021-12-27T12:00:00Z","totalCount":90,"onlineCount":90},{"windowStart":"2021-12-28T00:00:00Z","totalCount":65,"onlineCount":65},{"windowStart":"2021-12-28T12:00:00Z","totalCount":77,"onlineCount":77},{"windowStart":"2021-12-29T00:00:00Z","totalCount":83,"onlineCount":83},{"windowStart":"2021-12-29T12:00:00Z","totalCount":99,"onlineCount":99},{"windowStart":"2021-12-30T00:00:00Z","totalCount":74,"onlineCount":74},{"windowStart":"2021-12-30T12:00:00Z","totalCount":84,"onlineCount":84},{"windowStart":"2021-12-31T00:00:00Z","totalCount":70,"onlineCount":70},{"windowStart":"2021-12-31T12:00:00Z","totalCount":93,"onlineCount":93},{"windowStart":"2022-01-01T00:00:00Z","totalCount":120,"onlineCount":120},{"windowStart":"2022-01-01T12:00:00Z","totalCount":112,"onlineCount":112},{"windowStart":"2022-01-02T00:00:00Z","totalCount":46,"onlineCount":46}]},"priceModel":{"EgressBandwidth":2000,"RepairBandwidth":1000,"AuditBandwidth":1000,"DiskSpace":150},"nodeJoinedAt":"2021-05-11T20:11:14.910165Z"}

Sorry, not sure how to better format the huge json content here in a better way.

UPDATE #2

There is a result for http://localhost:14002/api/sno/satellites, I was not aware of before. The result is:

{
...
   "storageSummary": 6.8624392E13,
...
   "audits": [
      {
         "auditScore": 1,
         "suspensionScore": 1,
         "onlineScore": 0.99743587,
         "satelliteName": "us2.storj.io:7777"
      },
      {
         "auditScore": 1,
         "suspensionScore": 1,
         "onlineScore": 0.9992917,
         "satelliteName": "saltlake.tardigrade.io:7777"
      },
...
      {
         "auditScore": 1,
         "suspensionScore": 1,
         "onlineScore": 0.99930555,
         "satelliteName": "ap1.storj.io:7777"
      }
   ]
}

Your sample JSON isn't well-formed, which makes it difficult to make a concrete suggestion. Can you update the example and make it look like what you'd really expect to see? — Andy Dalton, Jan 1, 2022 at 22:42
I felt inspired coz there are other questions here about the same topic, with other example JSON. Hmm — bjoern, Jan 1, 2022 at 22:59
Andy, yes, from what I can see from the select, it’s possible to restructure: for sat in `docker exec -i storagenode wget -qO - localhost:14002/api/sno | jq .satellites[].id -r`; do docker exec -i storagenode wget -qO - localhost:14002/api/sno/satellite/$sat | jq .id,.audits; done — bjoern, Jan 1, 2022 at 23:00
You most recent comment refers to keys id and audit which is not part of the JSON document structure that you are showing in your question. Consider updating your data so that it corresponds to the data that you are actually working with. The thing you want to do is trivial with jq, but very difficult to help you with when you only show snippets of selected pieces data. — Kusalananda, Jan 2, 2022 at 6:33

Kusalananda · Accepted Answer · 2022-01-03 21:42:10Z

It's generally a good idea to avoid looping over the result of a command substitution. It's inelegant as the command in the substitution must finish executing before the loop can even start running, it's inefficient as the full result of the command in the substitution must be stored in memory, and it's prone to errors since the shell must be allowed to split the output of the command on whitespace and subject the resulting words to filename globbing.

It's better to use read in a while loop:

#!/bin/sh

curl -s 'localhost:14002/api/sno' |
jq -r '.satellites[].id' |
while IFS= read -r id; do
        curl -s 'localhost:14002/api/sno/satellite/'"$id"
done |
jq -r \
        --argjson auditScore 1 \
        --argjson suspensionScore 1 \
        --argjson onlineScore 0.9 \
        '.audits as $a | $a.satelliteName as $name |
        reduce ($ARGS.named|keys[]) as $key (
                [];
                if $a[$key] < $ARGS.named[$key] then (
                        . + ["\($key) below threshold: \($a[$key]) for \($name)"]
                ) else . end
        ) | .[]'

This script assumes that you can contact your REST endpoints on localhost:14002 (the Docker container might be made to expose that port, for example). If you need to use the docker exec command to access the API, then replace the plain calls to curl with, e.g.

docker exec -i curl -s 'localhost:14002/api/sno'

For the updated question, using the api/sno/satellites endpoint:

#!/bin/sh

curl -s 'localhost:14002/api/sno/satellites' |
jq -r \
        --argjson auditScore 1 \
        --argjson suspensionScore 1 \
        --argjson onlineScore 0.9 \
        '.audits[] as $a | $a.satelliteName as $name |
        reduce ($ARGS.named|keys[]) as $key (
                [];
                if $a[$key] < $ARGS.named[$key] then (
                        . + ["\($key) below threshold: \($a[$key]) for \($name)"]
                ) else . end
        ) | .[]'

Apart from a minor adjustment to the jq expression, this is essentially the same code as above, but bypassing the first curl call to fetch all the IDs, and the loop.

Thank you! Yes, I gave it a try and there is a sno/satellitesresult. I'll add it to the question above, as the comment field is limited by the number of characters. — bjoern, Jan 3, 2022 at 19:17
jq: error: ARGS/0 is not defined at <top-level>, line 2: reduce ($ARGS.named|keys[]) as $key ( jq: error: ARGS/0 is not defined at <top-level>, line 4: if $a[$key] < $ARGS.named[$key] then ( jq: 2 compile errors (23) Failed writing body — bjoern, Jan 4, 2022 at 9:22
@bjoern Are you using an older version of jq maybe? I'm on release 1.6 (released November 2018). — Kusalananda, Jan 4, 2022 at 9:25
@bjoern Use ($a.satelliteName | sub(":.*";"")) as $name instead of just $a.satelliteName as $name, then use \(100*$a[$key]|floor)% instead of \($a[$key]). — Kusalananda, Jan 4, 2022 at 22:17

bjoern · Accepted Answer · 2022-01-03 19:27:03Z

I‘ve found one possible answer on stackoverflow:


for sat in `docker exec -i storagenode wget -qO - localhost:14002/api/sno | jq .satellites[].id -r`
do
  docker exec -i storagenode wget -qO - localhost:14002/api/sno/satellite/$sat \
  | jq --raw-output \
    --argjson auditThreshold 1 \
    --argjson suspensionThreshold 1 \
    --argjson onlineThreshold 1 \
    '.audits
      | .satelliteName as $name
      | (
          [{auditScore}, $auditThreshold],
          [{suspensionScore}, $suspensionThreshold],
          [{onlineScore}, $onlineThreshold]
        )
      | select(.[0][] < .[1])
      | "\(.[0] | keys[]) (\(.[0][])) below threshold (\(.[1])) for \($name)"
    '
done

Stack Exchange Network

JSON with jq into bash array / loop through

2 Answers 2

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged
bash
jq
.

Linked

Hot Network Questions

JSON with jq into bash array / loop through

2 Answers 2

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged bashjq.

Linked

Related

Hot Network Questions

Not the answer you're looking for? Browse other questions tagged
bash
jq
.