19

I've got a JSON array like so:

{
  "SITE_DATA": {
    "URL": "example.com",
    "AUTHOR": "John Doe",
    "CREATED": "10/22/2017"
  }
}

I'm looking to iterate over this array using jq so I can set the key of each item as the variable name and the value as it's value.

Example:

  • URL="example.com"
  • AUTHOR="John Doe"
  • CREATED="10/22/2017"

What I've got so far iterates over the array but creates a string:

constants=$(cat ${1} | jq '.SITE_DATA' | jq -r "to_entries|map(\"\(.key)=\(.value|tostring)\")|.[]")

Which outputs:

URL=example.com
AUTHOR=John Doe
CREATED=10/22/2017

I am looking to use these variables further down in the script:

echo ${URL}

But this echos an empty output at the moment. I'm guessing I need an eval or something in there but can't seem to put my finger on it.

30

Your original version isn't going to be evalable because the author name has spaces in it - it would be interpreted as running a command Doe with the environment variable AUTHOR set to John. There's also virtually never a need to pipe jq to itself - the internal piping & dataflow can connect different filters together.

You can make a much simpler version of the jq program:

jq -r '.SITE_DATA | to_entries | .[] | .key + "=\"" + .value + "\""'

which outputs:

URL="example.com"
AUTHOR="John Doe"
CREATED="10/22/2017"

There's no need for a map: .[] deals with taking each object in the array through the rest of the pipeline as a separate item, so everything after the last | is applied to each one separately. At the end, we just assemble a valid shell assignment string with ordinary + concatenation, including quotes around the value.

All the pipes matter here - without them you get fairly unhelpful error messages, where parts of the program are evaluated in subtly different contexts.

This string is evalable as long as the characters `, $, newline and null don't appear in the data:

eval "$(jq -r '.SITE_DATA | to_entries | .[] | .key + "=\"" + .value + "\""' < data.json)"
echo "$AUTHOR"

As ever when using eval, be careful that you trust the data you're getting, since if it's malicious or just in an unexpected format things could go very wrong.

| improve this answer | |
15

Building on @Michael Homer's answer, you can avoid a potentially-unsafe eval entirely by reading the data into an associative array.

For example, if your JSON data is in a file called file.json:

#!/bin/bash

typeset -A myarray

while IFS== read -r key value; do
    myarray["$key"]="$value"
done < <(jq -r '.SITE_DATA | to_entries | .[] | .key + "=" + .value ' file.json)

# show the array definition
typeset -p myarray

# make use of the array variables
echo "URL = '${myarray[URL]}'"
echo "CREATED = '${myarray[CREATED]}'"
echo "AUTHOR = '${myarray[URL]}'"

Output:

$ ./read-into-array.sh 
declare -A myarray=([CREATED]="10/22/2017" [AUTHOR]="John Doe" [URL]="example.com" )
URL = 'example.com'
CREATED = '10/22/2017'
AUTHOR = 'example.com'
| improve this answer | |
  • 1
    You could also indirect the assignment alone with declare -- “$key=$value” and have $AUTHOR etc work as in the original, without an array. It’s still safer than eval, though changing PATH or something is still possible so less so than this version. – Michael Homer Dec 31 '17 at 5:32
  • 1
    yeah, the array nicely isolates the variables into a container of your choosing - no chance of accidentally/maliciously messing with important environment variables. you could make your declare -- version safe by comparing $key against a list of allowed variable names. – cas Dec 31 '17 at 5:47
1

Just realized that I can loop over the results and eval each iteration:

constants=$(cat ${1} | jq '.SITE_DATA' | jq -r "to_entries|map(\"\(.key)=\(.value|tostring)\")|.[]")

for key in ${constants}; do
  eval ${key}
done

Allows me to do:

echo ${AUTHOR}
# outputs John Doe
| improve this answer | |
0

I really like the @Michel suggestion. Sometimes, you may really just extract some variables value to execute a task in that specific server using BASH. So, desired variables are know. This using this approach is the way to avoid or multiple calls to jq to set a value per variable or even to using the read statement with multiple variables in which some can be valid and empty, leading to a value shift (that was my problem)

my previous approach that lead will lead to a value shift error if .svID[ ].ID="" (sv will get the slotID value

-rd '\n' getInfo sv slotID <<< $(jq -r '(.infoCMD // "no info"), (.svID[].ID // "none"), (._id // "eeeeee")' <<< $data)

If you downloaded the object using curl, here is my approach to rename some variables to a friendly name as extract data from data arrays

using eval and filters will solve the problem with one line and will produce variables with the desired name

eval "$(jq -r '.[0] | {varNameExactasJson, renamedVar1: .var1toBeRenamed, varfromArray: .array[0].var, varValueFilter: (.array[0].textVar|ascii_downcase)} | to_entries | .[] | .key + "=\"" + (.value | tostring) + "\""' <<< /path/to/file/with/object )"  

The advantage in this case, is the fact that it will filter, rename, format all the desired variables in the first step. Observe that in there is .[0] | that is very common to have if the source if from a RESTFULL API server using GET, response data as:

[{"varNameExactasJson":"this value", "var1toBeRenamed: 1500, ....}]

If your data is not from an array, ie. is an object like:

{"varNameExactasJson":"this value", "var1toBeRenamed: 1500, ....}

just remove the initial index:

eval "$(jq -r '{varNameExactasJson, renamedVar1: .var1toBeRenamed, varfromArray: .array[0].var, varValueFilter: (.array[0].textVar|ascii_downcase)} | to_entries | .[] | .key + "=\"" + (.value | tostring) + "\""' <<< /path/to/file/with/object )"  

This is an old question, but I felt sharing, since it was hard to find

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.