32

I have a JSON output that contains a list of objects stored in a variable. (I may not be phrasing that right)

[
  {
    "item1": "value1",
    "item2": "value2",
    "sub items": [
      {
        "subitem": "subvalue"
      }
    ]
  },
  {
    "item1": "value1_2",
    "item2": "value2_2",
    "sub items_2": [
      {
        "subitem_2": "subvalue_2"
      }
    ]
  }
]

I need all the values for item2 in a array for a bash script to be run on ubuntu 14.04.1.

I have found a bunch of ways to get the entire result into an array but not just the items I need

0

6 Answers 6

33

Using :

readarray arr < <(jq '.[].item2' json)
printf '%s\n' "${arr[@]}"

If you need a more hardened way:

readarray -td '' arr

for inputs with newlines or other special characters, avoiding word splitting.

Output:

value2
value2_2

Check:

Process Substitution >(command ...) or <(...) is replaced by a temporary filename. Writing or reading that file causes bytes to get piped to the command inside. Often used in combination with file redirection: cmd1 2> >(cmd2). See http://mywiki.wooledge.org/ProcessSubstitution http://mywiki.wooledge.org/BashFAQ/024

12
  • Is it possible to do this from a variable instead of a file? I try to avoid excess filesystem access if I don't need it. Once I pull this array I am done with the json output.
    – JpaytonWPD
    Commented Jan 7, 2015 at 0:10
  • I tried replacing json with $JSON but jq is only looking for files. The json output is from a API call gathered with curl. Your answer does work if I store the output in a file. I was just hoping to avoid it.
    – JpaytonWPD
    Commented Jan 7, 2015 at 0:17
  • 7
    Great jq command, but please don't parse command output into an array with arr=( $(...) ) (even though it happens to work with the sample input): it doesn't work as intended with embedded or leading/trailing whitespace and can result in accidental globbing.
    – mklement0
    Commented Jul 14, 2016 at 5:21
  • 2
    @GillesQuenot, ...would you an accept an edit fixing the shell end of this to either not depend on word-splitting, or explicitly setting IFS and disabling globbing to make that word-splitting reliable? Commented Feb 1, 2019 at 14:15
  • 1
    if item2 has value with space then this command separate them.
    – alhelal
    Commented Mar 12, 2020 at 9:58
9

The following is actually buggy:

# BAD: Output line of * is replaced with list of local files; can't deal with whitespace
arr=( $( curl -k "$url" | jq -r '.[].item2' ) )

If you have bash 4.4 or newer, a best-of-all-worlds option is available:

# BEST: Supports bash 4.4+, with failure detection and newlines in data
{ readarray -t -d '' arr && wait "$!"; } < <(
  set -o pipefail
  curl --fail -k "$url" | jq -j '.[].item2 | (., "\u0000")'
)

...whereas with bash 4.0, you can have terseness at the cost of failure detection and literal newline support:

# OK (with bash 4.0), but can't detect failure and doesn't support values with newlines
readarray -t arr < <(curl -k "$url" | jq -r '.[].item2' )

...or bash 3.x compatibility and failure detection, but without newline support:

# OK: Supports bash 3.x; no support for newlines in values, but can detect failures
IFS=$'\n' read -r -d '' -a arr < <(
  set -o pipefail
  curl --fail -k "$url" | jq -r '.[].item2' && printf '\0'
)

...or bash 3.x compatibility and newline support, but without failure detection:

# OK: Supports bash 3.x and supports newlines in values; does not detect failures
arr=( )
while IFS= read -r -d '' item; do
  arr+=( "$item" )
done < <(curl --fail -k "$url" | jq -j '.[] | (.item2, "\u0000")')
8
  • 1
    But does not handle values with a newline in them :-(
    – jrw32982
    Commented Jan 21, 2021 at 16:05
  • But I want the values in my array, not the encoded strings. See my answer below.
    – jrw32982
    Commented Jan 21, 2021 at 16:54
  • @jrw32982, gotcha. You've got two choices, then, either using "\u0000" (NULs) as separators, or @sh. Commented Jan 21, 2021 at 17:05
  • @jrw32982, ...edited to show the former, since there are already two answers (one of them yours) with the latter. (Also, I consider using eval unnecessarily a nonstarter). Commented Jan 21, 2021 at 17:08
  • Note that bash's readarray now supports readarray -td '' to read NUL-delimited data into an array. Commented Jan 21, 2021 at 17:13
5

Use jq to produce a shell statement that you evaluate:

eval "$( jq -r '@sh "arr=( \([.[].item2]) )"' file.json )"

Given the JSON document in your question, the call to jq will produce the string

arr=( 'value2' 'value2_2' )

which is then evaluated by your shell. Evaluating that string will create the named array arr with the two elements value2 and value2_2:

$ eval "$( jq -r '@sh "arr=( \([.[].item2]) )"' file.json )"
$ printf '"%s"\n' "${arr[@]}"
"value2"
"value2_2"

The @sh operator in jq takes care to properly quote the data for the shell.

Alternatively, move the arr=( ... ) part out of the jq expression:

eval "arr=( $( jq -r '@sh "\([.[].item2])"' file.json ) )"

Now, jq only generates the quoted list of elements, which is then inserted into arr=( ... ) and evaluated.

If you need to read data from a curl command, use curl ... | jq -r ... in place of jq -r ... file.json in the commands above.

10
  • Somehow, this question sat here for a couple of years, without an accurate answer. I just discovered the solution (answered separately here) and now I see you've posted almost the exact same answer only a few minutes before me. If only you'd posted this before, it would have saved me figuring it out on my own!
    – jrw32982
    Commented Jan 21, 2021 at 16:52
  • @jrw32982 I don't think I saw this question before, but Charles just edited their answer which bumped the question to the top of the home page, where I spotted it.
    – Kusalananda
    Commented Jan 21, 2021 at 16:54
  • 1
    Yes, that's the only approach so far that handles arbitrary values. jq even happens to support non-text values in its strings as an extension over standard json. It seems to be using the safest form of quoting ('...'). I note that it transforms a NUL byte to \0 and doesn't quote numbers nor false/true though that should be fine. As usual, note that it may transform numbers (like change 1e2 to 100 or infinity to 1.7976931348623157e+308). Commented Jan 21, 2021 at 17:01
  • @StéphaneChazelas Do you think it's an issue that some values are left unquoted? If so, one could pass the data through tostring before letting @sh format them.
    – Kusalananda
    Commented Jan 21, 2021 at 17:06
  • 1
    I don't expect it would here. There are context where not quoting numbers can be an issue (like in echo '2'>file vs echo 2>file), but no here I'd say. Commented Jan 21, 2021 at 17:10
3

To handle arbitrary values:

#!/bin/bash
cat <<EOF >json
[
  {
    "item1": "value1",
    "item2": "  val'\"ue2  ",
    "sub items": [
      {
        "subitem": "subvalue"
      }
    ]
  },
  {
    "item1": "value1_2",
    "item2": "  value\n2_2  ",
    "sub items_2": [
      {
        "subitem_2": "subvalue_2"
      }
    ]
  }
]
EOF
eval "arr=( $(jq -r ' .[].item2 | @sh ' json) )"
printf '<%s>\n' "${arr[@]}"

Output:

<  val'"ue2  >
<  value
2_2  >

Putting together some of the other solutions in this question mentioned in comments and answers from @CharlesDuffy, @StéphaneChazelas, @Kusalananda:

#!/bin/bash
v1='a\nb  \n'   
v2='c'\''\"d  '   # v2 will contain <c'\"d  >  
printf '$v1=<%s>\n$v2=<%s>\n\n' "$v1" "$v2"

>json printf "%s\n" "[ \"$v1\", \"$v2\" ]" 
printf 'JSON data: '; cat json
printf '\n'

eval "arr=( $( cat json | jq -r '.[] | @sh ' ) )" 
printf '$arr[0]:<%s>\n$arr[1]:<%s>\n\n' "${arr[@]}"

set -- 
eval "set -- $( cat json | jq -r '[.[]] | @sh ' )" 
printf '$1:<%s>\n$2:<%s>\n\n' "$1" "$2"

{ readarray -td '' arr2 && wait "$!"; } < <( 
   cat json | jq -j '.[] | (., "\u0000") '
)
printf 'rc=%s\n$arr[0]:<%s>\n$arr[1]:<%s>\n\n' "$?" "${arr2[@]}"

{ readarray -td '' arr3 && wait "$!"; } < <( 
   { echo x; cat json; } | jq -j '.[] | (., "\u0000") '
)
printf 'rc=%s\n' "$?"

Output:

$v1=<a\nb  \n>
$v2=<c'\"d  >

JSON data: [ "a\nb  \n", "c'\"d  " ]

$arr[0]:<a
b  
>
$arr[1]:<c'"d  >

$1:<a
b  
>
$2:<c'"d  >

rc=0
$arr[0]:<a
b  
>
$arr[1]:<c'"d  >

parse error: Invalid numeric literal at line 2, column 0
rc=4
3
  • See also @Kusalananda's answer, posted a few minutes before mine.
    – jrw32982
    Commented Jan 21, 2021 at 16:53
  • What's the purpose of the echo x? Is that to trigger a failure case, and thus test whether rc is set correctly? Commented Jan 21, 2021 at 21:47
  • @CharlesDuffy Correct; to cause a failure in the call to jq
    – jrw32982
    Commented Jan 21, 2021 at 23:58
2

Thanks to sputnick I got to this:

arr=( $(curl -k https://localhost/api | jq -r '.[].item2') )

The JSON I have is the output from an API. All I needed to do wans remove the file argument and pipe | the output of curl to jq. Works great and saved some steps.

2
  • That code's actually a bit buggy. Look at what happens if you have a result element of * -- it'll get replaced with a list of files in your current directory. Commented Oct 5, 2016 at 2:55
  • 1
    Similarly, an item2 value with whitespace in it would become more than one array element. Commented Oct 5, 2016 at 3:02
0

as an easy alternative, look at jtc tool (at https://github.com/ldn-softdev/jtc), to achieve the same thing (as in jq's example):

bash $ arr=( $(jtc -w '<item2>l+0' file.json) )
bash $ printf '%s\n' "${arr[@]}"
"value2"
"value2_2"
bash $ 

explanation on -w option: angular brackets <...> specify search entire json, suffix l instructs to search labels rather than values, +0 instructs to find all occurrences (rather than just first one).

1
  • Same bug as all the arr=( $(jq ...) ) answers, insofar as the contents are being string-split and glob-expanded to populate the array -- meaning spaces (not just newlines) create new elements, and elements that look like a glob expression are replaced by files that expression matches. Commented Feb 1, 2019 at 14:14

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .