23

I have a JSON output that contains a list of objects stored in a variable. (I may not be phrasing that right)

[
  {
    "item1": "value1",
    "item2": "value2",
    "sub items": [
      {
        "subitem": "subvalue"
      }
    ]
  },
  {
    "item1": "value1_2",
    "item2": "value2_2",
    "sub items_2": [
      {
        "subitem_2": "subvalue_2"
      }
    ]
  }
]

I need all the values for item2 in a array for a bash script to be run on ubuntu 14.04.1.

I have found a bunch of ways to get the entire result into an array but not just the items I need

0
26

Using :

$ cat json
[
  {
    "item1": "value1",
    "item2": "value2",
    "sub items": [
      {
        "subitem": "subvalue"
      }
    ]
  },
  {
    "item1": "value1_2",
    "item2": "value2_2",
    "sub items_2": [
      {
        "subitem_2": "subvalue_2"
      }
    ]
  }
]

CODE:

arr=( $(jq -r '.[].item2' json) )
printf '%s\n' "${arr[@]}"

OUTPUT:

value2
value2_2
13
  • 1
    Have you tried something ? – Gilles Quenot Jan 7 '15 at 0:13
  • 2
    jq . <<< "$json" it's shell (bash) related, non specific to jq – Gilles Quenot Jan 7 '15 at 0:23
  • 1
    Missing parentheses : arr=( $(...) ) – Gilles Quenot Jan 7 '15 at 0:35
  • 5
    Great jq command, but please don't parse command output into an array with arr=( $(...) ) (even though it happens to work with the sample input): it doesn't work as intended with embedded or leading/trailing whitespace and can result in accidental globbing. – mklement0 Jul 14 '16 at 5:21
  • 2
    @GillesQuenot, ...would you an accept an edit fixing the shell end of this to either not depend on word-splitting, or explicitly setting IFS and disabling globbing to make that word-splitting reliable? – Charles Duffy Feb 1 '19 at 14:15
8

The following is actually buggy:

# BAD: Output line of * is replaced with list of local files; can't deal with whitespace
arr=( $( curl -k "$url" | jq -r '.[].item2' ) )

If you have bash 4.4 or newer, a best-of-all-worlds option is available:

# BEST: Supports bash 4.4+, with failure detection and newlines in data
{ readarray -t -d '' arr && wait "$!"; } < <(
  set -o pipefail
  curl --fail -k "$url" | jq -j '.[].item2 | (., "\u0000")'
)

...whereas with bash 4.0, you can have terseness at the cost of failure detection and literal newline support:

# OK (with bash 4.0), but can't detect failure and doesn't support values with newlines
readarray -t arr < <(curl -k "$url" | jq -r '.[].item2' )

...or bash 3.x compatibility and failure detection, but without newline support:

# OK: Supports bash 3.x; no support for newlines in values, but can detect failures
IFS=$'\n' read -r -d '' -a arr < <(
  set -o pipefail
  curl --fail -k "$url" | jq -r '.[].item2' && printf '\0'
)

...or bash 3.x compatibility and newline support, but without failure detection:

# OK: Supports bash 3.x and supports newlines in values; does not detect failures
arr=( )
while IFS= read -r -d '' item; do
  arr+=( "$item" )
done < <(curl --fail -k "$url" | jq -j '.[] | (.item2, "\u0000")')
8
  • 1
    But does not handle values with a newline in them :-( – jrw32982 Jan 21 at 16:05
  • But I want the values in my array, not the encoded strings. See my answer below. – jrw32982 Jan 21 at 16:54
  • @jrw32982, gotcha. You've got two choices, then, either using "\u0000" (NULs) as separators, or @sh. – Charles Duffy Jan 21 at 17:05
  • @jrw32982, ...edited to show the former, since there are already two answers (one of them yours) with the latter. (Also, I consider using eval unnecessarily a nonstarter). – Charles Duffy Jan 21 at 17:08
  • Note that bash's readarray now supports readarray -td '' to read NUL-delimited data into an array. – Stéphane Chazelas Jan 21 at 17:13
2

Use jq to produce a shell statement that you evaluate:

eval "$( jq -r '@sh "arr=( \([.[].item2]) )"' file.json )"

Given the JSON document in your question, the call to jq will produce the string

arr=( 'value2' 'value2_2' )

which is then evaluated by your shell. Evaluating that string will create the named array arr with the two elements value2 and value2_2:

$ eval "$( jq -r '@sh "arr=( \([.[].item2]) )"' file.json )"
$ printf '"%s"\n' "${arr[@]}"
"value2"
"value2_2"

The @sh operator in jq takes care to properly quote the data for the shell.

Alternatively, move the arr=( ... ) part out of the jq expression:

eval "arr=( $( jq -r '@sh "\([.[].item2])"' file.json ) )"

Now, jq only generates the quoted list of elements, which is then inserted into arr=( ... ) and evaluated.

If you need to read data from a curl command, use curl ... | jq -r ... in place of jq -r ... file.json in the commands above.

10
  • Somehow, this question sat here for a couple of years, without an accurate answer. I just discovered the solution (answered separately here) and now I see you've posted almost the exact same answer only a few minutes before me. If only you'd posted this before, it would have saved me figuring it out on my own! – jrw32982 Jan 21 at 16:52
  • @jrw32982 I don't think I saw this question before, but Charles just edited their answer which bumped the question to the top of the home page, where I spotted it. – Kusalananda Jan 21 at 16:54
  • 1
    Yes, that's the only approach so far that handles arbitrary values. jq even happens to support non-text values in its strings as an extension over standard json. It seems to be using the safest form of quoting ('...'). I note that it transforms a NUL byte to \0 and doesn't quote numbers nor false/true though that should be fine. As usual, note that it may transform numbers (like change 1e2 to 100 or infinity to 1.7976931348623157e+308). – Stéphane Chazelas Jan 21 at 17:01
  • @StéphaneChazelas Do you think it's an issue that some values are left unquoted? If so, one could pass the data through tostring before letting @sh format them. – Kusalananda Jan 21 at 17:06
  • 1
    I don't expect it would here. There are context where not quoting numbers can be an issue (like in echo '2'>file vs echo 2>file), but no here I'd say. – Stéphane Chazelas Jan 21 at 17:10
1

Thanks to sputnick I got to this:

arr=( $(curl -k https://localhost/api | jq -r '.[].item2') )

The JSON I have is the output from an API. All I needed to do wans remove the file argument and pipe | the output of curl to jq. Works great and saved some steps.

2
  • That code's actually a bit buggy. Look at what happens if you have a result element of * -- it'll get replaced with a list of files in your current directory. – Charles Duffy Oct 5 '16 at 2:55
  • 1
    Similarly, an item2 value with whitespace in it would become more than one array element. – Charles Duffy Oct 5 '16 at 3:02
1

To handle arbitrary values:

#!/bin/bash
cat <<EOF >json
[
  {
    "item1": "value1",
    "item2": "  val'\"ue2  ",
    "sub items": [
      {
        "subitem": "subvalue"
      }
    ]
  },
  {
    "item1": "value1_2",
    "item2": "  value\n2_2  ",
    "sub items_2": [
      {
        "subitem_2": "subvalue_2"
      }
    ]
  }
]
EOF
eval "arr=( $(jq -r ' .[].item2 | @sh ' json) )"
printf '<%s>\n' "${arr[@]}"

Output:

<  val'"ue2  >
<  value
2_2  >

Putting together some of the other solutions in this question mentioned in comments and answers from @CharlesDuffy, @StéphaneChazelas, @Kusalananda:

#!/bin/bash
v1='a\nb  \n'   
v2='c'\''\"d  '   # v2 will contain <c'\"d  >  
printf '$v1=<%s>\n$v2=<%s>\n\n' "$v1" "$v2"

>json printf "%s\n" "[ \"$v1\", \"$v2\" ]" 
printf 'JSON data: '; cat json
printf '\n'

eval "arr=( $( cat json | jq -r '.[] | @sh ' ) )" 
printf '$arr[0]:<%s>\n$arr[1]:<%s>\n\n' "${arr[@]}"

set -- 
eval "set -- $( cat json | jq -r '[.[]] | @sh ' )" 
printf '$1:<%s>\n$2:<%s>\n\n' "$1" "$2"

{ readarray -td '' arr2 && wait "$!"; } < <( 
   cat json | jq -j '.[] | (., "\u0000") '
)
printf 'rc=%s\n$arr[0]:<%s>\n$arr[1]:<%s>\n\n' "$?" "${arr2[@]}"

{ readarray -td '' arr3 && wait "$!"; } < <( 
   { echo x; cat json; } | jq -j '.[] | (., "\u0000") '
)
printf 'rc=%s\n' "$?"

Output:

$v1=<a\nb  \n>
$v2=<c'\"d  >

JSON data: [ "a\nb  \n", "c'\"d  " ]

$arr[0]:<a
b  
>
$arr[1]:<c'"d  >

$1:<a
b  
>
$2:<c'"d  >

rc=0
$arr[0]:<a
b  
>
$arr[1]:<c'"d  >

parse error: Invalid numeric literal at line 2, column 0
rc=4
3
  • See also @Kusalananda's answer, posted a few minutes before mine. – jrw32982 Jan 21 at 16:53
  • What's the purpose of the echo x? Is that to trigger a failure case, and thus test whether rc is set correctly? – Charles Duffy Jan 21 at 21:47
  • @CharlesDuffy Correct; to cause a failure in the call to jq – jrw32982 Jan 21 at 23:58
0

as an easy alternative, look at jtc tool (at https://github.com/ldn-softdev/jtc), to achieve the same thing (as in jq's example):

bash $ arr=( $(jtc -w '<item2>l+0' file.json) )
bash $ printf '%s\n' "${arr[@]}"
"value2"
"value2_2"
bash $ 

explanation on -w option: angular brackets <...> specify search entire json, suffix l instructs to search labels rather than values, +0 instructs to find all occurrences (rather than just first one).

1
  • Same bug as all the arr=( $(jq ...) ) answers, insofar as the contents are being string-split and glob-expanded to populate the array -- meaning spaces (not just newlines) create new elements, and elements that look like a glob expression are replaced by files that expression matches. – Charles Duffy Feb 1 '19 at 14:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.