11

I have a JSON fragment.

The following does not work:

VALUE=<<PERSON
{
  "type": "account",
  "customer_id": "1234",
  "customer_email": "jim@gmail.com"  
}
PERSON
echo -n "$VALUE" | python -m json.tool

The result is:

No JSON object could be decoded

Doing the same with jq, i. e.

echo -n "$VALUE" | jq '.'

There is no output.

There is the same behavior for the following:

VALUE=<<PERSON
'{
  "type": "account",
  "customer_id": "1234",
  "customer_email": "jim@gmail.com"  
}'
PERSON
echo -n "$VALUE" | python -m json.tool

Response:

No JSON object could be decoded

But the following works:

VALUE='{
  "type": "account",
  "customer_id": "1234",
  "customer_email": "jim@gmail.com"
}'
echo -n "$VALUE" | jq '.'
echo -n "$VALUE" | python -m json.tool
  • 5
    I don't know what bash is doing, but there's a trailing comma after the email string in your first two but not on the third, which would make the first couple illegal JSON – Nick T Apr 11 '18 at 14:53
  • @NickT you should make that an answer as I think that is precisely the problem. – rrauenza Apr 11 '18 at 18:20
  • If that's the (sole) answer it should probably be closed as "can't be reproduced (a typo)". However, it looks like Kusa's and terdon's answer mention the assignment + redirection is totally broken so you get an empty string, so there are two problems, both of which would give the same "No JSON..." error. It's very good practice to bisect problems by checking your assumptions in the middle: a simple echo $VALUE without ... | jq would be informative. – Nick T Apr 11 '18 at 18:27
  • @NickT:That was a copy/paste issue. Sorry for the confusion – Jim Apr 11 '18 at 21:00
19
VALUE=<<PERSON
some data
PERSON

echo "$VALUE"

No output.

A here-document is a redirection, you can't redirect into a variable.

When the command line is parsed, redirections are handled in a separate step from variable assignments. Your command is therefore equivalent to (note the space)

VALUE= <<PERSON
some data
PERSON

That is, it assigns an empty string to your variable, then redirects standard input from the here-string into the command (but there is no command, so nothing happens).

Note that

<<PERSON
some data
PERSON

is valid, as is

<somefile

It's just that there is no command whose standard input stream can be set to contain the data, so it's just lost.

This would work though:

VALUE=$(cat <<PERSON
some data
PERSON
)

Here, the command that receives the here-document is cat, and it copies it to its standard output. This is then what is assigned to the variable by means of the command substitution.

In your case, you could instead use

python -m json.tool <<END_JSON
JSON data here
END_JSON

without taking the extra step of storing the data in a variable.

  • 2
    You could also just do PERSON=" followed by a newline and the multi-line data, then another " at the end. – R.. Apr 11 '18 at 14:14
  • 1
    @R.. Yes, but a here-document allows you to bypass the quoting rules of the shell. It is therefore often safer to use a here-document instead of a quoted string for multi-line data, especially if the data contains single or double quotes (or both). – Kusalananda Apr 11 '18 at 14:20
  • 2
    @R.. Given it's JSON we're talking about, it might be better to use single quotes to not have to escape the double quotes of each property name. PERSON='. That's unless the OP wants to interpolate variables later. – JoL Apr 11 '18 at 15:39
  • (backslash)(newline) seems to vanish in a here document, even if you quote/escape the delimiter word. That might be desirable, but is there any way to disable it? – Scott Apr 11 '18 at 18:17
  • @Scott If that question hasn't been asked on this site before, it would be an excellent question in its own right. – Kusalananda Apr 11 '18 at 18:24
11

Because the variable isn't being set by your heredoc:

$ VALUE=<<PERSON  
> {    
>   "type": "account",  
>   "customer_id": "1234",  
>   "customer_email": "jim@gmail.com",  
> }  
> PERSON
$ echo "$VALUE" 

$

If you want to use a heredoc to assign a value to a variable, you need something like:

$ read -d '' -r VALUE <<PERSON  
{    
  "type": "account",  
  "customer_id": "1234",  
  "customer_email": "jim@gmail.com",  
}   
PERSON
  • 1
    Why are you wrapping the JSON data in single quotes?  It doesn’t really look like the OP wants them to be part of his input string.  Aside from that, +1 for cutting down on the homeless cat population.  As with Kusalananda’s answer, you might want to suggest << \PERSON to protect against $s in the input and backslashes at the ends of lines. – Scott Apr 11 '18 at 18:17
  • @Scott um, because I just blindly copied the text from the OP. Thanks – terdon Apr 11 '18 at 18:29
  • 3
    This is the right answer. $(cat <<EOF ... EOF) is a weird construct: running a subshell and then sending a heredoc to cat just so it sends it to STDOUT and then assigning the result of that subshell to a variable? I wish people would think about what they're saying about their thought processes. Assigning a heredoc to a variable via read, by comparison, is sane. – Rich Apr 11 '18 at 19:18
  • I wouldn’t say that $(cat << EOF … (data) … EOF ) is weird. It’s awkward and convoluted, but so is read -d … << EOF — especially read -d '' << EOF . I appreciate terdon’s answer because it uses only builtins, no programs. But, more importantly, the $(cat << EOF … (data) … EOF ) fails if any lines end with \ (backslash) — see the comments under Kusalananda’s answer. – Scott Apr 12 '18 at 1:53
4

It is because the way you have defined a here-doc to use with a JSON is wrong. You need to use it as

VALUE=$(cat <<EOF
{  
  "type": "account",  
  "customer_id": "1234",  
  "customer_email": "jim@gmail.com",  
}
EOF
)

and doing printf "$VALUE" should dump the JSON as expected.

3

Heredocs and variables don't mix well or at least not in this way. You can either…

Pass the heredoc as the standard input of an application

python -m json.tool <<PERSON  
{
  "type": "account",
  "customer_id": "1234",
  "customer_email": "jim@gmail.com",
}
PERSON

or…

Store multi-line text in a shell variable

VALUE='{
  "type": "account",
  "customer_id": "1234",
  "customer_email": "jim@gmail.com",
}'

I used single quotes to avoid the need to escape the inner double quotes. Of course you can also use double quotes, e. g. if you need to expand parameters:

VALUE="{
  \"type\": \"account\",
  \"customer_id\": ${ID},
  \"customer_email\": \"${EMAIL}\",
}"

Then you can use the variable value later on.

echo -n "$VALUE" | python -m json.tool

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.