1

I'm trying to run curl to POST JSON data with x-www-form-urlencode inside a docker-compose file. For this example, I'm using this public website and I need to response to look like this

# sh -c 'curl http://httpbin.org/post -F "json={\"key\": \"value\"}"'
{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {
    "json": "{\"key\": \"value\"}"
  }, ...

That works fine with single quotes on the outside, but I need double quotes because I will have variables inside. So this command does not work, and I can't figure out the right syntax for escaping the nested quotes.

# sh -c "curl http://httpbin.org/post -F "json={\"key\": \"$VAR\"}""
{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {
    "json": "{key:"
  }, 
5
  • sh -c "curl http://httpbin.org/post -F \'json={\"key\": \"value\"}\'" Mar 4, 2021 at 6:24
  • Do you need the sh -c there? I.e. can't just put curl http://.. -F "whatever" as the command? Because if you can drop the sh -c, you get to drop one set of quoting which makes it a lot easier.
    – ilkkachu
    Mar 4, 2021 at 12:31
  • @berndbausch, nope, "... \' ..." leaves both backslash and single-tick in the result, and you don't want that sh -c to see the single tick escaped.
    – ilkkachu
    Mar 4, 2021 at 12:34
  • 1
    Had I tested it, I would have written an answer. Might be better to compose the string one piece after the other, then hand it over to the shell. And true, why is sh -c needed? Mar 4, 2021 at 13:23
  • The sub shell is required because it’s running the command line in a docker-compose script; there can only be one command.
    – Elliott B
    Mar 4, 2021 at 16:40

3 Answers 3

2

The data you want to insert into the argument sent to curl is a JSON-encoded string encoding a JSON document.

This string can be created separately like so:

json_string=$( jq -n --arg 'my key' 'my "value"' '$ARGS.named|@json' )

By using the JSON-aware tool jq to convert the data into a JSON-encoded string, we ensure that the contents of the string is both a valid JSON string and that it may be decoded into a valid JSON document containing the given key, my key, and value, my "value".

The exact command above will assign the string "{\"my key\":\"my \\\"value\\\"\"}" to json_string.

We may then call curl directly using this string in an argument, like so,

curl 'http://httpbin.org/post' -F "json=$json_string"

... or, we may call it from an in-line sh -c script by passing the string as an argument to that script, like so,

sh -c 'curl "http://httpbin.org/post" -F "json=$1"' sh "$json_string"

Either of these ways of calling curl would result in a reply like the following:

{
  "args": {},
  "data": "",
  "files": {},
  "form": {
    "json": "{\"my key\":\"my \\\"value\\\"\"}"
  },
  "headers": {
    "Accept": "*/*",
    "Content-Length": "160",
    "Content-Type": "multipart/form-data; boundary=------------------------f805cf070f665e7e",
    "Host": "httpbin.org",
    "User-Agent": "curl/7.84.0",
    "X-Amzn-Trace-Id": "Root=1-83012651-2995681c270059da2906c40f"
  },
  "json": null,
  "origin": "xxx.xxx.xxx.xxx",
  "url": "http://httpbin.org/post"
}

The encoding of the JSON document into a JSON string is not strictly necessary, as curl seems to be doing this on its own accord, so you could instead use

json_string=$( jq -n -c --arg 'my key' 'my "value"' '$ARGS.named' )

... to create the JSON string, or, if you happen to have jo installed, the shorter

json_string=$( jo 'my key'='my "value"' )
0

Like so?

$ var=hello
$ sh -c "curl http://httpbin.org/post -F 'json={\"key\": \"$var\"}'"
{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {
    "json": "{\"key\": \"hello\"}"
  }, 

Annotated:

$ sh -c "curl http://httpbin.org/post -F 'json={\"key\": \"$var\"}'"
        1                                2       3    3   34    3 21

(1) outer double-quoting, (2) single-quotes, not special within the double-quotes, so no need to escape, (3) inner double-quotes escaped for the outer ones. (4) variable expansion within the outer double-quotes, before running sh -c (note that single quotes within the variable value would cause issues for the inner shell, and double-quotes there would mess with the JSON syntax).

If sh is Bash, and you run that with -x, you'll see what it runs is this:

+ curl http://httpbin.org/post -F 'json={"key": "hello"}'
                                  2      3   3  3     3 2 

which is pretty much exactly what that sh -c sees, and where, (2) is a quoted string, and (3) is non-special double-quotes within that string.

(I say "in Bash", because Dash's set -x output is not unambiguous. And in Bash, in general, the output from -x redoes quoting, e.g. if you give the command echo "foo bar", it shows it as echo 'foo bar'. But that doesn't really change anything in this particular case.)

0

Heredocs make this quoting hell much more manageable

sh <<END_SCRIPT
curl http://httpbin.org/post -F 'json={"key": "$VAR"}'
END_SCRIPT

The heredoc itself is not quoted, so variables will be substituted.

If VAR="foo bar", we get

...
  "form": {
    "json": "{\"key\": \"foo bar\"}"
  },
...

The only thing you'll have to worry about is if the variable value contains quotes.

With VAR='She said "Hello"' we see this invalid JSON

...
  "form": {
    "json": "{\"key\": \"She said \"Hello\"\"}"
  },
...

We can use shell parameter expansion to escape the inner quotes:

sh <<END_SCRIPT
curl http://httpbin.org/post -F 'json={"key": "${VAR//\"/\\\"}"}'
END_SCRIPT

So that

...
  "form": {
    "json": "{\"key\": \"She said \\\"Hello\\\"\"}"
  },
...

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .