3

I'm writing some scripts for testing an API. I'm interested in the JSON response as well as the HTTP status code of the request. I definitely want to pretty-print the JSON response in order to make it easier to read.

I'm using curl to perform the request and want to use python -m json.tool to pretty-print the json result.

Curl has a nice option -w that can be used to surface information about the request, like %{http_code}. Unfortunately, that information prints to stdout and confuses python -m json.tool. It seems it isn't possible to configure it to ignore trailing non-json data.

When I do

curl \
'--silent' \
'--insecure' \
'-L' \
'-w' \
'\n%{http_code}\n' \
'--user' \
<REDACTED> \
'-X' \
'GET' \
'--' \
'https://somecompany.com/some_api_endpoint' \
| python -m json.tool

I get

$ bash call_api_endpoint_script.sh 
Extra data: line 2 column 1 - line 3 column 1 (char 203 - 207)
Exit 1

Is there a way to configure curl to write the status code to a file? The -w option in the man page doesn't seem to mention the possibility of redirecting this information elsewhere.

  • Is something like PycURL not an option? That should let you interface directly with the response data provided by libcurl. – thrig Feb 6 '17 at 19:22
2

I found a way around this issue by using -o to redirect just the content to a temporary file, leaving just the status code in the curl's output.

I can then read the contents of the temporary file from python -m json.tool and pretty print them.

For example,

content=$(mktemp)

curl \
...
-w \
'HTTP_STATUS_CODE: %{http_code}\n' \
...
-o \
"$content" \
-- \
'https://somecompany.com/some_api_endpoint'

<"$content" python -m json.tool
1

If you use jq instead of python -m json.tool, you’ll find it parses JSON even with trailing text:

$ echo '{ "foo": "bar" } text' | ./jq
{
  "foo": "bar"
}
parse error: Invalid literal at line 2, column 0

Of course if you want to ignore the error message you can just redirect that to /dev/null.

Also note that with jq if you want to use its output in a pipe or redirect it to a file, I think you need to give it a “filter” argument:

echo '{ "foo": "bar" } trailing text' | jq . > OUT.json

. is just the simplest possible filter. It means “take the input and produce it unchanged as output”.

0

I definitely want to pretty-print the JSON response in order to make it easier to read.

That's exactly what https://httpie.org/ is for. sending GET, POST, PUT, DELETE, PATCH etc to API and pretty print JSON answer.


As for the curl part of your question, I have no clue, sorry.

0

Whilst the write-out from -w only goes to stdout the output from -o can be redirected to any file. In particular you could direct it to stderr with -o /dev/stderr. If you now swap over stderr and stdout (with 3>&1- 1>&2- 2>&3-) you'll be in a position to see your HTTP return code (through stderr) whilst piping your output to your desired program.

In other words, something along the lines of:

curl ... -w 'HTTP_STATUS_CODE: %{http_code}\n' -o /dev/stderr ... | 3>&1- 1>&2- 2>&3- | python -m json.tool

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.