I am writing a shell script that has the following construct:

job_url=$(curl -X POST -u ${run_user} -s -i http://${server_addr}:8080/webportal/rest/v4/repository/users/BIRT/report_problem:jobs | grep Location | grep -o -E '[^ ]+$')
job_state=$(curl -X POST -u ${run_user} -H "Content-Type: application/json" -s -i ${job_url} | grep state |  grep -o -E '[^ ]+$')

The first variable job_url must evaluate to a URL which is then used in the second command. The issue is with the resulting URL. When I run it from in the terminal directly, I get a normal link:


...but, when this is used in a script - it fails. The bash debug mode shows the issue:

++ curl -X POST -u user:password -s -i http://address:8080/webportal/rest/v4/repository/users/BIRT/report_problem:jobs
++ grep -o -E '[^ ]+$'
++ grep Location
+ job_url=$'http://address:8080/webportal/rest/v4/jobs/e0356d1c-ce69-489d-8f3c-40ae240cae6d\r'

For some reason the URL has the regex symbols: $'http://address:8080/webportal/rest/v4/jobs/e0356d1c-ce69-489d-8f3c-40ae240cae6d\r' and this breaks all further script functionality.

What is the reason for this? I have tried changing quoting in cURL, switching between awk and grep and checking the bash options. I'm running bash-4.1.2-41.el6_8.x86_64 on RHEL 6.

Any help appreciated!


It sounds like whatever server you are querying is configured to return Windows-style line endings (\r\n) instead of the "normal" *nix style \n. Unfortunately, unlike \n, \r aren't stripped from command output when assigning to a variable. To illustrate:

$ var=$(printf 'hello\n')
$ printf  '%s' "$var" | od -c
0000000   h   e   l   l   o
$ var=$(printf 'hello\r\n')
$ printf  '%s' "$var" | od -c
0000000   h   e   l   l   o  \r

As you can see, the \n is automatically removed but the \r is not. So, if you can't change the configuration of the server, you'll just need to remove the \r yourself:

job_url=$(curl ... | grep Location | grep -o -E '[^ ]+$' | tr -d '\r')
  • Thank you for the kind explanation and the example. This worked for me. – qugu Jun 8 '17 at 9:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.