1

I have this section of curl command inside my shell script which caused a error 500 repsonse from the server. However, this code works when I manually run the command.

curl -qgsSkH "Content-Type: multipart/form-data" --no-progress-bar --header "Token: $authKey" -F "filename=@$compressedFileName" -F "options={\"application\":\"2\",\"timeout\":\"5000\"}" https://www.domain.com > results.txt

Anyone have any idea why?

Edit: I did some manual debugging and found out that the issue is caused by the variable $authKey. This variable is fetched by the following code in my script.

# Get authentication key from FireEye AX response
while read line
do
    if [[ $line =~ ^SOME-PATTERN:[[:space:]](.*) ]]
    then
        authKey="${BASH_REMATCH[1]}"
    fi
done <auth.txt

If I redefine this variable by using something like the following then the script will work.

authKey="TheAuthenticationKey"

But why is this happening? The earlier part where I pass the string from the file when I read it line by line is already passing the correct value (I double checked by echoing the output to a txt file).

Edit I managed to drill down somemore. When I try to echo the value of the token to a file and add " to the start and the end of the string I got this result.

"X-FeApi-Token: IDR+jxU1kB5ZpXsRNAgqfjTxkjZ0L2atH+l0H/NeatWDAJs=
"

I am not sure how to remove that new line thingy. Anyone can advise?

  • Please add an extract from your script to your question? – garethTheRed Nov 5 '15 at 6:27
  • Thanks for sharing the link @cas but this is not what caused the issue. – Norman Nov 5 '15 at 6:44
  • @garethTheRed I have updated the questions with more updated findings. – Norman Nov 5 '15 at 6:45
  • 1
    Does whatever sends you the auth key add a line-feed or CR-LF? or some non-essential verbosity. if so, try stripping them with sed or tr. hmmm.. the =~ should handle that. try adding printf debugging statements or run your script with bash -x to see what's going on. – cas Nov 5 '15 at 7:03
1

The problem seems to be caused by a newline or carriage-return and newline in $authKey, as supplied by "FireEye AX"

You can strip carriage returns and line feeds by piping it through tr -d '\r\n'

e.g.

# Get authentication key from FireEye AX response
while read line
do
    if [[ $line =~ ^SOME-PATTERN:[[:space:]](.*) ]]
    then
        authKey="${BASH_REMATCH[1]}"
        authKey=$(printf "%s" "$authKey" | tr -d '\r\n')
    fi
done <auth.txt

I can't remember if $BASH_REMATCH will be passed to a subshell...you could try it in one line instead, maybe it will work:

authKey=$(printf "%s" "${BASH_REMATCH[1]}" | tr -d '\r\n')
  • I used something similar to solve. My exact code can be seen in the following. Code: authKey=$(echo $line | tr -d '\r\n') – Norman Nov 6 '15 at 2:02
  • glad to hear it worked for you. BTW, printf "%s" "$line" is safer because a) the variable is properly quoted, and b) echo will interpret some string sequences (e.g. \t, \n) and change the output, while using printf "%s" will not. you have no control over the value of the auth token received so have no way to ensure that those sequences won't be in the string. – cas Nov 6 '15 at 3:48
  • Thanks for the detailed explanation and teaches me which method is a better one! – Norman Nov 9 '15 at 6:47
1

I'm guessing that the data file you have been provided has Windows-style line endings (CR/LF) rather than Unix/Linux line endings (LF).

This alternative pattern match will exclude any possible trailing CR:

[[ $line =~ ^SOME-PATTERN:[[:space:]](([^\015]*) ]]
authKey="${BASH_REMATCH[1]}"

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.