1

I have this section of curl command inside my shell script which caused a error 500 repsonse from the server. However, this code works when I manually run the command.

curl -qgsSkH "Content-Type: multipart/form-data" --no-progress-bar --header "Token: $authKey" -F "filename=@$compressedFileName" -F "options={\"application\":\"2\",\"timeout\":\"5000\"}" https://www.domain.com > results.txt

Anyone have any idea why?

Edit: I did some manual debugging and found out that the issue is caused by the variable $authKey. This variable is fetched by the following code in my script.

# Get authentication key from FireEye AX response
while read line
do
    if [[ $line =~ ^SOME-PATTERN:[[:space:]](.*) ]]
    then
        authKey="${BASH_REMATCH[1]}"
    fi
done <auth.txt

If I redefine this variable by using something like the following then the script will work.

authKey="TheAuthenticationKey"

But why is this happening? The earlier part where I pass the string from the file when I read it line by line is already passing the correct value (I double checked by echoing the output to a txt file).

Edit I managed to drill down somemore. When I try to echo the value of the token to a file and add " to the start and the end of the string I got this result.

"X-FeApi-Token: IDR+jxU1kB5ZpXsRNAgqfjTxkjZ0L2atH+l0H/NeatWDAJs=
"

I am not sure how to remove that new line thingy. Anyone can advise?

5
  • stackoverflow.com/questions/7172784/…
    – cas
    Nov 5, 2015 at 6:32
  • Thanks for sharing the link @cas but this is not what caused the issue.
    – Norman
    Nov 5, 2015 at 6:44
  • 1
    Does whatever sends you the auth key add a line-feed or CR-LF? or some non-essential verbosity. if so, try stripping them with sed or tr. hmmm.. the =~ should handle that. try adding printf debugging statements or run your script with bash -x to see what's going on.
    – cas
    Nov 5, 2015 at 7:03
  • Actually, looking at your =~ line again, I was right the first time. The (.*) will capture everything to the end of the line, including the EOL marker (LF or CR-LF). remove them with ... | tr -d '\r\n'.
    – cas
    Nov 5, 2015 at 9:39
  • re: your updateL - when you echo that auth token, use printf "%s" "$authKey" | sed -n l.
    – cas
    Nov 5, 2015 at 9:42

2 Answers 2

1

The problem seems to be caused by a newline or carriage-return and newline in $authKey, as supplied by "FireEye AX"

You can strip carriage returns and line feeds by piping it through tr -d '\r\n'

e.g.

# Get authentication key from FireEye AX response
while read line
do
    if [[ $line =~ ^SOME-PATTERN:[[:space:]](.*) ]]
    then
        authKey="${BASH_REMATCH[1]}"
        authKey=$(printf "%s" "$authKey" | tr -d '\r\n')
    fi
done <auth.txt

I can't remember if $BASH_REMATCH will be passed to a subshell...you could try it in one line instead, maybe it will work:

authKey=$(printf "%s" "${BASH_REMATCH[1]}" | tr -d '\r\n')
3
  • I used something similar to solve. My exact code can be seen in the following. Code: authKey=$(echo $line | tr -d '\r\n')
    – Norman
    Nov 6, 2015 at 2:02
  • glad to hear it worked for you. BTW, printf "%s" "$line" is safer because a) the variable is properly quoted, and b) echo will interpret some string sequences (e.g. \t, \n) and change the output, while using printf "%s" will not. you have no control over the value of the auth token received so have no way to ensure that those sequences won't be in the string.
    – cas
    Nov 6, 2015 at 3:48
  • Thanks for the detailed explanation and teaches me which method is a better one!
    – Norman
    Nov 9, 2015 at 6:47
1

I'm guessing that the data file you have been provided has Windows-style line endings (CR/LF) rather than Unix/Linux line endings (LF).

This alternative pattern match will exclude any possible trailing CR:

[[ $line =~ ^SOME-PATTERN:[[:space:]](([^\015]*) ]]
authKey="${BASH_REMATCH[1]}"

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .