Important note: I'm using Cygwin

I retrieve JSON from a file that I alter before sending it to a server using curl. By now everything is ok except one thing, when the server receive the JSON, every special characters (with accents, etc) aren't well encoded. I suppose this was due to the fact that my JSON wasn't encoded in UTF-8 before being sent but I don't manage to convert it.

Here is the code:

sed -i 's/\r//g' $file
fileContent=`cat $file`

result=$(jq -c -j ".docs[$docIndex] + { \"_rev\": \"$rev\" }"<<<"$fileContent")            result="{\"docs\":[$result]}"

result=$result | sed 's/\r//g'
result=$result | iconv -t "UTF-8"
s=$(curl -H "Content-Type: application/json; charset=UTF-8" -H "Cache-Control: no-cache" -d "$result" $2/$3/_bulk_docs --silent)

My bash script and my JSON files are both encoded in UTF-8. My LANG variable seems to be UTF-8. I checked with this: [[ $LANG =~ UTF-8$ ]] && echo "Uses UTF-8 encoding.."

Any idea?


Here is the full script:


# $1 = directory containing JSON files
# $2 = server url (+ port)
# $3 = database name

#Loop on every file of the given directory
for file in "$1"/*; do

    #Try to insert every document of a specific JSON file and retrieve their status (200 = OK; 409 = conflict)
    allStatus=$(curl -H "Content-Type: application/json" -H "Cache-Control: no-cache" --data-binary "@$file" $2/$3/_bulk_docs --silent | jq '.[] |.status' | tr -d '\r')

    #Loop on every status (one status = one document)
    while IFS=' ' read -ra statusArray; do
      for status in "${statusArray[@]}"; do

        #Remove unwanted windows characters of the file
        sed -i 's/\r//g' $file
        fileContent=`cat $file`

        #Retrieve the id of the current document
        id=`jq -r -j ".docs[$docIndex]._id"<<<"$fileContent" | tr -d '\r'`

        if [ "$status" = "409" ]
            #Retrieve the revision of the current document and add it to the document 
            rev=$(curl -X GET --header 'Accept: application/json' $2/$3/$id?revs=true --silent | jq -r -j '._rev' | tr -d '\r')
            result=$(jq -c -j ".docs[$docIndex] + { \"_rev\": \"$rev\" }"<<<"$fileContent")

            #Wrap the document inside {"docs":[]}

            #Remove unwanted windows characters before sending the document (again)
            result=$result | sed 's/\r//g'
            result=$result | iconv -t "UTF-8"
            s=$(curl -H "Content-Type: application/json; charset=UTF-8" -H "Cache-Control: no-cache" -d "$result" $2/$3/_bulk_docs --silent)

            #Echo result
            echo $s
          #Status should be 200.
          echo 'status: '$status ' - ' $id


    done <<< "$allStatus"

This script aims to upsert documents inside a NoSQL database. If first try to insert and if if fails it retrieves a property (the revision) of the document, append to it and then retry once.

I know this script could be improved but this is my first bash script and it won't be used in production (only for testing, etc.).

  • Some questions: 1) what is the "jq" command?, 2) What Windows are you running on? 3) What happens if you don't use 'iconv'? 4) You use "$2" and "$3" in your curl command. What are they (where do they come from?) If they are parameters to your script what happened to "$1"? 5) where are you getting the json file from? 6) where is "$file" set? I might have more, but it seems you are missing much detail from your question. You'll need to supply more information to fill in the gaps. – Astara Jun 22 '17 at 16:52
  • Is my answer incorrect? – Cyclic3 Jun 22 '17 at 17:18
  • Utf-8 is now the default encoding in Unixes. If you have it set up as such, then there is no need to convert. – ctrl-alt-delor Jun 22 '17 at 17:22
  • 1
    What is result=$result | …. should it be result="$( echo "$result" | …" — I added some safety quotes as well. – ctrl-alt-delor Jun 22 '17 at 17:23
  • 1
    @cat I'm not sure if comments persist on deleted answers, but thanks for pointing out that my answer was incorrect! – Cyclic3 Jun 22 '17 at 20:57

You can use this in Bash on Cygwin, as long as you at least have Python 2.7 installed:

utf8_result="$(echo "$result" | python -c "import sys;[sys.stdout.write((i).decode('utf-8')) for i in sys.stdin]")"

Partial answer:

According to man curl, -d or --data does a POST using the content-type application/x-www-form-urlencoded, and is equivalent to --data-ascii. I don't think the attempts to override the header will change this. As the standard says, the charset for urlencoding depends on form elements, which are missing completely here.

If I understand your script correctly, $result is UTF-8, and not urlencoded. So you probably should use --data-urlencode (see man curl for details), and hope that it does encode UTF-8 correctly, or you should use --form instead, which is more flexible, and where you can include a charset field.

So the main problem seems to be in how the standard prescribes the encoding, which is sufficiently difficult to understand (at least for me). Maybe you can get a better answer on stackoverflow.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.