3

Important note: I'm using Cygwin to run this script.


I got a json string by executing

result=$(jq -c ".docs[$docIndex] + { \"_rev\": \"rev\" }"<<<"$fileContent")

Here is the JSON I get:

{"_id":"VT_CONSULTATION","name":"External Consultation","type":"VISIT_TYPE","sections":["HS_SECTION_AMBU"],"displayFields":[{"eventTypeId":"ET_CONSULTATION","elementValueTypeId":"EVT_IS_ACNC"},{"eventTypeId":"ET_CONSULTATION","elementValueTypeId":"EVT_MAIN_DIAGNOSTIC"},{"eventTypeId":"ET_CONSULTATION","elementValueTypeId":"EVT_IS_CONTROL_CONSULTATION","displayTrueValue":"Control visit","displayFalseValue":""}],"_rev":"rev"}

After beautifing:

{
    "_id": "VT_CONSULTATION",
    "name": "External Consultation",
    "type": "VISIT_TYPE",
    "sections": [
        "HS_SECTION_AMBU"
    ],
    "displayFields": [
        {
            "eventTypeId": "ET_CONSULTATION",
            "elementValueTypeId": "EVT_IS_ACNC"
        },
        {
            "eventTypeId": "ET_CONSULTATION",
            "elementValueTypeId": "EVT_MAIN_DIAGNOSTIC"
        },
        {
            "eventTypeId": "ET_CONSULTATION",
            "elementValueTypeId": "EVT_IS_CONTROL_CONSULTATION",
            "displayTrueValue": "Control visit",
            "displayFalseValue": ""
        }
    ],
    "_rev": "rev"
}

And then I want to wrap this string inside {"docs":[]} so that my previous JSON is injected inside the docs[] array.

Here is what I did:

result="{\"docs\":["$result"]}"

But here is the result I get

{"docs":[{"_id":"VT_CONSULTATION","name":"External Consultation","type":"VISIT_TYPE","sections":["HS_SECTION_AMBU"],"displayFields":[{"eventTypeId":"ET_CONSULTATION","elementValueTypeId":"EVT_IS_ACNC"},{"eventTypeId":"ET_CONSULTATION","elementValueTypeId":"EVT_MAIN_DIAGNOSTIC"},{"eventTypeId":"ET_CONSULTATION","elementValueTypeId":"EVT_IS_CONTROL_CONSULTATION","displayTrueValue":"Control visit","displayFalseValue":""}],"_rev]}"rev"}

Where the "failed" part is ,"_rev]}"rev"} -> it should be ,"_rev":"rev"}]}

What could have gone wrong during the string concatenation?


UPDATE Here is the full script if it can help:

#!/bin/bash
for file in "$1"/*; do
    allStatus=$(curl -H "Content-Type: application/json" -H "Cache-Control: no-cache" --data-binary "@$file" $2/$3/_bulk_docs --silent | jq '.[] |.status' | tr -d '\r')
    docIndex=0
    while IFS=' ' read -ra statusArray; do
      for status in "${statusArray[@]}"; do
        if [ "$status" = "409" ]
        then
            sed -i 's/\r//g' $file
            fileContent=`cat $file`
            id=`jq -r ".docs[$docIndex]._id"<<<"$fileContent" | tr -d '\r'`
            rev=$(curl -X GET --header 'Accept: application/json' $2/$3/$id?revs=true --silent | jq -r '._rev' | tr -d '\r')
            result=$(jq -c ".docs[$docIndex] + { \"_rev\": \"rev\" }"<<<"$fileContent")
            result="{\"docs\":[$result]}"
            #Here result is broken
            s=$(curl -H "Content-Type: application/json" -H "Cache-Control: no-cache" -d $result $2/$3/_bulk_docs --silent)
        else
          echo "No Conflict"
        fi
        docIndex=$((docIndex+1))
      done
    done <<< "$allStatus"
done

Update 2 Here's what I have after I made a call to echo "$result" | od -tx1 (after the string concatenation)

0000000 7b 22 64 6f 63 73 22 3a 5b 7b 22 5f 69 64 22 3a
0000020 22 56 54 5f 43 4f 4e 53 55 4c 54 41 54 49 4f 4e
0000040 22 2c 22 6e 61 6d 65 22 3a 22 45 78 74 65 72 6e
0000060 61 6c 20 43 6f 6e 73 75 6c 74 61 74 69 6f 6e 22
0000100 2c 22 74 79 70 65 22 3a 22 56 49 53 49 54 5f 54
0000120 59 50 45 22 2c 22 73 65 63 74 69 6f 6e 73 22 3a
0000140 5b 22 48 53 5f 53 45 43 54 49 4f 4e 5f 41 4d 42
0000160 55 22 5d 2c 22 64 69 73 70 6c 61 79 46 69 65 6c
0000200 64 73 22 3a 5b 7b 22 65 76 65 6e 74 54 79 70 65
0000220 49 64 22 3a 22 45 54 5f 43 4f 4e 53 55 4c 54 41
0000240 54 49 4f 4e 22 2c 22 65 6c 65 6d 65 6e 74 56 61
0000260 6c 75 65 54 79 70 65 49 64 22 3a 22 45 56 54 5f
0000300 49 53 5f 41 43 4e 43 22 7d 2c 7b 22 65 76 65 6e
0000320 74 54 79 70 65 49 64 22 3a 22 45 54 5f 43 4f 4e
0000340 53 55 4c 54 41 54 49 4f 4e 22 2c 22 65 6c 65 6d
0000360 65 6e 74 56 61 6c 75 65 54 79 70 65 49 64 22 3a
0000400 22 45 56 54 5f 4d 41 49 4e 5f 44 49 41 47 4e 4f
0000420 53 54 49 43 22 7d 2c 7b 22 65 76 65 6e 74 54 79
0000440 70 65 49 64 22 3a 22 45 54 5f 43 4f 4e 53 55 4c
0000460 54 41 54 49 4f 4e 22 2c 22 65 6c 65 6d 65 6e 74
0000500 56 61 6c 75 65 54 79 70 65 49 64 22 3a 22 45 56
0000520 54 5f 49 53 5f 43 4f 4e 54 52 4f 4c 5f 43 4f 4e
0000540 53 55 4c 54 41 54 49 4f 4e 22 2c 22 64 69 73 70
0000560 6c 61 79 54 72 75 65 56 61 6c 75 65 22 3a 22 43
0000600 6f 6e 74 72 6f 6c 20 76 69 73 69 74 22 2c 22 64
0000620 69 73 70 6c 61 79 46 61 6c 73 65 56 61 6c 75 65
0000640 22 3a 22 22 7d 5d 2c 22 5f 72 65 76 22 3a 22 72
0000660 65 76 22 7d 0d 5d 7d 0a
0000670

I do not see any "anormal" characters here.

  • I tested both "{\"docs\":["$result"]}" and "{\"docs\":[$result]}" and got ,"_rev":"rev"}]} in both! are you sure about the contents of $result? – RYN Jun 16 '17 at 13:50
  • When I test it on jqplay.org I got the correct result too but not while using my script.. Could it be because of some environmental issues? I'm using Windows10 and my script is running with Cygwin. And the content of $result is a copy/paste after echo it (the inline one). – MHogge Jun 16 '17 at 13:53
  • OK, first of all, please edit your question and clarify that you're using Cygwin. That's not a minor detail. That said, does it work as expected if you use result='{"docs":['"$result"']}'? – terdon Jun 16 '17 at 13:58
  • Maybe! I tested it on linux terminal; I don't know about how windows and cygwin modifies variables; try echoing the $result and see if it is OK or not; also try concatenation with "{\"docs\":[$result]}" – RYN Jun 16 '17 at 14:02
  • Neither "{\"docs\":[$result]}" or '{"docs":['"$result"']}' seems to work (same result). What is strange is that it works with other JSON contents but I'm (at 99%) sure the JSON I pasted on my question is the correct one. – MHogge Jun 16 '17 at 14:11
1

I think steeldriver is on the right track with his last comment. I'm a frequent user of Cygwin and have seen strange things happen with text that I mistakenly did not strip of carriage-returns.

Let's try an experiment. I have a very basic distillation of your script:

result="$(cat $1)"
result="{\"docs\":[$result]}"
echo $result

My input is

{"_id":"VT_CONSULTATION","name":"External Consultation","type":"VISIT_TYPE","sections":["HS_SECTION_AMBU"],"displayFields":[{"eventTypeId":"ET_CONSULTATION","elementValueTypeId":"EVT_IS_ACNC"},{"eventTypeId":"ET_CONSULTATION","elementValueTypeId":"EVT_MAIN_DIAGNOSTIC"},{"eventTypeId":"ET_CONSULTATION","elementValueTypeId":"EVT_IS_CONTROL_CONSULTATION","displayTrueValue":"Control visit","displayFalseValue":""}],"_rev":"rev"}

Running with proper line endings:

{"docs":[{"_id":"VT_CONSULTATION","name":"External Consultation","type":"VISIT_TYPE","sections":["HS_SECTION_AMBU"],"displayFields":[{"eventTypeId":"ET_CONSULTATION","elementValueTypeId":"EVT_IS_ACNC"},{"eventTypeId":"ET_CONSULTATION","elementValueTypeId":"EVT_MAIN_DIAGNOSTIC"},{"eventTypeId":"ET_CONSULTATION","elementValueTypeId":"EVT_IS_CONTROL_CONSULTATION","displayTrueValue":"Control visit","displayFalseValue":""}],"_rev":"rev"}]}

That's what we expect.

Now I append a carriage return to the input line (using Ctrl-K+Ctrl-M in vim) and run again:

{"docs":[{"_id":"VT_CONSULTATION","name":"External Consultation","type":"VISIT_TYPE","sections":["HS_SECTION_AMBU"],"displayFields":[{"eventTypeId":"ET_CONSULTATION","elementValueTypeId":"EVT_IS_ACNC"},{"eventTypeId":"ET_CONSULTATION","elementValueTypeId":"EVT_MAIN_DIAGNOSTIC"},{"eventTypeId":"ET_CONSULTATION","elementValueTypeI]}:"EVT_IS_CONTROL_CONSULTATION","displayTrueValue":"Control visit","displayFalseValue":""}],"_rev":"rev"}

Oops. Look at where the final ]} ended up (right before "EVT_IS_CONTROL_CONSULTATION"). They are not offset the exact same way as yours but this is a suspicious similarity.

Similar effects can be caused by other control characters. A simple way to check if such characters exists is to use grep '[^[:print:]]' which looks for any "non-printing" characters (i.e. invisible things like control characters). If grep returns anything that confirms it at which point there are various ways to locate and fix the offending character(s). You can use the same regex with sed for example. Or use a hex editor (or even vim/xxd). If it's a CR then the same sed command used by the OP should do: sed 's/\r//g'

  • I added dos2unix $file just under for file in "$1"/*; do at the beginning of my for loop but it does not change anything. Don't my sed -i 's/\r//g' $filealready apply this conversion? – MHogge Jun 19 '17 at 7:50
  • I was mostly letting you know about dos2unix in case you weren't familiar. But, yes, unless CR got introduced after that point it doesn't sound like the culprit. Other control characters can have similar effect. You might check with grep '[^[:print:]]' (off the top of my head). This will identify any non-printing characters (not including trailing newlines). – B Layer Jun 19 '17 at 11:50
  • 1
    The fundamental problem is likely due to the various intermixing of string and file operations. With all the 'here documents', quoted command strings and process substitution it's hard to follow. Also note that systems don't like text files that don't have some kind of line endings. Your last use of jc prior to the final assignment to result is a likely culprit. You may want to try rewriting your script to be more "string-centric". Sorry if that's somewhat vague. :) – B Layer Jun 19 '17 at 19:54
  • 1
    Hopefully you have enough information to resolve the problem. If you get stuck let me know and I'll drill down into your script. Otherwise if you do resolve it, since my answer addressed your question of "what could have gone wrong" by identifying the root cause and giving you a means of confirming it I'd appreciate if you accept the answer. Thanks. – B Layer Jun 19 '17 at 20:08
  • 1
    One other thing. I don't use jq but looking at the options I see --join-output / -j: Like -r but jq won’t print a newline after each output. Maybe that'll help. – B Layer Jun 20 '17 at 2:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.