This question already has an answer here:

I have the following script that reads through a file line by line

while IFS='' read -r line || [[ -n "$line" ]]; do
    echo $line >> file2.test
    CALL="${LOCATION_PATH} ${line} ${LETTER} 1234"
    echo $CALL >> file.test
    echo =========
    #echo $RESULT
done < "$1"

But while the input file for each line has NO ^M (carriage return) character in it, the output of the file.test file has them as follows:

/.../inventory.sh 00000e99-bce9-11e4-8418-06e8ce2b06d8^M l 1234
/.../inventory.sh 0001688b-bce7-11e4-8418-06e8ce2b06d8^M l 1234

The output to file2.test alos has no ^M character.

I have tried substituting as follows:

SP=" "

To no avail.

marked as duplicate by G-Man, Stephen Rauch, dhag, Romeo Ninov, Wouter Verhelst Oct 19 '17 at 14:17

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • Check this. You should be adding a command dos2unix to convert the file into an Unix readable file. – mathB Oct 19 '17 at 9:26
  • How do you know the input file doesn't have CRs? Your editor may hide it. See the output of sed -n l filename to make invisible characters visible. – Philippos Oct 19 '17 at 9:27
  • @Philippos - well I am capable of checking that file.test has these characters, I used the same technique to check the input files – Quintin Balsdon Oct 19 '17 at 9:32
  • (1) You are probably wrong about both the input and the file2.test output file.  Unless you present evidence that you have checked correctly, I’m going to assume that they all have CRs. (2) You should probably use more quotes. – G-Man Oct 19 '17 at 9:43
  • @QuintinBalsdon CRs inside a line a typically displayed ^M, while at the end of the line they are typically hidden. So please verify your assumption with the given sed command. – Philippos Oct 19 '17 at 10:07

^M is a sign that you created the input data providing the ${line} variable in Windows, for that's an end-of-line character in Windows. If you transfer a file edited in Windows to Linux, you need to run the "dos2unix" command on it before using it in Unix/Linux.

To see whether this is the case, you can run the "od -Xc" command on the input file to hexdump it -- and then to look for carriage-return/line-feed characters instead of newlines. If you see characters like \r\l rather than \n then you've found the culprit.

  • Thank you. I needed to do a | tr '\r' ' ' on the line and that removed it – Quintin Balsdon Oct 19 '17 at 10:13
  • That replaced it with a space, to be exact. So finally you believe that there were CRs in your input? – Philippos Oct 19 '17 at 10:46
  • Mr. Balsdon, glad you solved your problem. "tr" is definitely a solution to use in a streaming environment. The "dos2unix" solution would be performed against the entire file prior to moving it through the pipeline. – unclesmrgol dragon Oct 20 '17 at 15:10

Not the answer you're looking for? Browse other questions tagged or ask your own question.