When I copy my bash script across from Notepad++ into a new file with nano editor inside SSH and save. It runs fine. (sh ./install).

But if I save the file (Exact same contents), upload to my web server, download it using Wget on the same machine. I get syntax errors. I've checked the encoding and they seem to be the same. I have since used a plethora of character encoding to see if it will resolve the issue. I am also setting the file to executable once I download it using wget!

The file runs fine and I have zero errors when copy and pasting using nano. Any idea what this may be?

  • 1
    There are obviously changes there, probably whitespace or something that is difficult to notice. Use diff between the one that works and the one which doesn't to see what they are.
    – Graeme
    Mar 5 '14 at 0:59
  • How big is the script? If it is not too large, you can debug the not-working script by commenting out lines to narrow down the issue. Additionally, you can add set -x to the source to see how things are running line by line.
    – mkc
    Mar 5 '14 at 1:00
  • @Ketan, why debug when you can just diff? Doesn't make a lot of sense to me.
    – Graeme
    Mar 5 '14 at 1:06
  • @Graeme It is possible that many places have changed in the file. Diff will help find those places and debug will find which is causing problems.
    – mkc
    Mar 5 '14 at 1:09
  • 1
    In addition to diff cmp may be useful. You can then have a look at the differing byte with a hex editor or dd ... |od ... Mar 5 '14 at 1:20

I would be willing to bet that the problem is related to line endings. You're probably going through a non-*nix machine somewhere along the line. I also ran into a problem once where apache (running on Linux) was adding windows style line endings to uploaded text files so you might be seeing something similar.

To test, take the file you downloaded and pass it through od. If it's a long file, just grab the first few lines:

head script.sh | od -c

Look through the output and check if you have anything like this:

f   o   o  \r  \n

The \r is a carriage return and on Windows, lines are ended with \r\n as opposed to \n on *nix. If it turns out that this was indeed the problem, you can fix your file by removing the carriage returns:

sed -i 's/\r//g' script.sh
  • 1
    Just tried using sed to strip out carriage and it is now working! Looks like you were bang on. Excellent sir. Thank you! I hope this post serves others as I did try Google for help but to no avail. Seems like when the file is saved on Windows, uploaded using SFTP and then downloaded, somewhere something is addded or changed. Mar 5 '14 at 10:43
  • 2
    Excellent answer! I'd add that the reason that it worked when pasted into an editor is that the carriage returns issued through an SSH shell will be stored as Unix line endings - which is why it worked when cut-and pasted, but not when the original file was scp'd or ftp'd over.
    – Jenny D
    Mar 5 '14 at 13:22

As @graeme has astutely pointed out since you have the script in both forms on the server you could perform a simple diff to determine what's different between the working version and the problematic version.

$ diff working.sh broken.sh

You can also do a side-by-side diff like this:

$ diff -y working.sh broken.sh

If the script isn't working because of some sort of typo you can often times detect these by adding the -x switch to bash, which causes it to be verbose.

$ bash -x broken.sh

You can also incorporate this switch into the shebang (#!/bin/bash) at the top of your scripts like this:

#!/bin/bash -x

Line endings

This is often the issue when moving files from Windows to Unix/Linux systems. The issue has to do with how the ends of lines are denoted on the 2 platforms. You can read more about it here on Wikipedia, titled: Newlines.

make a sample file

$ echo -e "This is a file.\nThat I made on Unix.\n" > unixfile.txt

As @terdon has described in his answer, you can use sed to strip these out, you can also often use a tool called dos2unix to do the same thing too. You can use it in either 1 of 2 ways:

$ dos2unix unixfile.txt

or if you do not want to overwrite the existing file:

$ dos2unix -n oldfile.txt newfile.txt

When you use the above diff I mentioned earlier you'll get output like this when you compare these 2 files:

$ diff -y unixfile.txt winfile.txt 
This is a couple                            |   This is a couple
of lines of sample                          |   of lines of sample
text.                                       |   text.

You won't be able to discern the differences, just that they're there. Again @terdon's answer shows one method for routing out the issue using od. You can of course use a variety of ways to figure out what's up.

Using vim

with the file cmd.

$ file unixfile.txt
winfile.txt: ASCII text 

$ file winfile.txt 
unixfile.txt: ASCII text, with CRLF line terminators

The above is highlighting the issue, that the file from Windows has CRLF (aka. Carriage Return + Linefeed characters at the end of the lines). These characters are 0x0D & 0x0A in hex, again see the Wikipedia article about Newlines if you want to know more about it.

You can also use vim to see the issue too:

$ vim winfile.txt

Here's a little sequence that shows what to do in vim to see the issue. The CRLF characters typically show up in Unix as ^M, that's a Ctrl+M.

                                       ss of vim

The sequence is showing me re-opening the file, winfile.txt as a formatted Unix file (:e ++ff=unix). This tells vim not to auto-detect that the file is formatted for Windows, and so it will display the ^M line termination characters.

  • Thanks for the reply. I've given diff a go and cannot see any changes. I've compared them both side by side. Is there anything I should be looking for? Mar 5 '14 at 10:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.