Handling Bash script with CRLF (carriage return) in Linux as in MSYS2?

Question

Let's say I have the following trivial script, tmp.sh:

echo "testing"
stat .
echo "testing again"

Trivial as it is, it has \r\n (that is, CRLF, that is carriage return+line feed) as line endings. Since the webpage will not preserve the line endings, here is a hexdump:

$ hexdump -C tmp.sh 
00000000  65 63 68 6f 20 22 74 65  73 74 69 6e 67 22 0d 0a  |echo "testing"..|
00000010  73 74 61 74 20 2e 0d 0a  65 63 68 6f 20 22 74 65  |stat ...echo "te|
00000020  73 74 69 6e 67 20 61 67  61 69 6e 22 0d 0a        |sting again"..|
0000002e

Now, it has CRLF line endings, because the script was started and developed on Windows, under MSYS2. So, when I run it on Windows 10 in MSYS2, I get the expected:

$ bash tmp.sh
testing
  File: .
  Size: 0               Blocks: 40         IO Block: 65536  directory
Device: 8e8b98b6h/2391513270d   Inode: 281474976761067  Links: 1
Access: (0755/drwxr-xr-x)  Uid: (197609/      USER)   Gid: (197121/    None)
Access: 2020-04-03 10:42:53.210292000 +0200
Modify: 2020-04-03 10:42:53.210292000 +0200
Change: 2020-04-03 10:42:53.210292000 +0200
 Birth: 2019-02-07 13:22:11.496069300 +0100
testing again

However, if I copy this script to an Ubuntu 18.04 machine, and run it there, I get something else:

$ bash tmp.sh
testing
stat: cannot stat '.'$'\r': No such file or directory
testing again

In other scripts with the same line endings, I have also gotten this error in Ubuntu bash:

line 6: $'\r': command not found

... likely from an empty line.

So, clearly, something in Ubuntu chokes on the carriage returns. I have seen BASH and Carriage Return Behavior :

it doesn’t have anything to do with Bash: \r and \n are interpreted by the terminal, not by Bash

... however, I guess that is only for stuff typed verbatim on the command line; here the \r and \n are already typed in the script itself, so it must be that Bash interprets the \r here.

Here is the version of Bash in Ubuntu:

$ bash --version
GNU bash, version 4.4.20(1)-release (x86_64-pc-linux-gnu)

... and here the version of Bash in MSYS2:

$ bash --version
GNU bash, version 4.4.23(2)-release (x86_64-pc-msys)

(they don't seem all that much apart ...)

Anyways, my question is - is there a way to persuade Bash on Ubuntu/Linux to ignore the \r, rather than trying to interpret it as a (so to speak) "printable character" (in this case, meaning a character that could be a part of a valid command, which bash interprets as such)? EDIT: without having to convert the script itself (so it remains the same, with CRLF line endings, if it is checked in that way, say, in git)

EDIT2: I would prefer it this way, because other people I work with might reopen the script in Windows text editor, potentially reintroduce \r\n again into the script and commit it; and then we might end up with an endless stream of commits which might be nothing else than conversions of \r\n to \n polluting the repository.

EDIT2: @Kusalananda in comments mentioned dos2unix (sudo apt install dos2unix); note that just writing this:

$ dos2unix tmp.sh 
dos2unix: converting file tmp.sh to Unix format...

... will convert the file in-place; to have it output to stdout, one must setup stdin redirection:

$ dos2unix <tmp.sh | hexdump -C
00000000  65 63 68 6f 20 22 74 65  73 74 69 6e 67 22 0a 73  |echo "testing".s|
00000010  74 61 74 20 2e 0a 65 63  68 6f 20 22 74 65 73 74  |tat ..echo "test|
00000020  69 6e 67 20 61 67 61 69  6e 22 0a                 |ing again".|
0000002b

... and then, in principle, one could run this on Ubuntu, which seems to work in this case:

$ dos2unix <tmp.sh | bash
testing
  File: .
  Size: 20480       Blocks: 40         IO Block: 4096   directory
Device: 816h/2070d  Inode: 1572865     Links: 27
Access: (1777/drwxrwxrwt)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2020-04-03 11:11:00.309160050 +0200
Modify: 2020-04-03 11:10:58.349139481 +0200
Change: 2020-04-03 11:10:58.349139481 +0200
 Birth: -
testing again

However, - aside from the slightly messy command to remember - this also changes bash semantics, as stdin is no longer a terminal; this may have worked with this trivial example, but see e.g. https://stackoverflow.com/questions/23257247/pipe-a-script-into-bash for example of bigger problems.

Answer 1

As far as I’m aware, there’s no way to tell Bash to accept Windows-style line endings.

In situations involving Windows, common practice is to rely on Git’s ability to automatically convert line-endings when committing, using the autocrlf configuration flag. See for example GitHub’s documentation on line endings, which isn’t specific to GitHub. That way files are committed with Unix-style line endings in the repository, and converted as appropriate for each client platform.

(The opposite problem isn’t an issue: MSYS2 works fine with Unix-style line endings, on Windows.)

Answer 2

You should use binfmt_misc for that [1].

First, define a magic that handles files which start with #! /bin/bash<CR><LF>, then create an executable interpreter for it. The interpreter can be another script:

INTERP=/path/to/bash-crlf

echo ",bash-crlf,M,,#! /bin/bash\x0d\x0a,,$INTERP," > /proc/sys/fs/binfmt_misc/register
cat > "$INTERP" <<'EOT'; chmod 755 "$INTERP"
#! /bin/bash
script=$1; shift; exec bash <(sed 's/\r$//' "$script") "$@"
EOT

Test it:

$ printf '%s\r\n' '#! /bin/bash' pwd >/tmp/foo; chmod 755 /tmp/foo
$ cat -v /tmp/foo
#! /bin/bash^M
pwd^M
$ /tmp/foo
/tmp

---

The sample interpreter has two problems: 1. since it passes the script via a non-seekable file (a pipe), bash will read it byte by byte, very inefficiently, and 2. any error messages will refer to /dev/fd/63 or similar instead of the name of the original script.

[1] Of course, instead of using binfmt_misc you can just create a /bin/bash^M symbolic link to the interpreter, which would also work on other systems like OpenBSD:

ln -s /path/to/bash-crlf $'/bin/bash\r'

But on Linux, shebanged executables have no advantage over binfmt_misc, and putting garbage inside system directories is not the right strategy, and will leave any sysadmin shaking his or her head ;-)

Answer 3

Ok, I found somewhat of a workaround, via:

"Junctioned" symbolic links

Modern unix systems have a way to make arbitrary data appear as a file, independently of how it's stored: FUSE. With FUSE, every operation on a file (create, open, read, write, list directory, etc.) invokes some code in a program, and that code can do whatever you want. See Create a virtual file that is actually a command. You could try out scriptfs or fuseflt, or if you're feeling ambitious, roll your own.

... and Create a virtual file that is actually a command

You may be looking for a named pipe.

So, this is the approach: create a named pipe, have dos2unix output to it, then have bash call the named pipe.

Here I have the original tmp.sh with CRLF line endings in /tmp; fist, let's create the named pipe:

tmp$ mkfifo ftmp.sh

Now, if you run this command:

tmp$ dos2unix <tmp.sh >ftmp.sh

... you will notice it blocks; then if you do, say:

~$ cat /tmp/ftmp.sh | hexdump -C
00000000  65 63 68 6f 20 22 74 65  73 74 69 6e 67 22 0a 73  |echo "testing".s|
00000010  74 61 74 20 2e 0a 65 63  68 6f 20 22 74 65 73 74  |tat ..echo "test|
00000020  69 6e 67 20 61 67 61 69  6e 22 0a                 |ing again".|
0000002b

... you will notice the conversion has been done - and after the cat command has ran its course, the dos2unix <tmp.sh >ftmp.sh command, that blocked earlier, has exited.

So, we can set up the dos2unix write to named pipe in an "endless" while loop:

tmp$ while [ 1 ] ; do dos2unix <tmp.sh >ftmp.sh ; done

... and even if it's a "tight" loop, it should not be a problem, as most of the time the command inside the while loop is blocking.

Then I can do:

~$ bash /tmp/ftmp.sh
testing
  File: .
  Size: 4096        Blocks: 8          IO Block: 4096   directory
Device: 801h/2049d  Inode: 5276132     Links: 7
...
testing again
$

... and clearly, the script runs fine.

What is good about this approach, is that I can have the original tmp.sh open in a text editor; and write new code - with CRLF endings - then save tmp.sh; and running bash /tmp/ftmp.sh under Linux will run the latest saved version.

The problem with this is that commands like read -p "Enter user: " user that rely on actual terminal stdin will fail; or rather not fail, but if you try, say this as /tmp/tmp.sh

echo "testing"
stat .
echo "testing again"
read -p "Enter user: " user
echo "user is: $user"

... then this will be output:

$ bash /tmp/ftmp.sh
testing
  File: .
  Size: 4096        Blocks: 8          IO Block: 4096   directory
...
 Birth: -
testing again
Enter user: tyutyu
user is: tyutyu
testing
  File: .
  Size: 4096        Blocks: 8          IO Block: 4096   directory
...
 Birth: -
testing again
Enter user: asd
user is: asd
testing
...

... and so on - that is, stdin from keyboard in terminal is interpreted correctly, but for some reason the script starts looping, and executes from the start over and over again (which does not happen if we do not have a read -p ... command in the original tmp.sh). Maybe there is some redirection stuff (e.g. adding some 0>1& or whatever to the while loop command; actually, I had an .sh script with wget that also started looping like that, and simply adding an explicit exit at end of the .sh script seemed to work to stop the script looping) that could handle this as well, - but so far, the script I need to use does not have read -p like commands, so this approach might work for me.

Answer 4

I had somewhat the same issue with scripts that I shared with users - Some would copy/paste the content of a script through Windows, and thereby add CRLF line-endings to it, breaking it completely.

I found that making the script itself "self-heal" was the only way to fix this in my case, as modifying the environments or bash on someone else's systems was not an option (or a good idea).

I managed to create a oneliner that you place as the first line of code in the bash script itself (after the #!/bin/bash line), which detects if the script itself has CRLF line-endings, and if so, re-writes itself, and then executes itself again. This has the nice side-effect that if someone re-introduces CRLF line-endings again (as you mentioned), the script will just self-heal itself again.

The only hitch is that the script needs to be run with "bash tmp.sh" for it to work (when it has CRLF line-endings) - I have found that most users do this anyway as the first thing when "./tmp.sh" doesn't work, so this has saved me in most cases. After the script has fixed itself, and it no longer has CRLF line-endings, executing it with "./tmp.sh" as normal then works without issue. In your question you use "bash tmp.sh" so you shouldn't even notice this.

Check out my GitHub page for the oneliner (see line 3 in the code) and the readme.

Answer 5

You can insert an hash (#) just at the end of every lines in your bash scripts. In such a way, the shells on Unix will consider the CR just as a comment and won't care about it.

"Hex speaking", any line should terminate with

0x23 0x0D 0x0A

Example:

echo "testing" #
stat . #
echo "testing again" #