1

Let's say I have the following trivial script, tmp.sh:

echo "testing"
stat .
echo "testing again"

Trivial as it is, it has \r\n (that is, CRLF, that is carriage return+line feed) as line endings. Since the webpage will not preserve the line endings, here is a hexdump:

$ hexdump -C tmp.sh 
00000000  65 63 68 6f 20 22 74 65  73 74 69 6e 67 22 0d 0a  |echo "testing"..|
00000010  73 74 61 74 20 2e 0d 0a  65 63 68 6f 20 22 74 65  |stat ...echo "te|
00000020  73 74 69 6e 67 20 61 67  61 69 6e 22 0d 0a        |sting again"..|
0000002e

Now, it has CRLF line endings, because the script was started and developed on Windows, under MSYS2. So, when I run it on Windows 10 in MSYS2, I get the expected:

$ bash tmp.sh
testing
  File: .
  Size: 0               Blocks: 40         IO Block: 65536  directory
Device: 8e8b98b6h/2391513270d   Inode: 281474976761067  Links: 1
Access: (0755/drwxr-xr-x)  Uid: (197609/      USER)   Gid: (197121/    None)
Access: 2020-04-03 10:42:53.210292000 +0200
Modify: 2020-04-03 10:42:53.210292000 +0200
Change: 2020-04-03 10:42:53.210292000 +0200
 Birth: 2019-02-07 13:22:11.496069300 +0100
testing again

However, if I copy this script to an Ubuntu 18.04 machine, and run it there, I get something else:

$ bash tmp.sh
testing
stat: cannot stat '.'$'\r': No such file or directory
testing again

In other scripts with the same line endings, I have also gotten this error in Ubuntu bash:

line 6: $'\r': command not found

... likely from an empty line.

So, clearly, something in Ubuntu chokes on the carriage returns. I have seen BASH and Carriage Return Behavior :

it doesn’t have anything to do with Bash: \r and \n are interpreted by the terminal, not by Bash

... however, I guess that is only for stuff typed verbatim on the command line; here the \r and \n are already typed in the script itself, so it must be that Bash interprets the \r here.

Here is the version of Bash in Ubuntu:

$ bash --version
GNU bash, version 4.4.20(1)-release (x86_64-pc-linux-gnu)

... and here the version of Bash in MSYS2:

$ bash --version
GNU bash, version 4.4.23(2)-release (x86_64-pc-msys)

(they don't seem all that much apart ...)

Anyways, my question is - is there a way to persuade Bash on Ubuntu/Linux to ignore the \r, rather than trying to interpret it as a (so to speak) "printable character" (in this case, meaning a character that could be a part of a valid command, which bash interprets as such)? EDIT: without having to convert the script itself (so it remains the same, with CRLF line endings, if it is checked in that way, say, in git)

EDIT2: I would prefer it this way, because other people I work with might reopen the script in Windows text editor, potentially reintroduce \r\n again into the script and commit it; and then we might end up with an endless stream of commits which might be nothing else than conversions of \r\n to \n polluting the repository.

EDIT2: @Kusalananda in comments mentioned dos2unix (sudo apt install dos2unix); note that just writing this:

$ dos2unix tmp.sh 
dos2unix: converting file tmp.sh to Unix format...

... will convert the file in-place; to have it output to stdout, one must setup stdin redirection:

$ dos2unix <tmp.sh | hexdump -C
00000000  65 63 68 6f 20 22 74 65  73 74 69 6e 67 22 0a 73  |echo "testing".s|
00000010  74 61 74 20 2e 0a 65 63  68 6f 20 22 74 65 73 74  |tat ..echo "test|
00000020  69 6e 67 20 61 67 61 69  6e 22 0a                 |ing again".|
0000002b

... and then, in principle, one could run this on Ubuntu, which seems to work in this case:

$ dos2unix <tmp.sh | bash
testing
  File: .
  Size: 20480       Blocks: 40         IO Block: 4096   directory
Device: 816h/2070d  Inode: 1572865     Links: 27
Access: (1777/drwxrwxrwt)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2020-04-03 11:11:00.309160050 +0200
Modify: 2020-04-03 11:10:58.349139481 +0200
Change: 2020-04-03 11:10:58.349139481 +0200
 Birth: -
testing again

However, - aside from the slightly messy command to remember - this also changes bash semantics, as stdin is no longer a terminal; this may have worked with this trivial example, but see e.g. https://stackoverflow.com/questions/23257247/pipe-a-script-into-bash for example of bigger problems.

  • 2
    Yes, convert the file to a Unix text file with dos2unix. – Kusalananda Apr 3 at 9:04
  • Thanks @Kusalananda - that just reminded me to add an edit, because I specifically do not want to change the file itself, nor its CRLF line endings. – sdaau Apr 3 at 9:07
  • Thanks @StephenKitt - I'm aware that MSYS2 will handle usual \n, but the problem is if I work on a repository with Windows people who otherwise don't care, the repository will end up being polluted with commits that are a constant change of line endings, which I want to avoid (added edits to OP). – sdaau Apr 3 at 9:24
  • Thanks, @StephenKitt - I took a look at the link; indeed that might help with git in itself ... although then I'd have to possible come up with a policy of how to handle other files than .sh. In all, I was hoping there is some bash environment variable or option I could use - but if there isn't one, there isn't one; I'll have to live with it. Thanks again! – sdaau Apr 3 at 9:35
  • Also, for those wondering how can MSYS2 bash handle both \n and \r\n as line endings, it turns out, it is not trivial at all - see 0005-bash-4.3-msys2-fix-lineendings.patch for all the gory details. – sdaau Apr 3 at 9:51
2

As far as I’m aware, there’s no way to tell Bash to accept Windows-style line endings.

In situations involving Windows, common practice is to rely on Git’s ability to automatically convert line-endings when committing, using the autocrlf configuration flag. See for example GitHub’s documentation on line endings, which isn’t specific to GitHub. That way files are committed with Unix-style line endings in the repository, and converted as appropriate for each client platform.

(The opposite problem isn’t an issue: MSYS2 works fine with Unix-style line endings, on Windows.)

| improve this answer | |
1

You should use binfmt_misc for that [1].

First, define a magic that handles files which start with #! /bin/bash<CR><LF>, then create an executable interpreter for it. The interpreter can be another script:

INTERP=/path/to/bash-crlf

echo ",bash-crlf,M,,#! /bin/bash\x0d\x0a,,$INTERP," > /proc/sys/fs/binfmt_misc/register
cat > "$INTERP" <<'EOT'; chmod 755 "$INTERP"
#! /bin/bash
script=$1; shift; exec bash <(sed 's/\r$//' "$script") "$@"
EOT

Test it:

$ printf '%s\r\n' '#! /bin/bash' pwd >/tmp/foo; chmod 755 /tmp/foo
$ cat -v /tmp/foo
#! /bin/bash^M
pwd^M
$ /tmp/foo
/tmp

The sample interpreter has two problems: 1. since it passes the script via a non-seekable file (a pipe), bash will read it byte by byte, very inefficiently, and 2. any error messages will refer to /dev/fd/63 or similar instead of the name of the original script.

[1] Of course, instead of using binfmt_misc you can just create a /bin/bash^M symbolic link to the interpreter, which would also work on other systems like OpenBSD:

ln -s /path/to/bash-crlf $'/bin/bash\r'

But on Linux, shebanged executables have no advantage over binfmt_misc, and putting garbage inside system directories is not the right strategy, and will leave any sysadmin shaking his or her head ;-)

| improve this answer | |
  • Thanks @mosvy, that looks pretty neat! – sdaau Apr 3 at 12:24
0

Ok, I found somewhat of a workaround, via:

"Junctioned" symbolic links

Modern unix systems have a way to make arbitrary data appear as a file, independently of how it's stored: FUSE. With FUSE, every operation on a file (create, open, read, write, list directory, etc.) invokes some code in a program, and that code can do whatever you want. See Create a virtual file that is actually a command. You could try out scriptfs or fuseflt, or if you're feeling ambitious, roll your own.

... and Create a virtual file that is actually a command

You may be looking for a named pipe.

So, this is the approach: create a named pipe, have dos2unix output to it, then have bash call the named pipe.

Here I have the original tmp.sh with CRLF line endings in /tmp; fist, let's create the named pipe:

tmp$ mkfifo ftmp.sh

Now, if you run this command:

tmp$ dos2unix <tmp.sh >ftmp.sh

... you will notice it blocks; then if you do, say:

~$ cat /tmp/ftmp.sh | hexdump -C
00000000  65 63 68 6f 20 22 74 65  73 74 69 6e 67 22 0a 73  |echo "testing".s|
00000010  74 61 74 20 2e 0a 65 63  68 6f 20 22 74 65 73 74  |tat ..echo "test|
00000020  69 6e 67 20 61 67 61 69  6e 22 0a                 |ing again".|
0000002b

... you will notice the conversion has been done - and after the cat command has ran its course, the dos2unix <tmp.sh >ftmp.sh command, that blocked earlier, has exited.

So, we can set up the dos2unix write to named pipe in an "endless" while loop:

tmp$ while [ 1 ] ; do dos2unix <tmp.sh >ftmp.sh ; done

... and even if it's a "tight" loop, it should not be a problem, as most of the time the command inside the while loop is blocking.

Then I can do:

~$ bash /tmp/ftmp.sh
testing
  File: .
  Size: 4096        Blocks: 8          IO Block: 4096   directory
Device: 801h/2049d  Inode: 5276132     Links: 7
...
testing again
$

... and clearly, the script runs fine.

What is good about this approach, is that I can have the original tmp.sh open in a text editor; and write new code - with CRLF endings - then save tmp.sh; and running bash /tmp/ftmp.sh under Linux will run the latest saved version.

The problem with this is that commands like read -p "Enter user: " user that rely on actual terminal stdin will fail; or rather not fail, but if you try, say this as /tmp/tmp.sh

echo "testing"
stat .
echo "testing again"
read -p "Enter user: " user
echo "user is: $user"

... then this will be output:

$ bash /tmp/ftmp.sh
testing
  File: .
  Size: 4096        Blocks: 8          IO Block: 4096   directory
...
 Birth: -
testing again
Enter user: tyutyu
user is: tyutyu
testing
  File: .
  Size: 4096        Blocks: 8          IO Block: 4096   directory
...
 Birth: -
testing again
Enter user: asd
user is: asd
testing
...

... and so on - that is, stdin from keyboard in terminal is interpreted correctly, but for some reason the script starts looping, and executes from the start over and over again (which does not happen if we do not have a read -p ... command in the original tmp.sh). Maybe there is some redirection stuff (e.g. adding some 0>1& or whatever to the while loop command; actually, I had an .sh script with wget that also started looping like that, and simply adding an explicit exit at end of the .sh script seemed to work to stop the script looping) that could handle this as well, - but so far, the script I need to use does not have read -p like commands, so this approach might work for me.

| improve this answer | |
  • 1
    You're overcomplicating yourself IMHO. Using binfmt_misc is simpler, see the example. – mosvy Apr 3 at 12:01
  • Thanks @mosvy - had never heard about binfmt_misc before, good to know! – sdaau Apr 3 at 12:22
  • 1
    Using binfmt_misc is the way to go IMHO, but the quick & dirty <(sed ...) has problems with bash doing a read for each byte when reading a script from a pipe, and with the error messages referring to /dev/fd/N instead of the original filename. I'll update the answer with something better. – mosvy Apr 3 at 12:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.