3

I execute the following code in Bash:

bash /dev/fd/10 10<<-'SES'

    cat <<EMR >> /etc/apache2/apache2.conf
        #
        <Directory /var/www/>
        Options Indexes FollowSymLinks
        AllowOverride All
        Require all granted
        </Directory>
        EMR

SES

But the execution breaks and I get:

warning: here-document at line 2 delimited by end-of-file (wanted `EMR')

  • I don't understand what the error means and I already added - between the operator and the name of the containing heredoc to allow tabulations.
  • I made sure there are no spaces after the opener (operator+name) or after the delimiter.

Note: If you copy the code above to test it in your computer, you need to change all leading spaces to tabs (tabulations) as StackExchange changed the original tabulations to spaces.


Possible causes to rule out:

There are several possible causes for this, here are some I've ruled out:

1. Problematic hidden characters:

I edit the file with Notepad++ and I turned on the "Show all symbols" mode to make sure all indents are tabulation based (see red arrows) and that all End Of Line chars (EOLs) are LF unix based (there aren't any CR chars):

enter image description here

Moreover, the encoding is UTF-8, so it's not an encoding problem.

2. Pseudo dash symbol (<<-):

It seems as the added hyphen (<<-) doesn't do its job of stripping all leading tabs because when I manually delete all leading tabs whatsoever (or all leading tabs before each delimiter) the heredoc does work as expected. One can suggest that the hyphen isn't really a dash symbol, but why won't it be? I have no way to validate that.

3. Bash bug:

Other people not-using Windows, and using different Bash-containing Linux distros didn't have this, so this is most likely not a Bash bug. I've also tested this with the Bash development section at GNU.


Corrupted pasting of copied data from Notepad++ might be it:

If I paste the heredoc from Notepad++ into a nano file, it seems the leading tabs are all in place - they aren't deleted are transformed into spaces in pasting.

Moreover, cat script.sh | grep "^ " and cat script.sh | grep -x '\s*EMR' (when done in the files directory), come out empty.

Yet, in a later pasting I found another corruption problem that is most likley to cause this (see my answer).

  • 1
    Oops, sorry, it should work, provided you use hard tabs (not spaces). As you say, the outer <<- should eat all leading tabs and the inner one shouldn't see any – ilkkachu Apr 19 '17 at 11:46
  • Works for me (as @ikkachu mentioned, using tabs, not spaces). – Patrick Apr 19 '17 at 12:48
  • 2
    The carriage returns are quite visible in that image, aren't they? The shell expects the string EMR, but what it sees, is EMR\r. – ilkkachu Apr 19 '17 at 18:05
  • 1
    There is, but I have no desire to discuss this here. – JohnDoea Apr 20 '17 at 8:28
  • 2
    This question is more and more hard to follow. I suggest that you rather than adding further updates to the question (or carry on conversations in various comments), consolidate it into a coherent unity. At this point in time I'm not even sure it's on topic for this forum. – Kusalananda Apr 21 '17 at 14:35
9

The image you posted shows carriage returns (CR in the image). This is a DOS-formatted text file. Use dos2unix or some similar utility to convert the file to a proper Unix text file.

As per comment below: If you absolutely need to do development in Notepad++ on Windows (I would personally recommend doing the development on the same OS that you're targeting, if at all possible), then the program may be told to save text files with Unix newlines by selecting "Unix (LF)" in the "New Documents" preferences:

Notepad++ setting for new documents

Make sure that your file has a newline (LF) at the end of the last line. All lines in a Unix text file (and a shell script is a text file) needs a terminating newline at the end. If the last line doesn't have one, the file is technically actually not a text file but binary.

See also, for example, the answers to "Remove ^M character from log files"

  • @Benia It may be that, or it isn't a real dash (minus). In general, I'd advice to edit Unix text files on a Unix system. – Kusalananda Apr 20 '17 at 14:35
  • @Benia I'll retract that comment since you say that it does work when you manually remove the tabs. The inner here-document shouldn't need the dash though, right? – Kusalananda Apr 20 '17 at 14:41
  • 2
    @Benia Your script works here with tabs, but not with spaces. Did you check whether Notepad++ does something like expanding tabs to spaces on writing and vice versa on reading? That way you think there are tabs, but there are spaces indeed. Check with grep "^ " script.sh – Philippos Apr 21 '17 at 13:18
  • @Benia I'm on debian 8. If you get nothing, it's not spaces instead of tabs. Another try: what about grep -x '\s*EMR'? – Philippos Apr 21 '17 at 14:31
  • Your aiming was right --- These are CR chars created by Windows as a response to the TTY window size. Please see my answer below if you want to expand on this Windows behavior. – JohnDoea Apr 24 '17 at 21:57
7
+150

Unfortunately, the design of terminal emulators (such as the terminal window on your Ubuntu box) mostly doesn't allow for good paste support; in particular, for the most part, paste works exactly like typing the same thing (if you were a very quick typist). Most stuff you'd think is some special signal to the program running in the terminal is actually in-band data—that is, just a character. If your paste contains that character, the program can't tell the difference between you typing the character and pasting it.

So you can't really use paste to transfer files around, unless they contain no characters that could confuse the receiving program. In particular, the tabs your paste contain are confusing bash—it's seeing a tab and attempting to do file or command completion. Just like when you type a tab.

The cat > filename trick may well fail if there were a control-D (yes, that's a character) in the pasted contents. But if you avoid control characters (other than newline and tab), it should work.

Unix folks typically use file transfer programs to move files around. The basic one we mostly use is scp or sftp, both of these are part of SSH (and are thus fully encrypted, authenticated, etc.—same security guarantees as SSH provides).

The easiest way of working between Windows and an Ubuntu VM (or vice versa) is probably to use your hypervisor's (VM product's) file sharing features—most have them. You can pick a directory on the host, and it'll be visible inside the guest. With WSL, you can access all your Windows files under /mnt/<drive letter>/. If your hypervisor doesn't have that feature, the second-easiest is to use an ordinary file share. You can mount Windows file shares from Ubuntu, or install Samba on Ubuntu to make file shares to Windows.

Depending on what you're doing, there may be better approaches. For example, automated deployment. Maybe you want to have it some that you check the file in to git, and then it's automatically deployed on your Ubuntu box when you do a git push? This can be as simple as using git hooks, or much more complicated—there are entire software packages dedicated to it.

Finally, there is a big advantage to writing these scripts to files, not just pasting them in to a shell: the file serves as documentation of what you did. Did something go wrong? Unless it went really wrong, check the file to see why. That lets you determine exactly what went wrong, often the first step to repairing the damage. Need to do it again? Just source the file a second time. Need to do something similar? Edit the file, then source it.

If you must paste a here-doc:

If you want to paste in to bash directly, the best bet is to make sure it contains no control characters other than newline. Pasting this should work:

bash /dev/fd/10 10<<'SES'

cat <<EMR >> /etc/apache2/apache2.conf
#
<Directory /var/www/>
Options Indexes FollowSymLinks
AllowOverride All
Require all granted
</Directory>
EMR

SES

If you must have the indentation, you can use spaces and sed instead of the leading - to strip it (note here that went ahead and used stdin instead of fd 10 because the syntax is easier and I doubt it matters):

sed -e 's/^\s\+//' <<'SES' | bash

    cat <<EMR >> /etc/apache2/apache2.conf
    #
    <Directory /var/www/>
    Options Indexes FollowSymLinks
    AllowOverride All
    Require all granted
    </Directory>
    EMR

SES

That uses sed's search-replace functionality (s/PATTERN/REPLACEMENT/) to look at the beginning of the line (^) for whitespace (\s) repeated one or more times (\+) and replaces it with nothing. In other words, it trims off all leading spaces on each line. The -e option to sed explicitly identifies this is a sed script to run—it's not actually required, sed will interpret its first argument as script anyway—but I just personally find it clearer to always use -e.

Note that "whitespace" above includes spaces and tabs (and a few more characters that aren't really relevant here). That sed will trim them all off, regardless of the mix used.

Of course, I have no idea why you're pasting a command in to bash telling it to run bash to run cat.

  • 1
    @Benia edited, I think that answers all your questions. You might find looking at the revision history easier than re-reading the whole thing. Also, maybe you should ask a question about your logistical reasons—possibly they're fixable. I don't anything about them, so I can't really advise there. You could try our chat room to see if people think they'd make a good question. chat.stackexchange.com/rooms/26/dev-chat – derobert Apr 21 '17 at 17:38
  • 1
    @Benia last solution contains no tabs, only spaces. The ^ in the sed pattern means beginning of line. So that's one or more (\+) whitespace characters (\s) at the beginning of the line (^). (sorry for brief response, using SE phone app) – derobert Apr 21 '17 at 22:26
  • 1
    @Benia no need for a bigger bounty, I think I've got enough worthless Internet points already. Tabs don't work because bash does special things with them. Note how when you type a space in the shell, you get a space. Type a tab, you don't get a tab. – derobert Apr 21 '17 at 22:40
  • 1
    @Benia I added in a sed explanation, based loosely on your proposed edit and on my comment (I don't think your edit should have been rejected, but nothing I can do about that). Also—there is only one sed script there, not multiple. I just use -e because I think its clearer. – derobert Apr 22 '17 at 13:30
  • 2
    @Benia BTW: I think part of the problem is I've been confused about how much you know about shell scripting—that bash /dev/fd/10 10<<-'SES' bit isn't something I'd expect a novice/beginner to come up with. But I guess maybe that's copy & paste? Or you just somehow never came across sed. – derobert Apr 22 '17 at 13:40
2

The problem undoubtedly originates in Windows. It deals with the TTY window-size; The narrower the window size is, the more Line Feed (LF) char and Carriage Return (CR) char combos will be added in pasting per the Window's narrowness. Such combos are represented as CRLF.

Solution

If I understand Nano's maintainer Benno Schulenberg correct, Nano will naturally convert CRLF combos into Trailing Whitespace chars (^J chars).

As suggested to me by Benno, adding the following code in the end of /etc/nanorc solved it:

bind ^J enter main

Clarifying the solution

On the one hand, it will disable formation of ^J chars that break the code; on the other hand, it will add only Line Feeds (LF chars), applicable in Linux (unlike the Windows CRLF combos), to the copied data.

The solely added LF chars are good as they prevent the pasted data to appear as one long line of text.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.