9

I'm trying to bring HEREDOC text into a shell script variable in a POSIX compliant way. I tried like so:

#!/bin/sh

NEWLINE="
"

read_heredoc2() {
  while IFS="$NEWLINE" read -r read_heredoc_line; do
    echo "${read_heredoc_line}"
  done
}

read_heredoc2_result="$(read_heredoc2 <<'HEREDOC'

                        _                            _ _
                       | |                          | (_)
  _ __ ___  _   _ _ __ | | __ _  ___ ___  ___  _ __ | |_ _ __   ___
 | '_ ` _ \| | | | '_ \| |/ _` |/ __/ _ \/ _ \| '_ \| | | '_ \ / _ \
 | | | | | | |_| | |_) | | (_| | (_|  __/ (_) | | | | | | | | |  __/
 |_| |_| |_|\__, | .__/|_|\__,_|\___\___|\___/|_| |_|_|_|_| |_|\___|
             __/ | |
            |___/|_|



HEREDOC
)"

echo "${read_heredoc2_result}"

That produced the following which is wrong:

                        _                            _ _
                       | |                          | (_)
  _ __ ___  _   _ _ __ | | __ _  ___ ___  ___  _ __ | |_ _ __   ___
 | '_ ` _ \| | | | '_ \| |/ _` |/ __/ _ \/ _ \| '_ \| | | '_ \ / _  | | | | | | |_| | |_) | | (_| | (_|  __/ (_) | | | | | | | | |  __/
 |_| |_| |_|\__, | .__/|_|\__,_|\___\___|\___/|_| |_|_|_|_| |_|\___|
             __/ | |
            |___/|_|

The following works but I don't like how clunky it is by using a random output variable:

#!/bin/sh

NEWLINE="
"

read_heredoc1() {
  read_heredoc_first=1
  read_heredoc_result=""
  while IFS="$NEWLINE" read -r read_heredoc_line; do
    if [ ${read_heredoc_first} -eq 1 ]; then
      read_heredoc_result="${read_heredoc_line}"
      read_heredoc_first=0
    else
      read_heredoc_result="${read_heredoc_result}${NEWLINE}${read_heredoc_line}"
    fi
  done
}

read_heredoc1 <<'HEREDOC'

                        _                            _ _            
                       | |                          | (_)           
  _ __ ___  _   _ _ __ | | __ _  ___ ___  ___  _ __ | |_ _ __   ___ 
 | '_ ` _ \| | | | '_ \| |/ _` |/ __/ _ \/ _ \| '_ \| | | '_ \ / _ \
 | | | | | | |_| | |_) | | (_| | (_|  __/ (_) | | | | | | | | |  __/
 |_| |_| |_|\__, | .__/|_|\__,_|\___\___|\___/|_| |_|_|_|_| |_|\___|
             __/ | |                                                
            |___/|_|                                                



HEREDOC

echo "${read_heredoc_result}"

Correct output:

                        _                            _ _            
                       | |                          | (_)           
  _ __ ___  _   _ _ __ | | __ _  ___ ___  ___  _ __ | |_ _ __   ___ 
 | '_ ` _ \| | | | '_ \| |/ _` |/ __/ _ \/ _ \| '_ \| | | '_ \ / _ \
 | | | | | | |_| | |_) | | (_| | (_|  __/ (_) | | | | | | | | |  __/
 |_| |_| |_|\__, | .__/|_|\__,_|\___\___|\___/|_| |_|_|_|_| |_|\___|
             __/ | |                                                
            |___/|_|                                                

Any ideas?

  • If the banner is used only once, use cat with a here-document directly. If it's used in many places in the script, store it on file an cat it from there, just like /etc/motd is used on some systems. – Kusalananda Jan 28 '17 at 7:42
  • 1
    Note that the problem you're having is actually a Bash bug - you had a POSIX solution already in your original attempt, which will work fine in ksh, dash, ash, and the oldest Bourne shell I can find. Bash command substitution parsing is weird, and used to be even more broken. – Michael Homer Jan 29 '17 at 4:02
  • @MichaelHomer Oh, interesting! Wow, I'm on the latest Fedora 25, bash 4.3.43-4.fc25 – Kevin Jan 29 '17 at 4:08
  • 1
    Yeah. If you can get a 3-series version to try the script with, it breaks even sooner at the first backtick, so it's improving. I'm not sure whether they consider it a bug - arguably POSIX doesn't explicitly prohibit this behaviour, but it's fairly clear that command substitution contains all the characters until the ) and that quoted heredocs don't have expansions. – Michael Homer Jan 29 '17 at 4:13
  • 1
11

The problem is that, in Bash, inside $( ... ) escape (and other) sequences get parsed, even though the heredoc itself wouldn't have them. You get a doubled line because \ escapes the line break. What you're seeing is really a parsing issue in Bash - other shells don't do this. Backticks can also be a problem in older versions. I have confirmed that this is a bug in Bash, and it will be fixed in future versions.

You can at least simplify your function drastically:

func() {
    res=$(cat)
}
func <<'HEREDOC'
...
HEREDOC

If you want to choose the output variable it can be parameterised:

func() {
    eval "$1"'=$(cat)'
}
func res<<'HEREDOC'
...
HEREDOC

Or a fairly ugly one without eval:

{ res=$(cat) ; } <<'HEREDOC'
...
HEREDOC

The {} are needed, rather than (), so that the variable remains available afterwards.

Depending on how often you'll do this, and to what end, you might prefer one or another of these options. The last one is the most concise for a one-off.


If you're able to use zsh, your original command substitution + heredoc will work as-is, but you can also collapse all of this down further:

x=$(<<'EOT'
...
EOT
)

Bash doesn't support this and I don't think any other shell that would experience the problem you're having does either.

| improve this answer | |
  • One issue I'm finding is that $(cat) is stripping trailing newlines – Kevin Jan 28 '17 at 8:47
  • 1
    @Kevin Could the workaround here help? – Eliah Kagan Jan 28 '17 at 9:21
  • @EliahKagan Thanks, that might help, I'll try those out. – Kevin Jan 29 '17 at 3:39
  • @EliahKagan I've added a new answer below. – Kevin Jan 29 '17 at 5:31
5

About the OP solution:

  • You do not need an eval to assign a variable if you allow some constant variable to be used.

  • the general structure of calling a function that receives the HEREDOC could also be implemented.

A solution that works in all (reasonable) shells with both items solved is this:

#!/bin/bash
nl="
"

read_heredoc(){
    var=""
    while IFS="$nl" read -r line; do
        var="$var$line$nl"
    done 
}


read_heredoc <<'HEREDOC'

                        _                            _ _
                       | |                          | (_)
  _ __ ___  _   _ _ __ | | __ _  ___ ___  ___  _ __ | |_ _ __   ___
 | '_ ` _ \| | | | '_ \| |/ _` |/ __/ _ \/ _ \| '_ \| | | '_ \ / _ \
 | | | | | | |_| | |_) | | (_| | (_|  __/ (_) | | | | | | | | |  __/
 |_| |_| |_|\__, | .__/|_|\__,_|\___\___|\___/|_| |_|_|_|_| |_|\___|
             __/ | |
            |___/|_|

HEREDOC

read_heredoc2_result="$str"

printf '%s' "${read_heredoc2_result}"

A solution for the original question.

A solution that works since bash 2.04 (and recent zsh, lksh, mksh).
Look below for a more portable (POSIX) version.

#!/bin/bash
read_heredoc() {
    IFS='' read -d '' -r var <<'HEREDOC'

                        _                            _ _
                       | |                          | (_)
  _ __ ___  _   _ _ __ | | __ _  ___ ___  ___  _ __ | |_ _ __   ___
 | '_ ` _ \| | | | '_ \| |/ _` |/ __/ _ \/ _ \| '_ \| | | '_ \ / _ \
 | | | | | | |_| | |_) | | (_| | (_|  __/ (_) | | | | | | | | |  __/
 |_| |_| |_|\__, | .__/|_|\__,_|\___\___|\___/|_| |_|_|_|_| |_|\___|
             __/ | |
            |___/|_|



HEREDOC

}

read_heredoc
echo "$var"

The core command

IFS='' read -d '' -r var <<'HEREDOC'

works as follows:

  1. The word HEREDOC is (single) quoted to avoid any expansion of the text that follows.
  2. The "here doc" contents are served in the stdin with <<.
  3. The option -d '' forces read to slurp the whole content of the "here doc".
  4. The -r option avoids interpretation of backslash quoted characters.
  5. The core command is similar to read var.
  6. And the last detail is IFS='', which will avoid that read remove leading or trailing characters in the default IFS: spacetabnewline.

In ksh, the null value for the -d '' option doesn't work.
As a workaround, if the text has no "carriage return", a -d $'\r' works (if a $'\r' is added to the end of each line, of course).


An added (in comments) requirement is to generate a POSIX compliant solution.

POSIX

Extending the idea to make it run only with POSIX options.
That means mainly no -d for read. That forces a read for each line.
That, in turn forces the need to capture a line at a time.
Then, to build the var a trailing new line must be added (as the read removed it).

#!/bin/sh

nl='
'

read_heredoc() {
    unset var
    while IFS="$nl" read -r line; do
        var="$var$line$nl"
    done <<\HEREDOC

                        _                            _ _
                       | |                          | (_)
  _ __ ___  _   _ _ __ | | __ _  ___ ___  ___  _ __ | |_ _ __   ___
 | '_ ` _ \| | | | '_ \| |/ _` |/ __/ _ \/ _ \| '_ \| | | '_ \ / _ \ 
 | | | | | | |_| | |_) | | (_| | (_|  __/ (_) | | | | | | | | |  __/ 
 |_| |_| |_|\__, | .__/|_|\__,_|\___\___|\___/|_| |_|_|_|_| |_|\___| 
             __/ | | 
            |___/|_| 



HEREDOC

}

read_heredoc
printf '%s' "$var"

That works (and has been tested) in all reasonable shells.

| improve this answer | |
2

Useless use of cat (quote \ and `):

myplaceonline="
                       _                            _ _            
 _ __ ___  _   _ _ __ | | __ _  ___ ___  ___  _ __ | (_)_ __   ___ 
| '_ \` _ \\| | | | '_ \\| |/ _\` |/ __/ _ \\/ _ \\| '_ \\| | | '_ \\ / _ \\
| | | | | | |_| | |_) | | (_| | (_|  __/ (_) | | | | | | | | |  __/
|_| |_| |_|\\__, | .__/|_|\\__,_|\\___\\___|\\___/|_| |_|_|_|_| |_|\\___|
       |___/|_
"

Or without quoting:

myplaceonline="$(figlet myplaceonline)"
| improve this answer | |
  • Latter is not POSIX. – phk Feb 23 '17 at 16:40
  • you are right, now it should be? – ctx Feb 23 '17 at 17:13
  • 1
    @ctx Thanks for the answer, but an unstated requirement of mine is that I want to be able to bring in unquoted HEREDOCs (partly because I'm making this into a public API function for a library called posixcube). The accepted answer is ultimately correct that there is a Bash bug processing nested HEREDOCs in command substitution (see the comment from Michael Homer in the question which links to another question which links to the Bash mailing list where the bug is confirmed by a Bash maintainer). My answer above on Jan 29th is my workaround solution and works well. – Kevin Feb 23 '17 at 18:43
1

To support trailing newlines, I combined the answer from @MichaelHomer and my original solution. I didn't use the suggested workarounds from the link that @EliahKagan noted because the first one uses magical strings and the last two weren't POSIX compliant.

#!/bin/sh

NEWLINE="
"

read_heredoc() {
  read_heredoc_result=""
  while IFS="${NEWLINE}" read -r read_heredoc_line; do
    read_heredoc_result="${read_heredoc_result}${read_heredoc_line}${NEWLINE}"
  done
  eval $1'=${read_heredoc_result}'
}

read_heredoc heredoc_str <<'HEREDOC'

                        _                            _ _
                       | |                          | (_)
  _ __ ___  _   _ _ __ | | __ _  ___ ___  ___  _ __ | |_ _ __   ___
 | '_ ` _ \| | | | '_ \| |/ _` |/ __/ _ \/ _ \| '_ \| | | '_ \ / _ \
 | | | | | | |_| | |_) | | (_| | (_|  __/ (_) | | | | | | | | |  __/
 |_| |_| |_|\__, | .__/|_|\__,_|\___\___|\___/|_| |_|_|_|_| |_|\___|
             __/ | |
            |___/|_|




HEREDOC

echo "${heredoc_str}"
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.