Why does a 'sudo -i' login shell break a here-doc command string argument?

Question

In the sequence of five commands below, all depend on single-quotes to hand off possible variable substitution to the called bash shell rather than the calling shell. The calling user is xx, but the called shell will be run as user yy. The first command substitutes $HOME with the calling shell's value because the called shell is not a login shell. The second command substitutes the value of $HOME loaded by a login shell, so it is the value belonging to user yy. The third command does not rely on a $HOME value and creates a file in the guessed home directory of user yy.

Why does the fourth command fail? The intention is that it writes the same file, but relying on the $HOME variable belonging to user yy to ensure it actually does end up in her home directory. I don't understand why a login shell breaks the behaviour of a here-doc command passed in as a static single-quoted string. The failure of the fifth command verifies that this problem is not about variable substitution.

xx@host ~ $ sudo -u yy bash -c 'echo HOME=$HOME'
HOME=/home/xx
xx@host ~ $ sudo -iu yy bash -c 'echo HOME=$HOME'
HOME=/home/yy
xx@host ~ $ sudo -u yy bash -c 'cat > /home/yy/test.sh << "EOF"
> script-content
> EOF
> '
xx@host ~ $ sudo -iu yy bash -c 'cat > $HOME/test.sh << "EOF"
> script-content
> EOF
> '
bash: warning: here-document at line 0 delimited by end-of-file (wanted `EOFscript-contentEOF')
xx@host ~ $ sudo -iu yy bash -c 'cat > /home/yy/test.sh << "EOF"
> script-content
> EOF
> '
bash: warning: here-document at line 0 delimited by end-of-file (wanted `EOFscript-contentEOF')

These commands were issued on a Linux Mint 18.3 Cinnamon 64-bit system, which is based on Ubuntu 16.04 (Xenial Xerus).

Update: The here-doc aspect is just clouding the issue. Here's a simplification of the problem:

$ sudo bash -c 'echo 1
> echo 2'
1
2
$ sudo -i bash -c 'echo 1
> echo 2'
1echo 2

Why does the first of those two commands preserve the linebreak and the second does not? sudo is common to both commands, yet seems to be escaping/filtering/interpolating differently depending on nothing but the "-i" option.

Answer 1

The documentation for -i states:

The -i (simulate initial login) option runs the shell specified by the password database entry of the target user as a login shell. This means that login-specific resource files such as .profile or .login will be read by the shell. If a command is specified, it is passed to the shell for execution via the shell's -c option.

That is, it genuinely runs the user's login shell, and then passes whatever command you gave sudo to it using -c - unlike what sudo cmd arg arg usually does without the -i option. Ordinarily, sudo just uses one of the exec* functions directly to start the process itself, with no intermediate shell and all arguments passed through exactly as-is.

With -i, it sets up the environment, runs the user's shell as a login shell, and reconstructs the command you asked to run as an argument to bash -c. In your case, it runs (say, approximately) /bin/bash -c "bash -c ' ... '" (imagine quoting that works).

The problem lies in how sudo turns the command you wrote into something that -c can deal with, explained in the next section. The last section has some possible solutions, and in between is some debugging and verification technique,

---

Why does this happen?

When passing the command to -c, it needs some preprocessing to make it do the right thing, which sudo does in advance of running the shell. For example, if your command is:

sudo -iu yy echo 'three   spaces'

then those spaces need to be escaped in order for the command to mean the same thing (i.e., for the single argument not to be split into two words). What ends up being run is:

/bin/bash -c 'echo three\ \ \ spaces'

Let's start with your simplified command:

sudo bash -c 'echo 1
echo 2'

In this case, sudo changes user, then runs execvp("bash", \["bash", "-c", "echo 1\necho 2"\]) (for an invented array literal syntax).

With -i:

sudo -i bash -c 'echo 1
echo 2'

instead it changes user, then runs execv("/bin/bash", ["-bash", "-c", "bash -c echo\\ 1\\\necho\\ 2"]), where \\ equates to a literal \ and \n is a linebreak. It's escaped the spaces and the newline in your main command by preceding them with backslashes.

That is, there's an outer login shell, which has two arguments: -c and your entire command, reconstructed into a form the shell is expected to understand correctly. Unfortunately, it doesn't. The inner bash command ultimately tries to run:

echo 1\
echo 2

where the first physical line ends with a line continuation (backslash followed by newline), which is deleted entirely. The logical line is then just echo 1echo 2, which doesn't do what you wanted.

There's an argument that this is a flaw in sudo's escaping, given the standard behaviour of backslash-newline pairs. I think it should be safe to leave them unescaped here.

---

The same happens for your command with a here-document. It runs as, roughly:

/bin/bash -c 'bash -c cat\ \<\<\ \"EOF\"\\012script-content\\012EOF\\012'

where \012 represents an actual newline - sudo has inserted a backslash before each of them, just like the spaces. Note the double-escaping on \\012: that's ps's rendition of an actual backslash followed by newline, which I'm using here (see below). What eventually runs is:

bash -c 'cat << "EOF"\
script-content\
EOF\
'

with line continuations \ + newline everywhere, which are just removed. That makes it one long line, with no actual newlines in it, and an invalid heredoc:

bash -c 'cat << "EOF"script-contentEOF'

So that's your problem: the inner bash process gets only one line of command, and so the here-document never gets a chance to end (or start). I have some imperfect solutions at the bottom, but this is the underlying cause.

---

How can you check what's happening?

To get those commands out correctly quoted and validate what was happening I modified my login shell's profile file (.profile, .bash_profile, .zprofile, etc) to say just:

ps awx|grep $$

That shows me the command line of the running shell at the time and gives me an extra couple of lines of output before the warning.

hexdump -C /proc/$$/cmdline will also be helpful on Linux.

---

What can you do about it?

I don't see an obvious and reliable way of getting what you want out of this. You don't want sudo to touch your command at all if possible. One option that will largely work for a simple case is to pipe the commands into the shell, rather than specifying them on the command line:

printf 'cat > ... << ... \n ...' | sudo -iu yy

That requires careful internal escaping still.

Probably better is just to put them into a temporary script file and run them that way:

f=`mktemp`
printf 'command' > "$f"
chmod +r "$f"
sudo -iu yy "$f"
rm "$f"

A made-up filename of your own will work too. Depending on your sudoers settings you might be able to keep a file descriptor open and have it read from that as a file (/dev/fd/3) if you really don't want it on disk, but a real file is going to be easier.

Answer 2

The more general need that brought this to light is to document sequences of commands that can be copy 'n pasted directly to a terminal session. Amongst these are commands that create small files in a manner that is human readable and still documents every step. Creating temporary files or directing the reader to "use an editor" deviates from the goal of what is essentially a 100% playback possibility. – froage 4 mins ago

If that's all you're trying to do, just use:

sudo -u yy tee ~yy/test.sh >/dev/null <<'EOF'
script-content
EOF

Much simpler and you don't need to fiddle about with the quoting problems of sticking the whole thing inside a bash -c '...' construct. Especially if you have any quotes in your script, this will make it a lot easier.

(Note that the heredoc delimiter definition is quoted as in 'EOF' so that any variables you use in the script-content won't be expanded during the creation of test.sh, which would likely not be your intention.)

Also note the use of tilde expansion to get user yy's home directory: ~yy. See LESS='+/^ *Tilde Expansion' man bash for more on this. That's why you don't need sudo -i.