In the sequence of five commands below, all depend on single-quotes to hand off possible variable substitution to the called bash shell rather than the calling shell. The calling user is xx, but the called shell will be run as user yy. The first command substitutes $HOME with the calling shell's value because the called shell is not a login shell. The second command substitutes the value of $HOME loaded by a login shell, so it is the value belonging to user yy. The third command does not rely on a $HOME value and creates a file in the guessed home directory of user yy.

Why does the fourth command fail? The intention is that it writes the same file, but relying on the $HOME variable belonging to user yy to ensure it actually does end up in her home directory. I don't understand why a login shell breaks the behaviour of a here-doc command passed in as a static single-quoted string. The failure of the fifth command verifies that this problem is not about variable substitution.

xx@host ~ $ sudo -u yy bash -c 'echo HOME=$HOME'
xx@host ~ $ sudo -iu yy bash -c 'echo HOME=$HOME'
xx@host ~ $ sudo -u yy bash -c 'cat > /home/yy/test.sh << "EOF"
> script-content
> '
xx@host ~ $ sudo -iu yy bash -c 'cat > $HOME/test.sh << "EOF"
> script-content
> '
bash: warning: here-document at line 0 delimited by end-of-file (wanted `EOFscript-contentEOF')
xx@host ~ $ sudo -iu yy bash -c 'cat > /home/yy/test.sh << "EOF"
> script-content
> '
bash: warning: here-document at line 0 delimited by end-of-file (wanted `EOFscript-contentEOF')

These commands were issued on a Linux Mint 18.3 Cinnamon 64-bit system, which is based on Ubuntu 16.04 (Xenial Xerus).

Update: The here-doc aspect is just clouding the issue. Here's a simplification of the problem:

$ sudo bash -c 'echo 1
> echo 2'
$ sudo -i bash -c 'echo 1
> echo 2'
1echo 2

Why does the first of those two commands preserve the linebreak and the second does not? sudo is common to both commands, yet seems to be escaping/filtering/interpolating differently depending on nothing but the "-i" option.

  • @Tigger: That's not how here-docs work. Quoting the limit string (first EOF) means that parameter substitution is suppressed for the content of the here-doc. The quotes are not part of the limit string. See tldp.org/LDP/abs/html/here-docs.html. – froage Dec 13 '17 at 1:17
  • Another thing clouding this issue is that you're using peculiar formatting. Please use code blocks, not block quotes. Just take the original text copy-pasted from your terminal, and insert four spaces at the beginning of every line. (You can do it in a gVim buffer with :%s/^/ / where there are four spaces between the final two slashes.) – Wildcard Dec 13 '17 at 5:02
  • @Wildcard: I would normally do as you suggest. I recognize it was peculiar formatting, but it was a (possibly misguided) attempt to make it clear what was actually being typed in light of the fact that novice *nix users will not necessarily recognize the '> ' as a secondary prompt. Code blocks don't provide such flexibility, AFAIK. – froage Dec 13 '17 at 5:12
  • Usual practice on this site is to use one of two types of copy-paste: either just the commands you type without including any prompt (primary or secondary) or the output either, or include all prompts, output, etc. But either way put it in code blocks, not a block quote. – Wildcard Dec 13 '17 at 5:20
  • By the way, to get the $HOME value for another user, just use tilde expansion. You don't need sudo -i at all. See LESS='+/^ *Tilde Expansion' man bash and also see my answer below. – Wildcard Dec 13 '17 at 5:22

The documentation for -i states:

The -i (simulate initial login) option runs the shell specified by the password database entry of the target user as a login shell. This means that login-specific resource files such as .profile or .login will be read by the shell. If a command is specified, it is passed to the shell for execution via the shell's -c option.

That is, it genuinely runs the user's login shell, and then passes whatever command you gave sudo to it using -c - unlike what sudo cmd arg arg usually does without the -i option. Ordinarily, sudo just uses one of the exec* functions directly to start the process itself, with no intermediate shell and all arguments passed through exactly as-is.

With -i, it sets up the environment, runs the user's shell as a login shell, and reconstructs the command you asked to run as an argument to bash -c. In your case, it runs (say, approximately) /bin/bash -c "bash -c ' ... '" (imagine quoting that works).

The problem lies in how sudo turns the command you wrote into something that -c can deal with, explained in the next section. The last section has some possible solutions, and in between is some debugging and verification technique,

Why does this happen?

When passing the command to -c, it needs some preprocessing to make it do the right thing, which sudo does in advance of running the shell. For example, if your command is:

sudo -iu yy echo 'three   spaces'

then those spaces need to be escaped in order for the command to mean the same thing (i.e., for the single argument not to be split into two words). What ends up being run is:

/bin/bash -c 'echo three\ \ \ spaces'

Let's start with your simplified command:

sudo bash -c 'echo 1
echo 2'

In this case, sudo changes user, then runs execvp("bash", \["bash", "-c", "echo 1\necho 2"\]) (for an invented array literal syntax).

With -i:

sudo -i bash -c 'echo 1
echo 2'

instead it changes user, then runs execv("/bin/bash", ["-bash", "-c", "bash -c echo\\ 1\\\necho\\ 2"]), where \\ equates to a literal \ and \n is a linebreak. It's escaped the spaces and the newline in your main command by preceding them with backslashes.

That is, there's an outer login shell, which has two arguments: -c and your entire command, reconstructed into a form the shell is expected to understand correctly. Unfortunately, it doesn't. The inner bash command ultimately tries to run:

echo 1\
echo 2

where the first physical line ends with a line continuation (backslash followed by newline), which is deleted entirely. The logical line is then just echo 1echo 2, which doesn't do what you wanted.

There's an argument that this is a flaw in sudo's escaping, given the standard behaviour of backslash-newline pairs. I think it should be safe to leave them unescaped here.

The same happens for your command with a here-document. It runs as, roughly:

/bin/bash -c 'bash -c cat\ \<\<\ \"EOF\"\\012script-content\\012EOF\\012'

where \012 represents an actual newline - sudo has inserted a backslash before each of them, just like the spaces. Note the double-escaping on \\012: that's ps's rendition of an actual backslash followed by newline, which I'm using here (see below). What eventually runs is:

bash -c 'cat << "EOF"\

with line continuations \ + newline everywhere, which are just removed. That makes it one long line, with no actual newlines in it, and an invalid heredoc:

bash -c 'cat << "EOF"script-contentEOF'

So that's your problem: the inner bash process gets only one line of command, and so the here-document never gets a chance to end (or start). I have some imperfect solutions at the bottom, but this is the underlying cause.

How can you check what's happening?

To get those commands out correctly quoted and validate what was happening I modified my login shell's profile file (.profile, .bash_profile, .zprofile, etc) to say just:

ps awx|grep $$

That shows me the command line of the running shell at the time and gives me an extra couple of lines of output before the warning.

hexdump -C /proc/$$/cmdline will also be helpful on Linux.

What can you do about it?

I don't see an obvious and reliable way of getting what you want out of this. You don't want sudo to touch your command at all if possible. One option that will largely work for a simple case is to pipe the commands into the shell, rather than specifying them on the command line:

printf 'cat > ... << ... \n ...' | sudo -iu yy

That requires careful internal escaping still.

Probably better is just to put them into a temporary script file and run them that way:

printf 'command' > "$f"
chmod +r "$f"
sudo -iu yy "$f"
rm "$f"

A made-up filename of your own will work too. Depending on your sudoers settings you might be able to keep a file descriptor open and have it read from that as a file (/dev/fd/3) if you really don't want it on disk, but a real file is going to be easier.

  • That's a good explanation of how a multi-line command is collapsed into a single command line before execution and I knew that was happening, but I don't see why that is any sort of problem. As you have identified though, the \\012 seems to be stripped out. Why? That seems to me to be the real problem. Why does switching from a non-login shell to a login shell cause any characters to be stripped out? – froage Dec 13 '17 at 3:48
  • Because a heredoc starts on one line and ends on a subsequent line; you only have the one line. – Michael Homer Dec 13 '17 at 3:49
  • The login shell difference is because it’s not “non-login shell”, it’s (login) shell or no shell at all. – Michael Homer Dec 13 '17 at 3:50
  • 1
    I've edited to elaborate on where this is all happening: the issue is that sudo escapes everything it isn't sure about, and that includes newlines, but \ + newline actually means something different. Arguably it's a bug in sudo. I don't see a way around it within your command structure. – Michael Homer Dec 13 '17 at 4:39
  • I've updated my question with a simpler distillation of the problem. It certainly does feel like a bug. – froage Dec 13 '17 at 4:57

The more general need that brought this to light is to document sequences of commands that can be copy 'n pasted directly to a terminal session. Amongst these are commands that create small files in a manner that is human readable and still documents every step. Creating temporary files or directing the reader to "use an editor" deviates from the goal of what is essentially a 100% playback possibility. – froage 4 mins ago

If that's all you're trying to do, just use:

sudo -u yy tee ~yy/test.sh >/dev/null <<'EOF'

Much simpler and you don't need to fiddle about with the quoting problems of sticking the whole thing inside a bash -c '...' construct. Especially if you have any quotes in your script, this will make it a lot easier.

(Note that the heredoc delimiter definition is quoted as in 'EOF' so that any variables you use in the script-content won't be expanded during the creation of test.sh, which would likely not be your intention.)

Also note the use of tilde expansion to get user yy's home directory: ~yy. See LESS='+/^ *Tilde Expansion' man bash for more on this. That's why you don't need sudo -i.

  • 2
    You are quoting an informational side comment to Michael in response to his workaround, not answering the question I submitted to Stack Exchange. While I really appreciate the suggestion to use tee instead of cat and will definitely adopt it, I trust you understand why I can't accept this as an answer. – froage Dec 13 '17 at 5:26
  • 1
    You're absolutely right; this isn't an answer to the actual posted question, it just addresses the use case underlying your particular situation. I'm glad it will still help you, though! That's why I wrote it. :) And I agree it wouldn't be appropriate to accept this answer. (I considered just posting it as a comment but it would have been much too long. Maybe it will help someone else work around the bug in sudo -i as well, someday.) – Wildcard Dec 13 '17 at 6:37
  • It just goes to show that deviating from rigid rules/conventions to solve a problem sometimes serves a good purpose ;-) – froage Dec 13 '17 at 8:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.