Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I often end up issuing complex commands over ssh; these commands involve piping to awk or perl one-lines, and as a result contain single quotes and $'s. I have neither been able to figure out a hard and fast rule to do the quoting properly, nor found a good reference for it. For instance, consider the following:

# what I'd run locally:
CMD='pgrep -fl java | grep -i datanode | awk '{print $1}'
# this works with ssh $host "$CMD":
CMD='pgrep -fl java | grep -i datanode | awk '"'"'{print $1}'"'"

(Note the extra quotes in the awk statement.)

But how do I get this to work with, e.g. ssh $host "sudo su user -c '$CMD'"? Is there a general recipe for managing quotes in such scenarios?..

share|improve this question
add comment

4 Answers

up vote 25 down vote accepted

Dealing with multiple levels of quoting (really, multiple levels of parsing/interpretation) can get complicated. It helps to keep a few things in mind:

  • Each “level of quoting” can potentially involve a different language.
  • Quoting rules vary by language.
  • When dealing with more than one or two nested levels, it is usually easiest to work “from the bottom, up” (i.e. innermost to outermost).

Levels of Quoting

Let us look at your example commands.

pgrep -fl java | grep -i datanode | awk '{print $1}'

Your first example command (above) uses four languages: your shell, the regex in pgrep, the regex in grep (which might be different from the regex language in pgrep), and awk. There are two levels of interpretation involved: the shell and one level after the shell for each of the involved commands. There is only one explicit level of quoting (shell quoting into awk).

ssh host …

Next you added a level of ssh on top. This is effectively another shell level: ssh does not interpret the command itself, it hands it to a shell on the remote end (via (e.g.) sh -c …) and that shell interprets the string.

ssh host "sudo su user -c …"

Then you asked about adding another shell level in the middle by using su (via sudo, which does not interpret its command arguments, so we can ignore it). At this point, you have three levels of nesting going on (awk → shell, shell → shell (ssh), shell → shell (su user -c), so I advise using the “bottom, up” approach. I will assume that your shells are Bourne compatible (e.g. sh, ash, dash, ksh, bash, zsh, etc.). Some other kind of shell (fish, rc, etc.) might require different syntax, but the method still applies.

Bottom, Up

  1. Formulate the string you want to represent at the innermost level.
  2. Select a quoting mechanism from the quoting repertoire of the next-highest language.
  3. Quote the desired string according to your selected quoting mechanism.
    • There are often many variations how to apply which quoting mechanism. Doing it by hand is usually a matter of practice and experience. When doing it programatically, it is usually best to pick the easiest to get right (usually the “most literal” (fewest escapes)).
  4. Optionally, use the resulting quoted string with additional code.
  5. If you have not yet reached your desired level of quoting/interpretation, take the resulting quoted string (plus any added code) and use it as the starting string in step 2.

Quoting Semantics Vary

The thing to keep in mind here is that each language (quoting level) may give slightly different semantics (or even drastically different semantics) to the same quoting character.

Most languages have a “literal” quoting mechanism, but they vary in exactly how literal they are. The single quote of Bourne-like shells is actually literal (which means you can not use it to quote a single quote character itself). Other languages (Perl, Ruby) are less literal in that they interpret some backslash sequences inside single quoted regions non-literally (specifically, \\ and \' result in \ and ', but other backslash sequences are actually literal).

You will have to read the documentation for each of your languages to understand its quoting rules and the overall syntax.

Your Example

The innermost level of your example is an awk program.

{print $1}

You are going to embed this in a shell command line:

pgrep -fl java | grep -i datanode | awk …

We need to protect (at a minimum) the space and the $ in the awk program. The obvious choice is to use single quote in the shell around the whole program.

  • '{print $1}'

There are other choices though:

  • {print\ \$1} directly escape the space and $
  • {print' $'1} single quote only the space and $
  • "{print \$1}" double quote the whole and escape the $
  • {print" $"1} double quote only the space and $
    This may be bending the rules a bit (unescaped $ at the end of a double quoted string is literal), but it seems to work in most shells.

If the program used a comma between the open and close curly braces we would also need to quote or escape either the comma or the curly braces to avoid “brace expansion” in some shells.

We pick '{print $1}' and embed it in the rest of the shell “code”:

pgrep -fl java | grep -i datanode | awk '{print $1}'

Next, you wanted to run this via su and sudo.

sudo su user -c …

su user -c … is just like some-shell -c … (except running under some other UID), so su just adds another shell level. sudo does not interpret its arguments, so it does not add any quoting levels.

We need another shell level for our command string. We can pick single quoting again, but we have to give special handling to the existing single quotes. The usual way looks like this:

'pgrep -fl java | grep -i datanode | awk '\''{print $1}'\'

There are four strings here that the shell will interpret and concatenate: the first single quoted string (pgrep … awk), an escaped single quote, the single-quoted awk program, another escaped single quote.

There are, of course many alternatives:

  • pgrep\ -fl\ java\ \|\ grep\ -i\ datanode\ \|\ awk\ \'{print\ \$1} escape everything important
  • pgrep\ -fl\ java\|grep\ -i\ datanode\|awk\ \'{print\$1} the same, but without superfluous whitespace (even in the awk program!)
  • "pgrep -fl java | grep -i datanode | awk '{print \$1}'" double quote the whole thing, escape the $
  • 'pgrep -fl java | grep -i datanode | awk '"'"'{print \$1}'"'" your variation; a bit longer than the usual way due to using double quotes (two characters) instead of escapes (one character)

Using different quoting in the first level allows for other variations at this level:

  • 'pgrep -fl java | grep -i datanode | awk "{print \$1}"'
  • 'pgrep -fl java | grep -i datanode | awk {print\ \$1}'

Embedding the first variation in the sudo/*su* command line give this:

sudo su user -c 'pgrep -fl java | grep -i datanode | awk '\''{print $1}'\'

You could use the same string in any other single shell level contexts (e.g. ssh host …).

Next, you added a level of ssh on top. This is effectively another shell level: ssh does not interpret the command itself, but it hands it to a shell on the remote end (via (e.g.) sh -c …) and that shell interprets the string.

ssh host …

The process is the same: take the string, pick a quoting method, use it, embed it.

Using single quotes again:

'sudo su user -c '\''pgrep -fl java | grep -i datanode | awk '\'\\\'\''{print $1}'\'\\\'

Now there are eleven strings that are interpreted and concatenated: 'sudo su user -c ', escaped single quote, 'pgrep … awk ', escaped single quote, escaped backslash, two escaped single quotes, the single quoted awk program, an escaped single quote, an escaped backslash, and a final escaped single quote.

The final form looks like this:

ssh host 'sudo su user -c '\''pgrep -fl java | grep -i datanode | awk '\'\\\'\''{print $1}'\'\\\'

This is a bit unwieldy to type by hand, but the literal nature of the shell’s single quoting makes it easy to automate a slight variation:

#!/bin/sh

sq() { # single quote for Bourne shell evaluation
    # Change ' to '\'' and wrap in single quotes.
    # If original starts/ends with a single quote, creates useless
    # (but harmless) '' at beginning/end of result.
    printf '%s\n' "$*" | sed -e "s/'/'\\\\''/g" -e 1s/^/\'/ -e \$s/\$/\'/
}

# Some shells (ksh, bash, zsh) can do something similar with %q, but
# the result may not be compatible with other shells (ksh uses $'...',
# but dash does not recognize it).
#
# sq() { printf %q "$*"; }

ap='{print $1}'
s1="pgrep -fl java | grep -i datanode | awk $(sq "$ap")"
s2="sudo su user -c $(sq "$s1")"

ssh host "$(sq "$s2")"
share|improve this answer
4  
Great explanation! –  Gilles Dec 12 '10 at 15:52
add comment

See Chris Johnsen's answer for a clear, in-depth explanation with a general solution. I'm going to give a few extra tips that help in some common circumstances.

Single quotes escape everything but a single quote. So if you know the value of a variable doesn't include any single quote, you can interpolate it safely between single quotes in a shell script.

su -c "grep '$pattern' /root/file"  # assuming there is no ' in $pattern

If your local shell is ksh93 or zsh, you can cope with single quotes in the variable by rewriting them to '\''. (Although bash also has the ${foo//pattern/replacement} construct, its handling of single quotes doesn't make sense to me.)

su -c "grep '${pattern//'/'\''}' /root/file"  # if the outer shell is zsh
su -c "grep '${pattern//\'/\'\\\'\'}' /root/file"  # if the outer shell is ksh93

Another tip to avoid having to deal with nested quoting is to pass strings through environment variables as much as possible. Ssh and sudo tend to drop most environment variables, but they're often configured to let LC_* through, because these are normally very important for usability (they contain locale information) and are rarely considered security sensitive.

LC_CMD='what you would use locally' ssh $host 'sudo su user -c "$LC_CMD"'

Here, since LC_CMD contains a shell snippet, it must be provided literally to the innermost shell. Therefore the variable is expanded by the shell immediately above. The innermost-but-one shell sees "$LC_CMD", and the innermost shell sees the commands.

A similar method is useful to pass data to a text processing utility. If you use shell interpolation, the utility will treat the value of the variable as a command, e.g. sed "s/$pattern/$replacement/" won't work if the variables contain /. So use awk (not sed), and either its -v option or the ENVIRON array to pass data from the shell (if you go through ENVIRON, remember to export the variables).

awk -vpattern="$pattern" replacement="$replacement" '{gsub(pattern,replacement); print}'
share|improve this answer
add comment

How about using more double quotes?

Then your ssh $host $CMD should work just fine with this one:

CMD="pgrep -fl java | grep -i datanode | awk '{print $1}'"

Now to the more complex one, the ssh $host "sudo su user -c \"$CMD\"". I guess all you have to do is escape sensitive characters in CMD: $, \ and ". So I'd try and see if this works: echo $CMD | sed -e 's/[$\\"]/\\\1/g'.

If that looks OK, wrap the echo+sed into a shell function, and you are good to go with ssh $host "sudo su user -c \"$(escape_my_var $CMD)\"".

share|improve this answer
add comment

As Chris Johnson describes very well, you have a few levels of quoted indirection here; you instruct your local shell to instruct the remote shell via ssh that it should instruct sudo to instruct su to instruct the remote shell to run your pipeline pgrep -fl java | grep -i datanode | awk '{print $1}' as user. That kind of command necessitates a lot of tedious \'"quote quoting"\'.

If you'll take my advice, you'll forego all of the nonsense and do:

% ssh $host <<REM=LOC_EXPANSION <<'REMOTE_CMD' |
> localhost_data='$(commands run on localhost at runtime)' #quotes don't affect expansion
> more_localhost_data="$(values set at heredoc expansion)" #remote shell will receive m_h_d="result"
> REM=LOC_EXPANSION
> commands typed exactly as if located at 
> the remote terminal including variable 
> "${{more_,}localhost_data}" operations
> 'quotes and' \all possibly even 
> a\wk <<'REMOTELY_INVOKED_HEREDOC' |
> {as is often useful with $awk
> so long as the terminator for}
> REMOTELY_INVOKED_HEREDOC
> differs from that of REM=LOC_EXPANSION and
> REMOTE_CMD
> and here you can | pipeline operate on |\
> any output | received from | ssh as |\
> run above | in your local | terminal |\
> however | tee > ./you_wish.result
<desired output>

FOR MORE:

Check my (perhaps too-long-winded) answer to Piping paths with different types of quotes for slash substitution in which I discuss some of the theory behind why that works.

-Mike

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.