11

In Bash, when specifying command line arguments to a command, what characters are required to be escaped?

Are they limited to the metacharacters of Bash: space, tab, |, &, ;, (, ), <, and >?

  • Don't forget (possible) filename globbing with * and ? – Jeff Schaller Mar 20 '16 at 3:41
  • Thanks. Could you exhaustively list the kinds of characters which need to be escaped in cmd line args? – Tim Mar 20 '16 at 3:48
  • The list is good to have, but the most important thing to understand about quoting, is: Everything between single quotes is passed literally and without word splitting. No exceptions. (This means there is no way whatsoever to embed a single quote within single quotes, by the way, but that's easy to work around.) – Wildcard Mar 20 '16 at 8:19
20

The following characters have special meaning to the shell itself in some contexts and may need to be escaped in arguments:

Some of those characters are used for more things and in more places than the one I linked.


There are a few corner cases that are explicitly optional:


Escaping a newline requires quoting — backslashes won't do the job. Any other characters listed in IFS will need similar handling. You don't need to escape ] or }, but you do need to escape ) because it's an operator.

Some of these characters have tighter limits on when they truly need escaping than others. For example, a#b is ok, but a #b is a comment, while > would need escaping in both contexts. It doesn't hurt to escape them all conservatively anyway, and it's easier than remembering the fine distinctions.

If your command name itself is a shell keyword (if, for, do) then you'll need to escape or quote it too. The only interesting one of those is in, because it's not obvious that it's always a keyword. You don't need to do that for keywords used in arguments, only when you've (foolishly!) named a command after one of them. Shell operators ((, &, etc) always need quoting wherever they are.


1Stéphane has noted that any other single-byte blank character from your locale also needs escaping. In most common, sensible locales, at least those based on C or UTF-8, it's only the whitespace characters above. In some ISO-8859-1 locales, U+00A0 no-break space is considered blank, including Solaris, the BSDs, and OS X (I think incorrectly). If you're dealing with an arbitrary unknown locale, it could include just about anything, including letters, so good luck.

Conceivably, a single byte considered blank could appear within a multi-byte character that wasn't blank, and you'd have no way to escape that other than putting the whole thing in quotes. This isn't a theoretical concern: in an ISO-8859-1 locale from above, that A0 byte which is considered a blank can appear within multibyte characters like UTF-8 encoded "à" (C3 A0). To handle those characters safely you would need to quote them "à". This behaviour depends on the locale configuration in the environment running the script, not the one where you wrote it.

I think this behaviour is broken multiple ways, but we have to play the hand we're dealt. If you're working with any non-self-synchronising multibyte character set, the safest thing would be to quote everything. If you're in UTF-8 or C, you're safe (for the moment).

  • Other blanks in your locale would need escaping as well (except currently the multi-byte one because of a bug) – Stéphane Chazelas Mar 20 '16 at 7:25
  • You only need to escape ! when csh history expansion is enabled, typically not in scripts. [ ! -f a ] or find . ! -name... are fine. That's covered by your tighter limits section but maybe worth mentioning explicitly. – Stéphane Chazelas Mar 20 '16 at 7:30
  • Note that there are contexts where other characters need quoting like: hash[foo"]"]=, ${var-foo"}"}, [[ "!" = b ]], [[ a = "]]" ]], the regexp operators for [[ x =~ ".+[" ]]. Other keywords than { (if, while, for...) would need to be quoted so they're not recognised as such... – Stéphane Chazelas Mar 20 '16 at 7:48
  • To the extent that those are command-line arguments at all, the interpretation is up to the command in question (just like ]), so I'm not listing them. I don't think any keyword needs quoting in argument position. – Michael Homer Mar 20 '16 at 8:02
  • 2
    Quoting builtins, dashes, or % doesn't do anything. – Michael Homer Mar 20 '16 at 22:06
2

In GNU Parallel this is tested and used extensively:

$a =~ s/[\002-\011\013-\032\\\#\?\`\(\)\{\}\[\]\^\*\<\=\>\~\|\; \"\!\$\&\'\202-\377]/\\$&/go;
# quote newline as '\n'                                                                                                         
$a =~ s/[\n]/'\n'/go;

It is tested in bash,dash,ash,ksh,zsh, and fish. Some of the characters do not need quoting in some (versions) of the shells, but the above works in all tested shells.

If you simply want a string quoted, you can pipe it into parallel --shellquote:

printf "&*\t*!" | parallel --shellquote
  • How have I not heard of parallel before... – Tom H Feb 26 '18 at 0:21
  • @TomH It will be appreciated if you can spend 5 minutes thinking of how we could have reached you. – Ole Tange Feb 26 '18 at 8:45
  • I think it's a progression problem. most people don't need or understand parallel until they have progressed through some complexity stages. By which time they have come across xargs, nohup and stuff like that. Also I don't see many people using parallel to solve problems in stack exchange or when I google for solutions to bash problems – Tom H Feb 27 '18 at 2:49
1

For lightweight escaping solution in Perl, I'm following the principle of single quotes. A Bash-string in single quotes can have any character, except the single quote itself.

My code:

my $bash_reserved_characters_re = qr([ !"#$&'()*;<>?\[\\`{|~\t\n]);

while(<>) {
    if (/$bash_reserved_characters_re/) {
        my $quoted = s/'/'"'"'/gr;
        print "'$quoted'";
    } else {
        print $_;
    }
}

Example run 1:

$ echo -n "abc" | perl escape_bash_special_chars.pl
abc

Example run 2:

echo "abc" | perl escape_bash_special_chars.pl
'abc
'

Example run 3:

echo -n 'ab^c' | perl escape_bash_special_chars.pl
ab^c

Example run 4:

echo -n 'ab~c' | perl escape_bash_special_chars.pl
'ab~c'

Example run 5:

echo -n "ab'c" | perl escape_bash_special_chars.pl
'ab'"'"'c'

echo 'ab'"'"'c'
ab'c
  • I don't think this is actually an answer the question of which characters need quoting. – ilkkachu Jan 22 '18 at 12:54
  • Yes, valid point that. My view is that most people will land on this page, because they have a problem to solve. Not because this makes an interesting academic debate. That's why I'd like to offer solutions and discuss the merits of them, even while being slightly off-topic. – Jari Turkia Jan 23 '18 at 10:49
  • Well, of course this does answer the question, in a way, since you've listed a set of characters as part of the Perl script. But to write a tool for quoting, you do need that "academic" discussion to known which characters to include. You've needed some information to come up with the list. But this answer gives only your word that the list is enough, no sources or rationale whatsoever. Without any further information, one might as well quote anything that doesn't fit e.g. [a-zA-Z0-9_] (those being the usual "word characters", the ones valid in identifiers in e.g. C (minus $ of course)) – ilkkachu Jan 23 '18 at 11:24
  • My code is just an implementation of Michael Homer's answer. I didn't intent to bring any more information, than what he did. – Jari Turkia Jan 23 '18 at 11:43

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.