8

When I open up a bash prompt and type:

$ set -o xtrace
$ x='~/someDirectory'
+ x='~/someDirectory'
$ echo $x
+ echo '~/someDirectory'
~/someDirectory

I was hoping that the 5th line above would have went + echo /home/myUsername/someDirectory. Is there a way to do this? In my original Bash script, the variable x is actually being populated from data from an input file, via a loop like this:

while IFS= read line
do
    params=($line)
    echo ${params[0]}
done <"./someInputFile.txt"

Still, I'm getting a similar result, with the echo '~/someDirectory' instead of echo /home/myUsername/someDirectory.

  • In ZSH this is x='~'; print -l ${x} ${~x}. I gave up after digging through the bash manual for a while. – thrig Oct 20 '17 at 19:02
  • @thrig: This is not a bashism, this behavior is POSIX. – WhiteWinterWolf Oct 20 '17 at 21:35
  • Extremely closely related (if not a dupe): unix.stackexchange.com/questions/151850/… – Kusalananda Oct 20 '17 at 21:42
  • @Kusalananda: I'm not sure this is a dupe as the reason here is somewhat different: the OP did not enclose $x between quotes when echoing it. – WhiteWinterWolf Oct 20 '17 at 21:55
  • You don't put tildes in the input file. Problem solved. – chepner Oct 21 '17 at 1:24
8

The POSIX standard imposes word expansion to be done in the following order (emphasize is mine):

  1. Tilde expansion (see Tilde Expansion), parameter expansion (see Parameter Expansion), command substitution (see Command Substitution), and arithmetic expansion (see Arithmetic Expansion) shall be performed, beginning to end. See item 5 in Token Recognition.

  2. Field splitting (see Field Splitting) shall be performed on the portions of the fields generated by step 1, unless IFS is null.

  3. Pathname expansion (see Pathname Expansion) shall be performed, unless set -f is in effect.

  4. Quote removal (see Quote Removal) shall always be performed last.

The only point which interests us here is the first one: as you can see tilde expansion is processed before parameter expansion:

  1. The shell attempts a tilde expansion on echo $x, there is no tilde to be found, so it proceeds.
  2. The shell attempts a parameter expansion on echo $x, $x is found and expanded and the command-line becomes echo ~/someDirectory.
  3. Processing continues, tilde expansion having already been processed the ~ character remains as-is.

By using the quotes while assigning the $x, you were explicitly requesting to not expand the tilde and treat it like a normal character. A thing often missed is that in shell commands you don't have to quote the whole string, so you can make the expansion happen right during the variable assignment:

user@host:~$ set -o xtrace
user@host:~$ x=~/'someDirectory'
+ x=/home/user/someDirectory
user@host:~$ echo $x
+ echo /home/user/someDirectory
/home/user/someDirectory
user@host:~$

And you can also make the expansion occur on the echo command-line as long as it can happen before parameter expansion:

user@host:~$ x='someDirectory'
+ x=someDirectory
user@host:~$ echo ~/$x
+ echo /home/user/someDirectory
/home/user/someDirectory
user@host:~$

If for some reason you really need to affect the tilde to the $x variable without expansion, and be able to expand it at the echo command, you must proceed in two times to force two expansions of the $x variable to occur:

user@host:~$ x='~/someDirectory'
+ x='~/someDirectory'
user@host:~$ echo "$( eval echo $x )"
++ eval echo '~/someDirectory'
+++ echo /home/user/someDirectory
+ echo /home/user/someDirectory
/home/user/someDirectory
user@host:~$ 

However, be aware that depending on the context where you use such structure it may have unwanted side-effect. As a rule of thumb, prefer to avoid using anything requiring eval when you have another way.

If you want to specifically address the tilde issue as opposed to any other kind of expansion, such structure would be safer and portable:

user@host:~$ x='~/someDirectory'
+ x='~/someDirectory'
user@host:~$ case "$x" in "~/"*)
>     x="${HOME}/${x#"~/"}"
> esac
+ case "$x" in
+ x=/home/user/someDirectory
user@host:~$ echo $x
+ echo /home/user/someDirectory
/home/user/someDirectory
user@host:~$ 

This structure explicitly check the presence of a leading ~ and replaces it with the user home dir if it is found.

Following your comment, the x="${HOME}/${x#"~/"}" may indeed be surprising for someone not used in shell programming, but is in fact linked to the same POSIX rule I quoted above.

As imposed by the POSIX standard, quote removal happens last and parameter expansion happens very early. Thus, ${#"~"} is evaluated and expanded far before the evaluation of the outer quotes. In turns, as defined in Parameter expansion rules:

In each case that a value of word is needed (based on the state of parameter, as described below), word shall be subjected to tilde expansion, parameter expansion, command substitution, and arithmetic expansion.

Thus, the right side of the # operator must be properly quoted or escaped to avoid tilde expansion.

So, to state it differently, when the shell interpretor looks at x="${HOME}/${x#"~/"}", he sees:

  1. ${HOME} and ${x#"~/"} must be expanded.
  2. ${HOME} is expanded to the content of the $HOME variable.
  3. ${x#"~/"} triggers a nested expansion: "~/" is parsed but, being quoted, is treated as a literal1. You could have used single quotes here with the same result.
  4. ${x#"~/"} expression itself is now expanded, resulting in the prefix ~/ being removed from the value of $x.
  5. The result of the above is now concatenated: the expansion of ${HOME}, the literal /, the expansion ${x#"~/"}.
  6. The end-result is enclosed in double-quotes, functionally preventing word splitting. I say functionally here because these double quotes are not technically required (see here and there for instance), but as a personal style as soon as an assignments gets anything beyond a=$b I usually find it clearer add double-quotes.

By-the-way, if look more closely to the case syntax, you will see the "~/"* construction which relies on the same concept as x=~/'someDirectory' I explained above (here again, double and simple quotes could be used interchangeably).

Don't worry if these things may seem obscure at the first sight (maybe even at the second or later sights!). In my opinion, parameter expansion are, with subshells, one of the most complex concept to grasp when programming in shell language.

I know that some people may vigorously disagree, but if you would-like to learn shell programming more in depth I encourage you to read the Advanced Bash Scripting Guide: it teaches Bash-scripting, so with a lot of extensions and bells-and-whistles compared to POSIX shell scripting, but I found it well written with loads of practical examples. Once you manage this, it is easy to restrict yourself to POSIX features when you need to, I personally think that entering directly in the POSIX realm is an unnecessary steep learning curve for beginners (compare my POSIX tilde replacement with @m0dular's regex-like Bash equivalent to get an idea of what I mean ;) !).


1: Which leads me into finding a bug in Dash which don't implement tilde expansion here correctly (verifiable using x='~/foo'; echo "${x#~/}"). Parameter expansion is a complex field both for the user and the shell developers themselves!

  • How is the bash shell parsing the line x="${HOME}/${x#"~/"}"? It looks like a concatenation of 3 strings: "${HOME}/${x#", ~/, and "}". Does the shell allow for nested double-quotes when the inner pair of double quotes is inside a ${ } block? – Andrew Oct 21 '17 at 2:35
  • @Andrew: I have complete my answer with additional information hopefully addressing your comment. – WhiteWinterWolf Oct 21 '17 at 8:41
  • Thanks, this is a great answer. I've learned a ton from reading it. Wish I could upvote it more than once:) – Andrew Oct 21 '17 at 14:03
  • @WhiteWinterWolf: still, shell does not see the nested quotes whatever the result is. – avp Nov 22 '18 at 10:15
5

One possible answer:

eval echo "$x"

Since you're reading input from a file, I would not do this.

You could search and replace the ~ with the value of $HOME, like this:

x='~/.config'
x="${x//\~/$HOME}"
echo "$x"

Gives me:

/home/adrian/.config
  • Note that the ${parameter/pattern/string} expansion is a Bash extension and may not be available in other shells. – WhiteWinterWolf Oct 20 '17 at 21:53
  • True. The OP did mention he was using Bash, so I think it's an appropriate answer. – m0dular Oct 20 '17 at 22:06
  • I agree, as long as one is sticking to Bash why not fully take advantage of it (not everybody need portability everywhere), but it's just worth to note it for non-Bash users (several distro now ship with Dash instead of Bash for instance) so affected users are not surprised. – WhiteWinterWolf Oct 20 '17 at 22:13
  • I took the liberty to mention your post in my digression about differences between Bash extensions and POSIX shell scripting, as I think that your Bash single-line regex-like statement compared to my POSIX case structure illustrate well how Bash scripting is more user-friendly specially for beginners. – WhiteWinterWolf Oct 21 '17 at 8:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.