So, I deleted my home folder (or, more precisely, all files I had write access to). What happened is that I had

build="build"
...
rm -rf "${build}/"*
...
<do other things with $build>

in a bash script and, after no longer needing $build, removing the declaration and all its usages -- but the rm. Bash happily expands to rm -rf /*. Yea.

I felt stupid, installed the backup, redid the work I lost. Trying to move past the shame.

Now, I wonder: what are techniques to write bash scripts so that such mistakes can't happen, or are at least less likely? For instance, had I written

FileUtils.rm_rf("#{build}/*")

in a Ruby script, the interpreter would have complained about build not being declared, so there the language protects me.

What I have considered in bash, besides corraling rm (which, as many answers in related questions mention, is not unproblematic):

  1. rm -rf "./${build}/"*
    That would have killed my current work (a Git repo) but nothing else.
  2. A variant/parameterization of rm that requires interaction when acting outside of the current directory. (Could not find any.) Similar effect.

Is that it, or are there other ways to write bash scripts that are "robust" in this sense?

  • 3
    There is no reason to rm -rf "${build}/* no matter where the quote marks go. rm -rf "${build} will do the same thing because of the f. – Monty Harder Jul 11 at 14:36
  • 1
    Static checking with shellcheck.net is a very solid place to start. There's editor integration available, so you could get a warning from your tools as soon as you remove the definition of something that's still used. – Charles Duffy Jul 11 at 15:15
  • @CharlesDuffy To add to my shame, I wrote that script in an IDEA-style IDE with BashSupport installed, which does warn in that case. So yes, valid point, but I really needed a hard cancel. – Raphael Jul 11 at 15:18
  • 2
    Gotcha. Be sure to note the caveats in BashFAQ #112. set -u isn't nearly as frowned on as set -e is, but it still does have its gotchas. – Charles Duffy Jul 11 at 15:20
  • 2
    joke answer: Just put #! /usr/bin/env ruby at the top of every shell script and forget about bash ;) – Pod Jul 12 at 15:25
up vote 73 down vote accepted
set -u

or

set -o nounset

This would make the current shell treat expansions of unset variables as an error:

$ unset build
$ set -u
$ rm -rf "$build"/*
bash: build: unbound variable

set -u and set -o nounset are POSIX shell options.

An empty value would not trigger an error though.

For that, use

$ rm -rf "${build:?Error, variable is empty or unset}"/*
bash: build: Error, variable is empty or unset

The expansion of ${variable:?word} would expand to the value of variable unless it's empty or unset. If it's empty or unset, the word would be displayed on standard error and the shell would treat the expansion as an error (the command would not be executed, and if running in a non-interactive shell, this would terminate). Leaving the : out would trigger the error only for an unset value, just like under set -u.

${variable:?word} is a POSIX parameter expansion.

Neither of these would cause an interactive shell to terminate unless set -e (or set -o errexit) was also in effect. ${variable:?word} causes scripts to exit if the variable is empty or unset. set -u would cause a script to exit if used together with set -e.


As for your second question. There is no way to limit rm to not work outside of the current directory.

The GNU implementation of rm has a --one-file-system option that stops it from recursively delete mounted filesystems, but that's as close as I believe we can get without wrapping the rm call in a function that actually checks the arguments.


As a side note: ${build} is exactly equivalent to $build unless the expansion occurs as part of a string where the immediately following character is a valid character in a variable name, such as in "${build}x".

  • Very good, thanks! 1) Is it ${build:?msg} or ${build?msg}? 2) In the context of something like build scripts, I think using a different tool than rm for safer deletion would be fine: we know we don't want to work outside of the current directory, so we make that explicit by using a restricted command. There's no need to make rm safer in general. – Raphael Jul 11 at 13:31
  • 1
    @Raphael Sorry, should be with :. I'll correct that typo now and I'll mention its significance later when I'm back at my computer. – Kusalananda Jul 11 at 13:34
  • 2
    @Raphael Ok, I've added a short sentence about what happens without the : (it would trigger the error only for unset variables). I dare not try to write a tool that would cope with deleting files under a specific path exclusively, under all circumstances. Parsing paths and caring/not caring about symbolic links etc. is a bit too fiddly for me at the moment. – Kusalananda Jul 11 at 14:17
  • 1
    Thanks, the explanation helps! Regarding the "local rm": I was mostly musing, maybe fishing for a "sure, that's <toolname>" -- but certainly not trying to help-vampire you into writing it! O.O All good, you've helped enough! :) – Raphael Jul 11 at 14:25
  • 2
    @MateuszKonieczny I don't think I will, sorry. I believe that safety measures like these should be used when needed. As with everything that makes an environment safe, it will eventually make people more and more careless and dependent on the safety measures. It's better to know what each and every safety measure does and then apply them selectively as needed. – Kusalananda Jul 13 at 6:52

I'm going to suggest normal validation checks using test/[ ]

You would had been safe if you'd written your script as such:

build="build"
...
[ -n "${build}" ] || exit 1
rm -rf "${build}/"*
...

The [ -n "${build}" ] checks that "${build}" is a non-zero length string.

The || is the logical OR operator in bash. It causes another command to be run if the first one failed.

In this way, had ${build} been empty/undefined/etc. the script would have exited (with a return code of 1, which is a generic error).

This also would have protected you in case you removed all uses ${build} because [ -n "" ] will always be false.


The advantage of using test/[ ] is there are many other more meaningful checks that it can also use.

For example:

[ -f FILE ] True if FILE exists and is a regular file.
[ -d FILE ] True if FILE exists and is a directory.
[ -O FILE ] True if FILE exists and is owned by the effective user ID.
  • Fair, but a tad unwieldy. Am I missing something, or does it do pretty much the same as ${variable:?word} @Kusalananda proposes? – Raphael Jul 11 at 18:38
  • @Raphael it works similarly in the case of "does the string have a value" but test (i.e. [) has many other checks that are relevant, such as -d (the argument is a directory), -f (the argument is a file), -O (the file is owned by the current user) and so-on. – Centimane Jul 11 at 18:43
  • 1
    @Raphael also, validation should be a normal part of any code/scripts. Also note, if you removed all instances of build, would you not have also removed ${build:?word} along with it? The ${variable:?word} format doesn't protect you if you remove all instances of the variable. – Centimane Jul 11 at 18:50
  • 1
    1) I think there's a difference between making sure assumptions hold (files exist, etc) and checking if variables are set (imho job of a compiler/interpreter). If I have to do the latter by hand, there better be ultra-snazzy syntax for it -- which a full-blown if isn't. 2) "would you not have also removed ${build:?word} along with it" -- the scenario is that I missed a usage of the variable. Using ${v:?w} would have protected me from damage. Had I removed all usages, even plain access wouldn't have been harmful, obviously. – Raphael Jul 12 at 7:29
  • That all said, your answer is a fair and helful response to the general titular question: making sure assumptions hold is important for scripts that stay around. The specific issue in the question body is, imho, better answered by Kusalananda. Thanks! – Raphael Jul 12 at 7:31

In your specific case, I've reworked 'deletion' in the past to move files/directories instead (assuming /tmp is on the same partition as your directory):

# mktemp -d is also a good, reliable choice
trashdir=/tmp/.trash-$USER/trash-`date`
mkdir -p "$trashdir"
...
mv "${build}"/* "$trashdir"
...

Behind the scenes, this moves the toplevel file/dir references from source to the $trashdir destination directory structures all on the same partition, and doesn't spend time walking the directory structure and freeing up the per-file disk blocks right then and there. This produces much faster cleanup while the system is in active use, in exchange for a slightly slower reboot (/tmp is cleaned on reboots).

Alternatively, a cron entry to periodically clean /tmp/.trash-$USER will keep /tmp from filling up, for processes (e.g., builds) that consume a lot of disk space. If your directory is on a different partition as /tmp, you could create a similar /tmp-like directory on your partition and have cron clean that instead.

Most importantly, though, if you screw up the variables in any way, you can recover the contents before the cleanup happens.

  • 2
    "This produces much faster cleanup while the system is in active use" -- does it? I'd have thought both are about changing the affected inodes only. It certainly isn't faster if /tmp is on another partition (I think it always is for me, at least for scripts that run in user space); then, the trash folder needs to be changed (and won't profit from OS-handling of /tmp). – Raphael Jul 12 at 10:02
  • 1
    You could use mktemp -d to get that temporary directory without treading on another process's toes (and to correctly honour $TMPDIR). – Toby Speight Jul 12 at 15:03
  • Added your accurate and helpful notes from both of you, thanks. I think I originally posted this too quickly without considering the points you brought up. – user117529 Jul 12 at 18:54

Use bash paramater substitution to assign defaults when variable are uninitialized, eg:

rm -rf ${variable:-"/nonexistent"}

The general advice to check whether your variable is set, is a useful tool to prevent this sort of issue. But in this case there was a simpler solution.

There is most likely no need to glob the contents of the $build directory in order to delete them but not the actual $build directory itself. So if you skipped the extraneous * then an unset value would turn into rm -rf / which by default most rm implementations in the last decade will refuse to perform (unless you disable this protection with GNU rm's --no-preserve-root option).

Skipping the trailing / as well would result in rm '' which would result in the error message:

rm: can't remove '': No such file or directory

This works even if your rm command does not implement protection for /.

I always try to start my Bash scripts with a line #!/bin/bash -ue.

-e means "fail on first uncathed error";

-u means "fail on first usage of undeclared variable".

Find more details in great article Use the Unofficial Bash Strict Mode (Unless You Looove Debugging). Also author recommends using set -o pipefail; IFS=$'\n\t' but for my purposes this is overkill.

You're thinking in programming language terms, but bash is a scripting language :-) So, use a wholly different instruction paradigm for a wholly different language paradigm.

In this case:

rmdir ${build}

Since rmdir will refuse to remove a non-empty directory, you need to remove the members first. You know what those members are, right? They're probably parameters to your script, or derived from a parameter, so:

rm -rf ${build}/${parameter}
rmdir ${build}

Now, if you put some other files or directories in there like a temp file or something that you shouldn't've, the rmdir will throw an error. Handle it properly, then:

rmdir ${build} || build_dir_not_empty "${build}"

This way of thinking has served me well because... yep, been there, done that.

  • 6
    Thanks for your effort, but this is completely off the mark. The assumption "you know what those members are" is incorrect; think of compiler output. make clean will use some wildcards (unless a very diligent compiler creates a list of every file it creates). Also, it seems to me that having rm -rf ${build}/${parameter} only moves the issue downward a bit. – Raphael Jul 11 at 22:13
  • Hm. Look how that reads. "Thank you, but this is for something I didn't disclose in the original question so I'm going to not only reject your answer but downvote it, despite it being more applicable to the general case." Wow. – Rich Jul 17 at 23:29
  • "Let me make additional assumptions beyond what you wrote in the question and then write a specialized answer." *shrug* (FYI, not that it's any of your business: I didn't downvote. Arguably I should have, because the answer was not useful (to me, at least).) – Raphael Jul 18 at 12:44
  • Hm. I made no assumptions, and wrote a general answer. There's nothing at all about make in the question. Writing an answer that's only applicable to make would be a very specific and surprising assumption. My answer works for cases other than make, other than software development, other than your specific novice problem that you were having by deleting your stuff. – Rich Jul 23 at 15:25
  • 1
    Kusalananda taught me how to fish. You came along and said, "Since you live in the plains, why don't you eat beef instead?" – Raphael Jul 25 at 5:17

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.