How can I harden bash scripts against causing harm when changed in the future?

Question

So, I deleted my home folder (or, more precisely, all files I had write access to). What happened is that I had

build="build"
...
rm -rf "${build}/"*
...
<do other things with $build>

in a bash script and, after no longer needing $build, removing the declaration and all its usages -- but the rm. Bash happily expands to rm -rf /*. Yea.

I felt stupid, installed the backup, redid the work I lost. Trying to move past the shame.

Now, I wonder: what are techniques to write bash scripts so that such mistakes can't happen, or are at least less likely? For instance, had I written

FileUtils.rm_rf("#{build}/*")

in a Ruby script, the interpreter would have complained about build not being declared, so there the language protects me.

What I have considered in bash, besides corraling rm (which, as many answers in related questions mention, is not unproblematic):

1. rm -rf "./${build}/"*
   That would have killed my current work (a Git repo) but nothing else.
2. A variant/parameterization of rm that requires interaction when acting outside of the current directory. (Could not find any.) Similar effect.

Is that it, or are there other ways to write bash scripts that are "robust" in this sense?

Answer 1

set -u

or

set -o nounset

This would make the current shell treat expansions of unset variables as an error:

$ unset build
$ set -u
$ rm -rf "$build"/*
bash: build: unbound variable

set -u and set -o nounset are POSIX shell options.

An empty value would not trigger an error though.

For that, use

$ rm -rf "${build:?Error, variable is empty or unset}"/*
bash: build: Error, variable is empty or unset

The expansion of ${variable:?word} would expand to the value of variable unless it's empty or unset. If it's empty or unset, the word would be displayed on standard error and the shell would treat the expansion as an error (the command would not be executed, and if running in a non-interactive shell, this would terminate). Leaving the : out would trigger the error only for an unset value, just like under set -u.

${variable:?word} is a POSIX parameter expansion.

Neither of these would cause an interactive shell to terminate unless set -e (or set -o errexit) was also in effect. ${variable:?word} causes scripts to exit if the variable is empty or unset. set -u would cause a script to exit if used together with set -e.

---

As for your second question. There is no way to limit rm to not work outside of the current directory.

The GNU implementation of rm has a --one-file-system option that stops it from recursively delete mounted filesystems, but that's as close as I believe we can get without wrapping the rm call in a function that actually checks the arguments.

---

As a side note: ${build} is exactly equivalent to $build unless the expansion occurs as part of a string where the immediately following character is a valid character in a variable name, such as in "${build}x".

Answer 2

I'm going to suggest normal validation checks using test/[ ]

You would had been safe if you'd written your script as such:

build="build"
...
[ -n "${build}" ] || exit 1
rm -rf "${build}/"*
...

---

The [ -n "${build}" ] checks that "${build}" is a non-zero length string.

The || is the logical OR operator in bash. It causes another command to be run if the first one failed.

In this way, had ${build} been empty/undefined/etc. the script would have exited (with a return code of 1, which is a generic error).

This also would have protected you in case you removed all uses ${build} because [ -n "" ] will always be false.

---

The advantage of using test/[ ] is there are many other more meaningful checks that it can also use.

For example:

[ -f FILE ] True if FILE exists and is a regular file.
[ -d FILE ] True if FILE exists and is a directory.
[ -O FILE ] True if FILE exists and is owned by the effective user ID.

Answer 3

In your specific case, I've reworked 'deletion' in the past to move files/directories instead (assuming /tmp is on the same partition as your directory):

# mktemp -d is also a good, reliable choice
trashdir=/tmp/.trash-$USER/trash-`date`
mkdir -p "$trashdir"
...
mv "${build}"/* "$trashdir"
...

Behind the scenes, this moves the toplevel file/dir references from source to the $trashdir destination directory structures all on the same partition, and doesn't spend time walking the directory structure and freeing up the per-file disk blocks right then and there. This produces much faster cleanup while the system is in active use, in exchange for a slightly slower reboot (/tmp is cleaned on reboots).

Alternatively, a cron entry to periodically clean /tmp/.trash-$USER will keep /tmp from filling up, for processes (e.g., builds) that consume a lot of disk space. If your directory is on a different partition as /tmp, you could create a similar /tmp-like directory on your partition and have cron clean that instead.

Most importantly, though, if you screw up the variables in any way, you can recover the contents before the cleanup happens.

Answer 4

Use bash paramater substitution to assign defaults when variable are uninitialized, eg:

rm -rf ${variable:-"/nonexistent"}

Answer 5

I always try to start my Bash scripts with a line #!/bin/bash -ue.

-e means "fail on first uncathed error";

-u means "fail on first usage of undeclared variable".

Find more details in great article Use the Unofficial Bash Strict Mode (Unless You Looove Debugging). Also author recommends using set -o pipefail; IFS=$'\n\t' but for my purposes this is overkill.

Answer 6

The general advice to check whether your variable is set, is a useful tool to prevent this sort of issue. But in this case there was a simpler solution.

There is most likely no need to glob the contents of the $build directory in order to delete them but not the actual $build directory itself. So if you skipped the extraneous * then an unset value would turn into rm -rf / which by default most rm implementations in the last decade will refuse to perform (unless you disable this protection with GNU rm's --no-preserve-root option).

Skipping the trailing / as well would result in rm '' which would result in the error message:

rm: can't remove '': No such file or directory

This works even if your rm command does not implement protection for /.

Answer 7

You're thinking in programming language terms, but bash is a scripting language :-) So, use a wholly different instruction paradigm for a wholly different language paradigm.

In this case:

rmdir ${build}

Since rmdir will refuse to remove a non-empty directory, you need to remove the members first. You know what those members are, right? They're probably parameters to your script, or derived from a parameter, so:

rm -rf ${build}/${parameter}
rmdir ${build}

Now, if you put some other files or directories in there like a temp file or something that you shouldn't've, the rmdir will throw an error. Handle it properly, then:

rmdir ${build} || build_dir_not_empty "${build}"

This way of thinking has served me well because... yep, been there, done that.