I just deleted everything in my home directory. How? And why are some files still there?

Question

Never thought this would happen to me, but there you go. ¯\_(ツ)_/¯

I ran a build script from a repository inside the wrong directory without looking at the source first. Here's the script Scripts/BuildLocalWheelLinux.sh:

cd ../Dependencies/cpython
mkdir debug
cd debug
../configure --with-pydebug --enable-shared
make
cd ../../..

cd ..
mkdir -p cmake-build-local
cd cmake-build-local
rm -rf *
cmake .. -DMVDIST_ONLY=True -DMVPY_VERSION=0 -DMVDPG_VERSION=local_build
make -j
cd ..

cd Distribution
python3 BuildPythonWheel.py ../cmake-build-local/[redacted]/core.so 0
python3 -m ensurepip
python3 -m pip install --upgrade pip
[more pip install stuff]
python3 -m setup bdist_wheel --plat-name manylinux1_x86_64 --dist-dir ../dist
cd ..
cd Scripts

The dangerous part seems to be

mkdir -p cmake-build-local
cd cmake-build-local
rm -rf *

But thinking about it, it actually seems like it couldn't possibly go wrong.

The way you're supposed to run this script is cd Scripts; ./BuildLocalWheelLinux.sh. When I ran it the first time, it showed an error on the very last line (as I learned afterwards). I was in a hurry, so I though "maybe the docs are outdated, I'll try running from the project root instead. So I ran ./Scripts/BuildLocalWheelLinux.sh. Suddenly, vscodes theme and zoom level changed, my zsh terminal config was reset, terminal fonts were set to default, and I Ctrl+C'd once I realized what was happening.

There are some files remaining, but there's no obvious pattern to them:

$ ls -la
total 216
drwx------ 27 felix   felix   4096 May 12 18:08 .
drwxr-xr-x  3 root    root    4096 Apr 15 16:39 ..
-rw-------  1 felix   felix  12752 Apr 19 11:07 .bash_history
-rw-r--r--  1 felix   felix   3980 Apr 15 13:40 .bashrc
drwxrwxrwx  7 felix   felix   4096 May 12 18:25 .cache
drwx------  8 felix   felix   4096 May 12 18:26 .config
drwx------  3 root    root    4096 Apr 13 21:40 .dbus
drwx------  2 felix   felix   4096 Apr 30 12:18 .docker
drwxr-xr-x  8 felix   felix   4096 Apr 15 13:40 .dotfiles
-rw-------  1 felix   felix   8980 Apr 13 18:10 examples.desktop
-rw-r--r--  1 felix   felix    196 Apr 19 15:19 .gitconfig
-rw-r--r--  1 felix   felix     55 Apr 16 13:56 .gitconfig.old
-rw-r--r--  1 felix   felix   1040 Apr 15 13:40 .gitmodules
drwx------  3 felix   felix   4096 May  6 10:10 .gnupg
-rw-r--r--  1 felix   felix   1848 May  5 14:24 heartbeat.tcl
-rw-------  1 felix   felix   1610 Apr 13 20:36 .ICEauthority
drwxr-xr-x  5 felix   felix   4096 Apr 21 16:39 .ipython
drwxr-xr-x  2 felix   felix   4096 May  4 09:35 .jupyter
-rw-------  1 felix   felix    161 Apr 27 14:23 .lesshst
drwx------  3 felix   felix   4096 May 12 18:08 .local
-rw-r--r--  1 felix   felix    140 Apr 29 17:54 minicom.log
drwx------  5 felix   felix   4096 Apr 13 18:25 .mozilla
drwxr-xr-x  2 felix   felix   4096 Apr 13 18:10 Music
drwxr-xr-x  6 felix   felix   4096 May 12 17:16 Nextcloud
-rw-r--r--  1 felix   felix     52 Apr 16 11:43 .nix-channels
-rw-------  1 felix   felix   1681 Apr 20 10:33 nohup.out
drwx------  3 felix   felix   4096 Apr 15 11:16 .pki
-rw-------  1 felix   felix    946 Apr 16 11:43 .profile
drwxr-xr-x  2 felix   felix   4096 Apr 13 18:10 Public
drwxr-xr-x  2 felix   felix   4096 May 12 18:08 .pylint.d
-rw-------  1 felix   felix   1984 May 12 18:06 .pythonhist
-rw-r--r--  1 felix   felix   2443 Apr 19 13:40 README.md
drwxr-xr-x 13 felix   felix   4096 May 12 18:08 repos
drwxr-xr-x  6 felix   felix   4096 Apr 19 11:08 snap
drwx------  3 felix   felix   4096 May  5 15:33 .ssh
drwxr-xr-x  5 felix   felix   4096 Apr 26 17:39 .stm32cubeide
drwxr-xr-x  5 felix   felix   4096 May  5 15:52 .stm32cubemx
drwxr-xr-x  2 felix   felix   4096 Apr 23 11:44 .stmcube
drwxr-xr-x  2 felix   felix   4096 Apr 13 18:10 Templates
drwxr-xr-x  3 felix   felix   4096 Apr 19 11:57 test
drwxr-xr-x  2 felix   felix   4096 Apr 13 18:10 Videos
-rw-------  1 felix   felix  14313 May 12 10:45 .viminfo
-rw-r--r--  1 felix   felix    816 Apr 15 13:40 .vimrc
drwxr-xr-x  3 felix   felix   4096 Apr 16 12:08 .vscode
-rw-r--r--  1 felix   felix   2321 Apr 19 18:47 weird_bug.txt
-rw-r--r--  1 felix   felix    162 Apr 15 13:40 .xprofile

.config is gone, as well as some standard XDG dirs like Pictures and Desktop, but .bashrc is still there. .nix-channels is still there, but .nix-defexpr was nuked.

So, this leads me to two questions:

1. What went wrong? I'd like to fix this build script and make a PR to prevent this from happening in the future.
2. What order were the files deleted in? Obviously not in alphabetical order, but * expands in alphabetical order, so something else is going on here, it seems.

Answer 1

Ouch. You aren't the first victim.

What went wrong?

Starting in your home directory, e.g. /home/felix, or even in /home/felix/src or /home/felix/Downloads/src.

cd ../Dependencies/cpython

Failed because there is no ../Dependencies.

mkdir debug
cd debug

You're now in the subdirectory debug of the directory you started from.

../configure --with-pydebug --enable-shared
make

Does nothing because there's no ../configure or make.

cd ../../..
cd ..

If you started out no more than three directory levels deep, with cd debug reaching a fourth level, the current directory is now the root directory. If you started out four directory levels deep the current directory is now /home.

mkdir -p cmake-build-local

This fails since you don't have permission to write in / or /home.

cd cmake-build-local

This fails since there is no directory cmake-build-local.

We now get to…

What order were the files deleted in?

rm -rf *

This tries to recursively delete every file in the current directory, which is / or /home. The home directories are enumerated in alphabetical order, but the files underneath are enumerated in the arbitrary order of directory traversal. It's the same order as ls --sort=none (unless rm decides to use a different order for some reason). Note that this order is generally not preserved in backups, and can change when a file is created or removed in the directory.

How to fix the script

First, almost any shell script should have set -e near the top. set -e causes the script to abort if a command fails. (A command fails if its exit status is nonzero.) set -e is not a panacea, because there are circumstances where it doesn't go into effect. But it's the bare minimum you can expect and it would have done the right thing here.

(Also the script should start with a shebang line to indicate which shell to use, e.g. #!/bin/sh or #!/bin/bash. But that wouldn't help with this problem.)

rm -rf *, or variants like rm -rf $foo.* (what if $foo turns out to be empty?), are fragile. Here, instead of

mkdir -p cmake-build-local
cd cmake-build-local
rm -rf *

it would be more robust to just remove and re-create the directory. (This would not preserve the permissions on the directory, but here this is not a concern.)

rm -rf cmake-build-local
mkdir cmake-build-local
cd cmake-build-local

Another way is more robust against deleting the wrong files, but more fragile against missing files to delete: delete only files that are known to have been built, by running make clean which has rm commands for known build targets and for known extensions (e.g. rm *.o is ok).

Answer 2

Tracing your cd calls, assuming we are running the script in ~/Distribution/Scripts and assuming that every cd succeeds:

cd ../Dependencies/cpython

We are now in ~/Distribution/Dependencies/cpython.

mkdir debug
cd debug

We are now in ~/Distribution/Dependencies/cpython/debug.

cd ../../..

We are now in ~/Distribution.

cd ..

We are now in your home directory.

mkdir -p cmake-build-local
cd cmake-build-local

We are now in ~/cmake-build-local. This is where you run rm -rf *.

cd ..

We are now back in your home directory

cd Distribution

We are now in ~/Distribution.

cd ..
cd Scripts

We are now in ~/Scripts (you get an error from this because you are one level further up than what you expect).

---

Now then. You tried running the same script, but from ~/Distribution.

cd ../Dependencies/cpython

This fails. This leaves you still in ~/Distribution.

mkdir debug
cd debug

You are now in ~/Distribution/debug.

cd ../../..

You are now in ~/.. (probably in /home).

cd ..

You are most likely in / now.

cd Distribution
mkdir -p cmake-build-local
cd cmake-build-local

These probably fail due to "no such file or directory" and "permission denied".

rm -rf *

You are still in the / directory, and your rm command would try to delete every single file in the whole filesystem. Permissions only allows for deleting files that are located in directories that you have write permissions to, so you likely only loose files in /tmp and in your home directory.

The arguments listed on the command line of rm would have been processed the order that * expanded them (lexicographical, i.e., bin, boot, cdrom, dev, etc, etc.) Each directory listed would then have been processed recursively, and in "directory order" (unsorted).

---

What you should have done:

#!/bin/sh

topdir=$HOME/Distribution

mkdir -p "$topdir/Dependencies/cpython/debug"
(
    cd "$topdir/Dependencies/cpython/debug" || exit 1

    ../configure --with-pydebug --enable-shared
    make
)

rm -rf "$topdir/cmake-build-local"
mkdir -p "$topdir/cmake-build-local"
(
    cd "$topdir/cmake-build-local" || exit 1

    cmake .. -DMVDIST_ONLY=True -DMVPY_VERSION=0 -DMVDPG_VERSION=local_build
    make -j
)

(
    cd "$topdir" || exit 1

    python3 BuildPythonWheel.py ../cmake-build-local/[redacted]/core.so 0
    python3 -m ensurepip
    python3 -m pip install --upgrade pip
    python3 -m setup bdist_wheel --plat-name manylinux1_x86_64 --dist-dir ../dist
)

The working directory within each separate ( ... ) sub-shell is local to that sub-shell. The initial cd in a sub-shell will not affect the working directory of the "outer" environment. The rest of the code uses absolute pathnames that do not depend on the initial working directory of the user's shell session. Note particularly that the rm command does not blindly expand * but rather deletes a specific directory specified via an absolute path (this would not go horribly wrong if the directory does not exist).

Answer 3

If the script assumes that it is run from inside Scripts, then none of these will do what the script author intended:

cd ../Dependencies/cpython
mkdir debug
cd debug
cd ../../..
cd ..
mkdir -p cmake-build-local
cd cmake-build-local

The first cd will fail, the second cd moves into a folder, then the cd ../../.. and cd .. probably put you somewhere above your home directory (maybe in the /home directory, where you usually don't have rights to make anything, so the mkdir and subsequent cd fails). Then the * probably expanded to your home directory, so rm -rf operated on it, recursing into the contents, which would explain the random order of deletes (directory entries aren't sorted in any particular order).