2

Let's say I have the following structure :

$ mkdir d1
$ mkdir d1/d2
$ touch d1/f1
$ touch d1/d2/f2

$ chmod u-w d1/d2

If I try to remove d1, I can't because I don't have write permissions on d1/d2. But it still deletes d1/f1 :

$ rm -rf d1
rm: cannot remove 'd1/d2/f2': Permission denied

$ ls d1
d2          # f1 has been deleted

Is there a way to implement an atomic rm tool? E.g., if it can not delete everything, then it deletes nothing at all.

11
  • The words "atomic" and "transactional" come to mind.
    – muru
    Aug 31 at 15:16
  • Exactly. Question is : is there something like that already implemented in Linux ?
    – ChennyStar
    Aug 31 at 15:19
  • 1
    Not on a usual filesystem, not properly. There's no way to lock the directory tree so that another process can't change the permissions while rm is going through the tree to delete the files. Of course if you know no other process will come mess you up, you could scan through the tree to check the permissions beforehand. (something like find . -type d \! -perm -0300 would check for directories where the user's w or x bits aren't set.)
    – ilkkachu
    Aug 31 at 15:30
  • Thanks for the idea. So, basically the solution is to write my own delete script, checking all permissions before deleting (or not) the folder.
    – ChennyStar
    Aug 31 at 15:50
  • We are starting to discuss many alternatives with various pros and cons. However, we have not agreed on what we are trying to achieve. What is the use-case?, what is the success criteria?, what is the problem that we are trying to solve? Edit question to make this clear. Sep 1 at 11:04
2

I would write this program in such a way that it creates a hard-linked backup of the tree. Then if the operation fails, it restores the already deleted files from the backup. If the operation succeeds, it removes the hard-linked backup.

Of course, this is only logically atomic, not in the sense of "we can pull the power cord at any time" atomic; though that can be arranged too, with extra logic, and some hook that is run on boot-up.

Making two passes (one pass to check permissions, and the other pass to delete) is tricky. The logic has to be exhaustive to check all permissions, including extended attributes. For instance, if we do sudo chattr +i file , then file becomes unremovable, even though the regular Unix permissions look good: the directory is writable. The best litmus test for "can we delete this file" is to actually try it.

Here is a somewhat tested prototype of the concept, as a proof of concept, using rsync for the hard-link-based backup, restore. This script is called atomic-rm.sh:

#!/bin/sh

set -eu

if [ $# -ne 1 ] ; then
  echo "specify directory to remove"
  exit 1
fi

ar_src=$(realpath "$1")
ar_tmp=$(mktemp -d "$(dirname "$ar_src")/tmp-XXXXXX")

if [ $? -ne 0 ] ; then
  echo "unable to create temporary directory"
  exit 1
fi

cleanup()
{
  find "$ar_tmp" -type d -print0 | xargs -0 chmod +w

  if ! rm -rf "$ar_tmp" ; then
    echo "removal of temporary directory $ar_tmp failed"
    exit 2 # 2 indicates dirty failure
  fi
}

trap cleanup EXIT

if ! rsync -ar --link-dest="$ar_src" "$ar_src"/ "$ar_tmp"/ ; then
  echo "unable to create hard-linked backup of $ar_src in $ar_tmp"
  exit 1
fi

if ! rm -rf "$ar_src" ; then
  if ! rsync -ar --link-dest="$ar_tmp" "$ar_tmp"/ "$ar_src"/ ; then
    echo "removal of $ar_src failed; unfortunately, so did the rollback"
    exit 2 # 2 indicates dirty failure
  fi
  exit 1
fi

exit 0

With some additional effort, it could deposit some information somewhere which a boot-time recovery script could use to clean up loose temporary directories. The temporary directory is created as a sibling of the to-be-deleted directory in order to ensure that they are on the same filesystem; we cannot use /tmp.

If the removal fails and rollback is performed, it will not absolutely restore the exact state of the tree, because directories were messed with, and so their modification time stamps are altered.

Note that in cleanup, we have to make a find pass over the directories to make them writable. The reason is that if a removal fails because a directory isn't writable, it will likewise fail the same way in the back-up copy, because rsync will have replicated those directory permissions.

2
  • If the tree is not too big (and disk space is not an issue), an alternative could be to copy the tree before trying to delete it (see my answer below, inspired from your script). Advantage : the original tree remains untouched if it can not be deleted.
    – ChennyStar
    Sep 1 at 8:36
  • See also pax -rwlpe for a POSIX way to make a hardlinked copy of a directory. Sep 1 at 12:42
1

Option 1

One approach is to move the tree to a trash directory, then use rm to garbage-collect. The files are removed using mv, then deletion is triggered by rm (rm does not delete, it only removed directory entries. Deletion happens by a reference counting garbage-collector).

E.G.

#!/bin/bash -e
directory_tree_to_remove="$1"
#only works if trash directory is on same file-system (no checks done),
#  if not a very expensive copy will be done, followed by an `rm`
#  :todo: add checks
#uses gnu `mv`: uses safety feature
#does not do full input error checking

trash_dir="…/trash"

#Atomicly remove tree or file, form its current location
mv -t "$trash_dir" "$directory_tree_to_remove"

#Garbage collect 
rm -rf "${trash_dir}/${directory_tree_to_remove}" || echo "error: could not garbage collect. There may be garbage left in "$trash_dir"

Option 2

There may be transactional file-systems, that are suited to what you are doing.

5
  • sorry, but I don't understand your option 1
    – ChennyStar
    Sep 1 at 8:40
  • as for option 2 : I believe the word "transactional" in a FS-context has a different meaning (check docs.oracle.com/cd/E19253-01/819-5461/zfsover-2 : "ZFS is a transactional file system, which means that the file system state is always consistent on disk.").
    – ChennyStar
    Sep 1 at 9:02
  • @ChennyStar it is the same meaning, but used in a different context. A transactional/journaling FS ensures that atomic operations are atomic (even if there is a power failure, and some other situations). They don't make rm -r atomic, because the kernel does not consider it to be atomic (there are multiple sys-calls). There may be other types of transactional-FS, that allow atomic recursive remove. Sep 1 at 9:36
  • @ChennyStar I added an example for option 1. Tell me if it helps. Sep 1 at 10:38
  • I tried your example, but I don't see the point. At the end the original tree is lost (because you moved it to trash_dir), and the trashed tree lacks f1, that has been deleted. I don't see the point of your script. I'm looking for a result like Kaz's solution, where in the end I get my original tree back if it can not be deleted atomically
    – ChennyStar
    Sep 1 at 10:52
1

Here, trying to answer the question: "how to know in advance if a directory and the whole of its contents can be deleted".

The things that can prevent a file from being deleted by a regular user on Linux (and that I can think of):

  • a non-empty directory cannot be deleted, so obviously the files within have to be deleted first
  • You can't delete a file (unlink it from its parent directory) if you don't have write access or search access to the directory.
  • You can't unlink a file from a directory that has the t bit set if you own neither the file nor the directory.
  • You can't delete a file if it or the directory it's link to has the a (FS_APPEND_FL) or i (FS_IMMUTABLE_FL) flag.
  • you can't delete a file if it's a mount point nor / nor . nor ...
  • you can't delete a file if you don't know it's there. Like when it's a directory you have write and search access to but not read permission.
  • you can't delete a file if on a read-only filesystem
  • things like apparmor, selinux and other mandatory access control frameworks may get in the way, as well as things like user namespaces, uid namespaces...
  • Any file system driver may add its own constraints. For instance, some like zfs or nfs have special files/dirs that can't be removed.

Some of those you can't easily check from a script if at all other than by trying it out.

But for most of the more common cases above, there are ways. For instance:

  • [ -w dir ] GNU find -writable, to check whether a directory is writable. Should also take care of read-only filesystems. Similar ones for readable / searchable.
  • ls -nd (or various and incompatible implementations of stat), find -user/-uid to check file owner.
  • lsattr to retrieve file flags.
  • mountpoint -q to check if a file is a mountpoint. See also findmnt.

Putting it all together in a shell script and doing it reliably would be quite tricky though. Here's an attempt using zsh:

#! /bin/zsh -
file=${1?}
export LC_ALL=C

name=$file:t parent=$file:h

die() {
  print -ru2 -- "$@"
  exit 1
}

# can't delete / . .. ""
[[ $name = (/|.|..|) ]] && die "Can't delete $file"

[[ -e $file || -L $file ]] || die "Can't tell whether $file exists"

[[ -w $parent ]] || die "Parent dir not writable"
[[ -x $parent ]] || die "Parent dir not searchable"

[[ -k $parent && ! -O $parent && ! -O $file ]] &&
  die "Parent has t bit, is not ours, nor is $file"

[[ $(lsattr -d -- $parent 2> /dev/null) = ????(i|?a)* ]] &&
  die "parent has a or i file flag"

mountpoint -q -- $file &&
  die "$file is a mount point"

# at this point, we should be able to delete non-directory files
[[ -L $file || ! -d $file ]] && exit 0

# same for empty readable dirs
[[ -r $file ]] && ()((! $#)) $file(NF) && exit 0

case $file in
  (/*) cd -P -- $file;;
  (*) cd -P -- ./$file;; # workaround for -, -1, +1, CDPATH...
esac 2> /dev/null || die "Can't cd into $file"

lsattr -Ra .//. 2> /dev/null |
  grep -Eq '^.{4}(i|.a).*//' &&
  die "Files with a and/or i file flags seen in $file"

zmodload zsh/system
{
  ERRNO=0
  non_empty_dirs=({.,**/*}(NDFoN))
  # error encountered while getting that list. Takes care of 
  # non-readable dirs or non-lstat()able mountpoints...
  (( ERRNO )) && {
    syserror -p "Error encountered while traversing $file: " $ERRNO
    exit 1
  }
}

for dir ($non_empty_dirs) {
  [[ -w $dir ]] || die "non-writable non-empty directory found"

  [[ -k $dir && ! -O $dir ]] && ()(($#)) $dir/*(ND^OY1) &&
    die "directory with t bit I don't own found with files I don't own inside"
}

mountpoints=(${(f)"$(findmnt -nlo target --submount -T .)"})
set -o extendedglob
for mount ($mountpoints) {
  # unescape the \xXX parts:
  mount=${mount//(#b)\\(x??)/${(#):-0$match[1]}}

  [[ $mount = $PWD/* ]] &&
    die "Mount point found under $file"
}

# probably deletable.
exit 0

To be used as:

if that-script "$some_dir_or_file"; then
  rm -rf -- "$some_dir_or_file"
  # and hope for the best
else
  echo probably not fully deletable
fi

It can still give false positives or false negatives in some corner cases. And of course between the time of check and the time of calling rm, many things could have changed in there.

0

A modified version of Kaz's answer. Make a copy (cp -a) of the original tree, try to delete that copy, and if the deletion is successful then delete the original tree.

  • -- : depending on the size of the tree, it can take longer, and more disk space
  • ++ : the original tree remains untouched if it can not be atomically deleted
#!/bin/sh

if [ $# -ne 1 ] ; then
  echo "specify directory to remove"
  exit 1
fi

ar_src=$(realpath "$1")
ar_tmp=$(mktemp -d "$(dirname "$ar_src")/tmp-XXXXXX")

if [ $? -ne 0 ] ; then
  echo "unable to create temporary directory"
  exit 1
fi

if ! cp -a "$1" "$ar_tmp"; then
    echo "unable to copy"
    exit 1
fi

if rm -rf "$ar_tmp"; then
    rm -rf "$ar_src";
fi

exit 0

Execution :

$ ./atomic-rm2.sh d1/
rm: cannot remove '/tmp/testRMDIR/tmp-4sOBNS/d1/d2/f2': Permission denied

$ tree d1
d1
├── d2
│   └── f2
└── f1
5
  • ugh, that sounds like overkill. Also the issue that you might not be able to make the copy in the first place, if you're missing read or access permission somewhere. You'd get a partial copy which you could probably remove, but might not be able to remove the original...
    – ilkkachu
    Sep 1 at 8:42
  • "that sounds like overkill" : not necessarily, it depends on the tree size. Like I said, the advantage is that the tree remains untouched if it can not be deleted "you might not be able to make the copy" : good point, I added a test around the cp -a. The script works only if the whole tree is readable
    – ChennyStar
    Sep 1 at 8:53
  • It is not atomic. And does the cp maintain permissions || fail Sep 1 at 11:01
  • Just because the copy is removable doesn't mean that the original is; it is not air tight. Also, just because a copy cannot be made, doesn't mean the original is not removable. For instance, suppose that a file owned by user bob has r-------- permissions, but is sitting in a writable directory that you own. You can delete that file, because you own the directory, but you cannot copy it because you cannot read it.
    – Kaz
    Sep 1 at 20:25
  • Yes. Good point !
    – ChennyStar
    Sep 2 at 8:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.