I want to remove the contents of a zfs datasets subdir. It's a large amount of data. For the pool "nas", the path is /nas/dataset/certainFolder

$ du -h -d 1 certainFolder/
1.2T    certainFolder/

Rather than me have to wait for rm -rf certainFolder/ can't I just destroy the handle to that directory so its overwrite-able(even by the same dir name if I chose to recreate it) ??

So for e.g. not knowing much about zfs file system internals, specifically how it journals its files, I wonder if I was able to access that journal/map directly, for e.g., then remove the right entries, so that the dir would no longer display. That space dir holds has to be removed from some kind of audit as well.

Is there an easy way to do this? Even if on an ext3 fs, or is that already what the recursive remove command has to do in the first place, i.e. pilfer through and edit journals?

I'm just hoping to do something of the likes of kill thisDir to where it simply removes some kind of ID, and poof the directory no longer shows up in ls -la. The data is still there on the drive obviously, but the space will now be reused(overwritten), because ZFS is just that cool?

I mean I think zfs is really that cool, how can we do it? Ideally? rubbing hands together :-)

My specific use case (besides my love for zfs) is management of a backup archive. The data is pushed to zfs via freefilesync (AWESOME PROG) on/from win boxes across SMB to the zfs pool. When removing rm -rf /nas/dataset/certainFolder through a putty term, it stalls, the term is obviously unusable for a long time now. I of course then have to open another terminal, to continue. Thats gets old, plus its no fun to monitor the rm -rf, it can take hours.

Maybe I should set the command to just release the handle e.g. &, then print to std out, that might be nice. More realistically, recreate the data-set in a few seconds zfs destroy nas/dataset; zfs create -p -o compression=on nas/dataset after the thoughts from the response from @Gilles.

  • FYI, i ran this command to make the datasets which im using currently.. ` zfs create dataset -p -o compression=on yourPoolName/BackupRootDir/hostNameYourPc/somesubdir Aug 3, 2015 at 7:36
  • Please do accept an answer if one solved the issue described in your original question. The problem you just append to your question looks to be quite different so should really be asked in a new question.
    – jlliagre
    Aug 28, 2015 at 8:41

4 Answers 4


Tracking freed blocks is unavoidable in any decent file system and ZFS is no exception. There is however a simple way under ZFS to have a nearly instantaneous directory deletion by "deferring" the underlying cleanup. It is technically very similar to Gilles' suggestion but is inherently reliable without requiring extra code.

If you create a snapshot of your file system before removing the directory, the directory removal will be very fast because nothing will need to be explored/freed under it, all being still referenced by the snapshot. You can then destroy the snapshot in the background so the space will be gradually recovered.

zfs snapshot ${d}@quickdelete && { 
    rm -rf /${d}/certainFolder
    zfs destroy ${d}@quickdelete & 
  • ok, i have been unfamiliar with snapshots. that might help me. i have been deleting/moving all day still. I created datasets for not just the main backup directory, but top level directories inside, each starting with hostname, and a few top levels.., so i have a bit of flexibility there to just destroy and recreate a pool, but its not perfect, because i wont always want to delete these entire pool dir, i would have to create even more, and thats alot of dataset making, so i like your suggestion for that reason! Aug 3, 2015 at 7:29
  • 5
    If available, feature@async_destroy might also help speed this up (from a user's or administrator's perspective) if enabled; see zpool get all $pool. Note that at least last I looked, if there is a pending destroy in progress on pool import, then that destroy becomes synchronous and the pool import won't finish until the destroy finishes. Watch out if you need to reboot!
    – user
    Sep 28, 2016 at 15:38
  • I have customer with a freenas which lost SMB connection on large deletes. After enabling periodic snapshots (and automatic removal) the problem "disappeared". the freeing of the space takes longer in the background, but the SMB-Share remains accessible all the time. Mar 1, 2017 at 0:11
  • In hindsight, i just wanted to add, this suggestion is brilliant. Defer it. Thanks. I can also say, my data is still safe obviously, its ZFS! Thanks again! Sep 15, 2020 at 3:55

What you're asking for is impossible. Or, more precisely, there's a cost to pay when deleting a directory and its files; if you don't pay it at the time of the deletion, you'll have to pay it elsewhere.

You aren't just removing a directory — that would be near-instantaneous. You're removing a directory and all the files inside it and also recursively likewise removing all of its subdirectories. Removing a file means decrementing its link count, and then marking its resources (the blocks use for file contents and file metadata, and the inode if the filesystem uses an inode table) as free if the link count reaches 0 and the file isn't open. This is an operation that has to be done for every file in the directory tree, so the time it takes is at least proportional to the number of files.

You could delay the cost of marking the resources as free. For example, there are garbage-collected filesystems, where you can remove a directory without removing the files it contains. A run of the garbage collector will detect the files that aren't reachable via the directory structure and mark them as free. Doing rm -f directory; garbage-collect on a garbage collected filesystem does the same things as rm -rf on a traditional filesystem, with different triggers. There are few garbage-collected filesystems because the GC is additional complexity which is rarely needed. The GC time could come at any moment, when the filesystem needs some free blocks and doesn't find any, so the performance of an operation would be dependent on past history, not just on the operation, which is usually undesirable. You'd need to run the garbage collector just to get the actual amount of free space.

If you want to simulate the GC behavior on a normal filesystem, you can do it:

mv directory .DELETING; rm -rf .DELETING &

(I omitted many important details such as error checking, as resilience to power loss, etc.) The directory name becomes non-existent immediately; the space is reclaimed progressively.

A different approach to avoid paying the cost during removal without GC would be to pay it during allocation. Mark the directory tree as deleted, and go through deleted directories when allocating blocks. That would be hard to reconcile with hard links, but on a filesystem without hard links, it can be done with O(1) cost increase in allocation. However that would make a very common operation (creating or enlarging a file) more expensive, with the only benefit being a relatively rare operation (removing a large directory tree) cheaper.

You could bulk-remove a directory tree if that tree was stored as its own pool of blocks. (Note: I'm using the word “pool” in a different meaning from ZFS's “storage pool”. I don't know what the proper terminology is.) That could be very fast. But what do you do with the free space? If you reassign it to another pool, that has a cost, though a lot less than deleting files individually. If you leave the space as unused reserve space, you can't reclaim it immediately. Having an individual pool for a directory tree means added costs to increase or reduce the size of that pool (either on the fly or explicitly). Making the tree its own storage pool also increases the cost of moving files into and out of the tree.

  • Ok great answer! The first half of which is completely satisfiable on a normal system. ZFS has some tricks up its sleeve, for instance there is no need to format it, so if i destroyed the pool, which i think im going to do next time is just make pool's(plural) like im supposed to, then ti disappears off the radar instantly, and that space is immediately available. I guess im trying to recreate that on the zfs, on a directory inside a pool, and i think since its not a pool itself, the nature of it becomes more standard, and the method you mentioned seems to apply in that case. interesting. Aug 2, 2015 at 19:26
  • I think thats where i made my mistake, i read an article last night, ill see if i can find it, that demonstrates that pools shoudl be used like dirs limited to ~18,446,744 Trillion pools max on the FS. if i make my upper backup directories as pools each, when the backup goes to write to them, the dir will already be in tact, which is an easily deletable pool.. If the pool didnt exist the backup will just create the dir, and the pool will not be seen in the zfs list. Until then, hoping somebody else has some input on how to bulk delte on ZFS in a subdir of a pool. :-) Aug 2, 2015 at 19:34
  • Also, when reading your first response, my first thought was; "RIGHT!", "the cost"! thats what i was touching on when i was talking about deleting journal entries. so as i suspected. darn! However, your on the right track. Lets come up with something here, so we can get a script together that will do this maybe... a thought :-) Aug 2, 2015 at 19:37
  • Brian, beware not to confuse zpools and datasets. While there is indeed no reachable hard coded limit on the number of zpools you can create, you will be quickly limited by the number of underlying devices (eg partitions) available on your machine. Moreover, having pools dedicated to single directories will defeat some valuable zfs features and make move operations much slower.
    – jlliagre
    Aug 3, 2015 at 11:39
  • on this comment you made here @Gilles "But what do you do with the free space? If you reassign it to another pool, that has a cost, though a lot less than deleting files individually" im not sure, but i dont think there is a penaltyever creating a new pool, i think i deals with it during write time only. never needs to be partitioned for the same reason.. i believe this is the same mechanism.. Aug 20, 2015 at 6:30

If it has to be quick, I generate a new temporary directory, mv the directory below it and then recursively delete the temporary:

t=`mktemp -d`
mv certainFolder $t/
rm -rf $t &
  • does the & remove handle, or squash errors? Aug 3, 2015 at 7:26
  • 1
    This is not really different from Gilles' suggestion and has a same flaw. Should the OS is rebooted or the rm command does not complete for some other reason, the phantom directory is left over undeleted.
    – jlliagre
    Aug 3, 2015 at 8:18
  • ahh right, but the & is new to me, thats part of the puzzle... i wanted to get rid of the handle. however yes your right, dont want that garbage if there is a problem.. Aug 3, 2015 at 8:24
  • @BrianThomas & simply backgrounds the process, so you can keep doing other things in the same shell while the delete is running (subject to any relevant performance penalties).
    – user
    Sep 28, 2016 at 15:42

If the folder you want to delete and re-create quickly is in its own dataset (and not just a sub-directory of another dataset), you can do:

zfs rename pool/dataset pool/dataset.old
zfs create -o ...options... pool/dataset
zfs destroy -r pool/dataset.old

The new pool/dataset can be used immediately while the old one is being destroyed.

It's a little more complicated than that if there are any child datasets that you don't want to delete (e.g. pool/dataset/child, which will be renamed along with its parent as pool/dataset.old/child), but no more so than if there were sub-directories that you want to keep when deleting most of a sub-directory. Just rename them back into the new pool/dataset before destroying pool/dataset.old, e.g. zfs rename pool/dataset.old/child pool/dataset/child. Similarly, you mv a subdir of pool/dataset.old back into pool/dataset.

If it's just a sub-directory, you can do the same as you would on any other filesystem:

mv subdir subdir.old
mkdir subdir
chmod ... subdir ; chown ... subdir    # if and as required
rm -rf subdir.old/ &

This is pretty much the same as what Gilles said in his answer.

Again, if there are child sub-directories you want to keep, move them back into the new subdir before running the rm -rf, e.g. mv subdir/child subdir/.

I've been doing this for decades, since the 1990s at least - the zfs rename version is just an obvious evolution of the same method. I can't remember if directories could be renamed in MS-DOS but if they could, I was probably doing this with MS-DOS in the 1980s too.

BTW, for both datasets and sub-directories, you don't have to delete the .old one immediately. I tend to keep them around until I'm sure I've retrieved everything I want to keep from them, or until I need to recover the disk space they're using up. I like to delay the point-of-no-return for as long as possible.

BTW, it's often a good idea to use datasets instead of sub-directories ZFS, because you can have different settings (e.g. for compression type, quota, reserved, atime/relatime, encryption, etc) on each dataset, and each dataset can be snapshotted and backed up with zfs send individually.

The price of that, though, is that moving a file or subdirectory tree to another dataset is a copy-and-delete operation, same as it would be when moving them to another filesystem on a different disk or partition or LV etc. A zfs dataset is effectively a different, separate filesystem with its own mount point. See How to move files from one zfs filesystem to a different zfs filesystem in the same pool? - a comment there brought me here today.

Also worth noting: a dataset's mount-point can be anywhere in the filesystem hierarchy, it doesn't have to be mounted directly under its parent, and the mount-point can be changed as needed. e.g. if I have a smallish SSD root pool called "rpool" and a large HDD pool for bulk data called "export", I can move large sub-directories (and datasets) of rpool to export and still have them mounted in the same place. e.g.

zfs create export/share-doc
mv /usr/share/doc/* /export/share-doc/
zfs set mountpoint=/usr/share/doc export/share-doc

(this is a simplified example. In practice, I tend to duplicate the fs hierarchy - e.g. create datasets for export/usr, export/usr/share, export/usr/share/doc to a) keep the top level of a pool uncluttered and b) in case I need to move other subdirs or datasets there)

This distinction between dataset name & hierarchy vs mount-point is important to understand. If you recursively destroy a dataset, its children will also be destroyed no matter where they are mounted. So remember to rename any children that you want to keep.

  • Thanks for adding the extra knowledge, and backlinking, this is helpful! :-) Jun 1, 2021 at 20:57

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .