Classical situation: I ran a bad
rm and realized immediately afterwards that I had removed the wrong files. (Nothing critical and I had tolerably recent backups, but still annoying.)
Knowing that further disk activity was my enemy if I wanted to recover the files with
extundelete or such tools, I immediately powered the machine down physically (i.e., with the power button, not with
halt or any such command). This was a laptop with no important tasks running or anything open, so it was an acceptable operation. (By the way, I learned since then that the first thing to do in such a situation would be to estimate first if the missing files may still be opened by a process https://unix.stackexchange.com/a/101247 -- if they are, you should recover them this way rather than power down the machine.)
Still, once the machine was powered down I thought for a while and decided the files were not worth the time investment of booting a live system for proper forensics. So I powered the machine back up. And then I discovered that my files were still sitting on disk: the
rm hadn't been propagated to disk before I had powered down. I did a little dance and thanked the god of sysadmins for His unexpected forgiveness.
My question is now to understand how this was possible, and what is the typical delay before an
rm is actually propagated to disk. I know that disk IO isn't flushed immediately but that it sits in memory for some time, but I thought that the disk journal would make sure quickly that pending operations do not get entirely lost. https://unix.stackexchange.com/a/78766 seems to hint at a separate mechanism to flush dirty pages and to flush journal operations but does not give sufficient detail about how the journal would be involved for a
rm, and the expected delay before operations are flushed.
Some more details: the data was in an ext4 partition inside a LUKS volume, and when booting the machine back up I saw the following in
Sep 24 10:24:58 gamma kernel: [ 11.457007] EXT4-fs (dm-0): 1 orphan inode deleted Sep 24 10:24:58 gamma kernel: [ 11.458393] EXT4-fs (dm-0): recovery complete Sep 24 10:24:58 gamma kernel: [ 11.482475] EXT4-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null)
but I am not confident it is related to the
Another question would be whether there is a way to tell the kernel to not perform any of the pending disk operations (but rather, say, dump them somewhere), rather than powering the machine down. (Of course, it sounds dangerous to not perform the pending operations, but this is what would happen when powering the machine down anyway, and it some cases it could save you.) This would be "cleaner", of course, and also interesting for e.g. remote servers where physical powerdown is not an easy option.