33

Classical situation: I ran a bad rm and realized immediately afterwards that I had removed the wrong files. (Nothing critical and I had tolerably recent backups, but still annoying.)

Knowing that further disk activity was my enemy if I wanted to recover the files with extundelete or such tools, I immediately powered the machine down physically (i.e., with the power button, not with halt or any such command). This was a laptop with no important tasks running or anything open, so it was an acceptable operation. (By the way, I learned since then that the first thing to do in such a situation would be to estimate first if the missing files may still be opened by a process https://unix.stackexchange.com/a/101247 -- if they are, you should recover them this way rather than power down the machine.)

Still, once the machine was powered down I thought for a while and decided the files were not worth the time investment of booting a live system for proper forensics. So I powered the machine back up. And then I discovered that my files were still sitting on disk: the rm hadn't been propagated to disk before I had powered down. I did a little dance and thanked the god of sysadmins for His unexpected forgiveness.

My question is now to understand how this was possible, and what is the typical delay before an rm is actually propagated to disk. I know that disk IO isn't flushed immediately but that it sits in memory for some time, but I thought that the disk journal would make sure quickly that pending operations do not get entirely lost. https://unix.stackexchange.com/a/78766 seems to hint at a separate mechanism to flush dirty pages and to flush journal operations but does not give sufficient detail about how the journal would be involved for a rm, and the expected delay before operations are flushed.

Some more details: the data was in an ext4 partition inside a LUKS volume, and when booting the machine back up I saw the following in syslog:

Sep 24 10:24:58 gamma kernel: [   11.457007] EXT4-fs (dm-0): 1 orphan inode deleted
Sep 24 10:24:58 gamma kernel: [   11.458393] EXT4-fs (dm-0): recovery complete
Sep 24 10:24:58 gamma kernel: [   11.482475] EXT4-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null)

but I am not confident it is related to the rm.

Another question would be whether there is a way to tell the kernel to not perform any of the pending disk operations (but rather, say, dump them somewhere), rather than powering the machine down. (Of course, it sounds dangerous to not perform the pending operations, but this is what would happen when powering the machine down anyway, and it some cases it could save you.) This would be "cleaner", of course, and also interesting for e.g. remote servers where physical powerdown is not an easy option.

Improve this question

2 Answers 2

Reset to default
24

It sounds like you've got a decent grasp on what happened.

Yes, because you hard-powered-off the system before your changes were committed to disk, they were there when you booted back up.

The system caches all writes before flushing them out to disk. There are several options which control this behavior, all located at /proc/sys/vm/dirty_* [kernel doc]. Unless a flush is explicitly performed by an application via fsync() [man 2 fsync], the data is committed when it is either old enough, or the write cache is filled up.
The definition of "data" as used above includes modification to the directory entry to delete the file.

Now, as for the journal, that's one of the common misconceptions of what the journal is for. The purpose of a journal is not to ensure changes get replayed, or that data is not lost. The purpose of a journal is to prevent corruption of the filesystem itself, not the files in it. The journal simply contains information about the changes being made, and not (typically) the full data of the change itself. The exact details are dependent upon the filesystem, and journal mode. For ext3/4, see the data mount option in man 8 mount.

To answer your supplementary question of whether there's a way to prevent the pending writes without a reboot:

From doing a quick read through the kernel source code, it looks like you can use the magic sysrq u command ([wikipedia], [kernel doc]) to do an emergency remount-read-only operation. It appears this will immediately remount all volumes read-only without a sync operation.

To use this, simply press Alt+SysRq+u.

Improve this answer
6
  • 1
    Thanks for this answer! I am still confused a bit about the journal: should I think of it as something that only gets involved when the changes get flushed to disk, so that write caching is the only relevant mechanism to estimate the grace time before the rm gets written? In other words, things are committed to the journal only when a write is just about to be performed? Or is the picture more complex than that? As for alt-sysrq-u, this is a pretty neat idea. Do you have a reference to give for the "It appears" claim? (It doesn't seem to follow from the links that you gave.) Thanks! :)
    – a3nm
    Commented Sep 24, 2014 at 5:11
  • Also, magic sysrq also has the limitation that you still cannot do it on a remote machine.
    – a3nm
    Commented Sep 24, 2014 at 5:16
  • 3
    @a3nm You can use sysrq on a remote machine. echo u > /proc/sysrq-trigger (you may need to activate it first).
    – Paulo Almeida
    Commented Sep 24, 2014 at 10:00
  • The journal doesn't deal with the file contents (by default, it can be changed fully-journalled), only with filesystem metadata, but in this case it could have deleted the file, as we are dealing with removing directory entry. Thus the journal must ensure that either the file exists (with its previous contents, assuming they had no other changes) or it doesn't.
    – Ángel
    Commented Sep 24, 2014 at 12:50
  • @a3nm In regards to your journal comment. The write cache sits between the journal and the disk. When you write to the filesystem, the journal is updated, then the filesystem, but neither are committed to disk yet.
    – phemmer
    Commented Sep 24, 2014 at 21:17
2

From: https://www.kernel.org/doc/Documentation/filesystems/ext4.txt

commit=nrsec (*) Ext4 can be told to sync all its data and metadata every 'nrsec' seconds. The default value is 5 seconds. This means that if you lose your power, you will lose as much as the latest 5 seconds of work (your filesystem will not be damaged though, thanks to the journaling). This default value (or any low value) will hurt performance, but it's good for data-safety. Setting it to 0 will have the same effect as leaving it at the default (5 seconds). Setting it to very large values will improve performance.

Also see here on how to flush them: How do you empty the buffers and cache on a Linux system?

Quoted from the above link:

NOTE: clean up memory of unnecessary things (Kernerl 2.6.16 or newer). Always make sure to run sync first to flush useful things out to disk!!!

To free pagecache:

$ echo 1 > /proc/sys/vm/drop_caches

To free dentries and inodes:

$ echo 2 > /proc/sys/vm/drop_caches

To free pagecache, dentries and inodes:

$ echo 3 > /proc/sys/vm/drop_caches
Improve this answer
3
  • Thanks for this answer! However, I do not understand this: as for this "sync" that is mentioned in commit=nrsec, is it something that would take place after the kernel has decided to flush changes from memory to disk? Or does setting commit=1 guarantee that all changes will be flushed after 1 second regardless of the dirty_expire_centisecs and dirty_writeback_centisecs settings?
    – a3nm
    Commented Sep 24, 2014 at 10:59
  • The kernel will flush (sync) any cache/buffers to the disc every 1 second for commit=1. As far as I understand it, sync forces everything to happen regardless of Virtual Memory Settings although it can happen sooner.
    – David
    Commented Sep 24, 2014 at 23:17
  • Also for performance reasons, (and storage longevity) setting commit to lower than default is not recommended.
    – David
    Commented Sep 24, 2014 at 23:24

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .