File system that never breaks (data loss acceptable)

Question

There're several existing topics revolving around this issue, but what I seek is slightly different. I have an SD card on an embedded Linux and it suffers from power loss. I might be able to modify the hardware at some point, close down properly and so on, etc. But right now, I'd just like to find a file system that survives power loss without fuss. Data loss is acceptable. I'd prefer not to lose more than the file that I'm currently writing, but I'd still rather lose it all than facing an 'unable to mount', 'wait for this 10 minutes fsck' or 'unable to create new file due to this inode something something error'. The program MUST go on!

I'm making a lot of effort in ensuring this. I'm using industrial grade components, I got hardware watchdogs, software watchdogs, internal, external, init restarting the programs, daemons constantly checking memory, file descriptors and whatnot, I got watchdogs watching my watchdogs, which in turn are watched by other watchdogs... But I can't seem to guarantee that the SD card is able to mount and function?

My best bet right now, is to use JFS on the SD card, include fsck and fsck.jfs in my installation. (Adding 600kb+ eating my ram and my flash. Which is bad.) And run fsck at every startup (maybe adding a lot of boot time. Which is somewhat bad.). It seems a bit sad though.

Does anyone know of a better way or a better file system?

UPDATE: e2fsprogs-libs (dependency to jfsutils) seems to be hellishly difficult to compile in my distribution. I'll look into ZFS (it's not native to my distribution though. And it seems to do a lot that I don't need.)

UPDATE2: Some more info about my system and my tests: The SD card storage is a secondary, optional storage. The SD cards are 2Gb-8Gb industrial grade microSD. The SD card is mounted through my rc with a mount -t command. Options "noatime" but not "sync". My distribution is a custom Analog Device flavored uClinux, with a 3.10 kernel and a 1.21 busybox. My primary storage is a spi flash with jffs2. I've never had any issues with that. I don't even know if there is a fsck.jffs2 available. Nand flash on the other hand ... but that's a different story. The purpose of the SD card, is to store measurement data. The 'monitor' program will append results to a file and has strategic sync placements. When the file comes above a given size a new will be created. When a given number of files have been reached the oldest one will be deleted. If the current measurement file is lost due to power loss, it's no disaster. The files are usually at 50-100kb and 1 result is usually 1kb. This is just the initial development phase. Nothing is fixed. This is the first time I've been dealing with non-flash filesystems in embedded systems. (I got ext4 at my x86 servers.)

I started out with vfat. The default filesystem. (I figured that the factories might have a reason for choosing it. And if things work I don't really care that much.) I've never seen any power loss issues in my embedded vfat devices. I've experienced issues with FAT in WinCE though. However, when my 'monitor' program reached 100-200 files it refused to create any more. It seems that FAT has a special file limit issue in the root and a slightly bigger one in sub dirs. I need to be able to create 500-1000 files in 1 dir. So vfat won't do.

Then I switched to ext2. I didn't insert a fsck at startup though. (Didn't know I had to do so.) Within a day my 'monitor' program were unable to create more files due to a 'inode something something' error. Disaster!

My current solution is ext2 with a "e2fsck -y" at startup. So far it's seems promising. But the e2fsck and the whole concept of 'fsck at startup' is nagging at me. The e2fsck by it self is spending more than 350kb of my primary flash and ram. (When it's not running.) Which means that it's my biggest program. It's bigger than busybox. It's almost rivaling my kernel.

I've been considering ext3. It has journalled meta data, which wouldn't hurt. I'm in doubt as to how much it will help though. With my small files and controlled syncs I should be covered I think? It has a ordered write sequence. Meaning that data are also somewhat journalled. This however can lead to non-deterministic lags. Which is bad in my situation. (It's probably not an issue.) It also has a scheduled sync feature. Eg. commit every 5 sec. Which is interfering with my own syncs I think. Too many writes are bad for SD cards. Even industrial ones. I cannot find any documentation on how to disable this. And ext3 still requires fsck to be run at every startup! But ext3 is still a possibility.

Ext4. Will fix a lot of the performance issues of ext3. I don't really need performance though. And my distribution doesn't seem to have a builtin mkfs.ext4 and a fsck.ext4. Perhaps that's not a problem. It might though. Eg. the e2progs-libs (dependency to jfsutils) seems to have a lot of compile issues.

JFS, XFS, BRFSS. All supported by my kernel. Currently not included in my user space tool box. All seems to be rather big, complex systems. And they all seem to require a 'fsck' equivalent at startup?

I've also considered throwing my own filesystem: Always write 2 copies of the file table. When traversing, it pick the one with the correct CRC and the newest sequence number. Make a 2-stage write sequence. Allocate temporary, fix at commit. No fsck needed. I'm afraid that it might be a bit naive though.

UPDATE3: BTW, the nature of embedded systems (this one at least) is that they're autonomous, unattended, out of reach, and they have to run for years. Programs like fsck that may require human interaction creeps me out.

Answer 1

There's a bit of an inconsistency or at least, ambiguity, in your story here:

I'd still rather lose it all than facing an 'unable to mount', 'wait for this 10 minutes fsck'

Implies -- although you don't actually say it -- that this is a problem you are actually experiencing. But then:

e2fsprogs-libs (dependency to jfsutils) seems to be hellishly difficult to compile in my distribution.

Meaning you don't have any fsck at all, since e2fsprogs-libs is a dependency for e2fsprogs which provides e2fsck. So perhaps you are still in a planning stage here and have not even tested the system with, e.g., ext4, but instead jumped to the conclusion that you should start with JFS? Is there any particular reason for that?

I've noticed on the raspberry pi exchange (the pi's primary storage is also a SD card) that a significant number of users seem to be very frustrated by problems of this sort, even though the majority (including myself) have never had it at all. At first I assumed these were people ignorant of the fact that the system should be cleanly shut down, but that is not a hard point to grasp when explained, and there are people who report it even though the system HAS been shut down properly.

You've already said you need this to be able to tolerate power cuts (which is fair enough), but I mention this because it implies there are some pis, or some SD cards, or some combination of both, that are just prone to corrupting the filesystem due to some event (surge?) that occurs regularly either when the plug is pulled, or when it is put back in. I also have NOT seen -- and there's been plenty of time for plenty of people to try -- ANY reports of someone saying they've switched to btrfs or jfs or whatever and now the problem is solved.

The other mysterious thing about this is even if people are yanking the cord, this should not regularly result in an unusable filesystem. Certainly I've done it a bunch of times w/ the pi, and scores if not hundreds of times w/ a regular linux box (the power was cut, the system has become unresponsive, I'm exhausted and angry, etc.) and while I've seen minor data loss, I've never seen a filesystem corrupted to the point of being unusable after a quick fsck.

Again, presuming all these reports are true (I don't see why numbers of people would lie about it), there's something much more going on than just not cleanly unmounting, but it seems to only affect a small percentage of users, implying again some kind of common hardware defect.

On the pi I write -y to /forcefsck in a boot script, so that on the next boot it is run automatically and any problems are fixed, regardless of whether this appears to be necessary or not. On a 700 Mhz single core this takes ~10 seconds for a 12 GB filesystem containing ~4 GB of data. So "10 minutes" sounds like an incredibly long time, especially since you've already said "This is the small filesystem for write!".

You might also consider calling sync at regular intervals.

Finally, you should update the question with more factual, specific details of the problems you have actually encountered, and less hyperbole. Otherwise it just looks too much like a premature XY problem, which will likely get quickly skipped over by people with a lot of experience and potential advice for you.

Answer 2

The program MUST go on!

Well this is a common requirement and Linux systems are the best choice when it comes to choose a stable System.

Your efforts seem also not go to the right direction. However what can you do in order to get a stable system?

In the first level you can improve your file system:

• Use yournal_data_ordered on a ext3/ext4 filesystem, when creating or modifying with tune2fs.
• With JFS use --replay_journal_only when checking.
• Enable autorepair by setting FSCKFIX=yes in the initscripts.

If this is not enough, you can boot your system without mounting the buggy disk. Instead create a fresh ramdisk while you manually checking and repairing your buggy disk. This can be also automated by scripts.

In the next level you'll need to leave you embedded system and read some topics about High Availability