8

Following an unclean shutdown on an SD card based device, I took the SD card out to fsck the root filesystem. This led to variations on the following:

e2fsck 1.43.1 (08-Jun-2016)
/dev/sdc2: recovering journal
Superblock needs_recovery flag is clear, but journal has data.
Run journal anyway<y>? no
Clear journal<y>? no
e2fsck: unable to set superblock flags on /dev/sdc2

Here I've answered "no" both times but there is no sequence of yes/no that does not immediately lead to the same outcome.

The filesystem can be mounted and on casual inspection appears okay; it also works fine in the device, and that's the root filesystem (actually it turned out to be not quite fine, see comments; tldr some irretrievably corrupted directories).

I dd'd the partition (8 GB) to a file, and tried fsck on that. Interestingly:

e2fsck 1.43.1 (08-Jun-2016)
plush.rootfs: recovering journal
Clearing orphaned inode 18290 (uid=0, gid=0, mode=0100644, size=34096)
Clearing orphaned inode 18270 (uid=0, gid=0, mode=0100644, size=38916)
Clearing orphaned inode 18250 (uid=0, gid=0, mode=0100644, size=1128076)
Clearing orphaned inode 11411 (uid=0, gid=0, mode=0100644, size=293108)
Setting free inodes count to 406127 (was 408580)
Setting free blocks count to 1305622 (was 1347486)
plush.rootfs: clean, 60209/466336 files, 604906/1910528 blocks (check after next mount)

A subsequent fsck passed clean, the image can be mounted, and fsck -f after that passes as well.

But the filesystem on the card from which the raw block copy image was created still has the same problem -- except that the systemd-fsck which takes place during boot logs the filesystem as "clean". Subsequently though, a proper shutdown, taking the card out, and trying fsck again from another box presents the same error.

Whenever the original is mounted on another machine, syslog notes:

kernel: EXT4-fs (sdc2): 4 orphan inodes deleted
kernel: EXT4-fs (sdc2): recovery complete

Since I have it all backed up, I'm open to trying anything here. I could simply forget about this and reburn the partition from the apparently fixed image, but that does not seem like a very satisfactory solution, since it means assuming fsck cryptically failed at solving a minor looking problem.

I suspect this is going to turn into a "request for official documentation" question regarding stuff like needs recovery_flag (or just plain "What does this mean?" question), so any suggestions along those lines are appreciated.

  • Anything in the kernel logs about device errors? This wouldn't be the first time an SD card suddenly became read-only. – Mark Plotnick Dec 3 '16 at 23:22
  • @MarkPlotnick No, and it is writable. The last thing in the log from before the problem was systemd restarting (the device is headless and became unresponsive following a long apt upgrade). After that it logs a normal boot -- and the systemd-fsck says "clean" (I'll edit that in), but trying fsck outside of that context still fails. – goldilocks Dec 3 '16 at 23:36
  • Your fsck on the copy cleared 4 inodes, but fixed the free inode count by reducing it by 2453 inodes! That's enormous. Check that the device is getting enough power. – meuh Dec 4 '16 at 11:08
  • @meuh I notice whenever it's mounted on the big box syslog refers to those 4 inodes (edited in above). Some stuff on the fs did turn out to be messed up (updated kernel modules! \O/) so I've burned a new card and will hang on to the old one in case I have a chance to dig into this more. It isn't exactly new -- an unbranded bargain bin class 10 card, in (light duty) use 24/7 for possibly a few years so...I don't think there's any way to verify an SD card is definitively defunct, but I guess it could be that. The power should be okay but might be iffy under certain conditions. – goldilocks Dec 4 '16 at 16:48
  • 2
    Doesn't it really suck when the very tool that's supposed to fix your problem, doesn't work because of the nature of the problem? Conclusion: The tool is bad and should be fixed. – Marc.2377 Jul 7 '17 at 2:42
6

I just ran into this same problem. After debugging the issue with the e2fsck maintainer, we realised that the SD card was broken. It was accepting writes without error, but it wasn't actually writing the data to the card. The SD card was effectively read only.

It seems the card had gone into some sort of failsafe mode, where the data could still be read, but nothing written.

The e2fsck message unable to set superblock flags means it tried to write to the superblock to mark the journal as processed, which happened without error, but when it went to read the superblock back again it still indicated that the journal needed to be replayed. In other words, the changes written to the superblock were not saved on the storage medium.

The card I am using that has this problem is a Samsung Evo 16GB microSD, which I mention just in case it's a common problem with these cards.

I was able to test this by using dd to write 4096 bytes from /dev/zero onto the card at block 0, then I read back from the card and instead of getting all zeroes as I should, I still got the original unchanged ext4 superblock.

I'm now in the process of moving the data onto a new card and then seeing if I can get a replacement from Samsung, who appear to offer a 10 year warranty on SD cards.

UPDATE: Samsung replaced the 16GB card with a 32GB one in the same Evo series, so guess I can't complain too much!

  • "where the data could still be read, but nothing written" -> The fs was writable. – goldilocks Oct 1 '17 at 13:05
  • @goldilocks: Sounds like your fs superblock may not have been writable though. Also, my fs appeared writable thanks to caching, it was only after unmounting and remounting I noticed any changes were lost. – Malvineous Oct 2 '17 at 1:27
  • It was not an illusion due to caching. – goldilocks Oct 2 '17 at 12:57
6

I know this is an old thread, but, I figured I'd offer some insight.

This seems to be the way sd cards die a natural death. The number of read/write cycles sd cards can endure is substantially lower than most other mediums considered 'read/write'. When that has been exhausted, the card will go into read only mode, but won't inform you of such. Lots of things will think they're writing to the card thanks to OS caching, etc., but nothing ever sticks.

A great way to kill an sd card is to mount it as a swap partition or something that's very read/write intensive. You'd be surprised how quickly you can kill a card that way. I've found that running knoppix off of an sd card or usb thumb drive will only last a month or two, depending on the quality of the card and the intensity of knoppix usage. (I've since switched to running knoppix off of a usb SSD drive which has lasted a couple of years now).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.