2

After a breaker trip a Raspberry Pi of mine started to halt boot with kernel panic (same message as here). This is a Raspberry Pi running Raspbian, so it runs from a SD card, from a main ext4 partition, which I've tried repairing on my PC with:

sudo e2fsck -f -y -v /dev/sdx2

However, this eventually fails with some weird output:

Error writing block 137439060017 (Invalid argument) while getting next inode from scan.  Ignore error? yes
Error reading block 183472412950529 (Invalid argument).  Ignore error? yes
Force rewrite? yes
Error writing block 183472412950529 (Invalid argument) while getting next inode from scan.  Ignore error? yes
Inode 13329, i_size is 4096, should be 549755817984.  Fix? yes
Inode 13607, i_size is 69632, should be 137439023104.  Fix? yes
Error reading block 36983963385857 (Invalid argument).  Ignore error? yes
Force rewrite? yes
Error writing block 36983963385857 (Invalid argument) while getting next inode from scan.  Ignore error? yes
Error reading block 179632729097217 (Invalid argument).  Ignore error? yes
Force rewrite? yes
Error writing block 179632729097217 (Invalid argument) while getting next inode from scan.  Ignore error? yes
Error reading block 17592186080054 (Invalid argument) while reading directory block.  Ignore error? yes
Force rewrite? yes
Error writing block 17592186080054 (Invalid argument) while getting next inode from scan.  Ignore error? yes
Error storing directory block information (inode=17449, block=0, num=134507168): Memory allocation failed
/dev/sdx2: ***** FILE SYSTEM WAS MODIFIED *****
e2fsck: aborted
/dev/sdx2: ***** FILE SYSTEM WAS MODIFIED *****

There are two things that are worrying here:

  • the inode sizes and block sizes, which seem ridiculously high (we're talking ab out a 16GB SD card),
  • e2fsck ends with Memory allocation failed - on a PC with 32 GB of RAM, most of which are free. It does actually take up the free RAM before it fails.

I've tried configuring a scratch file directory with the same result (e2fsck does write some files there, and the target directory is on a mount with +250GB free space - it takes up the available RAM, and fails).

It looks like there's some corruption in the fundamental file system parameters on the affected partition. How to diagnose and eliminate it?

  • If it's exhausted 32 GB examining a 16 GB card, something's gone way out to lunch. I wouldn't bother trying to arrange more memory for it just to find out how far it will go, as this is not normal behavior -- possibly it would just continue to eat space indefinitely, and I really doubt at this point it is going to actually repair a filesystem. You need to test the card at a low level to make sure it isn't physically defunct. – goldilocks Nov 11 '13 at 19:40
  • WRT to that (testing the card), you can do a non-destructive test with badblocks , just use -n instead of -w. – goldilocks Nov 11 '13 at 19:43
  • @goldilocks : exactly, I just wanted to clarify what the behavior actually is. And good point on the physical damage, I'm running a read-only test now. – mikołak Nov 11 '13 at 20:23
1

I had a quick glance through the e2fsck source, and it seems to me there are places where the "Memory allocation failed" error can occur for reasons that might not really be memory allocation errors.

The error string is defined in [src]/lib/ext2fs/ext2_err.et.in in relation to the constant EXT2_ET_NO_MEMORY. This can be returned from various places in the code in [src]/e2fsck/. Here's an example from ea_refcount.c:

errcode_t ea_refcount_increment(ext2_refcount_t refcount, blk_t blk, int *ret)
{
    struct ea_refcount_el   *el;

    el = get_refcount_el(refcount, blk, 1);
    if (!el)
        return EXT2_ET_NO_MEMORY; 

get_refcount_el() is in the same file:

static struct ea_refcount_el *get_refcount_el(ext2_refcount_t refcount,
                          blk_t blk, int create)
{
    int low, high, mid;

    if (!refcount || !refcount->list)
        return 0;    

That's not the only reason it will return null, nor the only reason that looks like it is not directly related to a failed allocation.

To really prove that I'd have to do more digging, but it does fit with your assertion that it did not really exhaust the system memory.

This being the case, perhaps the problem is related to an obscure and unpredictable potential of deranged/damaged SD card controllers, but it still amounts to a bug in e2fsck in so far as some sort of sanity checking or something should be done to catch this, even if it's just to say, "Sorry, your device is screwed" (probably true) vs. "Out of memory" (probably not true). You may want to report this ("In case of bugs in these programs, please contact Ted Ts'o at tytso@mit.edu or tytso@alum.mit.edu" -- I believe T.T. is a linux kernel dev), and you can reference this Q&A.

Beyond that, IMO you might as well forget whatever is on that card and do a destructive read-write test on it:

badblocks -v -w -b 1048576 -c 16 /dev/sdx

Remember, that's a DESTRUCTIVE test -- you'll be loosing all your data. Badblocks is not useful for creating an actual badblocks list for an SD card (they do not report actual physical addresses because of wear levelling), but if the card is borked, it will probably let you know. Testing a 16 GB card this way takes less than an hour.

  • Sorry, I think my description might have been misleading - it does take up the free RAM before it crashes (checked with top). The only thing weird thing regarding resource usage is that it doesn't use up the space for the scratch file. I've edited the question to clarify that. – mikołak Nov 11 '13 at 19:21
  • If it is a series of allocations that are supposed to be very small but end up being very large because of some zany value that's snuck in, those allocations might intentionally not involve scratch files on the presumption that (sans zaniness) they should be very small. Still seems to fit the definition of "bug" to me; the fact that there don't seem to be other reports of it is why I think it is probably something unpredictable related specifically to SD controllers (or the driver for such). – goldilocks Nov 11 '13 at 19:36
  • Pass completed, 0 bad blocks found. (0/0/0 errors). Looks like your bug interpretation may be correct (which is a shame in the context of repairing the FS). I'll leave the question open for the next few days, just in case. – mikołak Nov 11 '13 at 22:54
  • @ Ted T. -- By "the event itself" I meant the one from the original question and not the (mistakenly hypothesized, according to the 1st comment) one from answer. – goldilocks Nov 12 '13 at 0:29
2
+50

When you run e2fsck -fy you really need to save the whole e2fsck transcript, and not just show the last couple of error messages. It may be that file system was very badly damaged, and the -y option means to keep going no matter what.

It looks like the block group descriptors were badly corrupted, so that the locations of the inode table were insane. E2fsck probably tried to repair it, but for some reason it wasn't able to fix it, and "-y" means it will keep on going despite that. So when people send bug reports, I always suggest that they send the full e2fsck transcript, and not just last couple of errors.

  • First of all, thank you for bringing this question to your attention! Regarding the transcript - of course the full bug report will include the full version, I've only used a snippet here to highlight the problem of the weirdly large block numbers. – mikołak Nov 20 '13 at 6:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.