Stress testing SD cards using linux

Question

I got into a little debate with someone yesterday regarding the logic and/or veracity of my answer here, vis., that logging and maintaining fs meta-data on a decent (GB+) sized SD card could never be significant enough to wear the card out in a reasonable amount of time (years and years). The jist of the counter-argument seemed to be that I must be wrong since there are so many stories online of people wearing out SD cards.

Since I do have devices with SD cards in them containing rw root filesystems that are left on 24/7, I had tested the premise before to my own satisfaction. I've tweaked this test a bit, repeated it (using the same card, in fact) and am presenting it here. The two central questions I have are:

1. Is the method I used to attempt to wreck the card viable, keeping in mind it's intended to reproduce the effects of continuously re-writing small amounts of data?
2. Is the method I used to verify the card was still okay viable?

I'm putting the question here rather than S.O. or SuperUser because an objection to the first part would probably have to assert that my test didn't really write to the card the way I'm sure it does, and asserting that would require some special knowledge of linux.

[It could also be that SD cards use some kind of smart buffering or cache, such that repeated writes to the same place would be buffered/cached somewhere less prone to wear. I haven't found any indication of this anywhere, but I am asking about that on S.U.]

The idea behind the test is to write to the same small block on the card millions of times. This is well beyond any claim of how many write cycles such devices can sustain, but presuming wear leveling is effective, if the card is of a decent size, millions of such writes still shouldn't matter much, as "the same block" would not literally be the same physical block. To do this, I needed to make sure every write was truly flushed to the hardware, and to the same apparent place.

For flushing to hardware, I relied on the POSIX library call fdatasync():

#include <stdio.h>
#include <string.h>
#include <fcntl.h>
#include <errno.h>
#include <unistd.h>
#include <stdlib.h>

// Compile std=gnu99

#define BLOCK 1 << 16

int main (void) {
    int in = open ("/dev/urandom", O_RDONLY);
    if (in < 0) {
        fprintf(stderr,"open in %s", strerror(errno));
        exit(0);
    }

    int out = open("/dev/sdb1", O_WRONLY);
    if (out < 0) {
        fprintf(stderr,"open out %s", strerror(errno));
        exit(0);
    }

    fprintf(stderr,"BEGIN\n");

    char buffer[BLOCK];
    unsigned int count = 0;
    int thousands = 0;
    for (unsigned int i = 1; i !=0; i++) {
        ssize_t r = read(in, buffer, BLOCK);
        ssize_t w = write(out, buffer, BLOCK);
        if (r != w) {
            fprintf(stderr, "r %d w %d\n", r, w);
            if (errno) {
                fprintf(stderr,"%s\n", strerror(errno));
                break;
            }
        }
        if (fdatasync(out) != 0) {
            fprintf(stderr,"Sync failed: %s\n", strerror(errno));
            break;
        }
        count++;
        if (!(count % 1000)) {
            thousands++;
            fprintf(stderr,"%d000...\n", thousands);
        }
        lseek(out, 0, SEEK_SET);
    }
    fprintf(stderr,"TOTAL %lu\n", count);
    close(in);
    close(out);

    return 0;
}                                 

I ran this for ~8 hours, until I had accumulated 2 million+ writes to the beginning of the /dev/sdb1 partition.¹ I could just have easily used /dev/sdb (the raw device and not the partition) but I cannot see what difference this would make.

I then checked the card by trying to create and mount a filesystem on /dev/sdb1. This worked, indicating the specific block I had been writing to all night was feasible. However, it does not mean that some regions of the card had not been worn out and displaced by wear levelling, but left accessible.

To test that, I used badblocks -v -w on the partition. This is a destructive read-write test, but wear levelling or not, it should be a strong indication of the feasibility of the card since it must still provide space for each rolling write. In other words, it is the literal equivalent of filling the card completely, then checking that all of that was okay. Several times, since I let badblocks work through a few patterns.

[Contra Jason C's comments below, there is nothing wrong or false about using badblocks this way. While it would not be useful for actually identifying bad blocks due to the nature of SD cards, it is fine for doing destructive read-write tests of an arbitrary size using the -b and -c switches, which is where the revised test went (see my own answer). No amount of magic or caching by the card's controller can fool a test whereby several megabytes of data can be written to hardware and read back again correctly. Jason's other comments seem based on a misreading -- IMO an intentional one, which is why I have not bothered to argue. With that head's up, I leave it to the reader to decide what makes sense and what does not.]

¹ The card was an old 4 GB Sandisk card (it has no "class" number on it) which I've barely used. Once again, keep in mind that this is not 2 million writes to literally the same physical place; due to wear leveling the "first block" will have been moved constantly by the controller during the test to, as the term states, level out the wear.

Answer 1

I think stress testing an SD card is in general problematic given 2 things:

1. wear leveling There are no guarantees that one write to the next is actually exercising the same physical locations on the SD. Remember that most of the SD systems in place are actively taking a block as we know it and moving the physical location that backs it around based on the perceived "wear" that each location has been subjected to.

2. different technologies (MLC vs. SLC) The other issue that I see with this is the difference in technologies. SLC types of SSD I would expect to have a far longer life vs. the MLC variety. Also there are much tighter tolerances on MLC that you just don't have to deal with on SLC's, or at least they're much more tolerant to failing in this way.

   • MLC - Multi Level Cell
   • SLC - Single Level Cell

The trouble with MLC is that a given cell can store multiple values, the bits are essentially stacked using a voltage, rather than just being a physical +5V or 0V, for example, so this can lead to much higher failure rate potential than their SLC equivalent.

Life expectancy

I found this link that discusses a bit about how long the hardware can last. It's titled: Know Your SSDs - SLC vs. MLC.

SLC

SLC ssds can be calculated, for the most part, to live anywhere between 49 years and 149 years, on average, by the best estimates. The Memoright testing can validate the 128Gb SSD having a write endurance lifespan in excess of 200 years with an average write of 100Gb per day.

MLC

This is where the mlc design falls short. None have been released as of yet. Nobody has really examined what kind of life expectancy is assured with the mlc except that, it will be considerably lower. I have received several different beliefs which average out to a 10 to 1 lifespan in favour of the slc design. A conservative guess is that most lifespan estimates will come between 7 and 10 years, depending on the advancement of ‘wear leveling algorythms ’ within the controllers of each manufacturer.

Comparisons

To draw comparison by way of write cycles, a slc would have a lifetime of 100,000 complete write cycles in comparison to the mlc which has a lifetime of 10,000 write cycles. This could increase significantly depending on the design of ‘wear leveling’ utilized.

Answer 2

There are a number of issues with your test, some fuzzy, some not. It also depends on your goal. Two subtle, sort of fuzzy issues are:

• You are not reading from the same area you are writing to, your read test effectively, then, does nothing (unless the controller has read disturb correction, in which case it may occasionally move the page being read to somewhere else, but this still does not affect your test).
• You assume (and it is likely, but not guaranteed) that a read/write to a bad block is detected and reported by the controller -- you would want to write data, read it back, and compare it for a guaranteed check.

However, those are arguably pedantic. More serious is:

• You cannot use badblocks to show you failed pages on flash memory; all failure detections and subsequent page mappings are done by the controller and are transparent to the OS. You could get some info from SMART if the drive supports it (I know of no SD cards that support it, maybe there are higher end thumb drives that do).
• Wear leveling, complicated by your test not taking into account prior TRIM commands, the free/used state of the drive during the test, and reserved space.

Wear Leveling: The main issue is that wear leveling is a major variable in your test. It happens on the controller (usually), and in any case its transparent to even direct device seek + read/write. In your example, you don't actually know the wear leveling state (in particular, have TRIM commands been issued to free blocks recently?) ...

For dynamic wear leveling (present in virtually all consumer grade storage devices) on your device, then, it could be in any state: At one extreme, none of the pages are marked as free, and so the only pages the controller has to work with are the ones in reserved space (if any). Note that if there is reserved space on the device, it will have to fail entirely before you start getting guaranteed fails on page writes (presuming there are no other pages marked as free remaining). At the other extreme, every page is marked as free, in which case you theoretically have to have every page on the device fail before you start seeing write failures.

For static wear leveling (which SSDs tend to have, SD cards tend to not have, and thumb drives vary): There is really no way around it, aside from repeatedly writing to every page on the device.

... In other words, there are wear leveling details that you have no way of knowing and certainly no way of controlling -- particularly whether or not dynamic wear leveling is in use, whether or not static wear leveling is in use, and the amount of space reserved on the device for wear leveling (which is not visible past the controller [or driver in some cases, like M-Systems old DiskOnChip]).

SLC/MLC: As for SLC vs. MLC, this has a very direct impact on the limits you would expect to see, but the general wear leveling procedure and testing procedure is the same for both. Many vendors do not publish whether or not their devices are SLC or MLC for their cheaper consumer products, although any flash drive that claims a 100k+ cycle limit per page is likely SLC (simplified tradeoff is SLC = endurance, MLC = density).

Caching: As for caching, it's a bit iffy. At the OS level, in the general case, of course, fsync/fdatasync does not guarantee that the data is actually written. However, I think it is safe to presume that it is (or at least the controller has committed to doing so i.e. the write won't get swallowed in cache) in this case, as removable drives are generally designed for the common use pattern of "eject" (unmount > sync) then remove (power cut). While we don't know for sure, an educated guess says that it's safe to assume that sync guarantees that the write will absolutely take place, especially in write -> sync -> read back (if it were not, the drives would be unreliable after eject). There is no other command beyond 'sync' that can be issued on eject.

At the controller anything is possible, but the assumption above also includes the assumption that the controller at least isn't doing anything "complicated" enough to risk data loss after a sync. It is conceivable that the controller may, say, buffer and group writes, or not write data if the same data is being rewritten (to a limited extent). In the program below, we alternate between two different blocks of data and perform a sync before the read back specifically to defeat a reasonable controller caching mechanism. Still, of course, there are no guarantees and no way of knowing but we can make reasonable assumptions based on normal usage of these devices and sane/common caching mechanisms.

Testing:

Unfortunately, the truth is, unless you know that the device has no reserved space and is not doing static leveling, there is no way to definitively test the cycle limit of a specific page. However, the closest you can get is as follows (presume no static wear leveling):

The first thing you need to do is fill the entire card with data. This is important, and is the main variable that was left in your original test. This marks as many block possible as used, aside from any reserved space (which you have no way of accessing). Note that we are working with an entire device (which this will destroy all data on), as working with a single partition only affects one specific area on the device:

dd if=/dev/urandom bs=512k of=/dev/sdb conv=fsync oflag=sync

If you're the progress bar type:

pv -pterb -s <device_size> /dev/urandom | dd bs=512k of=/dev/sdb conv=fsync oflag=sync

Edit: For cards with 4MB erase blocks, try this for a faster write:

dd if=/dev/urandom bs=4M of=/dev/sdb conv=fsync oflag=direct,sync iflag=fullblock

Next, then, you can write a cycle test program as follows, making use of O_DIRECT and O_SYNC (and possibly paranoid, redundant use of fsync()) to cut as much OS buffering and caching out of the picture as possible and, theoretically, write directly to the controller and wait until it reports that the operation has finished:

#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <cstdlib>
#include <cstdio>
#include <cstring>

using namespace std;

static const int BLOCK_SIZE = 512;
static const int ALIGNMENT = 512;
static const int OFFSET = 1024 * ALIGNMENT; // 1024 is arbitrary


int main (int argc, char **argv) {

    if (argc != 2) {
        fprintf(stderr, "usage: %s device\n", argv[0]);
        return 1;
    }

    int d = open(argv[1], O_RDWR | O_DIRECT | O_SYNC);
    if (d == -1) {
        perror(argv[1]);
        return 1;
    }

    char *block[2], *buffer;
    int index = 0, count = -1;

    // buffers must be aligned for O_DIRECT.
    posix_memalign((void **)&(block[0]), ALIGNMENT, BLOCK_SIZE);
    posix_memalign((void **)&(block[1]), ALIGNMENT, BLOCK_SIZE);
    posix_memalign((void **)&buffer, ALIGNMENT, BLOCK_SIZE);

    // different contents in each buffer
    memset(block[0], 0x55, BLOCK_SIZE);
    memset(block[1], 0xAA, BLOCK_SIZE);

    while (true) {

        // alternate buffers
        index = 1 - index;

        if (!((++ count) % 100)) {
            printf("%i\n", count);
            fflush(stdout);
        }

        // write -> sync -> read back -> compare
        if (lseek(d, OFFSET, SEEK_SET) == (off_t)-1)
            perror("lseek(w)");
        else if (write(d, block[index], BLOCK_SIZE) != BLOCK_SIZE)
            perror("write");
        else if (fsync(d))
            perror("fsync");
        else if (lseek(d, OFFSET, SEEK_SET) == (off_t)-1)
            perror("lseek(r)");
        else if (read(d, buffer, BLOCK_SIZE) != BLOCK_SIZE)
            perror("read");
        else if (memcmp(block[index], buffer, BLOCK_SIZE))
            fprintf(stderr, "memcmp: test failed\n");
        else
            continue;

        printf("failed after %i successful cycles.\n", count);
        break;

    }

}

Note that for O_DIRECT, buffers must be suitably aligned. 512-byte boundaries is generally sufficient. You may compile with:

g++ -O0 test.cpp -o test

Add -D_POSIX_C_SOURCE=200112L if necessary.

Then, after filling the device entirely as above, just leave it run over night:

./test /dev/sdb

512 byte, aligned writes are fine, that will give you one entire page erased and rewritten. You could significantly speed up the test by using a larger block size, but then it becomes complicated to arrive at concrete results.

I am currently testing on a rather beat-up looking 4GB PNY thumb drive that I found on the sidewalk yesterday (appeared to be what was left of a http://www3.pny.com/4GB-Micro-Sleek-Attach---Purple-P2990C418.aspx).

The above program is essentially a limited version of badblocks and you would not see failures until all reserved space was exhausted. Therefore, the expectation (with 1 page written per iteration) is that the above procedure, on average, should fail in reserved_page_count * write_cycle_limit iterations (again, the wear leveling is a major variable). It's too bad thumb drives and SD cards don't usually support SMART, which has the ability to report the reserved space size.

By the way, fsync vs fdatasync makes no difference for the block device writes you are doing, for the purposes of this test. Your open() modes are important.

If you are curious about technical details; here is everything you might want to know (plus more) about the inner workings of SD cards: https://www.sdcard.org/downloads/pls/simplified_specs/part1_410.pdf

Edit: Bytes vs Pages: In the context of these types of tests, it is important to think of things in terms of pages, not bytes. It can be very misleading to do the opposite. For example, on a SanDisk 8GB SD, the page size according to the controller (accessible via /sys/classes/mmc_host/mmc?/mmc?:????/preferred_erase_size) is a full 4MB. Writing 16MB (aligned to 4MB boundaries), then, erase/writes 4 pages. However, writing four single bytes each at 4MB offsets from each other also erase/writes 4 pages.

It is inaccurate, then to say "I tested with 16MB writes", as it is the same amount of wear as "I tested with 4 byte writes". More accurately, "I tested with 4 page writes".

Answer 3

Just adding some points to slm's answer - note these are more in place for SSDs than for "dumb" SD cards, since SSDs play much dirtier tricks with your data (e.g. de-duplication):

• you are writing 64KB to the beginning of the device - this itself has two problems:

  1. flash cells usually have erase blocks of size from 16KB up (more likely in the 128-512KB range, though). Which means that it needs cache of at least this size. Hence writing 64KB doesn't seem to be enough to me.

  2. for low-end (read "non-enterprise") solutions (and I would expect this even more for the SD/CF cards than for SSDs) manufacturers may choose to make the beginning of the device more resilient to wear than the rest since the important structures - the partition table and FAT on the single partition on the device (most memory cards are using this setup) - are located there. Thus testing the beginning of the card might be biased.

• fdatasync() doesn't really guarantee that the data get written to the physical medium (although it probably does the best what is under the control of the OS) - see the man page:

  The call blocks until the device reports that the transfer has completed

  I wouldn't be overly surprised if it turned out that there is a small capacitor, that is able to provide energy for writing cached data to the flash memory in case of losing external power.

  In any case, under the assumption of a cache being present on the card (see my answer to your question on SU), writing 64KB and syncing (with fdatasync()) doesn't seem to be convincing enough for this purpose. Even without any "power backup" the firmware might still play it unsafe and keep the data unwritten for a bit longer than one would expect (since in typical use cases it shouldn't create any problems).

• you might want to read the data before writing new block and comparing it, just to make sure it really works (and use a cleared buffer for the reading, if you are paranoid enough).

Answer 4

Peterph's answer did make me consider the issue of possible caching further. After digging around, I still can't say for sure whether any, some, or all SD cards do this, but I do think it is possible.

However, I don't believe that the caching would involve data larger than the erase block. To be really sure, I repeated the test using a 16 MB chunk instead of 64 kB. This is 1/250th the total volume of the 4 GB card. It took ~8 hours to do this 10,000 times. If wear leveling does its best to spread the load around, this means every physical block would have been used 40 times.

That's not much, but the original point of the test was to demonstrate the efficacy of wear leveling by showing that I could not easily damage the card through repeated writes of modest amounts of data to the same (apparent) location. IMO the previous 64 kB test was probably real -- but the 16 MB one must be. The system has flushed the data to the hardware and the hardware has reported the write without an error. If this were a deception, the card would not be good for anything, and it can't be caching 16 MB anywhere but in primary storage, which is what the test is intended to stress.

Hopefully, 10,000 writes of 16 MB each is enough to demonstrate that even on a bottom end name brand card (value: $5 CDN), running a rw root filesystem 24/7 that writes modest amounts of data daily will not wear the card out in a reasonable period of time. 10,000 days is 27 years...and the card is still fine...

If I were getting paid to develop systems that did heavier work than that, I would want to do at least a few tests to determine how long a card can last. My hunch is that with one like this, which has a low write speed, it could take weeks, months, or years of continuous writing at the max speed (the fact there aren't oodles of comparative tests of this sort online speaks to the fact that it would be a very prolonged affair).

With regard to confirming the card is still okay, I no longer think using badblocks in it's default configuration is appropriate. Instead, I did it this way:

badblocks -v -w -b 524288 -c 8

Which means to test using a 512 kB block repeated 8 times (= 4 MB). Since this is a destructive rw test, it would probably be good as my homespun one with regard to stressing the device if used in a continuous loop.

I've also created a filesystem on it, copied in a 2 GB file, diff'd the file against the original and then -- since the file was an .iso -- mounted it as an image and browsed the filesystem inside that.

The card is still fine. Which is probably to be expected, after all...

;) ;)