Udev triggers are not firing on insert of CF card into USB card reader (anymore)

Question

I've got a udev rule that triggers a shell script that copies photos off my CF card when I insert the card into my USB card reader. This worked fine for a while. But after a recent upgrade it has stopped working.

The problem (AFAICT) is that inserting the CF card into the USB reader is no longer triggering any udev events, so my script never runs. (Even the entries in /dev/disk/by-label/ don't show up on insert, and dmesg shows nothing about the insert.)

I can cause the udev events to trigger if I run fdisk -l /dev/sdf (or sdh or whatever device it is that seems to be where the CF card will appear). I've verified the udev events with udevadm monitor, and by running udevd --debug. After I manually trigger the system, the script works fine from this point.

I'm running Debian 6.0 (Squeeze), and I'm pretty confident it was the upgrade from Lenny a couple months back that changed the behavior, but I'm not sure if it was some subtle change in udev (perhaps I'm listening for the wrong events?) or some change in the kernel or USB stack? Anyway, anyone have suggestions for how to fix this? Or suggestions for getting more debugging info out of the usb-storage module?

This is the udev rule I'm using:

KERNEL=="sd??", SUBSYSTEM=="block", RUN+="/usr/local/bin/hot-add"

Update #1: Trigger on blockdev and partition changes

I added a rule to trigger on just the block device instead of on a partition (in case that might show up), but it doesn't trigger when I insert the CF card either:

KERNEL=="sd?", SUBSYSTEM=="block", RUN+="/usr/local/bin/hot-add-disk"

Oddly, if I run fdisk -l /dev/sdf to trigger recognition of the CF card, I get a "changed" event for /dev/sdf (even though I never got an "add" event).

Update #2: add uevent triggers

There is an open, incomplete Debian bug that covers some similar ground (skip to Update#25 for the relevant bits). That has some handy tips.

For my setup, if I do (as root) echo add > /sys/block/sdf/uevent with the CF card inserted but otherwise unrecognized, that will trigger all the udev events properly and all my scripts run correctly.

Running udevadm info --query=all --name=sdf before and after inserting the CF card shows no difference. If I inject the 'add' event and re-run the udevadm command I get a bit more output (there is some information about disk partitions).

Update #3: USB key vs. CF card in USB reader

Inserting a vanilla USB key triggers all the udev events immediately and works just fine. Inserting a CF card into the USB card reader does not. For the card reader, it seems that the USB device "exists" even when no CF card is inserted (so a /sys/block/sdf entry exists), but with the USB key, the /sys/block/sdj only exists while the USB key is plugged in.

Update #4: Maybe fixed in 2.6.38

This Ubuntu bug has an update (the last one) that claims the bug was no longer present in a 2.6.38 kernel.

Update #5: Not fixed (in 3.2.0)

I've upgraded the kernel a couple times, and even changed the motherboard I'm using, and I still see the same problem. I'm currently at a 3.2.0.0 kernel. And udev version 164.

Answer 1

tl;dr: See solution at Update 2 below.

This is not really an answer sadly, or at least not good news: I'm getting the feeling that this can never work anyway — I just find that a little hard to believe since I surely think this has worked in the past!

This is what usually works, and why: Most laptop SD card readers. You'll find that you plug in an SD card, udev sees it, Linux will scan it for partitions and automounting will just work. Why does this work? Look at lsusb before and after inserting the card. You'll note that with no SD card plugged in, the USB device doesn't exist. So when you insert an SD card, there's a full-blown USB hotplug event.

The card reader in my monitor doesn't seem to work like this, it's always there. And I can't get it to generate any event when I insert an SD card. I started with looking at syslogs, then on to udevadm monitor, then usbmon for raw USB event logging. No luck, there doesn't seem to be any insert/eject notification/event at all, from either the card reader in my monitor or the cheap DealExtreme one in my PC.

The moment I try to access the raw device, the kernel wakes up and the partition appears. Similarly, call udevadm trigger and it'll note the device. Take out the card and nothing happens until I try to access the device again and the kernel realises it's gone.

What I wonder myself now is, how could this ever have worked (that is, if I'm not making up this memory)? Maybe some daemon used to be polling the raw device every few seconds and stopped doing this? It's hard to find any documentation on this, really.

Update #1

Above is all with a Debian stock kernel. I've just done some experiments with an Ubuntu kernel, where SD card hotplugging appears to work. And not surprisingly, in usbmon I see polls every two seconds. Which does also explain why the cards aren't detected immediately but with a slight delay.

The polling seems to be done by the kernel, since it happens even in single-user mode with no userland process doing anything other than being stuck in a select() loop. I haven't been able to figure out where in the kernel this is done, I'll update this post if I find out.

Update #2

The important difference seems to be /sys/block/sd?/events_poll_msecs. On my Debian box it's -1, on the Ubuntu box it's 2000. This udev rule seems to do it:

# enable in-kernel media-presence polling
ACTION=="add", SUBSYSTEM=="module", KERNEL=="block", ATTR{parameters/events_dfl_poll_msecs}=="0", ATTR{parameters/events_dfl_poll_msecs}="2000"
ACTION=="add", ATTR{removable}=="1", ATTR{events_poll_msecs}=="-1", ATTR{events_poll_msecs}="2000"