f3 and a custom Bash script to test USB flash memory in big amounts.
A common problem I encounter is that some faulty drives would starve all the healthy ones for IO, effectively stalling the testing process.
For example - when leaving 50 USB drives for testing I often find out after an hour that 48 are doing nothing, and 2 are blinking their LEDs. Removing these two drives suddenly resumes all others drives testing.
There sometimes are more complex situations where 24 of the drives are stalled, and the rest seems to work fine. Except a few drives don't make any progress after 20 minutes. You plug them out, the rest is coming back to life, and the testing continues.
However - I also found out it's enough to stop testing the faulty drives to make the rest come to life.
I'm looking for a way to find out which drives are causing this file operation blockage on others so I can automatically stop them in my script.
I've been watching
dmesg to try and find a discriminating factor, but I can't see anything. I've found there's so called
usbmon kernel debugging interface, though it's so low-level that I don't really know how to use it. The raw USB packets don't tell me anything.
Are there any other tools I could use to tell which drives are misbehaving?
f3read programs to test the drives. The
f3write program creates 1GB files that the
f3read program then reads identifying any data damage that occurred in the process.
Also - it's strange, but when a misbehaving drive is present, the rest of the "healthy" drives will finish their work on the current file. Say - writing or reading a 1GB sized file - but will not create a new file until the misbehaving drives are removed. It's like opening a new file becomes impossible in the presence of an "IO hog" drive.
What can I do to tell them apart?