4

I'm working with massive amounts of USB flash memory. I'd like to be able to track events regarding the USB drives. Specifically: I perform f3 tests, format the drives and write data to them.

I'm looking for a way to mark the drives digitally (for example with a UUID) so I can perform various actions with the drive while having the serial numbers intact. Hopefully also have other people use and format the drives, still allowing me to read the serial number off of them later.

I have tried using the serial number reported by the USB device, but I've soon have found out that the majority of the drives have no uniquely identifying information stored in them.

Hence - I have tired generating a UUID concatenated with it's own CRC sum so I can write that string to the drive and read it back, verifying if it was stored correctly.

Writing the serial number to the filesystem however means that once the drive is filled up the every copy of the serial number will be overwritten.

My latest idea is to partition the drive to leave 1 MB of unpartitioned empty space so I can store the serial number there.

This is going to have a marginal impact on a drive's usefullness but should allow me to somewhat reliably identify drives, as long as the drive will not be repartitioned.

Ideally each drive would have a UUID written to the Mass Storage driver chip and would report that - but I don't manufacture the drives.

I thought about generating the serial numbers a bit like this (Bash):

UUID=$(uuidgen)
CK=$(echo "$UUID" | cksum | cut -c -8)
SN="SN:$UUID:$CK;"
echo $SN

This generates a UUID, calculates a CRC sum of that UUID, leaves just 8 first characters of that sum (because the length can vary and we need a constant length to effectively search for the pattern later). Then it concatenates that with some markers that can be used to extract the number form a data stream and verify if it wasn't damaged.

Example serial numbers:

SN:7199e42e-465b-4baf-96b6-a109247d3f4b:39272476;
SN:aa795142-fa2e-4929-b93b-e991f95a74cd:40619286;
SN:43f93702-eda9-4089-9a81-d21172e051bf:37174302;

I can then verify the serial number integrity like this:

SN="SN:7199e42e-465b-4baf-96b6-a109247d3f4b:39272476;"
UUID=$(echo "$SN" | cut -d':' -f2)
CK=$(echo "$SN" | cut -d':' -f3 | cut -d';' -f1)
if [[ $(echo "$UUID" | cksum | cut -c -8) == $CK ]]; then
    echo "SN is fine"
else echo "SN is corrupted"; fi

I can also extract a serial number from a drive with grep:

cat /dev/sdX | grep -E "SN:.{8}-.{4}-.{4}-.{4}-.{12}:.{8};" -o

Then I'd get a list of serial numbers, and evaluate them until I find one that has a matching checksum.

Because I work with potentially faulty drives I need to write the serial number in multiple copies so at least one copy will survive.

I wonder if there's a better approach to what I'm trying to do - without of course resorting to manufacturing my own flash memory,

EDIT: UDEVADM

I can try to get a unique identifier using various read-only information from the drive itself:

$ udevadm info /dev/sdd | grep -e "ID_MODEL" -e "ID_SERIAL" -e "ID_VENDOR"
E: ID_MODEL=Flash_Disk
E: ID_MODEL_ENC=Flash\x20Disk\x20\x20\x20\x20\x20\x20
E: ID_MODEL_ID=6387
E: ID_SERIAL=Generic_Flash_Disk_97C06F44-0:0
E: ID_SERIAL_SHORT=97C06F44
E: ID_VENDOR=Generic
E: ID_VENDOR_ENC=Generic\x20
E: ID_VENDOR_ID=058f

$ udevadm info /dev/sdd | grep -e "ID_MODEL" -e "ID_SERIAL" -e "ID_VENDOR" | md5sum | cut -d' ' -f1
7b7f63b829b6301cdf35d71eed9bb707

I am almost certain however that I'll encounter a lot of drives that return identical values here - making this approach useless.

EDIT2:

I have accumulated nearly 1600 datapoints on this, and most drives seem to produce unique hashes, but I got still around 400 drives that returned identical hashes, so this method alone is not enough.

  • 1
    Why do you need to complicate that so much? Instead of leaving unpartitioned space, which will be inevitably used by the controller to remap bad blocks, create a small partition and store your information there, separately from the rest of data. – ajeh Jun 8 '18 at 17:47
  • 1
    @sudodus - the drives would be distributed to other people (clients basically), so anything strange like two partitions detected by Windows or prompts to format the drive are unacceptable - that's why I wanted to try a small unpartitioned space so it doesn't result in any annoyances to the end users. I do my tests with f3 and it works on the file level, so it won't write or read beyond provided partitions. – unfa Jun 14 '18 at 9:37
  • 1
    @unfa, 1. I don't know Windows well enough to know if there is a way to make Windows ignore a partition entirely; 2. If you do not intend to make these drives boot in BIOS mode, you could use the drive space between the first 512 bytes and the beginnning of the second mibibyte (where grub-pc will put data for the bootloader, so most of the first mibibyte. This part of the drive will not be partitioned by linux tools (for example by gparted), and you can write things to it for example near the end of it using dd. This could be a good and safe place for your special identification data. – sudodus Jun 14 '18 at 12:22
  • 1
    @unfa, I tested gparted: Both when creating an MSDOS partition table and a GPT, and after those operations using 'the whole drive' to create a partition with a file system, the last kibibyte of the first mibibyte is untouched, so your own ID will be preserved (if you put it there). – sudodus Jun 29 '18 at 7:17
  • 1
    @sudodus - that sounds interesting. I have done a tests where I've written a drive full with dd and after formatting and writing the filesystem full - everything was overwriitten. However - I used mkntfs -QF and made the filesystem on the bare drive, without a partition. Maybe that's why it didn't work as you've experienced it. I'll research this. Thanks! – unfa Jun 29 '18 at 7:35
1

The ID_SERIAL of the device usually does the trick for verifying the identity of a device.

It is persistent across formatting because it's a property of the device itself; however, afaik, you cannot change it due to the same reason.

You can get it by running: udevadm info /dev/sdX | grep ID_SERIAL=

  • I'll have to check, but I think that many devices I work with return identical values. – unfa Jun 11 '18 at 7:43
0

Persistent device naming (a name that is the same across diferent mount points) is an elusive goal. There are several UUIDs created in several containers (filesystem, mount point, etc) as described in detail here.

An UUID (well, actually a World Wide Identifier (WWID)) given by the hardware of the USB device connected may be given in:

ls -l /dev/disk/by-id/

But understand that not all devices support such numbering.

  • 1
    I deal with many chineese USB flash memory modules and many report the vendor as "Generic" and serial as "1234". There are so called "MP tools" (Mass Production) that are vendor-specific programs used to write the metadata to the flash memory controller chip at the manufacturing facility. I wish they assigned UUIDs to all the drives, it'd make things much easier for me. – unfa Sep 20 '18 at 8:35
0

Some flash storage is not uniquely identifiable after a format and the best you can do is set custom partition UUIDs or upgrade to better storage.

https://superuser.com/questions/1247972/change-uuid-of-vfat-partition

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.