/dev/sda **sometimes** confused with /dev/sdb by smartctl

Question

I have a script - run from cron.daily that gathers SMART stats from two identical SATA SSD's. However, smartctl -A /dev/sda sometimes returns the stats for /dev/sdb - and if does so smartctl -A /dev/sdb returns the stats for /dev/sdb. However, sometimes it gets it right!

The system boots into / on a M2 nvme0n1 with /home on one of the SATA SSD's and all filesystems are mounted via fstab using UUID references.

I have tried inserting random sleep commands - but this makes no difference.

The output of smartctl doesn't include any notification of what it is the output of - example output:-

    smartctl 6.6 2017-11-05 r4594 [x86_64-linux-5.10.0-0.bpo.5-amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0032   099   099   000    Old_age   Always       -       2396
. . .

uname -a

Linux hal 5.10.0-0.bpo.5-amd64 #1 SMP Debian 5.10.24-1~bpo10+1 (2021-03-29) x86_64 GNU/Linux

Here is the script, which writes all the output as a single CSV line to a log file.

#!/bin/sh

# SMART DISK PROCESSING
# =====================
tmpfile=$(mktemp -q)
today=$(date -u +%d-%m-%Y)

smartctl -A /dev/sdb > $tmpfile

# Output log as a single line - note "Unknown_Attribute" is "POR_Recovery_Count" [unexpected shutdown]
echo -n $today ', ' >> /var/log/disk-monitor.d/sdb-errors.csv
awk 'NR>=8 && NR<=21 {print $1,",",$2,",",$10,",";}' $tmpfile | tr -d '\n' | sed 's/Unknown_Attribute/POR_Recovery_Count/;s/\,$/\n/' >> /var/log/disk-monitor.d/sdb-errors.csv
#------------------------------
smartctl -A /dev/sda > $tmpfile

# Output log as a single line - note "Unknown_Attribute" is "POR_Recovery_Count" [unexpected shutdown]
echo -n $today ', ' >> /var/log/disk-monitor.d/sda-errors.csv
awk 'NR>=8 && NR<=21 {print $1,",",$2,",",$10,",";}' $tmpfile | tr -d '\n' | sed 's/Unknown_Attribute/POR_Recovery_Count/;s/\,$/\n/' >> /var/log/disk-monitor.d/sda-errors.csv

exit 0

Answer 1

Device nodes for drives aren't guaranteed to be consistent across reboots. They're allocated on a first-seen basis, at boot time. This may vary due to hardware changes, kernel changes, module loading order, minor variations in timing, etc.

If you want persistent device node naming, use the symlinks under /dev/disk/*/. They will always point to the correct device node for the same device, no matter what order the kernel found it in.

I prefer to use the symlinks in /dev/disk/by-id/ because they provide the device type (e.g. nvme or ata or usb), the device brand, model, and serial number. I print sticky labels with the serial numbers for each drive, so I can easily find one if it needs to be replaced without risking confusion with device node names.

e.g. some of the SATA SSDs on one of my systems (partitions from these are used for its zfs rootfs pool):

# ls -lF /dev/disk/by-id/ata-Crucial* | grep -v part
lrwxrwxrwx 1 root root  9 May  9 20:06 /dev/disk/by-id/ata-Crucial_CT275MX300SSD1_163313AAxxx -> ../../sdl
lrwxrwxrwx 1 root root  9 May  9 20:06 /dev/disk/by-id/ata-Crucial_CT275MX300SSD1_163313AAExxx -> ../../sdq
lrwxrwxrwx 1 root root  9 May  9 20:06 /dev/disk/by-id/ata-Crucial_CT275MX300SSD1_163313AAFxxx -> ../../sdo
lrwxrwxrwx 1 root root  9 May  9 20:06 /dev/disk/by-id/ata-Crucial_CT275MX300SSD1_163313AB0xxx -> ../../sdp

# zpool status ganesh
  pool: ganesh
 state: ONLINE
  scan: scrub repaired 0B in 00:22:42 with 0 errors on Sun May  9 00:46:44 2021
config:

        NAME                                               STATE     READ WRITE CKSUM
        ganesh                                             ONLINE       0     0     0
          mirror-0                                         ONLINE       0     0     0
            ata-Crucial_CT275MX300SSD1_163313AADxxx-part5  ONLINE       0     0     0
            ata-Crucial_CT275MX300SSD1_163313AAExxx-part5  ONLINE       0     0     0
          mirror-1                                         ONLINE       0     0     0
            ata-Crucial_CT275MX300SSD1_163313AAFxxx-part5  ONLINE       0     0     0
            ata-Crucial_CT275MX300SSD1_163313AB0xxx-part5  ONLINE       0     0     0

errors: No known data errors

These symlinks will be 100% consistent across every reboot (unless you remove or replace the drive, of course). Whenever a given drive is in the system, the exact same symlinks will be created. And symlinks for each partition on it, too.

BTW, these symlinks are made by udev rules. On my Debian system, /lib/udev/rules.d/60-persistent-storage.rules. You can write your own rules if you want your own naming scheme instead of, or in addition to, these ones. There's not many reasons to want to do that, but you can if you need to.

Answer 2

To highlight a couple of points made by @cas

Previously, all my PC's have had one or two SATA/IDE disks, with '/' being on one of them. Obviously, under these conditions it is impossible for jobs to be run before the proper /dev/sdX identifications have been made.

However, using a separate boot drive containing '/', especially one of a different kind (nvme) containing scripts, programs etc, that isn't in the /dev/sdX naming scheme isn't going to be restrained by any timing constraints.

The use of /dev/disk/by-id/ as well as using the proper hardware name (including serial number) prevents problems caused by improper identifications. It's a pity that disks don't have easily accessible UUID's (although partitions do).

BTW There are numerous examples (especially on youtube) where some very dangerous commands such as dd if=/dev/sdX... are recommended. Perhaps these should be treated with even more caution?