I have a script - run from cron.daily that gathers SMART stats from two identical SATA SSD's. However, smartctl -A /dev/sda sometimes returns the stats for /dev/sdb - and if does so smartctl -A /dev/sdb returns the stats for /dev/sdb. However, sometimes it gets it right!

The system boots into / on a M2 nvme0n1 with /home on one of the SATA SSD's and all filesystems are mounted via fstab using UUID references.

I have tried inserting random sleep commands - but this makes no difference.

The output of smartctl doesn't include any notification of what it is the output of - example output:-

    smartctl 6.6 2017-11-05 r4594 [x86_64-linux-5.10.0-0.bpo.5-amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0032   099   099   000    Old_age   Always       -       2396
. . .

uname -a

Linux hal 5.10.0-0.bpo.5-amd64 #1 SMP Debian 5.10.24-1~bpo10+1 (2021-03-29) x86_64 GNU/Linux

Here is the script, which writes all the output as a single CSV line to a log file.


# =====================
tmpfile=$(mktemp -q)
today=$(date -u +%d-%m-%Y)

smartctl -A /dev/sdb > $tmpfile

# Output log as a single line - note "Unknown_Attribute" is "POR_Recovery_Count" [unexpected shutdown]
echo -n $today ', ' >> /var/log/disk-monitor.d/sdb-errors.csv
awk 'NR>=8 && NR<=21 {print $1,",",$2,",",$10,",";}' $tmpfile | tr -d '\n' | sed 's/Unknown_Attribute/POR_Recovery_Count/;s/\,$/\n/' >> /var/log/disk-monitor.d/sdb-errors.csv
smartctl -A /dev/sda > $tmpfile

# Output log as a single line - note "Unknown_Attribute" is "POR_Recovery_Count" [unexpected shutdown]
echo -n $today ', ' >> /var/log/disk-monitor.d/sda-errors.csv
awk 'NR>=8 && NR<=21 {print $1,",",$2,",",$10,",";}' $tmpfile | tr -d '\n' | sed 's/Unknown_Attribute/POR_Recovery_Count/;s/\,$/\n/' >> /var/log/disk-monitor.d/sda-errors.csv

exit 0
  • 7
    Does this happen after reboots? Because device nodes aren't guaranteed to remain the same across reboots (admittedly, it's abnormal for them to change as frequently as you say unless the hardware changes frequently). If you want persistent naming (and you do!), use the symlinks in /dev/disk/by-id/ or /dev/disk/by-label/ etc. – cas May 11 at 14:41
  • 1
    @cas I will try this; I'd suggest that this is definitely the answer - ls /dev/disk/by-id/ shows (amongst other links) ata-Samsung_SSD_870_QVO_1TB_S5SVNG0N955653H and ata-Samsung_SSD_870_QVO_1TB_S5SVNG0NB22319L – Jeremy Boden May 11 at 15:08
  • Actually, I always get correct results for sda & sdb - but they do not necessarily correspond with the smartctl parameter. The two SSD's are connected directly to the motherboard SATA ports, by their own signal cables - nothing complicated! – Jeremy Boden May 11 at 21:49

Device nodes for drives aren't guaranteed to be consistent across reboots. They're allocated on a first-seen basis, at boot time. This may vary due to hardware changes, kernel changes, module loading order, minor variations in timing, etc.

If you want persistent device node naming, use the symlinks under /dev/disk/*/. They will always point to the correct device node for the same device, no matter what order the kernel found it in.

I prefer to use the symlinks in /dev/disk/by-id/ because they provide the device type (e.g. nvme or ata or usb), the device brand, model, and serial number. I print sticky labels with the serial numbers for each drive, so I can easily find one if it needs to be replaced without risking confusion with device node names.

e.g. some of the SATA SSDs on one of my systems (partitions from these are used for its zfs rootfs pool):

# ls -lF /dev/disk/by-id/ata-Crucial* | grep -v part
lrwxrwxrwx 1 root root  9 May  9 20:06 /dev/disk/by-id/ata-Crucial_CT275MX300SSD1_163313AAxxx -> ../../sdl
lrwxrwxrwx 1 root root  9 May  9 20:06 /dev/disk/by-id/ata-Crucial_CT275MX300SSD1_163313AAExxx -> ../../sdq
lrwxrwxrwx 1 root root  9 May  9 20:06 /dev/disk/by-id/ata-Crucial_CT275MX300SSD1_163313AAFxxx -> ../../sdo
lrwxrwxrwx 1 root root  9 May  9 20:06 /dev/disk/by-id/ata-Crucial_CT275MX300SSD1_163313AB0xxx -> ../../sdp

# zpool status ganesh
  pool: ganesh
 state: ONLINE
  scan: scrub repaired 0B in 00:22:42 with 0 errors on Sun May  9 00:46:44 2021

        NAME                                               STATE     READ WRITE CKSUM
        ganesh                                             ONLINE       0     0     0
          mirror-0                                         ONLINE       0     0     0
            ata-Crucial_CT275MX300SSD1_163313AADxxx-part5  ONLINE       0     0     0
            ata-Crucial_CT275MX300SSD1_163313AAExxx-part5  ONLINE       0     0     0
          mirror-1                                         ONLINE       0     0     0
            ata-Crucial_CT275MX300SSD1_163313AAFxxx-part5  ONLINE       0     0     0
            ata-Crucial_CT275MX300SSD1_163313AB0xxx-part5  ONLINE       0     0     0

errors: No known data errors

These symlinks will be 100% consistent across every reboot (unless you remove or replace the drive, of course). Whenever a given drive is in the system, the exact same symlinks will be created. And symlinks for each partition on it, too.

BTW, these symlinks are made by udev rules. On my Debian system, /lib/udev/rules.d/60-persistent-storage.rules. You can write your own rules if you want your own naming scheme instead of, or in addition to, these ones. There's not many reasons to want to do that, but you can if you need to.

  • 1
    Checking journalctl, the job ran within the same second as mounting of the drives took place, during booting of the system. So have to suspect that udev was still working. – Jeremy Boden May 11 at 21:55

To highlight a couple of points made by @cas

Previously, all my PC's have had one or two SATA/IDE disks, with '/' being on one of them. Obviously, under these conditions it is impossible for jobs to be run before the proper /dev/sdX identifications have been made.

However, using a separate boot drive containing '/', especially one of a different kind (nvme) containing scripts, programs etc, that isn't in the /dev/sdX naming scheme isn't going to be restrained by any timing constraints.

The use of /dev/disk/by-id/ as well as using the proper hardware name (including serial number) prevents problems caused by improper identifications. It's a pity that disks don't have easily accessible UUID's (although partitions do).

BTW There are numerous examples (especially on youtube) where some very dangerous commands such as dd if=/dev/sdX... are recommended. Perhaps these should be treated with even more caution?

  • 2
    When a guide tells you to check that you have to right disk for dd of=/dev/sdX or whatever, they really mean check, now, during this reboot, before running the command. e.g. check /proc/partitions to make sure the partitions on it are the sizes you were expecting, check /proc/mounts, or check lsblk output. And check dmesg or hdparm -i, or smartctl -a, to see that it's the right disk. Guides don't tend to reiterate that, but getting a definitive ID in one or more of those ways is how you should interpret it before running any destructive command. – Peter Cordes May 12 at 4:02
  • 1
    Disks do have WWNs (world wide names), which serve pretty much the same purpose as UUIDs, with some added benefits (just like MAC addresses they're globally unique but non-random so you can actually tell the vendor and perhaps even the model from them). They're also often printed on the label. – TooTea May 12 at 7:16
  • That explains the existence of links in ls -l /dev/disk/by-id/ such as:- wwn-0x5002538f70b0e020 -> ../../sda – Jeremy Boden May 12 at 15:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.