USB attached SATA - partitions table not being read at boot

Question

Linux apparently fails to identify partitions in a USB-attached SATA disk, if the system is booted with said disk plugged in. This causes partitions (and therefore filesystems) to be invisible, requiring either a partition rescan (partprobe, blockdev --rereadpt, etc), a UAS module re-load, or to unplug the drive & re-plug it back in. None of those are feasible if the disk in question is intended to be the boot device, with ESP and root filesystem in it.

Some experiments were done. Results:

1. Plugging in disk with host online and UAS module loaded -> works
2. Plugging in disk with host online and UAS & usb_storage modules NOT loaded -> works
3. With disk plugged in, booting the system from an another drive -> fails Partitions not detected. After giving it plenty of time to settle, the following is observed: Both UAS and usb_storage modules loaded; /dev/sda and /sys/class/block/sda exist; Interestingly lsblk does not list it, neither does /proc/partitions.
   1. From this state, unloading and re-loading UAS module -> works (lsblk lists it, /dev/sda{1,2} are created, everything "normal")
   2. Also from this state, rescanning partition via blockdev --rereadpt -> works
   3. Repeating test with and without UAS module pre-loaded via kernel command line yields same result.
4. With disk plugged in, attempt to boot the system from same disk -> fails (system does not boot. Dropped to rescue shell.)
   1. Once dropped to the shell, attempting to blockdev --rereadpt -> works
   2. Repeating test with and without UAS module pre-loaded via kernel command line yields same result.

Output of dmesg | grep sd executed from test case #3 (at 42 seconds a blockdev --rereadpt was executed). Output of same from case #4 is identical.

[   13.831953] sd 6:0:0:0: [sda] Read Capacity(10) failed: Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[   13.831955] sd 6:0:0:0: [sda] Sense Key : Not Ready [current] 
[   13.831956] sd 6:0:0:0: [sda] Add. Sense: Logical unit is in process of becoming ready
[   13.832435] sd 6:0:0:0: [sda] Test WP failed, assume Write Enabled
[   13.832850] sd 6:0:0:0: [sda] Asking for cache data failed
[   13.832853] sd 6:0:0:0: [sda] Assuming drive cache: write through
[   15.038494] sd 6:0:0:0: [sda] Read Capacity(10) failed: Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[   15.038497] sd 6:0:0:0: [sda] Sense Key : Not Ready [current] 
[   15.038498] sd 6:0:0:0: [sda] Add. Sense: Logical unit is in process of becoming ready
[   15.039379] sd 6:0:0:0: [sda] Attached SCSI disk
[   42.833001] sd 6:0:0:0: [sda] 1953525168 512-byte logical blocks: (1.00 TB/932 GiB)
[   42.834148] sda: detected capacity change from 0 to 1000204886016
[   42.956764]  sda: sda1 sda2

Any hints? I assume the device takes a bit to spin-up and become ready (from the Logical unit is in process of becoming ready messages), and maybe the partition scan happens while the device is not ready, failing or getting invalid data. I do not know when this scan happens, or which subsystem causes it. Does it have to do anything with udev? Any other possible culprits? At first I thought that, perhaps, the on-demand module loading caused it to be unavailable when it was needed. So I tried to force it to be loaded - apparently something that can be accomplished by listing the module's name on the kernel command line - but to no avail.

rootwait and rootdelay have been tested. Both fail to address the problem.

I find it interesting to remark that a run-of-the-mill USB memory stick works for the intended purpose of booting off of it (ESP and root filesystem). Also, USB-attached SATA disk is detected by BIOS, UEFI executable (GRUB) found & launched, GRUB read and booted kernel from it, etc.

--

Question's background:

I have a USB to SATA cable I'm using to connect a 2,5in HDD to a machine. The machine is a little home server with 4 internal SATA ports, all in use for RAIDZ member disks. I intend to boot off the external drive, and have root filesystem in there. Up until this point, I had ESP and root filesystem in a 8GB USB stick and it works great... until it doesn't. I've found that a USB stick will very quickly shit its pants if put through the workload a typical root filesystem endures, even if journald logs are redirected to RAM and everything but essential processes are run in LXC containers, which have their root filesystems in the ZFS pool.

Answer 1

After some more digging around, I found a hack-ish solution to this problem. It is far from perfect, and it does not address the root cause, but it will at least make the system bootable.

The hack is to have a script in initramfs that executes just before trying to mount root, but after device initialization, and sleeps a moment before executing blockdev --rereadpt.

In a debian-based system, simply put the following file in /etc/initramfs-tools/scripts/local-top/rereadpt:

#!/bin/sh
PREREQ=""

prereqs()
{
    echo "$PREREQ"
}

case $1 in
# get pre-requisites
prereqs)
    prereqs 
    exit 0
    ;;
esac

sleep 5
blockdev --rereadpt /dev/<yourdisknode>

You can use /dev/sda-like names if your site environment has a single SCSI disk, or /dev/disk/by-id/..., or whatever other logic to ensure you're triggering the rescan on the correct drive.

UPDATE: If you don't want update-initramfs to execute the aforementioned script every time it rebuilds the image (even though it supposedly, shouldn't??), you must not omit the prereqs and case snippet at the beginning