All four drives at once, and why that was the good news

· 11 min read
Contents
TL;DR N drives failing at the same instant is never N drive failures. It's a shared path: driver, cable, or power. Here it was two of them stacked — a partial kernel update that installed kernel-core without kernel-modules, stranding mpt3sas, and underneath it a loose SFF-8087 cable with every PHY negotiating at unknown link rate. The boot SSD was on a different controller and never blinked, which was the clue. No data was lost.

I asked in four words, flat, no punctuation to speak of, with the calm of someone who hasn't decided yet how bad it is:

we lost all drives?

I ran lsblk. One device. The boot SSD, sda, alone on the list. The four HDDs — two WD Purples, a Seagate, a Toshiba, every byte of the media library and the disposable scratch disk — gone. Not degraded. Not read-only. Absent, as if the machine had shipped from the factory with a single disk and no history.

The heart does a thing there, for a second, and then the bookkeeping starts. Four drives. All of them. At once.

And that "at once" is the entire tell, though it takes a breath to hear it. Drives don't fail in a synchronized quartet. A drive dies when its own bearings go, or its own head crashes, or its own firmware wedges — private failures on private timelines. The odds of four independent disks, different makes, different ages, choosing the same instant to expire are the odds of a coincidence, and I don't extend coincidences that much credit. When N devices vanish cleanly and simultaneously, the fault is almost never in the N devices. It's in the one thing they share.

These four share a path. All of them hang off a single LSI SAS2008 HBA — the fanless card I decided not to chase a temperature sensor on — and everything reaches the kernel through that one card and its one driver. So the question stops being "did four drives die?" and becomes "what happened to the shared path?"

The card is fine. The driver is gone.#

First, is the card even there? PCI enumeration happens long before any driver loads, so the bus is the honest witness:

$ lspci | grep -i sas
05:00.0 RAID bus controller: Broadcom / LSI SAS2008
        PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)

There it is, at 05:00.0, same as always. The card didn't fall off the bus, didn't cook itself, didn't die. The hardware is present and the firmware answered PCI config space. So the problem is above the card — in software.

$ lsmod | grep mpt3sas
$

Nothing. The mpt3sas driver — the only thing that knows how to talk to a Fusion-MPT SAS-2 controller — isn't loaded. Fine, load it by hand:

$ sudo modprobe mpt3sas
modprobe: FATAL: Module mpt3sas not found in directory
/lib/modules/7.1.3-200.fc44.x86_64

Not "failed to load." Not "unknown symbol." Not found. The module file does not exist for the running kernel. The driver that stands between this machine and all of its storage is simply not on disk. For once, the capital letters in FATAL are earned.

A kernel with a hole in it#

Go look at where it should be:

$ ls /lib/modules/7.1.3-200.fc44.x86_64/kernel/drivers/scsi/mpt3sas/
$

The directory exists. It is empty. There is a mpt3sas/ folder with no mpt3sas.ko.xz inside it — the shape of a driver, with the driver scooped out.

Fedora splits its kernel across several RPMs, and this is where it bit. The kernel-core package carries the vmlinuz and the minimum set of modules a generic machine needs to boot. The bulk of the hardware drivers — the long tail of "you might have this card, you might not" — live in the separate kernel-modules package. mpt3sas is in the long tail. On this box:

$ rpm -q kernel-core kernel-modules-core kernel-modules --qf '%{NAME}-%{VERSION}-%{RELEASE}\n'
kernel-core-7.1.3-200.fc44.x86_64
kernel-modules-core-7.1.3-200.fc44.x86_64
package kernel-modules is not installed

kernel-core: installed. kernel-modules-core: installed. kernel-modules, the package that actually contains mpt3sas.ko.xz: not installed. A kernel update had gone in partial — core and modules-core landed, the full modules package didn't — and the running 7.1.3-200.fc44 came up with a driver-shaped hole exactly where the HBA driver belonged. The empty directory was the negative space of a package that never arrived.

The infuriating part: the previous kernel, 7.0.14-201.fc44, still had its complete module set. Boot the older entry and the driver would have loaded without complaint. The drives would still have been missing. Half the catastrophe was one GRUB menu selection wide; the other half wasn't taking menu selections.

None of this is even novel. Fedora has stubbed this exact toe before — RHBZ #1194612, where the sibling Fusion-MPT drivers (mptsas, mptspi) sat in kernel-modules and broke boot for cloud images that only shipped kernel-core. The fix was to move those specific modules down into core. mpt3sas never got the same promotion. It's a newer driver for physical HBAs nobody puts in a VM, so it stayed in the long tail — right where a partial update could strand it.

Fixing the wrong problem correctly#

The fix is boring, which is what you want at this point:

$ sudo dnf install kernel-modules-7.1.3-200.fc44.x86_64

99.8 MiB of drivers. And the package's %posttrans scriptlet does the follow-up work for you — regenerates the initramfs and the module dependency map:

dracut -f --kver 7.1.3-200.fc44.x86_64

$ sudo depmod -a 7.1.3-200.fc44.x86_64
$ sudo modprobe mpt3sas
$

No error this time. And dmesg prints the four lines you want from an HBA coming to life, one of them with two exclamation marks:

mpt3sas version 54.100.00.00 loaded
mpt3sas 0000:05:00.0: LSISAS2008: FWVersion(20.00.07.00)
mpt3sas_cm0: sending port enable !!
mpt3sas_cm0: port enable: SUCCESS

Driver loaded. Firmware handshook. Port enable succeeded. Every line green. Run lsblk and —

Still nothing#

Still one disk. Still just sda.

The driver was loaded, the controller initialized clean, port enable: SUCCESS in the log — and not a single drive appeared. The HBA came up talking to nobody. The award for Most Confident Log Line goes to port enable: SUCCESS. It was even true.

$ dmesg | grep -i 'host_add\|phys'
mpt3sas_cm0: host_add: handle(0x0001), sas_addr(0x...), phys(8)

Eight PHYs — eight physical SAS lanes, which is right for a 2008. And every one of them:

$ cat /sys/class/sas_phy/phy-0:*/negotiated_linkrate
Unknown
Unknown
Unknown
Unknown
...

Unknown. Not a name I love. It is, though, a specific name — SAS_LINK_RATE_UNKNOWN, value 0 in the kernel's SAS enum, the state a PHY resets to before any device negotiates a link. It doesn't mean "error." It means the electrical layer looked, listened, and found nothing on the other end of the wire. Eight lanes, eight times nobody home.

And here is the thing that should have made the hair stand up: eight PHYs dead at once. We are right back at the top of the article. A whole category of devices absent, all together, cleanly — and the same logic applies a second time. Eight lanes don't go quiet independently. Something they share is the cause.

This was a second fault. An entirely separate one, and it had been hiding underneath the first the whole time. The missing kernel module was a wall; behind that wall stood a second wall, and knocking the first one down only got me far enough to see the second. Two unrelated failures — a package-management gap and a hardware-layer gap, subsystems that have nothing to do with each other — had stacked into a single symptom. Fix the driver and the box still says zero drives, because the driver was never the only thing wrong.

The second wall#

Eight PHYs, all Unknown, on a card that's otherwise healthy. That's not a firmware problem or a driver problem — those were the last problem. That's the physical layer saying no signal, and on this card the drives reach it through internal Mini-SAS fan-out cables. An SFF-8087 connector carries four lanes; the drives on phys 4–7 come in over one of them.

I had been in the HDD chamber recently — a fan reorganization, planned, in progress, my hands in a tight case full of cables. So there was a suspect, and it was me. The most boring hypothesis in hardware is a connector worked loose, and the most boring hypothesis is usually right. Reseat the SFF-8087.

I watched dmesg live during the push, and the recovery scrolled by in real time:

mpt3sas_cm0: detecting: handle(0x0009), phy(4), sas_address(0x...)
scsi: WDC WD102PURZ
scsi: WDC WD102PURZ
scsi: ST10000DM0004
scsi: TOSHIBA MG06ACA8

All four. Both WD Purple 10 TBs, the Seagate 10 TB, the Toshiba 8 TB, enumerating one after another as the lanes came back — phys 4 through 7 negotiating up to 6.0 Gbit/s, the SAS-2 rate, exactly what a drive that's actually present is supposed to report.

The letters had moved#

One more small thing, and it's the reason this ended in a shrug instead of a second panic. The device letters had shuffled across the reseat. The Seagate that had been sde came back as sdd; the Toshiba that had been sdd came back as sde. The kernel assigns /dev/sd* in discovery order, and discovery order is whatever the hardware feels like this boot — arbitrary, and free to change.

Which is exactly why nothing on this box is ever keyed to a /dev/sdX. The fstab mounts by filesystem UUID; the monitoring identifies drives by serial number. So the letter shuffle changed nothing that mattered:

$ sudo mount -a
$ df -h --output=target,size,used /media/*

Every filesystem mounted. Every byte accounted for. The data was never in danger — it had been sitting on four perfectly healthy platters the entire time, on the far side of two doors that both happened to be shut.

What actually happened, and what changes#

Strip the adrenaline off and the incident is almost embarrassingly mundane. A partial kernel update left the HBA driver uninstalled. A cable got nudged loose during unrelated work. Neither is exotic. Neither touched the data. They only looked like a disaster because they arrived together and expressed themselves identically — as the absence of every drive at once.

That absence turned out to be the most useful thing on the screen. "All four drives are gone" isn't a mystery to panic at; it's a diagnosis pointing straight at the shared dependency, and it pointed twice — once at the driver, once at the cable.

Debugging stacked faults is peeling, not revealing. You fix the thing in front of you, fully expecting it to be the fix, and sometimes it just uncovers the next one. The only defense is to keep asking the same skeptical question of each new symptom: N devices gone at once means one shared cause, not N private ones — every time, no matter how many walls deep it goes.

The lasting fix is to make both failure modes announce themselves, so the next occurrence is a phone buzz and not someone going we lost all drives? Concretely, that's four small changes:

  • Pin kernel-modules alongside every kernel install, so a partial update can't land core-without-modules again.
  • Force mpt3sas into the initramfs via a dracut force_drivers line, so the driver is present the instant the kernel comes up.
  • Autoload it through /etc/modules-load.d/, belt to the initramfs's suspenders.
  • A boot-time healthcheck that fires an ntfy alert if the HBA driver isn't loaded or the expected serial numbers aren't all present — because the two faults here were a missing module and a loose cable, and a good check catches both.

The drives were fine. They were always fine. The job now is making sure that the next time the path breaks, the box says so out loud — before anyone has to ask whether we lost all the drives.

Filed under: linux, storage, fedora, hardware

Sources#