Designing alerts you'll actually understand at 3am

· 6 min read
Contents
TL;DR A digest that dumps raw SMART counters is complete and useless: re=344 means nothing at 3am. Alert on growth — compare today's counters against yesterday's row for the same serial, and push urgently only when a number moved. Seed the first sighting of a drive silently, or every new disk pages you on day one. Two channels, two intents: a calm daily digest that reassures, and an urgent push that interrupts.

Here is the notification my daily SMART logger pushed on day one. Four drives, one line each, everything the tool knew:

sdb VXA1EXMP 38C poh=28805h re=0 pend=0 unc=0
sdc VXA2EXMP 39C poh=28798h re=0 pend=0 unc=0
sdd 62K0B0EXAMPL 38C poh=54631h re=0 pend=160 unc=126
sde ZA4EXMP7 38C poh=24745h re=344 pend=0 unc=4003

Technically complete. Every counter that matters is in there. I read it once and knew I would never remember what to look for.

That's the whole bug. Is re=344 bad? Is unc=4003 an emergency or ancient history? Is unc=126 on sdd worse than unc=4003 on sde? To answer, you have to remember that Reallocated_Sector_Ct being high might be fine but Current_Pending_Sector being anything is not, and that read-error counts on Seagates read differently than on everyone else. At 3am, woken by a phone buzz, you will remember none of this. The alert fails at its one job: telling a half-asleep human whether to get out of bed. It answers what are the numbers? instead, a question nobody has ever asked at 3am.

The redesign#

An alert's audience is a tired human, not a parser. So I stopped emitting facts and started emitting verdicts. Same data, same drives, next morning:

✅ WD Purple 9.1T (sdb) — healthy
   38°C · 1200 days powered on
   no bad sectors

⚠️ Seagate 9.1T (sde) — worn, not worsening
   38°C · 1031 days powered on
   344 remapped, 4003 read-errors (flat for months)

🔴 Toshiba 7.3T (sdd) — GETTING WORSE
   pending sectors 160 → 172 since yesterday
   copy data off, plan replacement

Every line now carries three things the raw version made you supply yourself: a verdict, plain words, and an action. re=344 became "344 remapped ... flat for months." pend climbing became "GETTING WORSE ... copy data off." You don't recall a threshold; the message already applied it.

And the part that matters most at 3am is the part you see before you open anything — the notification title:

  • ✅ host: all drives healthy
  • ⚠️ host: all stable (some worn)
  • 🔴 host: a drive is getting worse

Green title, roll over and go back to sleep. Red title, sit up. One line legend at the bottom of the body for the one time you forget what the colors mean: ✅ healthy · ⚠️ worn but not worsening · 🔴 worsening = act.

Alert on the slope, not the value#

The second principle is why the Seagate's 344 is a shrug and the Toshiba's 160 → 172 is a siren. For a spinning disk the delta tells you more than the absolute count. A reallocated count of 344 that hasn't moved in a year is old, settled damage — the drive hit some bad sectors once, remapped them, and has run clean since. A pending count that grew since yesterday is a defect happening right now, and the rate almost always accelerates from there.

So the system keeps history. A systemd timer runs the logger once a day and writes one row per drive into sqlite, keyed by serial — because /dev/sde is a lie that changes when you recable, but ZA4EXMP7 is forever:

CREATE TABLE smart (
  ts INTEGER, serial TEXT, model TEXT,
  poh INTEGER, temp INTEGER,
  realloc INTEGER, pending INTEGER,
  uncorr INTEGER, cmdto INTEGER, reported INTEGER,
  PRIMARY KEY (ts, serial)
);

Each run compares today's counters against yesterday's row for the same serial. A high but flat number is history and stays in the calm daily digest. A number that grew triggers a separate push at urgent priority. Two channels, two intents: the digest reassures, the urgent one interrupts.

Delivery is one curl. No SMTP, no Gmail relay, no sendmail that silently drops mail — ntfy turns a POST into a phone buzz:

curl \
  -H "Title: 🔴 host: a drive is getting worse" \
  -H "Priority: urgent" \
  -H "Tags: rotating_light" \
  -d "$body" \
  https://ntfy.sh/hddbox-smart-XXXXXXXXXXXX

smartd still runs alongside this for live, event-driven failures — it polls every 30 minutes and shouts the instant a self-test fails. But smartd doesn't do "here is a calm sentence about each drive every morning." That's what the timer is for.

The gotcha: don't scream about wear you were born with#

The first version of the delta check was proud and wrong. "Did the counter grow since last time?" — but on the very first run there is no last time, so it compared against zero. Result: sde's baked-in 344 remapped and 4003 read-errors looked like they'd all appeared "since yesterday," and the tool declared a healthy old drive DEGRADED on its first breath. It had known the drive for one run and already had concerns.

The fix is one rule: the first time you see a serial, seed it silently. Write the baseline row, emit nothing. Only start comparing once a baseline exists.

prior=$(sqlite3 "$db" \
  "SELECT pending FROM smart WHERE serial='$serial'
   ORDER BY ts DESC LIMIT 1")

if [ -z "$prior" ]; then
  :                       # first sighting — seed, no alert
elif [ "$pending" -gt "$prior" ]; then
  alert "$serial" "$prior" "$pending"
fi

The nice side effect: when you add a fifth drive next year, it joins the same way — one quiet baseline row, no 3am false alarm about wear it came with.

The takeaway#

The counter that matters is the one that changed, and the alert that works is the one that already did the thinking for you. Everything else is a puzzle you've handed to the worst possible version of yourself: tired, half-awake, every threshold forgotten. Write the notification for that person. If reading it requires remembering which SMART attribute is load-bearing, you built a log line and called it an alert.

The firmware-level mechanism underneath this — why a high count can be harmless but a moving one is fatal, and what the drive actually does on each write — is its own post: Reallocated sectors: is your drive dying or just scarred?.

Filed under: linux, monitoring, smart, bash

Sources#