The alert about the blackout needs the internet too

· 14 min read
Contents

Part 5 left the observer able to shut machines down and unable to tell me anything, with one item still open: an alert for an observer that stops. The alerts looked like one function. Log the line, and if a notifier is configured, hand it over:

func alertf(format string, a ...any) {
	msg := fmt.Sprintf(format, a...)
	log.Print(msg)
	if alerts != nil {
		alerts.Send(msg)
	}
}

That function hasn't changed since. The file around it, notify.go, is now 217 lines, and its tests are another 197. Most of it comes from one inconvenient fact: a message about a power cut leaves through an internet uplink, and the cut can take the uplink with it. I can keep the router running on battery. I can't vouch for anything upstream of it.

Every alert channel needs something the outage can take#

Here is what each way out depends on:

ChannelNeedsStill works whenFails when
ntfy and Telegram, from the observerobserver, router, internetthe grid goes: the router is on backup powerthe uplink is down, hence the retries
Watchdog script on the routerrouter, internetthe observer stopsthe router stops; it also doesn't retry
LED on the smart plugmains at the plug, observerthe bridge goes silent, a station is unpluggedthe grid goes: the LED is dark
upsmon on each hostrouter, LANthe internet is downthe router stops: shutdown after DEADTIME if the UPS was last seen on battery

The last row is the shutdown path from part 3. The other three are for me, and each has its hole in a different place.

The observer sends its own start, grid lost and grid back per station, a bridge that goes silent while on grid and comes back, a tier hitting LOW BATTERY (its hosts are shutting down), and that tier's latch clearing. ntfy priority comes from the text: 5 for low battery, 4 for on battery or a silent bridge, 3 for everything else.

Retry until delivered, give up only on a 4xx#

An alert about a grid cut goes out at exactly the moment the uplink is most at risk. A sender that tries three times and gives up has thrown away the one message that mattered. So each message retries after 5 s, 30 s and 2 min, then every 5 min for as long as it takes:

retry:  []time.Duration{5 * time.Second, 30 * time.Second, 2 * time.Minute, 5 * time.Minute},

wait := n.retry[min(i, len(n.retry)-1)]

Forever has one exception. A revoked bot token earns the same refusal on every attempt, and retrying it every five minutes until the heat death of the router helps nobody. A 4xx gives up, except 429, which means "later" rather than "no":

if resp.StatusCode/100 == 4 && resp.StatusCode != http.StatusTooManyRequests {
	return permanentError{err}
}

Callers never wait: Send drops the message into each channel's queue of 64, and a full queue drops it with a log line. Failures are logged during the backoff steps, then only every tenth.

Retrying forever blocked the second channel#

Retry-forever was the right call, and the first version put it in the worst possible place. One queue, one sender, both channels in a row:

func (n *Notifier) Run(ctx context.Context) {
	for {
		select {
		case <-ctx.Done():
			return
		case msg := <-n.ch:
			if n.NtfyURL != "" {
				n.deliver(ctx, "ntfy", func() error { return n.ntfy(ctx, msg) })
			}
			if n.TGToken != "" {
				n.deliver(ctx, "telegram", func() error { return n.telegram(ctx, msg) })
			}
		}
	}
}

deliver keeps retrying until the message is out, and returns early only on a permanent refusal or shutdown. With ntfy unreachable and Telegram fine, the first alert would sit in ntfy's retry loop for as long as ntfy stayed down. Telegram would get nothing, and every later alert would queue behind the first. The second channel existed for the case where the first one fails, and that was precisely the case where it went quiet.

It turned up on a read-through of the code, not in an outage. Now each channel gets its own queue and its own goroutine:

type sink struct {
	name string
	send func(context.Context, string) error
	ch   chan string
}

A test pins it: an ntfy server that accepts the connection and never answers, and a Telegram fake that must still receive both messages.

The event time goes inside the message#

A message retried for an hour arrives an hour late, and the phone shows when it arrived. So Send stamps the time as it queues the message:

msg = time.Now().UTC().Format("15:04 UTC ") + msg

The resolution is a minute, and the time is when the observer decided, not when the grid went. Part 5's two-reading debounce sits between the two.

The Telegram bot token is part of the URL#

Telegram's Bot API takes the token in the path: https://api.telegram.org/bot<token>/METHOD_NAME. Go's http.Client returns every error as a *url.Error, and that type prints the method and the full URL in front of the cause. The first failed DNS lookup during an outage would write the bot token into the router log, which is exactly the kind of file that ends up pasted into a blog post.

So every logged error goes through redact first:

for _, secret := range []string{n.TGToken, n.NtfyToken, n.NtfyURL} {
	if secret != "" {
		s = strings.ReplaceAll(s, secret, "<redacted>")
	}
}

The ntfy topic URL is on the list because ntfy's own docs say it plainly: with no sign-up, "the topic is essentially a password". The test feeds deliver two failures shaped like a real Telegram url.Error, lets the third attempt succeed, and checks that redact strips the token from that kind of string.

CA roots for a FROM scratch image#

The observer ships in a FROM scratch image: no OS, so no CA bundle for HTTPS to ntfy and Telegram. A blank import embeds a set of roots in the binary and registers them as the fallback for a system that has none:

_ "golang.org/x/crypto/x509roots/fallback"

Its docs ask for binaries only and regular updates: a root set compiled into a binary ages with the binary.

The observer can't report its own death#

Everything above runs inside the observer, so it stops when the observer does. That job went to the router: a RouterOS script on a one-minute scheduler that checks the two containers the setup depends on, the observer and the MQTT broker, and sends one message whenever the set of stopped ones changes:

:foreach n in={"observer";"mosquitto"} do={
  :local id [/container/find name=$n]
  :if ([:len $id]=0) do={:set down ($down . $n . " missing; ")} else={
    :if ([/container/get $id running]!=true) do={:set down ($down . $n . " not running; ")}
  }
}
:if ($down != $upsWdState) do={
  :set upsMsg "router: observer and mosquitto running again"
  :if ($down != "") do={:set upsMsg ("router: " . $down)}
  /system/script/run ups-notify
  :set upsWdState $down
}

It stays quiet for the first three minutes after boot, while containers are still starting. It reads the ntfy and Telegram settings from the observer's container envlist, so the script file holds no secrets. An early run of the installer checked RouterOS's dry-run import for anything matching syntax error, and stopped on this:

No syntax errors found in the import file
ABORT: dry-run found errors

It now checks for the success line instead.

The watchdog is the weaker sender, and stays that way for now: one /tool/fetch per channel, an on-error that only logs, and the state updated either way. A container that stops while the uplink is down produces a message nobody receives, and the next one comes only when the set changes again. The install sent a test message through the same notify script, with no fetch failure in the router log. I haven't stopped a container yet to watch the real path fire. Two short ones went unreported, as designed: the observer was re-created twice that day, under ten seconds each, both between two of the scheduler's once-a-minute checks. If the router itself dies, nothing reports it: the watchdog and the observer go with it, and hosts on battery shut down once DEADTIME runs out.

The status LED goes dark in exactly this outage#

The observer also drives the LED on the smart plug from the OpenBeken post. The worst state across all six UPSes wins:

func ledMode(snaps map[string]Snapshot) int {
	mode := LEDSteady
	for _, sn := range snaps {
		switch {
		case strings.HasSuffix(sn.Status, " LB"):
			return LEDFastBlink
		case strings.HasPrefix(sn.Status, "OB"):
			mode = LEDPulse
		case (sn.Status == "" || !sn.Fresh) && mode == LEDSteady:
			mode = LEDSlowBlink
		}
	}
	return mode
}

Steady means all fine on wall power, pulse a station on battery, fast blink a tier at low battery, slow blink a lost bridge. The mode is re-sent every 60 s, because the plug resets it to steady when it reboots.

That re-send made the first version a two-writer bug. The plug's voltage guard set fast blink when it tripped the relay and steady when it restored it. Fast blink also meant low battery in the observer's table, and within a minute the observer would have painted over the trip with its own opinion. This was caught before it was deployed. The guard no longer touches the LED, and its trip now has a pattern of its own that the observer can't override (the plug post has the script).

The plug post already admits the catch: the plug runs on mains, so in a real blackout the LED is off and pulse and fast blink never appear. In a real outage, low battery reaches me through ntfy and Telegram or not at all. A status light needs no internet. This one needs the grid instead.

Pulling the relay with the alerts on#

At 11:35:09 UTC I set the plug's upper voltage limit to 200 V, below the mains, so the guard opened the relay. Its channels, polled every 10 s, showed the relay open and the trip flag set at 11:35:20. The AC180 went to battery. The observer, in its own UTC timestamps:

2026/10/16 11:35:49.324546 station ac180: OL -> OB
2026/10/16 11:35:49.324837 station ac180: grid lost, on battery
2026/10/16 11:37:29.434648 station ac180: OB -> OL
2026/10/16 11:37:29.435290 station ac180: grid back, on line power

"Grid lost" was logged 40 s after I forced the trip: the bridge's 20 s readings plus the two-reading debounce. I put the limit back at 11:36:11. The first poll that saw the relay closed again was 11:36:41, since the guard waits 30 s in range before restoring, and "grid back" came 48 s after that. Telegram showed the two messages at 2:35 PM and 2:37 PM local time, the first one reading nut-observer: 11:35 UTC station ac180: grid lost, on battery. No notify lines in the log for that window: nothing needed a retry. The test ran on the build before the per-channel queues; with the uplink up, the difference can't show.

Two seconds after "grid lost", the observer switched the LED to its on-battery pattern (on the LED map of that day). The plug's trip flag outranks anything the observer sends, so the plug's script showed its own trip pattern instead. The one outage I can stage on demand is the one where the LED is busy saying something else.

The uplink stayed up the whole time. The retry path has run in a unit test, nineteen failures and then a delivery against a simulated unreachable network, and not yet against a real dead uplink.

Update, 17–18 October: a relay off that passed for a blackout#

All times below are UTC, as the observer logs them, unless marked local (UTC+3). Log excerpts are observer lines, filtered; the plug's address is swapped for a placeholder.

On 17 October the alerts got an outage that wasn't one:

2026/10/17 06:06:36.250848 station ac180: grid lost, on battery
2026/10/17 06:06:37.448183 led 192.0.2.28:80: mode 5
2026/10/17 06:11:36.397330 ups ac180-t1: LOW BATTERY, its hosts shut down (on battery 5m0s)
2026/10/17 06:11:37.455903 led 192.0.2.28:80: mode 4

Tier t1 also fires on time alone: five minutes on battery, whatever the charge. So at 06:11 it declared low battery and the LED went to fast blink. About two hours later I noticed the blinking and wondered whether that was intended. It was. The grid, however, was fine.

The plug's relay was off. Its meter still read normal mains, the guard's trip flag on channel 12 was 0, and its uptime showed no reboot. The guard keeps that flag set until its own restore, so it wasn't the guard. At the time the observer only sent the plug STATUS 8 reads and LED modes; nothing in that set touches the relay. The command history had nothing near 06:06. What's left is a ghost press of the plug's button, where a short press toggles the relay. Nothing proves it. The cause is still unknown.

I switched the relay back on over HTTP at 08:15 and misread the next four minutes. Right after, I opened the BLUETTI app to check the battery, and the AC180 accepts one BLE central, so the bridge logged bridge poll failed: station_not_found from 08:16:03 to 08:19:03 while NUT kept serving the last on-battery reading. Relay on, still no input: I took it for a dead outlet and moved the AC180 to another wall socket to charge. The first fresh read, at 08:19:30, said grid back. Those minutes proved nothing about the outlet, because the bridge never read the station during them. The next day the AC180 charged through the plug again, outlet and all.

The LED told the truth. The gap was that a relay switched off under good mains reads the same as an outage, and nothing said why.

The plug now undoes it and keeps count#

prot.bat got one more branch. Relay off without a trip for 10 s: relay back on, and channel 14 counts it since boot:

relayoff:
	addChannel 15 1
	if $CH15<10 then goto watch
	setChannel 1 1
	addChannel 14 1
	setChannel 15 0
	goto watch

The test, polled about once a second, local time:

12:44:35 {"POWER":"OFF"} ch15=9 ch14=0
12:44:36 {"POWER":"ON"} ch15=0 ch14=1

The price: a manual POWER OFF no longer sticks. Keeping the relay off on purpose now means stopScript 2 first, and a reboot starts the script again.

The observer reads POWER and GetChannel 14 after each 10 s voltage probe. It alerts on relay off (ntfy priority 4), on relay back on, and on any rise in the counter. A plug reboot resets the counter, which becomes a silent new baseline. The live test, a few minutes after the plug test:

2026/10/17 09:47:01.089589 plug 192.0.2.28:80: relay OFF (229.8 V at the plug)
2026/10/17 09:47:11.085469 plug 192.0.2.28:80: relay ON again
2026/10/17 09:47:11.137104 plug 192.0.2.28:80: relay was switched off without a voltage trip; the plug turned it back on (2 since plug boot)

All three reached Telegram and ntfy. The 2 is the 12:44 test plus this one. A relay-off shorter than the poll can slip past the OFF alert, but not past the counter: the restore still raises it, and that alert fires.

The next morning, a real trip#

At 06:43 on 18 October the guard tripped for real, on over-voltage:

2026/10/18 06:43:22.108346 station ac180: grid lost, on battery
2026/10/18 06:43:31.100815 plug 192.0.2.28:80: relay OFF (255.0 V at the plug)
2026/10/18 06:47:41.106194 plug 192.0.2.28:80: relay ON again
2026/10/18 06:48:22.975845 ups ac180-t1: LOW BATTERY, its hosts shut down (on battery 5m0s)
2026/10/18 06:48:42.378425 station ac180: grid back, on line power
2026/10/18 06:53:42.975802 station ac180: grid back for 5m0s, tier LB cleared

The limit is 253 V. There's no "without a voltage trip" line, correctly: this one was the guard's own trip. It restores only after 30 s inside 185–245 V, so the mains didn't hold inside that window for 30 s until about 06:47. "Grid back" arrived twenty seconds after tier t1 had already called low battery. This time, though, the alert said why.

Filed under: go, ups, power, nut, mikrotik, monitoring

Sources#