A null battery reading cleared the low-battery alarm

· 10 min read
Contents

Here is the first version of the observer handling a station at 12 % charge, on battery, followed by a reading where the bridge couldn't decode the charge:

=== RUN   TestProbeNullSoCClearsLB
    probe_test.go:20: soc 12 -> "OB DISCHRG LB"
    probe_test.go:25: then soc null -> "OB DISCHRG", battery.charge present=false
--- PASS: TestProbeNullSoCClearsLB (0.00s)

OB DISCHRG LB is the line that makes every machine on that tier shut down. One reading later, it was gone. A null in the bridge's JSON means "I couldn't read this field", and the observer had treated it as "nothing to worry about". With no charge in the latest reading, the low-battery check had nothing to compare against its threshold. LB dropped and battery.charge vanished from the replies. In a real outage the hosts would have been stood down while the battery kept draining.

The fix keeps the last valid charge in its own field, and the low-battery check and battery.charge now read that kept value instead of whatever the latest reading carried:

if r.SoC != nil {
	s.lastSoC, s.lastSoCAt, s.haveSoC = *r.SoC, now, true
}

A null now changes nothing. A station on battery that keeps sending readings without a valid charge for ten minutes goes to low battery by itself: not knowing the charge for that long is a reason to shut down, not a reason to wait.

NUT for six virtual UPSes#

Part 3 was the design: a Go service on the router that reads the stations over MQTT and speaks the NUT protocol itself, so every machine runs the stock upsmon as a read-only secondary. It refuses PRIMARY, FSD, SET and INSTCMD, so no host can tell its neighbours to shut down. Each station appears once per shutdown tier. With the DJI on the bridge as well, that makes six UPSes:

BEGIN LIST UPS
UPS ac180-t1 "BLUETTI AC180, tier t1, via nut-observer"
UPS ac180-t2 "BLUETTI AC180, tier t2, via nut-observer"
UPS ac180-t3 "BLUETTI AC180, tier t3, via nut-observer"
UPS dji2000-t1 "DJI Power 2000, tier t1, via nut-observer"
UPS dji2000-t2 "DJI Power 2000, tier t2, via nut-observer"
UPS dji2000-t3 "DJI Power 2000, tier t3, via nut-observer"
END LIST UPS

The null-charge bug came out of a round of probe tests written against the first working version, each one trying to make the observer say something false. It wasn't alone, and most of the others failed in the same direction: open.

Five more ways the first version could fail#

One reading after a restart. The debounce wanted two on-battery readings before declaring an outage, but only once it already knew the grid state. A fresh observer had no state, and its first reading went straight through:

    probe_test.go:13: single first reading OB soc10 -> "OB DISCHRG LB"

One bad voltage decode right after a restart would have shut down a tier. Now the first reading can only say OL, the safe direction. OB always takes two.

An interval the bridge controls. The observer judges a reading fresh if it is younger than three poll intervals, and the bridge reports its interval in every message. The interval can be changed remotely, so a bridge set to an hour stretched "fresh" to three hours:

    probe_test.go:34: interval 3600, blind 2h, bridge online -> "OB DISCHRG" fresh=true
    probe_test.go:38: interval 1e12 -> interval=-2562047h47m16.854775808s fresh=false status "OB DISCHRG"

Two hours with no data during an outage still counted as fresh, so the countdown for a silent bridge never started. A bridge setting changed over MQTT could stall the shutdown of every host on that station. The second line is Go's time.Duration, an int64 of nanoseconds that tops out at about 290 years, overflowing and wrapping negative. That one happened to fail closed, by luck. The interval is now clamped to 10–120 s, and the comparison is made on the float seconds, so an overflowed conversion can't slip under the limit:

func (s *Station) clampInterval(sec float64) time.Duration {
	d := time.Duration(sec * float64(time.Second))
	if sec > s.pol.MaxInterval.Seconds() || d > s.pol.MaxInterval {
		return s.pol.MaxInterval
	}
	if d < s.pol.MinInterval {
		return s.pol.MinInterval
	}
	return d
}

One accept error. A single transient error from accept, injected by the probe as "too many open files", and the NUT server stopped serving for good:

    probe_test.go:48: Serve returned on transient accept error: accept4: too many open files

On battery, every host would then shut down after its dead-UPS timeout, for a problem that would have cleared a second later. Accept errors now back off and retry, connections are capped at 64 with extras closed at once, and every client has read and write deadlines.

Half a line, executed. Adding those deadlines turned up the next one. The server read commands with bufio.Reader.ReadLine, whose documentation is precise about what happens when input stops mid-line: "No indication or error is given if the input ends without a final line end." A client that timed out halfway through a command had the half executed. The replacement reads with ReadSlice('\n') and treats anything short of a full line as an error:

func readLine(r *bufio.Reader) (string, error) {
	b, err := r.ReadSlice('\n')
	if errors.Is(err, bufio.ErrBufferFull) {
		return "", errTooLong
	}
	if err != nil {
		return "", err
	}
	if len(b) > maxLine+1 {
		return "", errTooLong
	}
	return strings.TrimRight(string(b), "\r\n"), nil
}

A subscription nobody checked. The MQTT client fired off its subscribe and moved on. A broker that refused it with a SUBACK of 0x80 would have left the observer connected and deaf, with nothing in the log to say why. Subscribe results are now checked and logged.

All of this is pinned in 40 unit tests, which pass under -race, and an integration run against real upsmon 2.8.5 in Docker. That run's 11 cases, all passing, include low battery, grid return, the observer killed on battery and on grid, the bridge going silent, a restart over a retained on-battery message, and a broker restart.

Two canaries, and both must go quiet#

The bridge is a single board on a power bank. If it dies mid-outage, the observer counts down blind from the last good reading. If it dies on grid, the observer needs another opinion on whether the grid is still there. Part 3 left open which device that would be. It became two devices on wall power, neither behind a station: a Xiaomi lamp that answers a TCP connect, and a smart plug that reports mains voltage over HTTP. The plug counts as up only if it reads 180 V or more, so a brownout counts as an outage.

The grid counts as down only when both go quiet. A restart of the plug, on the morning of the test below, shows why:

2026/10/16 06:48:55.791531 canary <plug>:80: down (Get "http://<plug>:80/cm?cmnd=STATUS%208": context deadline exceeded)
2026/10/16 06:49:22.842538 canary <plug>:80: up (242.6 V)

The plug was marked down for 27 seconds, and off the network for longer than that, since a canary only flips after three equal results. The bridge was reporting the whole time, so the canaries were only a second opinion and nothing changed. Had the bridge been silent too, a plug-only rule would have declared an outage for a restart. With the lamp still answering, the pair would have held the grid up. Either device alone turns a Wi-Fi hiccup into a false outage on exactly the day the bridge is down.

Deploying on the router#

The observer runs on the MikroTik that hosts the MQTT broker, which is on backup power itself. RouterOS runs third-party code only as containers, so the observer ships as a static arm64 binary in a FROM scratch image, written by docker buildx build --output type=docker as a docker-save tar and copied to the router's SSD. I built the image with uncompressed layers rather than find out whether RouterOS imports gzipped ones. The MQTT and NUT passwords go into a container envlist, never into flags or a file on the SSD. After deployment the container used 7.7 MiB of RAM.

One RouterOS default nearly undid the point of the exercise. Both containers, the observer and the broker, came up with restart-policy=no. A crash would have left them down until the next router reboot, and the hosts would only log a comms warning that nobody reads. Both now restart after 10 s. That covers a crash, not an observer that is stopped by hand or wedged in some new way; an alert for those is still missing. The rest of the RouterOS container quirks are in their own post.

Stock upsmon on the hosts#

Each host gets three lines that matter in upsmon.conf, written by an install script:

MONITOR $UPS@$OBSERVER 1 $NUT_USER $NUT_PASS secondary
MINSUPPLIES 1
SHUTDOWNCMD "/usr/sbin/shutdown -h +0"

DEADTIME 300 sits below them. A router reboot takes longer than the 15 s the integration tests use, and on battery a lost observer means shutdown once DEADTIME runs out.

Fedora had one trap of its own. Enabling nut-monitor.service isn't enough to start it at boot, because of its install section:

[Install]
WantedBy=nut.target

Nothing pulls in nut.target unless you enable that too. A host that looks configured and active after the install would boot unprotected next time. The script enables both.

The first outage was a test#

The AC180's wall input now runs through that smart plug, which can cut its own relay when the voltage goes out of range. Lowering its over-voltage limit below the mains voltage makes it trip on purpose. The relay opened a few seconds after 06:46:28 UTC and closed again at 06:47:05. With the bridge reporting every 20 s and a two-reading debounce, the observer logged:

2026/10/16 06:47:00.072448 station ac180: OL -> OB
2026/10/16 06:48:00.981834 station ac180: OB -> OL

And upsmon on my workstation, two to three seconds behind each time:

2026-10-16T09:47:03+03:00 fedora nut-monitor[3121592]: UPS ac180-t2@<router> on battery
2026-10-16T09:48:03+03:00 fedora nut-monitor[3121592]: UPS ac180-t2@<router> on line power

The charge went from 100 % to 99 %, nowhere near that tier's 35 % or 20-minute limits, so nothing else happened, which is the correct amount of drama. Grid return took three readings, just under a minute after the relay closed. That is the debounce doing its job: a grid that flickers back for one reading doesn't count.

A cut earlier that morning, the plug script's first test and of a similar length, never moved the observer off OL at all. Whether a half-minute outage registers depends on where the 20-second readings fall, since two in a row are needed. Blips that short are meant to stay out of the shutdown path either way.

The observer is public at gitlab.com/random-home-utils/nut-observer under MIT, with generic example addresses and a README covering the config, the tiers and the RouterOS container setup.

Filed under: go, ups, power, nut, mqtt, mikrotik

Sources#