Pulling plugs on purpose

· 7 min read
Contents

A test that passes tells you the thing works. That is the reading everyone defaults to, and the first power-bank test of the ESP32 bridge was a clean example of why it's wrong.

The bridge from the second UPS post runs off a USB power bank, which charges from the wall. The worry was simple: when the grid goes and the power bank stops charging, does it hold its output steady, or does the board reboot right at the start of the outage it is supposed to report? I cut the power bank's charging, and the board's uptime counter kept going, from 14160 seconds to 14180. No reboot.

It proved nothing. The board was also plugged into the PC's USB port, the cable used to flash it. Any dip from the power bank would have been covered by the PC. For the result to mean anything, the cable had to come out first.

Watching a reboot without a cable#

Pulling the USB cable also removes the obvious way to watch the board: its serial console. The test used MQTT instead. A subscriber on the PC, logged in as the read-only user, followed the bridge's topics through the broker on the router. Two fields carried the answer:

  • uptime_s in every state message. It counts up by 20 seconds per poll. If it drops to a small number, the board restarted.
  • reset_reason in the boot line the bridge sends to its log topic. 1 is ESP-IDF's code for a power-on event, as opposed to a watchdog or a software restart.

Neither depends on the cable under test, so unplugging it changes nothing about what can be seen.

The clean test: both transitions reboot#

I unplugged the PC cable first; that alone caused no reboot. With the power bank as the only supply, I turned its charging back on:

2026-10-14T19:03:46+0000 ups/ac180/bridge offline
2026-10-14T19:03:46+0000 ups/ac180/bridge online
[…]
2026-10-14T19:03:46+0000 ups/ac180/log boot: fw b1095de-dirty reset_reason=1 target=AC180 interval=20s

The last reading before it said 14240 seconds; the first one after it said 12. Restoring the charging had cut the output long enough for a power-on reset. Less than a minute later I pulled the charger, and the log at 19:04:23 showed the same: offline, online, uptime 12, reset_reason=1. That second case is the outage itself. The power bank reboots the board at the exact moment the board is needed. I looked at what that costs, a reboot and about twelve seconds without data, and accepted the gap.

I tried a second power bank a few minutes later. It rebooted the board when I plugged its charger in. I never pulled that one's charger. Of the four reboots whose first poll made it into a log, that was the only one where the poll failed:

2026-10-14T19:09:32+0000 ups/ac180/bridge offline
2026-10-14T19:09:32+0000 ups/ac180/bridge online
2026-10-14T19:09:32+0000 ups/ac180/state {"ok":false,"error":"handshake_timeout","ts":null,"interval_s":20,"uptime_s":24}
2026-10-14T19:09:32+0000 ups/ac180/log boot: fw b1095de-dirty reset_reason=1 target=AC180 interval=20s
2026-10-14T19:09:32+0000 ups/ac180/log write failed
2026-10-14T19:09:32+0000 ups/ac180/log write failed
2026-10-14T19:09:32+0000 ups/ac180/log poll failed: handshake_timeout

The first poll after the reboot failed. Both writes to the station failed and the handshake timed out. The next poll, twelve seconds later, read normally. The likely explanation is that the AC180 was still holding the dead board's connection until its own timeout ran out, but nothing in the log proves it. One reboot in four shows the failure can happen; it doesn't show how often, or why.

The first test, the one that proved nothing about the power bank, turned out to be useful after all. With the PC cable connected, cutting the charging caused no reboot: a valid test of the two supplies together. When I put the PC cable back in later, uptime went from 92 to 112 to 132 seconds without a break.

That is the setup that stayed: power bank on one port, PC on the other. In an outage the PC keeps running off the AC180, so its USB port covers the power bank's dip. When the PC later shuts down on its own schedule, the board is already on the power bank and doesn't notice. The one remaining gap is an outage that begins while the PC is already off. That case costs a reboot and about twelve seconds, sometimes a failed poll on top.

When the evidence is what's missing#

The broker's access list was tested before it went anywhere near the router. A local copy of Mosquitto ran in a container on the PC, with the real config, the real access list and the real users. Then six messages were published:

  • the admin user, the bridge and the read-only user each published to the command topic;
  • the bridge and the read-only user each published a state message;
  • the bridge published to an unrelated topic.

Two subscribers listened, the bridge on its command topic and the read-only user on everything under ups/:

== bridge sees on cmd:
ups/ac180/cmd from-admin
== reader sees on ups/#:
ups/ac180/cmd from-admin
ups/ac180/state state-from-bridge

Two of the six arrived. Three of the missing four are the result: the read-only user's command, its fake state message, and the bridge's attempt to command itself would each have reached a subscriber if the broker had let them through. The sixth proves nothing. Neither subscriber listened outside ups/, so the bridge's write to an unrelated topic couldn't have shown up either way. A test like this only works if the person reading it knows what was sent, and what could have been seen.

On the router, later, the same check ran against the live broker: a publish to the command topic from the read-only user went nowhere.

Tests that say what they can't prove#

The earlier tests in this project were written the same way, and the UPS posts kept their limits in the text.

The first UPS post pulled the AC180's wall plug while the workstation logging it ran off the station, and the switchover held. It also missed a brief replug that fell between two reads. It can't tell whether a blip that short falls between samples or never gets reported by the station at all.

The bridge post's last-will test unplugged the bridge and timed the broker's offline: 137 seconds after the last packet. The faster timestamp rule appears there as arithmetic, because no consumer was running yet to observe it.

The test nobody ran#

No test drained the AC180. A battery that has been run to the floor for a test is a battery that isn't there if the grid goes an hour later.

The plan for the shutdown side works on the grid instead. The failure-mode part of the design has a fault-injection plan where a simulated feed stands in for the bridge, on its own topic or on a local broker like the access-list test. Every host's shutdown command is swapped for a log line that says it would have shut down. Tiers can then be walked through every threshold, and the observer can be frozen mid-countdown, or the feed and the broker stopped under it, without anything turning off. The only real discharge in the plan is optional: a shallow one, from 80 % or above, to calibrate the runtime estimate.

When this was written, none of it had run, because the thing it tests didn't exist yet. It was written down first, so that the first shutdown the logic ever orders ends up in a log file and nothing in the house turns off.

Filed under: testing, esp32, mqtt, ups

Sources#