Bash on tsvinc

My drift check said 0 differences. It checked 4 arrays.

Sat, 31 Oct 2026 00:00:00 +0000

My LAN proxy for local LLMs keeps its own catalog of how to serve each model, models.json. Each of the two GPU boxes behind it keeps the same settings again in its own models.sh, a bash script. I wrote a drift check to compare the two, and its first run said "0 differences".

I read that as a sentence about the config files. It was a sentence about my parser, which had pulled four arrays out of two bash scripts of about 500 and 600 lines, compared them, and reported back with total calm.

An exception in one layer deleted a block in another

Mon, 26 Oct 2026 00:00:00 +0000

Builds on My guard blocked bash -c. Models write bash -lc., which explains the guard's command-position rules and lists $(echo rm) -rf victim among the things a regex can't see.

At 20:01 I committed a convenience: my coding agents could now delete reverse-engineering output without asking. At 20:14 I committed a new deny rule for a command my own guard post had filed under "what a regex still can't see". In the thirteen minutes between them, one end-to-end run showed that the convenience had quietly switched off the only thing covering that command, in the one agent where anything covered it at all.

My guard blocked bash -c. Models write bash -lc.

Mon, 26 Oct 2026 00:00:00 +0000

Assumes you've run a coding agent with auto-approval on, or written a hook that vets its shell commands.

The guard looked at bash -lc 'rm -rf build' and said allow.

It denied bash -c 'rm -rf build'. It denied bash -l -c 'rm -rf build', the same two flags passed as separate arguments. Pack the l and the c into one argument and the guard waved the recursive delete through without so much as a question. The guard had applied its rule exactly as written. The rule had been written for a different spelling of the same command.

My unit tests never typed `; echo`

Mon, 26 Oct 2026 00:00:00 +0000

Builds on My guard blocked bash -c. Models write bash -lc., which explains the guard and its command-position rules.

Both guards agreed on more than 700 cases. Four coding agents then failed one end-to-end case, and the failures were right.

The guards are the TypeScript and Python halves of the guard in my agent-config templates: one shared file of regex rules that hard-denies destructive shell commands even when an agent runs with auto-approval. Both halves have their own unit tests, and the two must agree. The case that failed was git log -S 'DROP TABLE' --oneline, which searches history and destroys nothing. It has been in the end-to-end suite as a near miss, a harmless command that has to keep running, ever since a harness's own *DROP TABLE* permission glob blocked it. I had just made the SQL rule stricter, every unit case was green, and the agents now refused to run a search.

Stop docker compose from orphaning anonymous volumes

Sun, 25 Oct 2026 00:00:00 +0000

Tear a script's compose stack down with docker compose rm -fsv, then docker compose down, in one EXIT trap, so each run's anonymous volumes are deleted with their containers and the named volumes stay.

Tested on: a Fedora 44 workstation (x86_64), Docker Engine 29.9.1 with the containerd image store, Docker Compose v5.6.0, GNU bash 5.3.9, GNU coreutils 9.10 for timeout (2026-10-25), against a throwaway test stack.
Not verified: Docker Desktop, rootless Docker, Podman, other Compose versions, one-off containers from docker compose run, moving data out of an anonymous volume.
Needs: Docker Engine with the Compose plugin and permission to run docker; the script that brings your stack up and down; network access to pull alpine:3.24.2 for the test stack.
Time: 15 min.
Risk: rm -v deletes the anonymous volumes of the containers it removes, and there is no undo for that. Anything you want to keep must sit in a named volume or a bind mount before you change the teardown (checked in Before you start). Named volumes and bind mounts are not touched.

Before you start#

Docker, Compose and bash versions, and whether the test image tag is already on the box (0 = no; Undo removes the tag only in that case):

$ docker version --format '{{.Server.Version}}'
29.9.1
$ docker compose version
Docker Compose version v5.6.0
$ bash --version | head -1
GNU bash, version 5.3.9(1)-release (x86_64-redhat-linux-gnu)
$ docker image ls --filter 'reference=alpine:3.24.2' --format '{{.Repository}}:{{.Tag}}' | wc -l
0

Placeholders used below:
- <SCRIPT> — the script that runs your stack (benchmark, test run).
- <PROJECT> — the Compose project name it uses: the -p value; without -p, Compose takes COMPOSE_PROJECT_NAME, then a top-level name: in the compose file, then the name of the directory holding the compose file.
- <CONTAINER> — one container of that stack, as listed by docker compose -p <PROJECT> ps.
- <VOLUME> — the name of one anonymous volume (64 hex characters).
Back up the script: cp <SCRIPT> <SCRIPT>.bak. Edit <SCRIPT> only while no run of it is in progress: bash reads a script as it executes it.
With your stack up, run this for each container listed by docker compose -p <PROJECT> ps. Every volume row with a 64-character hex name is anonymous, and the new teardown deletes it:
```
docker inspect -f '{{.Name}}{{range .Mounts}}{{println}}  {{.Type}} {{.Name}} -> {{.Destination}}{{end}}' <CONTAINER>
```
Output for a stand-in image that declares MongoDB's two VOLUME paths, run with a named volume on /data/db only (how it was built: Field notes):
```
$ docker inspect -f '{{.Name}}{{range .Mounts}}{{println}}  {{.Type}} {{.Name}} -> {{.Destination}}{{end}}' anonvol2-db-1
/anonvol2-db-1
  volume 5f8dc2fb359c192091a7e14bd087a2251ed275b05247a9d3289028cfe992a54a -> /data/configdb
  volume anonvol2_dbdata -> /data/db
```
Anonymous volumes come from two places: a Compose volumes: entry with only a container path (- /anon), and every VOLUME path in the image that nothing is mounted onto. Current official images declare, for example: mongo VOLUME /data/db /data/configdb (7.0 to 9.0); postgres VOLUME /var/lib/postgresql/data (16, 17) and VOLUME /var/lib/postgresql (18); mysql and mariadb VOLUME /var/lib/mysql.

If a row you need is anonymous, stop here. Pointing that path at a new named volume does not move the data: in a probe on the same box (a - /data entry, and an image VOLUME path), the recreated container saw an empty directory both times, and in the - /data case the old data was still in the previous anonymous volume, now attached to nothing, which is the kind of volume docker volume prune deletes. Copy the data into a named volume first; that procedure is not covered here.

"Terminating immediately" took 90 seconds

Sat, 24 Oct 2026 00:00:00 +0000

llama-server keeps a message for the moment you run out of patience with it: Received second interrupt, terminating immediately. On the 3090 box it printed that at 09:45:04. The next thing systemd had to say about the unit came at 09:46:34, and it was Aborting.

That stop was half of an engine switch. My LAN proxy for local models stops one llama.cpp build and starts another when a request names a model that only the other build has (why it does that). The proxy logged the new engine up after 1m36s. A switch between those two builds takes 2 to 4 seconds. Two signals did this, the second one came from my own script, and the "immediately" path deadlocked inside glibc.

Every way my GPU job queue went quiet

Fri, 16 Oct 2026 00:00:00 +0000

Six days, one GPU, and every way the queue went quiet: idle with work waiting, stuck with nothing logged, or failing in words that pointed somewhere else.

A model download on the net lane: GPU idle 3 h 56 min, 3 h 32 min of it behind one 56.3 GB file at about 3 MB/s.
A llama-server left behind by the previous job: GPU lane blocked 3 h 44 min. One event, then nothing.
A REQUIRES naming a file nothing would ever write: job 84c could not run for 1 h 07 min and logged nothing. The GPU kept busy; the job just disappeared.
One model's launch flags breaking every model's smoke test: the bug lived 16 h 56 min, and three intakes failed their gate, each one looking like the model's fault.
A seed script that stops at the first bad job: every job after it is never written. Caught once by a scratch render and reproduced below; no count of what it would have dropped.
The watcher that reads the queue: froze on a hung ssh overnight. Not measured, by construction.

TL;DR — A queue that logs state changes cannot report a state that doesn't change. Give every kind of waiting an event or a watcher condition: a download that runs too long, a GPU held by a process you didn't start, a REQUIRES that has been false for hours, and nothing running at all. Make jobs kill their own servers. Render the seed before it reaches the box. Put timeouts on the thing that watches.

The benchmark was talking to the previous server

Tue, 13 Oct 2026 00:00:00 +0000

Strata v0.1.42 is an inference engine for one model family. Its loader accepts exactly one architecture, qwen4exp: Qwen3.8-Flash-Next and the models built on it. On the A9 its own setup.sh compiled the engine against ROCm 7.14 nightly wheels in about seven minutes, no root, and reused the GGUFs already sitting in ~/models. No new model download.

So far, a short and happy engine review. Then I asked it to compare its own modes, and the second mode's result was measured on the first mode's server.

Designing alerts you'll actually understand at 3am

Wed, 08 Jul 2026 00:00:00 +0000

TL;DR A digest that dumps raw SMART counters is complete and useless: re=344 means nothing at 3am. Alert on growth — compare today's counters against yesterday's row for the same serial, and push urgently only when a number moved. Seed the first sighting of a drive silently, or every new disk pages you on day one. Two channels, two intents: a calm daily digest that reassures, and an urgent push that interrupts.

Here is the notification my daily SMART logger pushed on day one. Four drives, one line each, everything the tool knew:

Four Bugs, Three Exit Codes From a Benchmark Orchestrator

Thu, 02 Jul 2026 00:00:00 +0000

The orchestrator reported ALL_DONE after one branch. The map held fourteen branches and 36 scenarios.

Twenty-four framework branches. Docker Compose up, k6 ramp to 200 VUs, Prometheus scrape, tear down, next branch. The loop script is a few dozen lines of bash — read a branch map, check out worktrees, run benchmark-all.sh, collect results. None of this is complicated, which is exactly why all four bugs took so long to find.