I kept measuring the thing next door

· 8 min read
Contents

Most of the signals I misread during two days of disk cleanup were accurate. They described the thing next door.

The cleanup was destructive on purpose: about seventy stale projects archived, Docker cleared out, btrfs measured, the identical parts of a stale copy of another machine's files deleted.

The short version first. The loop held: every archive was test-restored and compared before its original went. One thing was lost for real, two test fixtures, and they were rebuilt from a README.

What's worth writing down is who caught what. My own mistakes were mostly about the present: the disk as it was. A review of the cleanup plan caught a different kind entirely. And one mistake nobody could have measured.

Workflow: every deletion as a list of literal paths#

The loop kept its shape while rules were added to it. Inventory, read-only. Stop at anything irreplaceable and decide it by hand. Build the archive. Restore it into scratch and compare every entry. Delete. Record what happened. Every destructive step was printed with its exact paths, and read, before it ran.

Part of that was enforced for me. The tool I ran the commands through refuses rm -rf $(…), because it can't resolve what the substitution will expand to. So deletions became a printed list of literal paths, which is also the only form anyone can check by reading it.

What the loop did was make each wrong reading visible, usually before anything was gone.

A dormant Go proxy, misread in both directions#

The best specimen is a small API-capture proxy written in Go. Its source hadn't changed in 98 days, so I archived it with a batch of other stale projects.

It wasn't only source. Its installed binary was still on disk, along with a disabled user systemd unit and a 5 GB data directory. Faced with those, I swung the other way and declared the tool still in use. My evidence: the SQLite database's -shm and -wal side files carried yesterday's mtime. SQLite keeps both files while a connection has the database open and removes them on a clean last close, so their presence said the database had been opened and was either still open or never cleanly closed. The WAL was 0 bytes, which fits an open with nothing written. I took the claim back.

One tool, two readings in opposite directions, two timestamps that were both accurate. The source really was 98 days old. The side files really were touched yesterday. Neither answered the question, which was whether anything still depended on the tool. The places that answer that are ~/.local/bin, the user units, the data directories and the shell aliases.

The resolution was boring on purpose: restore the source, stop the service cleanly, archive source, config, unit and binary together, archive the data as a second archive, then remove all of it. There was one scare on the way. When the proxy was started again, its data directory shrank from 5.1 GB to 2.0 GB. That was the proxy's own seven-day rule: it packs idle sessions with tar and zstd, writes the archive, commits the index, and only then deletes the loose files. An old record still loaded through its API.

Wrong readings: the thing next door#

Three more of my mistakes have the same shape:

QuestionWhat got readWhat it actually measures
Was this image pulled more than 30 days ago?Createdwhen someone upstream built it
Is this directory empty?absent from the restorethat nothing in it was archived
Did those fonts get deleted?a shell wrapper's du outputwhat the wrapper chose to print

Every middle column was true. The images really were built more than 30 days earlier, and 71 of the 93 removed in that pass had been pulled within the month. The directory really was missing from the restore, because everything in it had been excluded, and that cost the only copies of two test fixtures. The wrapper really did print fewer lines; it had truncated the listing, and eight font families looked deleted until a plain Python listing showed them all there.

Some of the rules I worked by came out of near-misses of a similar shape. Ranking stale projects by atime was ruled out because this desktop's file indexer reads everything, and under relatime, the kernel default, any read refreshes an atime that is more than a day old. The archives also carry each project's session logs, which my tools file in directories named after the working directory: its path, with non-alphanumerics turned into -. So a log is matched to a project by the working directory recorded inside it, never by that name, because a project app prefix-matches its sibling app_1. One gotcha, not a rule, has the shape too. When I compared the stale copy against its source, my first script to classify rsync's output looked for > markers. In push direction rsync writes <, and the script counted zero differences.

Fixes: a second signal that can disagree#

It would be neat to say every fix replaced a proxy with the thing itself. "Needed", "in use" and "stale" have no field to read.

Metadata.LastTagTime is the local pull or tag time, a better clock than Created and still only a stand-in for "a project needs this". Docker keeps no last-used history, so I started keeping one. What came out of the cleanup is a weekly Docker tidy, a user-level timer meant to remove only regenerable things, with a companion service that logs every container start. Newest mtime outside .git, half of the replacement for atime, is the same family of signal as the 98 days that misjudged the proxy; the other half is the newest session log for that directory. What made those safe was a second, independent signal that could disagree with the first. A pull time plus references in project files. A truncated du listing plus a re-listing in Python. Six unknown Postgres volumes, where about 7.5 MB per database said empty and strings on the catalog named the databases. 12.4 GB of apparent duplicates, plus compsize per set to find the 3.4 GB that was real.

Four Green Signals, Four Lies ended on opening the artifact a tool claims to have produced. Here there was often no artifact to open, only a second witness to call.

Reviewing the cleanup plan: later and elsewhere#

A review of the cleanup plan raised notes I had missed:

  • one git-only archive depended on a hosting account that won't be around much longer. It was rebuilt as a full archive;
  • upstream clones had been swapped for re-clone lists (path, URL, branch, commit) before I made it a rule to archive any clone whose upstream is gone. Those early lists had never been rechecked. They were, and every upstream was still alive;
  • the weekly Docker tidy would have deleted three stopped MQTT watcher containers, with their captured telemetry, on its first Sunday;
  • the temporary SSH key used to compare against an older laptop of mine, the source of that stale copy, was still installed. Left there, it would have kept the workstation's passwordless access to that laptop open indefinitely. It was removed, and a fresh login was checked to fail.

Every one of those notes is about later or elsewhere: what depends on a remote, what runs next Sunday, what access outlives the job. I was busy reading the disk as it was. The review asked what happens next.

Off-limits projects: the mistake nobody could measure#

Two projects were off-limits: an active benchmark repo with a job queue, and an ongoing hardware review. By the time that was on record, I had already pushed a teardown fix to one and deduplicated files in the other.

No signal on disk would have caught that. The boundary lived in my head and nowhere else. The fix was to write decisions down where they are read first. The cleanup project's instructions file now lists the off-limits projects, the candidates I declined, and the other choices already settled, so the next session doesn't reopen them. The long archive history moved into its own file, which took the instructions from 123 lines to 83. That file is still only something read at the start. It blocks nothing by itself; a rule that has to hold no matter what belongs in a hook that refuses the command.

I was never short of a command that could check what I claimed about the disk. Each time I misread the disk, I had read one witness and stopped. Destructive work wants two that can disagree. The facts that aren't on the disk at all have to be written down before it starts.

Filed under: linux, docker

Sources#