A directory-only diff is not an empty directory

· 6 min read
Contents

Archiving a dormant project should be the dull end of disk cleanup. Pack it, restore the pack somewhere else, compare the restore with the original, and only then delete the original. Over two days I built about seventy of those archives on my workstation, and no original was deleted without my explicit yes.

One project came back without its test data. The compare that exists to catch exactly that caught it, printed it, and was read as nothing.

The archive loop, and the one knob that mattered#

Each archive is a <name>.tar.xz, a raw/ folder for blobs that are already compressed, a SHA256SUMS, and a generated restore-<name>.sh that checks the sums and refuses to overwrite anything. Verification restores into a scratch directory with a fake HOME and compares every entry by type, permission bits and SHA-256. Right before a deletion, it runs again.

The knob that mattered was the exclude list. Not everything in a project folder deserves cold storage, and one rule in that list dropped untracked *.log files. An untracked log is usually run output nobody will open again.

How a test fixture ends up untracked#

The project was a small log analyzer in Haskell. Its tests read testdata/app.log and testdata/access.log. A log analyzer's fixtures are logs, and logs end in .log.

My global gitignore, the file core.excludesFile points to, has *.log. The repo's own .gitignore never said otherwise, so git never saw the fixtures: not when they were created and not on any commit. In git 2.56.0, bulk adds skip ignored files without a word; only git add testdata/app.log, naming the file, gets a refusal. The working tree held the only copy, and the everyday commands don't say so. A toy re-creation, with a fake HOME whose global ignore holds that one line:

$ git add -A
$ git commit -q -m 'parser and tests'
$ git status
On branch main
nothing to commit, working tree clean
$ git ls-files
src/Main.hs
$ git status --ignored --short
!! testdata/
$ git check-ignore -v testdata/app.log
~/.config/git/ignore:1:*.log	testdata/app.log

"Working tree clean" means clean among the files git was allowed to look at. git status --ignored lists what it isn't looking at, collapsed to testdata/ because everything inside is ignored, and git check-ignore -v names the file and line responsible. A global ignore had done the same to a vendored JS library once before, in the aside of Deleting a GitLab project gives it a new name.

The exclude agreed with the gitignore#

Two reasonable rules now lined up. The gitignore kept the fixtures out of git, which made them untracked. The archive's exclude dropped untracked logs. Between them, they removed precisely the one copy that existed.

A missing directory, read as an empty one#

The two fixtures never showed up in the compare as missing files. Their directory did: testdata/ present in the source, absent in the restore.

That line was read as an empty directory, and the archive loop moved on. This one wasn't empty. Everything in it had been excluded. The project was deleted.

Found a batch later#

The same exclude met a Go project in a later batch. This time it was caught before anything was deleted, and that is when the log analyzer's loss surfaced.

Nothing held the originals anymore: not git, not the archive, not the disk.

Rebuilding test data from a README#

Two things survived. The README documented the exact output of the tool's --by level report: ERROR 3, INFO 3, WARN 2, DEBUG 1, FATAL 1. The parser source defined which line shapes it accepts.

That was enough to rebuild the fixtures as files that satisfy the specification. I wrote both again so that the tool reproduced the README's output byte for byte, ran every command in the README, and ran cabal test: 14 examples, 0 failures. Then I committed them, with a repo-level exception so git would finally see them:

!**/testdata/*.log

In the toy repo, that one line flips the verdict:

$ printf '!**/testdata/*.log\n' > .gitignore
$ git status --short
?? .gitignore
?? testdata/
$ git check-ignore -v testdata/app.log
.gitignore:1:!**/testdata/*.log	testdata/app.log

They are labelled as reconstructions, because that is what they are. With them, the tool prints the documented output and all 14 examples pass. Anything the originals exercised that neither the README nor the tests described is gone, and nobody can say whether that was anything.

Everything a clone doesn't carry#

The loop came out of this with two new rules: logs under test and data directories are data, and a directory-only diff means its children were excluded, never that it was empty.

The rest of the archive rules turned out to be the same lesson in other clothes. The small, git-only archives store only untracked and ignored files and restore by cloning the remote at the archived commit, which works only for what git carries. So a project gets the full tar, .git included, if any of these hold:

  • it has a stash (a clone brings none back);
  • a tracked file has a mode other than 0644 or 0755 (git records only those two for regular files, and under the usual umask 022 a clone checks them out as 0644 or 0755, so a tracked 0400 key and a 0444 certificate both failed the mode check);
  • any branch has unpushed commits, or a tracked file is dirty;
  • the remote is gone, no longer has the commit, or lives on an account that won't last.

The !**/testdata/*.log line is in the repo now. Of all the rules in this story, it is the only one that lives in the project itself, next to the files it protects.

Filed under: git, testing, linux

Sources#

  • gitignore(5): precedence of command line, .gitignore, info/exclude and core.excludesFile; ! negation; **/ patterns; the default ~/.config/git/ignore location.
  • git-check-ignore(1): -v prints source:line:pattern for the rule that matched, including negated ones.
  • git-status(1): --ignored and the !! marker.
  • gitformat-index(5): "Only 0755 and 0644 are valid for regular files."
  • git-stash(1) and git-clone(1): stashes live in refs/stash and its reflog; a clone creates remote-tracking branches for the source's branches.