Storage on tsvinc

12.4 GB of duplicates, 3.4 GB on disk

Sun, 25 Oct 2026 00:00:00 +0000

$ compsize -x model.bin model-copy.bin
model.bin: SEARCH_V2: Operation not permitted

That is compsize, the tool that reports what a file actually costs on btrfs, refusing to talk to a normal user. It reads each file's extent records through a btrfs ioctl, and the kernel lets only CAP_SYS_ADMIN call it. I did the measuring on the workstation I was clearing out without sudo.

What it did have was rootful Docker, through the docker group. So I ran the host's own compsize through a read-only chroot:

Measure real btrfs disk use without sudo

Sun, 25 Oct 2026 00:00:00 +0000

Run the host's own compsize as root in a throwaway Docker container to get the real on-disk size of a path on btrfs, with compression and reflinked copies counted correctly, when you are in the docker group but have no sudo.

Tested on: Fedora 44 workstation, kernel 7.2.10-200.fc44, btrfs /home mounted with compress=zstd:1, compsize 1.5^git20250123.d79eacf-15.fc44, coreutils 9.10, Docker CE 29.9.1 (rootful), alpine image 3.24.2 (2026-10-25, on test files). Used on real data on the same workstation on 2026-10-23 or 2026-10-24 (tool versions not recorded).
Not verified: a Docker daemon with SELinux support enabled (selinux in docker info security options); rootless Docker or Podman themselves (only the user-namespace mechanism they rely on, Failure modes row 2); the All empty or still-delalloced files. message compsize prints for data not yet written out.
Needs: a btrfs filesystem; compsize installed on the host as /usr/bin/compsize; access to a rootful Docker daemon, usually through the docker group; the alpine image, or network access to pull it. No sudo.
Time: 5 min.
Risk: nothing on disk changes (except a root-owned ro-test file if the read-only check in Before you start fails), but this is not a low-privilege method. The docker group grants root-level privileges (Docker's own docs). The container runs as root with CAP_SYS_ADMIN, and -v /:/host:ro guards against your own mistakes, not against hostile code in the image. Run it only with an image you trust. The image provides only chroot; the compsize binary and its libraries come from the host. Nothing to undo afterwards.

Before you start#

Optional: a test directory with known content, to check the numbers before you trust them. Every transcript in this manual ran in one scratch directory on the same btrfs /home; the measurements use these three 64 MiB files:
```
$ head -c 64M /dev/urandom > random.bin
$ cp random.bin random-copy.bin
$ yes 'compsize demo: the quick brown fox jumps over the lazy dog' | head -c 64M > text.txt
```
compsize fails as your user, on any file on the btrfs filesystem (here text.txt from the test directory). This is the error the manual works around:
```
$ compsize text.txt
text.txt: SEARCH_V2: Operation not permitted
```

The path is on btrfs, and compression is on:

$ findmnt -no FSTYPE,OPTIONS -T .
btrfs rw,relatime,seclabel,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=257,subvol=/home

Expected: btrfs and a compress= option. Use -T <path> for a path that is not the current directory.

compsize is installed on the host, at the path the container will call:
```
$ rpm -q compsize
compsize-1.5^git20250123.d79eacf-15.fc44.x86_64
$ command -v compsize
/usr/bin/compsize
```
Without it the chroot has nothing to run, and installing it needs root.
The Docker daemon is rootful:
```
$ docker info --format '{{.SecurityOptions}}'
[name=seccomp,profile=builtin name=cgroupns]
```
Expected: no rootless entry. A rootless daemon runs containers in a user namespace, where compsize fails the same way as for your user (Failure modes, row 2; rootless Docker itself not tested).
The read-only mount reaches the filesystem that holds your path. /home can be a separate mount under /; on this box it is (subvol=/home above). Docker makes the submounts of a read-only bind read-only too only as a best effort, on kernel 5.12 or newer:
```
$ docker run --rm -v /:/host:ro -w "/host$PWD" alpine touch ro-test
touch: ro-test: Read-only file system
```
Expected: Read-only file system. If touch succeeds instead, the guard does not cover your path, and a root-owned ro-test file is left behind. Remove it through a container, for example docker run --rm -v "$PWD":/w alpine rm /w/ro-test [UNVERIFIED: not exercised, the touch failed here].
The alpine image is present, or docker run pulls it on first use:
```
$ docker run --rm alpine cat /etc/alpine-release
3.24.2
```
Expected: an Alpine version. The measurement uses only its chroot.
Placeholders: <path> is the file or directory to measure, as an absolute path ("$PWD/<name>" or ~/…; a relative path prints No files., Failure modes row 3). <name> is a file or directory in the current directory. <dir> is your test directory.
Backups: none needed. Nothing is written.

Steps#

Flush pending writes.
```
sync
```
Expected: no output. compsize counts only data that already has extents on disk; its man page says recently written files can show as taking no space until they are synced or written out, typically within about 30 seconds.

Run the host's compsize through the container. This is the command as used on real data:

docker run --rm --cap-add SYS_ADMIN -v /:/host:ro alpine chroot /host /usr/bin/compsize -x <path>

On the test directory:

$ docker run --rm --cap-add SYS_ADMIN -v /:/host:ro alpine chroot /host /usr/bin/compsize -x "$PWD"
Processed 3 files, 513 regular extents (514 refs), 0 inline, 325 fragments.
Type       Perc     Disk Usage   Uncompressed Referenced  
TOTAL       51%       66M         128M         192M       
none       100%       64M          64M         128M       
zstd         3%      2.0M          64M          64M

Expected: a Processed N files line, a TOTAL row, and one row per compression type. --cap-add SYS_ADMIN grants the capability that btrfs checks for the search ioctl compsize uses. -v /:/host:ro shows the host's root read-only. chroot /host makes /usr/bin/compsize the host's binary. -x stops at filesystem boundaries, and to compsize a nested btrfs subvolume is one (Failure modes, row 5).

Read the TOTAL row against du.
```
$ du -h random.bin random-copy.bin text.txt
64M	random.bin
64M	random-copy.bin
64M	text.txt
$ du -sh .
192M	.
```
Expected: du matches Referenced (192M), not Disk Usage (66M). From the compsize man page and README: Disk Usage is what the files cost on disk, and every referenced extent is counted once; Uncompressed is the same extents before compression; Referenced is the apparent size of every file, so each reflinked copy counts again. A none row is data btrfs stored uncompressed.

Price duplicates as a set: all copies, then one.

$ docker run --rm --cap-add SYS_ADMIN -v /:/host:ro alpine chroot /host /usr/bin/compsize -x "$PWD/random.bin" "$PWD/random-copy.bin"
Processed 2 files, 1 regular extents (2 refs), 0 inline, 2 fragments.
Type       Perc     Disk Usage   Uncompressed Referenced  
TOTAL      100%       64M          64M         128M       
none       100%       64M          64M         128M       
$ docker run --rm --cap-add SYS_ADMIN -v /:/host:ro alpine chroot /host /usr/bin/compsize -x "$PWD/random.bin"
Processed 1 file, 1 regular extents (1 refs), 0 inline, 1 fragments.
Type       Perc     Disk Usage   Uncompressed Referenced  
TOTAL      100%       64M          64M          64M       
none       100%       64M          64M          64M

Expected: Disk Usage of all copies minus Disk Usage of one copy is what the extra copies really cost. Here it is 64M − 64M = 0, because cp made a reflink: coreutils 9.0 and later default to --reflink=auto, and this box has 9.10. A duplicate finder that adds up file sizes would report 64M wasted.

Verify#

A file with known content gives the expected cost. 64 MiB of repeated text takes 2.0M on disk:

$ docker run --rm --cap-add SYS_ADMIN -v /:/host:ro alpine chroot /host /usr/bin/compsize -x "$PWD/text.txt"
Processed 1 file, 512 regular extents (512 refs), 0 inline, 323 fragments.
Type       Perc     Disk Usage   Uncompressed Referenced  
TOTAL        3%      2.0M          64M          64M       
zstd         3%      2.0M          64M          64M

64 MiB of random data takes 64M, type none (step 4, second command).

Undo / rollback#

Nothing on the filesystem changes: compsize only reads, and the host root is mounted read-only (checked in Before you start).

--rm removes the container when it exits. Checked with a named run:

$ docker run --rm --name compsize-manual-rmcheck-20261025 --cap-add SYS_ADMIN -v /:/host:ro alpine chroot /host /usr/bin/compsize -x "$PWD/text.txt" > /dev/null; docker ps -a --filter name=compsize-manual-rmcheck-20261025 --format '{{.Names}}' | wc -l
0

If docker run pulled alpine only for this, docker image rm alpine removes it [UNVERIFIED: not exercised, the image was already present].
Remove the test directory with rm -r <dir> if you made one [UNVERIFIED: not exercised, the test files are kept as evidence].

Failure modes#

Symptom (exact message)	Cause	Fix
`text.txt: SEARCH_V2: Operation not permitted` (compsize run as your user; the file name comes first)	The btrfs `TREE_SEARCH_V2` ioctl returns EPERM without CAP_SYS_ADMIN (kernel 7.2.10 `fs/btrfs/ioctl.c`).	Step 2.
The same message from root in a user namespace: `unshare -Ur` shows `CapEff: 000001ffffffffff` and compsize still fails	The kernel checks the capability in the initial user namespace. Rootless Docker runs the daemon and its containers inside a user namespace (Docker docs); tested here with `unshare -Ur` only.	A rootful daemon, or an admin who runs compsize.
`No files.`	A relative `<path>`. After `chroot /host` the working directory is `/` (checked with `pwd`), and compsize skips paths that do not exist without an error.	Pass an absolute path: `"$PWD/<name>"` or `~/…`. A relative path given together with absolute ones is skipped the same way, so the total is partial with no error.
`du: cannot read directory './locked': Permission denied` and a total that is too small	`du` runs as your user and skips directories it cannot read, such as a container's database directory with mode 700.	Use the container total: root in the container reads them. On the test directory, a 16 MiB file in a `chmod 000` directory was missing from `du -sh .` (192M) and counted by compsize (Disk Usage 66M → 82M).
No message, but files in a nested subvolume are missing: `Processed 4 files` instead of 5	`-x` skips anything whose device number differs from its parent's, and every btrfs subvolume has its own.	Drop `-x` when the path holds subvolumes you want counted. On the test directory a 16 MiB file in a subvolume made 82M (with `-x`) into 98M (without). Without `-x` compsize also enters other mounts below the path, and per its source it stops at a non-btrfs one with `Not btrfs (or SEARCH_V2 unsupported).` (not exercised).

Field notes#

2026-10-25, workstation: Fedora 44, kernel 7.2.10-200.fc44, btrfs compress=zstd:1 (/home); compsize 1.5^git20250123.d79eacf-15.fc44; Docker CE 29.9.1 (rootful), alpine 3.24.2; coreutils 9.10, overlayfs storage, SELinux enforcing: Every block in this manual was run on test files in a scratch directory on the same /home, not on real data. As the normal user, compsize failed with the file name in front of the error: text.txt: SEARCH_V2: Operation not permitted. The container run worked first time. du said 192M for the directory, compsize said 66M on disk. The 64 MiB cp copy cost nothing (one extent, two references). A compsize run started within a second of writing 64 MiB of text was expected to show the unflushed-data message, but all 512 extents were already counted, so the sync in step 1 is a precaution from the man page, not an observed fix. A touch through the read-only mount failed, so the read-only flag reached the /home submount on this kernel. Root in a user namespace (unshare -Ur, all capabilities) still got EPERM. A relative path printed No files. A directory with mode 000 hid 16 MiB from du but not from the container, and -x silently left out a nested test subvolume. The daemon has no SELinux support enabled, so this run says nothing about one that has.
2026-10-23 or 2026-10-24 (the day was not recorded), same workstation and /home filesystem, tool versions not recorded: On the real disk, compsize ran without sudo, through the rootful Docker daemon (docker group), and each destructive step of the housekeeping waited for my approval. du reported uncompressed sizes. compsize run as my normal user failed on SEARCH_V2 with Operation not permitted, and the container command in step 2 worked. Result: 144 GB of data in 94 GB of disk. 78 GB of it was stored uncompressed: zips, ROMs, maps and model weights, data that btrfs's compression heuristic skips. A Postgres data directory owned by a container's uid (mode 700) made du print Permission denied and undercount by 15.8 GB. A duplicate finder listed 268 sets of identical files of at least 5 MB, 12.4 GB of apparent waste. compsize per set (all copies, then one) put the real figure at 3.4 GB, because cp had already made reflinks: a 2.9 GB speech model and its copy took 2.8 GB together. Recompressing with btrfs filesystem defragment -czstd -L 9 worked without root but gained about 5%, so it was dropped. A live SQLite database had the m (no-compress) attribute on purpose.

Notes#

Without Docker: btrfs filesystem du -s <path> runs as a normal user and separates Exclusive from Set shared, so it does see reflinked copies. It reports uncompressed sizes, though: on the test directory it gave text.txt 64.00MiB, against compsize's 2.0M.
Why chroot into the host instead of a compsize image: no package install inside a root container, and the binary is the host's own. Docker keeps the SYS_CHROOT capability by default.
~/… paths work for <path> because the host shell expands them before docker runs (checked); relative paths do not (Failure modes, row 3).
Higher zstd level: about 5% on the original box, which was put down to btrfs compressing each 128 KiB chunk on its own (the chunk size is in the btrfs docs). btrfs-filesystem(8) warns that defragmenting breaks up reflinks on kernels 3.14-rc2 and later, so do not defragment files whose copies share extents.
The story behind the numbers in the field notes, and why du misled: the btrfs duplicates that weren't.

Sources#

compsize source, compsize.c: the %s: SEARCH_V2: %m error, -x as a device-number comparison, missing paths skipped silently, No files.. Fedora's package is built from commit d79eacf, byte-identical to master when fetched.
compsize man page, compsize.8: meaning of Disk Usage, Uncompressed and Referenced; -x; the caveat about unsynced files; "The ioctls used by this program require root."
compsize README: "every referenced extent is counted whole, exactly once"; the ioctl requires root.
Linux 7.2.10 fs/btrfs/ioctl.c: btrfs_ioctl_tree_search_v2() returns -EPERM unless capable(CAP_SYS_ADMIN).
GNU coreutils NEWS, release 9.0: "cp now uses --reflink=auto mode by default."
Docker: Linux post-installation steps: "The docker group grants root-level privileges to the user."
Docker: security: the daemon needs root unless in rootless mode; only trusted users should control it.
Docker: bind mounts: read-only binds make submounts read-only as a best effort, on kernel 5.12 or newer.
Docker: running containers: default capabilities (DAC_OVERRIDE and SYS_CHROOT kept, SYS_ADMIN only through --cap-add).
Docker: rootless mode: a rootless daemon lists rootless under Security Options in docker info.
btrfs documentation: compression: data is split into 128 KiB chunks before compression; if the first portion compressed does not shrink, compression is disabled for the whole file.
btrfs-filesystem(8) from btrfs-progs 7.1 (man btrfs-filesystem): the defragment warning about breaking up reflinks.

Prune a stale synced copy with an rsync checksum dry run

Sun, 25 Oct 2026 00:00:00 +0000

Compare an old one-way copy of another machine's tree with that machine by checksum, without moving any file contents, delete locally only the top-level entries whose files all exist on the source with the same checksum (git's .git/index and FETCH_HEAD aside), and revoke the temporary SSH key.

Tested on: Fedora 44 (kernel 7.2.10), rsync 3.5.1 protocol 33, GNU findutils 4.10.0, GNU Awk 5.3.2, against a loopback remote shell on synthetic trees (2026-10-25). Original run: a Fedora workstation against an older laptop over SSH (2026-10-23 or 2026-10-24, the day was not recorded); the rsync versions and the laptop's OS were not recorded.
Not verified: steps 1, 10 and 11 against a real SSH server, and step 2 over ssh (it ran through the loopback shell). No second machine was reachable for this manual; the key-removal command ran only against a fake authorized_keys through the loopback shell, and step 11's identity check only as ssh -G.
Needs: bash, rsync on both machines, an SSH login on the source, and the source owner's permission to add a temporary key and remove it again. GNU find and xargs, awk and coreutils on this machine.
Time: about 30 min of hands-on steps (an estimate; the original run was not timed), plus one checksum pass over both trees.
Risk: step 8 deletes local entries for good. Every file in them matched the source by checksum at the step 7 re-check, except .git/index and .git/FETCH_HEAD in git-noise entries: Undo brings back the source's versions of those two, so the copy's own index, and anything staged in it, cannot be restored. Undo needs access to the source again. On the source, step 1 adds one key and step 10 removes it; the dry runs write nothing there, but they make it read every file that has the same size as its local counterpart.

Before you start#

Placeholders used below:
- <ABSOLUTE_PATH_OF_COPY>: root of the stale copy on this machine.
- <USER>@<HOST>: the SSH login on the source machine.
- <PATH_ON_SOURCE>: the tree the copy was made from, relative to that login's home or absolute.
- <KEY>: file name for the temporary key, for example tmp-compare.
- <ENTRY>: one top-level entry name (Undo only).
- <ABSOLUTE_PATH_OF_SHIM>: where you saved the test shim (Notes only).
Set them once, in the bash shell you use for every step (the commands rely on bash splitting an unquoted $RSYNC_RSH into words):
```
LOCAL=<ABSOLUTE_PATH_OF_COPY>
SRC=<USER>@<HOST>
REMOTE=<PATH_ON_SOURCE>
```
Work from a directory outside $LOCAL, for example mkdir -p ~/copy-compare && cd ~/copy-compare. The steps write dry.txt, verdicts.tsv and similar files to the current directory; inside the copy they would turn up in the comparison.

rsync, GNU find, awk, and a UTF-8 character set (in the C locale rsync escapes non-ASCII names; see Failure modes):

$ rsync --version | head -1; find --version | head -1; awk --version | head -1; locale charmap
rsync  version 3.5.1-g315819c5  protocol version 33
find (GNU findutils) 4.10.0
GNU Awk 5.3.2, API 4.0, PMA Avon 8-g1, (GNU MPFR 4.2.2, GNU MP 6.3.0)
UTF-8

Expected: the last line is UTF-8.

Nothing under $LOCAL is unreadable to you:
```
$ find "$LOCAL" $ ! -type l ! -readable -o -type d ! -executable $ -prune -printf '%U %m %P\n'
```
Expected: no output. Each line it prints is an owner uid, a mode and a path that rsync cannot read or enter either. On a test tree with a mode-000 file and a mode-444 directory:
```
$ find "$LOCAL" $ ! -type l ! -readable -o -type d ! -executable $ -prune -printf '%U %m %P\n'
1000 0 both-locked/secret.bin
1000 444 noexec-dir
```
Both kinds can make an entry look identical. rsync prints no itemize line for the contents of a directory it cannot open, which shows up in step 3 as an error. A file it cannot read gets no error at all: when the source's copy of it is unreadable too and has the same size, rsync prints nothing for it (Failure modes, rows 2 and 3). Make the listed paths readable, or move them out of $LOCAL and decide on them separately, before step 3. ! -readable alone misses the mode-444 directory, hence the -executable test. ! -type l leaves symlinks out: rsync compares a symlink's target as text, so one that dangles on this machine is fine, and without that test the check listed a dangling symlink in the same test tree. The original copy held an unreadable directory, a Postgres data directory written by a container: owned by the container's uid, mode 700, and unreadable to du as well.
Backups: none. Nothing is deleted before step 8, and step 8 deletes only entries that step 7 has just re-checked against the source.

Steps#

Create a temporary key and install it on the source.
```
ssh-keygen -t ed25519 -N '' -C tmp-compare -f ~/.ssh/<KEY>
ssh-copy-id -i ~/.ssh/<KEY>.pub "$SRC"
export RSYNC_RSH="ssh -i $HOME/.ssh/<KEY> -o IdentitiesOnly=yes"
```
ssh-copy-id appends the public key to ~/.ssh/authorized_keys on the source, creating the file if needed; it logs in with the account's password (or a key that already works) to do so. RSYNC_RSH is the remote shell rsync uses when no -e is given, so every rsync command below goes through this ssh command. rsync splits the value into separate arguments: a logging shim set as RSYNC_RSH received -i, the key path, -o and IdentitiesOnly=yes as four arguments, followed by the host and rsync. IdentitiesOnly=yes keeps keys held by ssh-agent out of the login; IdentityFile lines from ~/.ssh/config still apply, which is why step 11 tests the key with -F /dev/null. Expected: ssh-copy-id reports one key added. [UNVERIFIED: steps 1 and 2 not run against an SSH server for this manual. The original run used a temporary key; its setup commands were not recorded.]
Check the path and rsync on the source without listing anything. The loopback run (its "source" is a second directory on the same machine, so it reports the local rsync):
```
$ $RSYNC_RSH "$SRC" "test -d $REMOTE && echo dir-ok; command -v rsync && rsync --version | head -1"; echo "rc=$?"
dir-ok
/usr/bin/rsync
rsync  version 3.5.1-g315819c5  protocol version 33
rc=0
```
Expected: three lines, dir-ok, the path of rsync on the source and its version. No dir-ok line: $REMOTE is not a directory there (in the loopback run, a path that does not exist printed only the two rsync lines). No path and version: rsync is not installed there, or not in the remote shell's PATH. The final rc comes from the last command and says nothing about the directory.
Run the checksum dry run, from the copy to the source. The command from the original run, with LOCAL, host and REMOTE as its placeholders: rsync --dry-run -rlc --itemize-changes --out-format='%i %n' LOCAL/ host:REMOTE/ The same command with the variables from Before you start, saving both output streams:
```
$ rsync --dry-run -rlc --itemize-changes --out-format='%i %n' "$LOCAL"/ "$SRC":"$REMOTE"/ > dry.txt 2> dry.err; echo "rc=$?"
rc=0
```
Expected: rc=0 and an empty dry.err (step 4 shows its line count). Anything else: find the cause under Failure modes before going on; step 5 refuses to classify while dry.err has a line.
- --dry-run: rsync works out what it would send and reports it; no file contents are sent and nothing is written on the source. The copy's file list, with names, sizes and checksums, does go to the source (Notes). In the loopback runs, a listing of the source tree with sizes, mtimes and ctimes was the same before and after seven dry runs.
- -c: decide by checksum, not by size and modification time. The local side checksums its files while it builds the file list; the source checksums each of its files that has the same size as the local one, and a file whose size differs is reported without one. The algorithm is negotiated between the two ends (xxh128 between the two rsync 3.5.1 ends of the loopback run), so "identical" below means the same checksum. Without -c, the loopback run missed a same-size edit with an unchanged mtime and flagged a file whose bytes matched but whose mtime did not (Notes).
- -r -l: recurse, and compare symlinks as links instead of following them. Not -a: its -t -p -o -g add attribute-only lines, such as .f..t...... notes/touched.txt for a file with identical bytes, and those would turn an identical entry into differs.
- --itemize-changes --out-format='%i %n': one line per item that would change, made of an 11-character change code, a space and the path relative to $LOCAL. The explicit format leaves out the -> target that --itemize-changes alone appends to symlinks, so every line splits the same way.
- "$LOCAL"/ with the trailing slash: compare the contents of the copy with the contents of $REMOTE. The form without the slash is Failure modes, row 4.
- No --delete: it would itemize files that exist only on the source, and without --dry-run it would delete them there.

Read the itemized lines.

$ cat dry.txt; wc -l dry.txt dry.err
<fcsT...... edited/changed-size.txt
<fc.T...... edited/same-size.txt
cd+++++++++ gone-there/
<f+++++++++ gone-there/NOTES
cd+++++++++ gone-there/src/
<f+++++++++ gone-there/src/main.c
<f+++++++++ grown/new-here.txt
<fcsT...... repo-index/.git/FETCH_HEAD
<fc.T...... repo-index/.git/index
  9 dry.txt
  0 dry.err
  9 total

In this direction the local side is the sender, so every file line starts with <, sent to the remote host. On the loopback trees, notes/, repo-same/, top-file.txt and .dotfile print nothing: every file in them matched by checksum, including one whose mtime differs on the source, and the symlink in notes/ points at the same target on both sides.

Line	Meaning here
no line	the item exists on the source with the same checksum (a symlink: the same target)
`<f+++++++++ path`	file missing on the source
`cd+++++++++ dir/`	directory missing on the source; everything under it follows as new
`cL+++++++++ link`	symlink missing on the source
`<fcsT...... path`	content (`c`) and size (`s`) differ
`<fc.T...... path`	same size, different content

A leading c means the receiver would create the item, a directory or symlink here. The T only says the receiver would set the transfer time as mtime, because times are not preserved; it carries no information here. Line codes by kind:

$ cut -c1-11 dry.txt | sort | uniq -c
      2 cd+++++++++
      3 <f+++++++++
      2 <fcsT......
      2 <fc.T......

GotchaCount <f, not >f

> marks an item received by the local side, which happens in a pull. In the original run my first classifier looked for > and counted zero.

$ grep -c '^>f' dry.txt
0
$ grep -c '^<f' dry.txt
7

Give each top-level entry a verdict. Save this as classify.awk in the working directory:

# classify.awk: one verdict per top-level entry of LOCAL, from a push-direction
#   rsync --dry-run -rlc --itemize-changes --out-format='%i %n' LOCAL/ host:REMOTE/ > dry.txt 2> dry.err
# usage: awk -f classify.awk entries.txt dry.txt dry.err > verdicts.tsv
#   identical   no line: every file under it exists on the source with the same checksum
#   git-noise   only .git/index or .git/FETCH_HEAD differ
#   only-local  the entry itself is missing on the source
#   differs     anything else (new or changed files inside)
# Any line in dry.err, or any line of dry.txt it cannot map to an entry,
# aborts with exit 2 and no verdicts: an unreadable directory prints no line.
FILENAME == ARGV[1] { order[++n] = $0; v[$0] = "identical"; next }
FILENAME == ARGV[3] { print "rsync stderr: " $0 > "/dev/stderr"; bad = 1; next }
{
    code = substr($0, 1, 11); path = substr($0, 13); sub(/\/$/, "", path)
    top = path; sub(/\/.*/, "", top)
    if (substr($0, 12, 1) != " " || !(top in v)) {
        print "unmapped line: " $0 > "/dev/stderr"; bad = 1; next
    }
    new = (substr(code, 3) == "+++++++++")
    if (path == top && new) v[top] = "only-local"
    else if (v[top] == "only-local") next
    else if (!new && path ~ /\/\.git\/(index|FETCH_HEAD)$/) {
        if (v[top] == "identical") v[top] = "git-noise"
    }
    else v[top] = "differs"
}
END {
    if (bad) exit 2
    for (i = 1; i <= n; i++) printf "%s\t%s\n", v[order[i]], order[i]
}

List the entries, then classify:

$ find "$LOCAL" -mindepth 1 -maxdepth 1 -printf '%f\n' | sort > entries.txt; cat entries.txt
.dotfile
edited
gone-there
grown
notes
repo-index
repo-same
top-file.txt
$ awk -f classify.awk entries.txt dry.txt dry.err > verdicts.tsv; echo "rc=$?"; cat verdicts.tsv
rc=0
identical	.dotfile
differs	edited
only-local	gone-there
differs	grown
identical	notes
git-noise	repo-index
identical	repo-same
identical	top-file.txt

Expected: rc=0 and one verdict per entry. rc=2 with rsync stderr: or unmapped line: messages and an empty verdicts.tsv: see Failure modes.

identical: no line under the entry; every file in it exists on the source with the same checksum, and every symlink with the same target.
git-noise: the only differing files are .git/index and .git/FETCH_HEAD, and every object, ref and work-tree file matched by checksum. The index caches each tracked file's stat data, which differs between two machines even for identical files, and git status rewrites it when it refreshes that data; FETCH_HEAD records the last git fetch. On the loopback tree, one git status on the source side and a fetch there from a second URL produced exactly this. The index is also git's staging area, so whatever the copy had staged goes with it; the objects it names are on the source, because no file under .git/objects differed. Any other differing file under .git leaves the entry as differs.
only-local: the entry itself is missing on the source; the copy is the only one.
differs: new or changed files inside; either side may hold the newer version.

List the deletion candidates and look at what stays.

$ awk -F'\t' '$1 == "identical" || $1 == "git-noise" { print $2 }' verdicts.tsv | sort > planned.txt; cat planned.txt
.dotfile
notes
repo-index
repo-same
top-file.txt
$ awk -F'\t' '$1 == "only-local" || $1 == "differs"' verdicts.tsv
differs	edited
only-local	gone-there
differs	grown

Expected: planned.txt holds only identical and git-noise entries. Remove from it anything you want to keep anyway. The other entries stay: an only-local entry exists nowhere else, and a differs entry may hold the newer copy of a file.

Re-run the dry run right before deleting, and keep only entries that still qualify. The source can change between step 3 and the deletion:

rsync --dry-run -rlc --itemize-changes --out-format='%i %n' "$LOCAL"/ "$SRC":"$REMOTE"/ > dry2.txt 2> dry2.err; echo "rc=$?"; cat dry2.err
awk -f classify.awk entries.txt dry2.txt dry2.err > verdicts2.tsv; echo "rc=$?"
awk -F'\t' '$1 == "identical" || $1 == "git-noise" { print $2 }' verdicts2.tsv | sort | comm -12 planned.txt - > delete.txt
comm -23 planned.txt delete.txt | sed 's/^/dropped: /'; cat delete.txt

In the loopback run, a line was appended to notes/b.txt on the source between the two dry runs (a harness step, to show what a change does):

$ rsync --dry-run -rlc --itemize-changes --out-format='%i %n' "$LOCAL"/ "$SRC":"$REMOTE"/ > dry2.txt 2> dry2.err; echo "rc=$?"; cat dry2.err
rc=0
$ awk -f classify.awk entries.txt dry2.txt dry2.err > verdicts2.tsv; echo "rc=$?"
awk -F'\t' '$1 == "identical" || $1 == "git-noise" { print $2 }' verdicts2.tsv | sort | comm -12 planned.txt - > delete.txt
comm -23 planned.txt delete.txt | sed 's/^/dropped: /'; cat delete.txt
rc=0
dropped: notes
.dotfile
repo-index
repo-same
top-file.txt

Expected: rc=0 twice, and delete.txt lists the planned entries that are still identical or git-noise; dropped: lines name those that are not. When the classifier refuses (exit 2), verdicts2.tsv is left empty (0 bytes in the slashless test), so delete.txt comes out empty.

Delete the entries in delete.txt from the copy.

WarningThis step is irreversible on this machine

The entries are gone from the copy. Their files matched the source by checksum at step 7, apart from .git/index and FETCH_HEAD in git-noise entries, whose versions in the copy cannot be brought back. Undo pulls everything else back from the source while the source keeps it.

Size first:

$ (cd "$LOCAL" && xargs -d '\n' du -shc --) < delete.txt
4.0K	.dotfile
132K	repo-index
128K	repo-same
4.0K	top-file.txt
268K	total

Then delete:

while IFS= read -r e; do
  if [ -z "$e" ] || [ "$e" = . ] || [ "$e" = .. ] || [ "${e#*/}" != "$e" ]; then echo "skip: $e" >&2; continue; fi
  rm -rf -- "${LOCAL:?}/$e" && printf 'removed\t%s\n' "$e"
done < delete.txt | tee -a removed.log

$ while IFS= read -r e; do
  if [ -z "$e" ] || [ "$e" = . ] || [ "$e" = .. ] || [ "${e#*/}" != "$e" ]; then echo "skip: $e" >&2; continue; fi
  rm -rf -- "${LOCAL:?}/$e" && printf 'removed\t%s\n' "$e"
done < delete.txt | tee -a removed.log
removed	.dotfile
removed	repo-index
removed	repo-same
removed	top-file.txt

Expected: one removed line per entry in delete.txt and no skip: line. The loop refuses empty names, ., .. and anything containing /, and ${LOCAL:?} makes the shell refuse the rm when $LOCAL is empty or unset instead of running it against /<entry>.

Re-check the copy against the source while you still have access.

$ find "$LOCAL" -mindepth 1 -maxdepth 1 -printf '%f\n' | sort > entries-after.txt
rsync --dry-run -rlc --itemize-changes --out-format='%i %n' "$LOCAL"/ "$SRC":"$REMOTE"/ > dry3.txt 2> dry3.err; echo "rc=$?"
awk -f classify.awk entries-after.txt dry3.txt dry3.err
rc=0
differs	edited
only-local	gone-there
differs	grown
differs	notes

Expected: rc=0, and only only-local and differs verdicts, plus any entry you took out of planned.txt on purpose. notes is differs here because of the source-side edit from step 7.

Remove the temporary key from the source. This is the one change on the source after step 1; ask its owner first.
```
BLOB=$(cut -d' ' -f2 ~/.ssh/<KEY>.pub)
$RSYNC_RSH "$SRC" "cd ~/.ssh && cp -p authorized_keys authorized_keys.tmpkey-bak && { grep -vF '$BLOB' authorized_keys.tmpkey-bak > authorized_keys; [ \$? -le 1 ]; } && rm authorized_keys.tmpkey-bak; grep -cF '$BLOB' authorized_keys"
```
Expected: 0, the number of lines still holding the key (the command exits 1 because that last grep matched nothing). This edits authorized_keys only, the file ssh-copy-id appended to; sshd also reads authorized_keys2 by default, which step 1 did not touch. > rewrites the file in place, so it keeps its mode; the backup is deleted only after the filter succeeded. If grep reports an error instead, authorized_keys may be cut short and the backup is still there: cp -p ~/.ssh/authorized_keys.tmpkey-bak ~/.ssh/authorized_keys on the source puts it back (not run). The same command, run through the loopback shell against a fake home, left a file that had held two other keys plus this one with exactly the two others, and a file that had held only this key empty (0 bytes); both kept mode 600, and the backup was gone both times. It needs a POSIX-style login shell on the source and was tested with /bin/sh only. [UNVERIFIED: not run against a real sshd.]
Close any shared connection and check that the key no longer logs in.
```
ssh -O exit "$SRC"
ssh -G -F /dev/null -i ~/.ssh/<KEY> -o IdentitiesOnly=yes -o BatchMode=yes -o ControlPath=none "$SRC" true | grep -E '^(identityfile|identitiesonly|batchmode|controlpath) '
ssh -F /dev/null -i ~/.ssh/<KEY> -o IdentitiesOnly=yes -o BatchMode=yes -o ControlPath=none "$SRC" true; echo "rc=$?"
```
ssh -O exit closes a master connection if one exists. Without one it fails harmlessly; with a control path that has no master behind it, a local test printed:
```
$ ssh -F /dev/null -o ControlPath=/nonexistent/master.sock -O exit "$SRC"; echo "rc=$?"
Control socket connect(/nonexistent/master.sock): No such file or directory
rc=255
```
ssh -G prints the configuration the test login will use and exits without connecting. With -F /dev/null, ssh reads no ~/.ssh/config and no system-wide config, so the temporary key is the only identity:
```
$ ssh -G -F /dev/null -i keytest/tmp-key -o IdentitiesOnly=yes -o BatchMode=yes -o ControlPath=none "$SRC" true | grep -E '^(identityfile|identitiesonly|batchmode|controlpath) '
batchmode yes
identitiesonly yes
identityfile keytest/tmp-key
```
Without -F /dev/null, an IdentityFile line in a config still joins the list, even with IdentitiesOnly=yes:
```
$ cat keytest/ssh_config; ssh -G -F keytest/ssh_config -i keytest/tmp-key -o IdentitiesOnly=yes "$SRC" true | grep '^identityfile '
Host *
    IdentityFile keytest/other-key
identityfile keytest/tmp-key
identityfile keytest/other-key
```
Expected: the test login fails with a Permission denied message and rc=255. Any other error (Could not resolve hostname, Connection refused) means the test never reached authentication; if the source needs settings from ~/.ssh/config, such as Port or ProxyJump, add them as -o options and test again. Only after a Permission denied, delete the local key files: rm ~/.ssh/<KEY> ~/.ssh/<KEY>.pub; unset RSYNC_RSH.

GotchaAn open master connection logs in without the key

With ControlMaster and ControlPersist in ~/.ssh/config, a master connection stays open in the background after the rsync runs, and later sessions reuse its network connection instead of opening a new one. A test login over it reuses a connection that was authenticated before the key was removed, so it says nothing about the key. -O exit closes the master, ControlPath=none keeps the test off any master, and BatchMode=yes stops ssh from falling back to a password prompt.

[UNVERIFIED: the test login was not run against a real sshd. In the original run the multiplexed connection was closed and a fresh login was checked to fail.]

All four drives at once, and why that was the good news

Wed, 08 Jul 2026 00:00:00 +0000

TL;DR N drives failing at the same instant is never N drive failures. It's a shared path: driver, cable, or power. Here it was two of them stacked — a partial kernel update that installed kernel-core without kernel-modules, stranding mpt3sas, and underneath it a loose SFF-8087 cable with every PHY negotiating at unknown link rate. The boot SSD was on a different controller and never blinked, which was the clue. No data was lost.

I asked in four words, flat, no punctuation to speak of, with the calm of someone who hasn't decided yet how bad it is:

Reallocated sectors: is your drive dying or just scarred?

Wed, 08 Jul 2026 00:00:00 +0000

TL;DR A high Reallocated_Sector_Ct is history, not prognosis. The drive with 344 reallocations was healthy — scarred, but its firmware was still successfully remapping. The drive with zero reallocations was the dying one: Current_Pending_Sector climbing while the reallocated count stayed pinned at 0 means the firmware has stopped remapping, which is the failure. Judge by whether the counters move under a forced write pass, not by the raw number.

Two drives on the same HBA, same evening, same test. Here are the SMART lines that matter, before I touched either of them:

Salvaging a dying disk by partitioning around the wound

Wed, 08 Jul 2026 00:00:00 +0000

TL;DR badblocks tells you how many bad blocks; its -o logfile tells you where, and that changes the verdict. All 58 failures on this 8 TB drive sat in the first 5.23% of the platter. Partition past the wound, leave the damaged region unallocated, and the healthy 94.8% — about 6.7 TiB — goes back to work as disposable scratch space. Not for anything you'd miss.

Every badblocks run leaves two artifacts, and the useful one is the file almost nobody opens. There's the summary line everyone reads:

Stop chasing a sensor that doesn't exist

Wed, 08 Jul 2026 00:00:00 +0000

TL;DR The LSI SAS2008 has no temperature sensor. Not an unsupported one, not one behind a proprietary tool — the silicon was never fabricated with it. No driver, lsiutil option, or sysfs path will summon it. Once you can tell "hard to monitor" from "impossible to monitor," the answer stops being another tool and starts being airflow. The same night's fan experiment made the point twice: the null result was the diagnosis.

The storage box runs every HDD through one LSI SAS2008 HBA -- a 9211-8i-class card in IT mode, mpt3sas, sitting at PCI 05:00.0. These things are fanless, run hot, and are famous for slow-cooking themselves in dead-air cases. So the instinct is obvious: monitor the chip temperature. Graph it, alert on it, normal ops hygiene.