Prune a stale synced copy with an rsync checksum dry run

Sun, 25 Oct 2026 00:00:00 +0000

Compare an old one-way copy of another machine's tree with that machine by checksum, without moving any file contents, delete locally only the top-level entries whose files all exist on the source with the same checksum (git's .git/index and FETCH_HEAD aside), and revoke the temporary SSH key.

Tested on: Fedora 44 (kernel 7.2.10), rsync 3.5.1 protocol 33, GNU findutils 4.10.0, GNU Awk 5.3.2, against a loopback remote shell on synthetic trees (2026-10-25). Original run: a Fedora workstation against an older laptop over SSH (2026-10-23 or 2026-10-24, the day was not recorded); the rsync versions and the laptop's OS were not recorded.
Not verified: steps 1, 10 and 11 against a real SSH server, and step 2 over ssh (it ran through the loopback shell). No second machine was reachable for this manual; the key-removal command ran only against a fake authorized_keys through the loopback shell, and step 11's identity check only as ssh -G.
Needs: bash, rsync on both machines, an SSH login on the source, and the source owner's permission to add a temporary key and remove it again. GNU find and xargs, awk and coreutils on this machine.
Time: about 30 min of hands-on steps (an estimate; the original run was not timed), plus one checksum pass over both trees.
Risk: step 8 deletes local entries for good. Every file in them matched the source by checksum at the step 7 re-check, except .git/index and .git/FETCH_HEAD in git-noise entries: Undo brings back the source's versions of those two, so the copy's own index, and anything staged in it, cannot be restored. Undo needs access to the source again. On the source, step 1 adds one key and step 10 removes it; the dry runs write nothing there, but they make it read every file that has the same size as its local counterpart.

Before you start#

Placeholders used below:
- <ABSOLUTE_PATH_OF_COPY>: root of the stale copy on this machine.
- <USER>@<HOST>: the SSH login on the source machine.
- <PATH_ON_SOURCE>: the tree the copy was made from, relative to that login's home or absolute.
- <KEY>: file name for the temporary key, for example tmp-compare.
- <ENTRY>: one top-level entry name (Undo only).
- <ABSOLUTE_PATH_OF_SHIM>: where you saved the test shim (Notes only).
Set them once, in the bash shell you use for every step (the commands rely on bash splitting an unquoted $RSYNC_RSH into words):
```
LOCAL=<ABSOLUTE_PATH_OF_COPY>
SRC=<USER>@<HOST>
REMOTE=<PATH_ON_SOURCE>
```
Work from a directory outside $LOCAL, for example mkdir -p ~/copy-compare && cd ~/copy-compare. The steps write dry.txt, verdicts.tsv and similar files to the current directory; inside the copy they would turn up in the comparison.

rsync, GNU find, awk, and a UTF-8 character set (in the C locale rsync escapes non-ASCII names; see Failure modes):

$ rsync --version | head -1; find --version | head -1; awk --version | head -1; locale charmap
rsync  version 3.5.1-g315819c5  protocol version 33
find (GNU findutils) 4.10.0
GNU Awk 5.3.2, API 4.0, PMA Avon 8-g1, (GNU MPFR 4.2.2, GNU MP 6.3.0)
UTF-8

Expected: the last line is UTF-8.

Nothing under $LOCAL is unreadable to you:
```
$ find "$LOCAL" $ ! -type l ! -readable -o -type d ! -executable $ -prune -printf '%U %m %P\n'
```
Expected: no output. Each line it prints is an owner uid, a mode and a path that rsync cannot read or enter either. On a test tree with a mode-000 file and a mode-444 directory:
```
$ find "$LOCAL" $ ! -type l ! -readable -o -type d ! -executable $ -prune -printf '%U %m %P\n'
1000 0 both-locked/secret.bin
1000 444 noexec-dir
```
Both kinds can make an entry look identical. rsync prints no itemize line for the contents of a directory it cannot open, which shows up in step 3 as an error. A file it cannot read gets no error at all: when the source's copy of it is unreadable too and has the same size, rsync prints nothing for it (Failure modes, rows 2 and 3). Make the listed paths readable, or move them out of $LOCAL and decide on them separately, before step 3. ! -readable alone misses the mode-444 directory, hence the -executable test. ! -type l leaves symlinks out: rsync compares a symlink's target as text, so one that dangles on this machine is fine, and without that test the check listed a dangling symlink in the same test tree. The original copy held an unreadable directory, a Postgres data directory written by a container: owned by the container's uid, mode 700, and unreadable to du as well.
Backups: none. Nothing is deleted before step 8, and step 8 deletes only entries that step 7 has just re-checked against the source.

Steps#

Create a temporary key and install it on the source.
```
ssh-keygen -t ed25519 -N '' -C tmp-compare -f ~/.ssh/<KEY>
ssh-copy-id -i ~/.ssh/<KEY>.pub "$SRC"
export RSYNC_RSH="ssh -i $HOME/.ssh/<KEY> -o IdentitiesOnly=yes"
```
ssh-copy-id appends the public key to ~/.ssh/authorized_keys on the source, creating the file if needed; it logs in with the account's password (or a key that already works) to do so. RSYNC_RSH is the remote shell rsync uses when no -e is given, so every rsync command below goes through this ssh command. rsync splits the value into separate arguments: a logging shim set as RSYNC_RSH received -i, the key path, -o and IdentitiesOnly=yes as four arguments, followed by the host and rsync. IdentitiesOnly=yes keeps keys held by ssh-agent out of the login; IdentityFile lines from ~/.ssh/config still apply, which is why step 11 tests the key with -F /dev/null. Expected: ssh-copy-id reports one key added. [UNVERIFIED: steps 1 and 2 not run against an SSH server for this manual. The original run used a temporary key; its setup commands were not recorded.]
Check the path and rsync on the source without listing anything. The loopback run (its "source" is a second directory on the same machine, so it reports the local rsync):
```
$ $RSYNC_RSH "$SRC" "test -d $REMOTE && echo dir-ok; command -v rsync && rsync --version | head -1"; echo "rc=$?"
dir-ok
/usr/bin/rsync
rsync  version 3.5.1-g315819c5  protocol version 33
rc=0
```
Expected: three lines, dir-ok, the path of rsync on the source and its version. No dir-ok line: $REMOTE is not a directory there (in the loopback run, a path that does not exist printed only the two rsync lines). No path and version: rsync is not installed there, or not in the remote shell's PATH. The final rc comes from the last command and says nothing about the directory.
Run the checksum dry run, from the copy to the source. The command from the original run, with LOCAL, host and REMOTE as its placeholders: rsync --dry-run -rlc --itemize-changes --out-format='%i %n' LOCAL/ host:REMOTE/ The same command with the variables from Before you start, saving both output streams:
```
$ rsync --dry-run -rlc --itemize-changes --out-format='%i %n' "$LOCAL"/ "$SRC":"$REMOTE"/ > dry.txt 2> dry.err; echo "rc=$?"
rc=0
```
Expected: rc=0 and an empty dry.err (step 4 shows its line count). Anything else: find the cause under Failure modes before going on; step 5 refuses to classify while dry.err has a line.
- --dry-run: rsync works out what it would send and reports it; no file contents are sent and nothing is written on the source. The copy's file list, with names, sizes and checksums, does go to the source (Notes). In the loopback runs, a listing of the source tree with sizes, mtimes and ctimes was the same before and after seven dry runs.
- -c: decide by checksum, not by size and modification time. The local side checksums its files while it builds the file list; the source checksums each of its files that has the same size as the local one, and a file whose size differs is reported without one. The algorithm is negotiated between the two ends (xxh128 between the two rsync 3.5.1 ends of the loopback run), so "identical" below means the same checksum. Without -c, the loopback run missed a same-size edit with an unchanged mtime and flagged a file whose bytes matched but whose mtime did not (Notes).
- -r -l: recurse, and compare symlinks as links instead of following them. Not -a: its -t -p -o -g add attribute-only lines, such as .f..t...... notes/touched.txt for a file with identical bytes, and those would turn an identical entry into differs.
- --itemize-changes --out-format='%i %n': one line per item that would change, made of an 11-character change code, a space and the path relative to $LOCAL. The explicit format leaves out the -> target that --itemize-changes alone appends to symlinks, so every line splits the same way.
- "$LOCAL"/ with the trailing slash: compare the contents of the copy with the contents of $REMOTE. The form without the slash is Failure modes, row 4.
- No --delete: it would itemize files that exist only on the source, and without --dry-run it would delete them there.

Read the itemized lines.

$ cat dry.txt; wc -l dry.txt dry.err
<fcsT...... edited/changed-size.txt
<fc.T...... edited/same-size.txt
cd+++++++++ gone-there/
<f+++++++++ gone-there/NOTES
cd+++++++++ gone-there/src/
<f+++++++++ gone-there/src/main.c
<f+++++++++ grown/new-here.txt
<fcsT...... repo-index/.git/FETCH_HEAD
<fc.T...... repo-index/.git/index
  9 dry.txt
  0 dry.err
  9 total

In this direction the local side is the sender, so every file line starts with <, sent to the remote host. On the loopback trees, notes/, repo-same/, top-file.txt and .dotfile print nothing: every file in them matched by checksum, including one whose mtime differs on the source, and the symlink in notes/ points at the same target on both sides.

Line	Meaning here
no line	the item exists on the source with the same checksum (a symlink: the same target)
`<f+++++++++ path`	file missing on the source
`cd+++++++++ dir/`	directory missing on the source; everything under it follows as new
`cL+++++++++ link`	symlink missing on the source
`<fcsT...... path`	content (`c`) and size (`s`) differ
`<fc.T...... path`	same size, different content

A leading c means the receiver would create the item, a directory or symlink here. The T only says the receiver would set the transfer time as mtime, because times are not preserved; it carries no information here. Line codes by kind:

$ cut -c1-11 dry.txt | sort | uniq -c
      2 cd+++++++++
      3 <f+++++++++
      2 <fcsT......
      2 <fc.T......

GotchaCount <f, not >f

> marks an item received by the local side, which happens in a pull. In the original run my first classifier looked for > and counted zero.

$ grep -c '^>f' dry.txt
0
$ grep -c '^<f' dry.txt
7

Give each top-level entry a verdict. Save this as classify.awk in the working directory:

# classify.awk: one verdict per top-level entry of LOCAL, from a push-direction
#   rsync --dry-run -rlc --itemize-changes --out-format='%i %n' LOCAL/ host:REMOTE/ > dry.txt 2> dry.err
# usage: awk -f classify.awk entries.txt dry.txt dry.err > verdicts.tsv
#   identical   no line: every file under it exists on the source with the same checksum
#   git-noise   only .git/index or .git/FETCH_HEAD differ
#   only-local  the entry itself is missing on the source
#   differs     anything else (new or changed files inside)
# Any line in dry.err, or any line of dry.txt it cannot map to an entry,
# aborts with exit 2 and no verdicts: an unreadable directory prints no line.
FILENAME == ARGV[1] { order[++n] = $0; v[$0] = "identical"; next }
FILENAME == ARGV[3] { print "rsync stderr: " $0 > "/dev/stderr"; bad = 1; next }
{
    code = substr($0, 1, 11); path = substr($0, 13); sub(/\/$/, "", path)
    top = path; sub(/\/.*/, "", top)
    if (substr($0, 12, 1) != " " || !(top in v)) {
        print "unmapped line: " $0 > "/dev/stderr"; bad = 1; next
    }
    new = (substr(code, 3) == "+++++++++")
    if (path == top && new) v[top] = "only-local"
    else if (v[top] == "only-local") next
    else if (!new && path ~ /\/\.git\/(index|FETCH_HEAD)$/) {
        if (v[top] == "identical") v[top] = "git-noise"
    }
    else v[top] = "differs"
}
END {
    if (bad) exit 2
    for (i = 1; i <= n; i++) printf "%s\t%s\n", v[order[i]], order[i]
}

List the entries, then classify:

$ find "$LOCAL" -mindepth 1 -maxdepth 1 -printf '%f\n' | sort > entries.txt; cat entries.txt
.dotfile
edited
gone-there
grown
notes
repo-index
repo-same
top-file.txt
$ awk -f classify.awk entries.txt dry.txt dry.err > verdicts.tsv; echo "rc=$?"; cat verdicts.tsv
rc=0
identical	.dotfile
differs	edited
only-local	gone-there
differs	grown
identical	notes
git-noise	repo-index
identical	repo-same
identical	top-file.txt

Expected: rc=0 and one verdict per entry. rc=2 with rsync stderr: or unmapped line: messages and an empty verdicts.tsv: see Failure modes.

identical: no line under the entry; every file in it exists on the source with the same checksum, and every symlink with the same target.
git-noise: the only differing files are .git/index and .git/FETCH_HEAD, and every object, ref and work-tree file matched by checksum. The index caches each tracked file's stat data, which differs between two machines even for identical files, and git status rewrites it when it refreshes that data; FETCH_HEAD records the last git fetch. On the loopback tree, one git status on the source side and a fetch there from a second URL produced exactly this. The index is also git's staging area, so whatever the copy had staged goes with it; the objects it names are on the source, because no file under .git/objects differed. Any other differing file under .git leaves the entry as differs.
only-local: the entry itself is missing on the source; the copy is the only one.
differs: new or changed files inside; either side may hold the newer version.

List the deletion candidates and look at what stays.

$ awk -F'\t' '$1 == "identical" || $1 == "git-noise" { print $2 }' verdicts.tsv | sort > planned.txt; cat planned.txt
.dotfile
notes
repo-index
repo-same
top-file.txt
$ awk -F'\t' '$1 == "only-local" || $1 == "differs"' verdicts.tsv
differs	edited
only-local	gone-there
differs	grown

Expected: planned.txt holds only identical and git-noise entries. Remove from it anything you want to keep anyway. The other entries stay: an only-local entry exists nowhere else, and a differs entry may hold the newer copy of a file.

Re-run the dry run right before deleting, and keep only entries that still qualify. The source can change between step 3 and the deletion:

rsync --dry-run -rlc --itemize-changes --out-format='%i %n' "$LOCAL"/ "$SRC":"$REMOTE"/ > dry2.txt 2> dry2.err; echo "rc=$?"; cat dry2.err
awk -f classify.awk entries.txt dry2.txt dry2.err > verdicts2.tsv; echo "rc=$?"
awk -F'\t' '$1 == "identical" || $1 == "git-noise" { print $2 }' verdicts2.tsv | sort | comm -12 planned.txt - > delete.txt
comm -23 planned.txt delete.txt | sed 's/^/dropped: /'; cat delete.txt

In the loopback run, a line was appended to notes/b.txt on the source between the two dry runs (a harness step, to show what a change does):

$ rsync --dry-run -rlc --itemize-changes --out-format='%i %n' "$LOCAL"/ "$SRC":"$REMOTE"/ > dry2.txt 2> dry2.err; echo "rc=$?"; cat dry2.err
rc=0
$ awk -f classify.awk entries.txt dry2.txt dry2.err > verdicts2.tsv; echo "rc=$?"
awk -F'\t' '$1 == "identical" || $1 == "git-noise" { print $2 }' verdicts2.tsv | sort | comm -12 planned.txt - > delete.txt
comm -23 planned.txt delete.txt | sed 's/^/dropped: /'; cat delete.txt
rc=0
dropped: notes
.dotfile
repo-index
repo-same
top-file.txt

Expected: rc=0 twice, and delete.txt lists the planned entries that are still identical or git-noise; dropped: lines name those that are not. When the classifier refuses (exit 2), verdicts2.tsv is left empty (0 bytes in the slashless test), so delete.txt comes out empty.

Delete the entries in delete.txt from the copy.

WarningThis step is irreversible on this machine

The entries are gone from the copy. Their files matched the source by checksum at step 7, apart from .git/index and FETCH_HEAD in git-noise entries, whose versions in the copy cannot be brought back. Undo pulls everything else back from the source while the source keeps it.

Size first:

$ (cd "$LOCAL" && xargs -d '\n' du -shc --) < delete.txt
4.0K	.dotfile
132K	repo-index
128K	repo-same
4.0K	top-file.txt
268K	total

Then delete:

while IFS= read -r e; do
  if [ -z "$e" ] || [ "$e" = . ] || [ "$e" = .. ] || [ "${e#*/}" != "$e" ]; then echo "skip: $e" >&2; continue; fi
  rm -rf -- "${LOCAL:?}/$e" && printf 'removed\t%s\n' "$e"
done < delete.txt | tee -a removed.log

$ while IFS= read -r e; do
  if [ -z "$e" ] || [ "$e" = . ] || [ "$e" = .. ] || [ "${e#*/}" != "$e" ]; then echo "skip: $e" >&2; continue; fi
  rm -rf -- "${LOCAL:?}/$e" && printf 'removed\t%s\n' "$e"
done < delete.txt | tee -a removed.log
removed	.dotfile
removed	repo-index
removed	repo-same
removed	top-file.txt

Expected: one removed line per entry in delete.txt and no skip: line. The loop refuses empty names, ., .. and anything containing /, and ${LOCAL:?} makes the shell refuse the rm when $LOCAL is empty or unset instead of running it against /<entry>.

Re-check the copy against the source while you still have access.

$ find "$LOCAL" -mindepth 1 -maxdepth 1 -printf '%f\n' | sort > entries-after.txt
rsync --dry-run -rlc --itemize-changes --out-format='%i %n' "$LOCAL"/ "$SRC":"$REMOTE"/ > dry3.txt 2> dry3.err; echo "rc=$?"
awk -f classify.awk entries-after.txt dry3.txt dry3.err
rc=0
differs	edited
only-local	gone-there
differs	grown
differs	notes

Expected: rc=0, and only only-local and differs verdicts, plus any entry you took out of planned.txt on purpose. notes is differs here because of the source-side edit from step 7.

Remove the temporary key from the source. This is the one change on the source after step 1; ask its owner first.
```
BLOB=$(cut -d' ' -f2 ~/.ssh/<KEY>.pub)
$RSYNC_RSH "$SRC" "cd ~/.ssh && cp -p authorized_keys authorized_keys.tmpkey-bak && { grep -vF '$BLOB' authorized_keys.tmpkey-bak > authorized_keys; [ \$? -le 1 ]; } && rm authorized_keys.tmpkey-bak; grep -cF '$BLOB' authorized_keys"
```
Expected: 0, the number of lines still holding the key (the command exits 1 because that last grep matched nothing). This edits authorized_keys only, the file ssh-copy-id appended to; sshd also reads authorized_keys2 by default, which step 1 did not touch. > rewrites the file in place, so it keeps its mode; the backup is deleted only after the filter succeeded. If grep reports an error instead, authorized_keys may be cut short and the backup is still there: cp -p ~/.ssh/authorized_keys.tmpkey-bak ~/.ssh/authorized_keys on the source puts it back (not run). The same command, run through the loopback shell against a fake home, left a file that had held two other keys plus this one with exactly the two others, and a file that had held only this key empty (0 bytes); both kept mode 600, and the backup was gone both times. It needs a POSIX-style login shell on the source and was tested with /bin/sh only. [UNVERIFIED: not run against a real sshd.]
Close any shared connection and check that the key no longer logs in.
```
ssh -O exit "$SRC"
ssh -G -F /dev/null -i ~/.ssh/<KEY> -o IdentitiesOnly=yes -o BatchMode=yes -o ControlPath=none "$SRC" true | grep -E '^(identityfile|identitiesonly|batchmode|controlpath) '
ssh -F /dev/null -i ~/.ssh/<KEY> -o IdentitiesOnly=yes -o BatchMode=yes -o ControlPath=none "$SRC" true; echo "rc=$?"
```
ssh -O exit closes a master connection if one exists. Without one it fails harmlessly; with a control path that has no master behind it, a local test printed:
```
$ ssh -F /dev/null -o ControlPath=/nonexistent/master.sock -O exit "$SRC"; echo "rc=$?"
Control socket connect(/nonexistent/master.sock): No such file or directory
rc=255
```
ssh -G prints the configuration the test login will use and exits without connecting. With -F /dev/null, ssh reads no ~/.ssh/config and no system-wide config, so the temporary key is the only identity:
```
$ ssh -G -F /dev/null -i keytest/tmp-key -o IdentitiesOnly=yes -o BatchMode=yes -o ControlPath=none "$SRC" true | grep -E '^(identityfile|identitiesonly|batchmode|controlpath) '
batchmode yes
identitiesonly yes
identityfile keytest/tmp-key
```
Without -F /dev/null, an IdentityFile line in a config still joins the list, even with IdentitiesOnly=yes:
```
$ cat keytest/ssh_config; ssh -G -F keytest/ssh_config -i keytest/tmp-key -o IdentitiesOnly=yes "$SRC" true | grep '^identityfile '
Host *
    IdentityFile keytest/other-key
identityfile keytest/tmp-key
identityfile keytest/other-key
```
Expected: the test login fails with a Permission denied message and rc=255. Any other error (Could not resolve hostname, Connection refused) means the test never reached authentication; if the source needs settings from ~/.ssh/config, such as Port or ProxyJump, add them as -o options and test again. Only after a Permission denied, delete the local key files: rm ~/.ssh/<KEY> ~/.ssh/<KEY>.pub; unset RSYNC_RSH.

GotchaAn open master connection logs in without the key

With ControlMaster and ControlPersist in ~/.ssh/config, a master connection stays open in the background after the rsync runs, and later sessions reuse its network connection instead of opening a new one. A test login over it reuses a connection that was authenticated before the key was removed, so it says nothing about the key. -O exit closes the master, ControlPath=none keeps the test off any master, and BatchMode=yes stops ssh from falling back to a password prompt.

[UNVERIFIED: the test login was not run against a real sshd. In the original run the multiplexed connection was closed and a fresh login was checked to fail.]

Rsync on tsvinc

Prune a stale synced copy with an rsync checksum dry run

Before you start#

Steps#