Prune a stale synced copy with an rsync checksum dry run

Steps#

1. Create a temporary key and install it on the source.

   ssh-keygen -t ed25519 -N '' -C tmp-compare -f ~/.ssh/<KEY>
   ssh-copy-id -i ~/.ssh/<KEY>.pub "$SRC"
   export RSYNC_RSH="ssh -i $HOME/.ssh/<KEY> -o IdentitiesOnly=yes"
   

   ssh-copy-id appends the public key to ~/.ssh/authorized_keys on the source, creating the file if needed; it logs in with the account's password (or a key that already works) to do so. RSYNC_RSH is the remote shell rsync uses when no -e is given, so every rsync command below goes through this ssh command. rsync splits the value into separate arguments: a logging shim set as RSYNC_RSH received -i, the key path, -o and IdentitiesOnly=yes as four arguments, followed by the host and rsync. IdentitiesOnly=yes keeps keys held by ssh-agent out of the login; IdentityFile lines from ~/.ssh/config still apply, which is why step 11 tests the key with -F /dev/null. Expected: ssh-copy-id reports one key added. [UNVERIFIED: steps 1 and 2 not run against an SSH server for this manual. The original run used a temporary key; its setup commands were not recorded.]

2. Check the path and rsync on the source without listing anything. The loopback run (its "source" is a second directory on the same machine, so it reports the local rsync):

   $ $RSYNC_RSH "$SRC" "test -d $REMOTE && echo dir-ok; command -v rsync && rsync --version | head -1"; echo "rc=$?"
   dir-ok
   /usr/bin/rsync
   rsync  version 3.5.1-g315819c5  protocol version 33
   rc=0
   

   Expected: three lines, dir-ok, the path of rsync on the source and its version. No dir-ok line: $REMOTE is not a directory there (in the loopback run, a path that does not exist printed only the two rsync lines). No path and version: rsync is not installed there, or not in the remote shell's PATH. The final rc comes from the last command and says nothing about the directory.

3. Run the checksum dry run, from the copy to the source. The command from the original run, with LOCAL, host and REMOTE as its placeholders: rsync --dry-run -rlc --itemize-changes --out-format='%i %n' LOCAL/ host:REMOTE/ The same command with the variables from Before you start, saving both output streams:

   $ rsync --dry-run -rlc --itemize-changes --out-format='%i %n' "$LOCAL"/ "$SRC":"$REMOTE"/ > dry.txt 2> dry.err; echo "rc=$?"
   rc=0
   

   Expected: rc=0 and an empty dry.err (step 4 shows its line count). Anything else: find the cause under Failure modes before going on; step 5 refuses to classify while dry.err has a line.

   • --dry-run: rsync works out what it would send and reports it; no file contents are sent and nothing is written on the source. The copy's file list, with names, sizes and checksums, does go to the source (Notes). In the loopback runs, a listing of the source tree with sizes, mtimes and ctimes was the same before and after seven dry runs.
   • -c: decide by checksum, not by size and modification time. The local side checksums its files while it builds the file list; the source checksums each of its files that has the same size as the local one, and a file whose size differs is reported without one. The algorithm is negotiated between the two ends (xxh128 between the two rsync 3.5.1 ends of the loopback run), so "identical" below means the same checksum. Without -c, the loopback run missed a same-size edit with an unchanged mtime and flagged a file whose bytes matched but whose mtime did not (Notes).
   • -r -l: recurse, and compare symlinks as links instead of following them. Not -a: its -t -p -o -g add attribute-only lines, such as .f..t...... notes/touched.txt for a file with identical bytes, and those would turn an identical entry into differs.
   • --itemize-changes --out-format='%i %n': one line per item that would change, made of an 11-character change code, a space and the path relative to $LOCAL. The explicit format leaves out the -> target that --itemize-changes alone appends to symlinks, so every line splits the same way.
   • "$LOCAL"/ with the trailing slash: compare the contents of the copy with the contents of $REMOTE. The form without the slash is Failure modes, row 4.
   • No --delete: it would itemize files that exist only on the source, and without --dry-run it would delete them there.

4. Read the itemized lines.

   $ cat dry.txt; wc -l dry.txt dry.err
   <fcsT...... edited/changed-size.txt
   <fc.T...... edited/same-size.txt
   cd+++++++++ gone-there/
   <f+++++++++ gone-there/NOTES
   cd+++++++++ gone-there/src/
   <f+++++++++ gone-there/src/main.c
   <f+++++++++ grown/new-here.txt
   <fcsT...... repo-index/.git/FETCH_HEAD
   <fc.T...... repo-index/.git/index
     9 dry.txt
     0 dry.err
     9 total
   

   In this direction the local side is the sender, so every file line starts with <, sent to the remote host. On the loopback trees, notes/, repo-same/, top-file.txt and .dotfile print nothing: every file in them matched by checksum, including one whose mtime differs on the source, and the symlink in notes/ points at the same target on both sides.

   Line	Meaning here
   no line	the item exists on the source with the same checksum (a symlink: the same target)
   <f+++++++++ path	file missing on the source
   cd+++++++++ dir/	directory missing on the source; everything under it follows as new
   cL+++++++++ link	symlink missing on the source
   <fcsT...... path	content (c) and size (s) differ
   <fc.T...... path	same size, different content

   A leading c means the receiver would create the item, a directory or symlink here. The T only says the receiver would set the transfer time as mtime, because times are not preserved; it carries no information here. Line codes by kind:

   $ cut -c1-11 dry.txt | sort | uniq -c
         2 cd+++++++++
         3 <f+++++++++
         2 <fcsT......
         2 <fc.T......
   

   GotchaCount <f, not >f

   > marks an item received by the local side, which happens in a pull. In the original run my first classifier looked for > and counted zero.

   $ grep -c '^>f' dry.txt
   0
   $ grep -c '^<f' dry.txt
   7
   

5. Give each top-level entry a verdict. Save this as classify.awk in the working directory:

   # classify.awk: one verdict per top-level entry of LOCAL, from a push-direction
   #   rsync --dry-run -rlc --itemize-changes --out-format='%i %n' LOCAL/ host:REMOTE/ > dry.txt 2> dry.err
   # usage: awk -f classify.awk entries.txt dry.txt dry.err > verdicts.tsv
   #   identical   no line: every file under it exists on the source with the same checksum
   #   git-noise   only .git/index or .git/FETCH_HEAD differ
   #   only-local  the entry itself is missing on the source
   #   differs     anything else (new or changed files inside)
   # Any line in dry.err, or any line of dry.txt it cannot map to an entry,
   # aborts with exit 2 and no verdicts: an unreadable directory prints no line.
   FILENAME == ARGV[1] { order[++n] = $0; v[$0] = "identical"; next }
   FILENAME == ARGV[3] { print "rsync stderr: " $0 > "/dev/stderr"; bad = 1; next }
   {
       code = substr($0, 1, 11); path = substr($0, 13); sub(/\/$/, "", path)
       top = path; sub(/\/.*/, "", top)
       if (substr($0, 12, 1) != " " || !(top in v)) {
           print "unmapped line: " $0 > "/dev/stderr"; bad = 1; next
       }
       new = (substr(code, 3) == "+++++++++")
       if (path == top && new) v[top] = "only-local"
       else if (v[top] == "only-local") next
       else if (!new && path ~ /\/\.git\/(index|FETCH_HEAD)$/) {
           if (v[top] == "identical") v[top] = "git-noise"
       }
       else v[top] = "differs"
   }
   END {
       if (bad) exit 2
       for (i = 1; i <= n; i++) printf "%s\t%s\n", v[order[i]], order[i]
   }
   

   List the entries, then classify:

   $ find "$LOCAL" -mindepth 1 -maxdepth 1 -printf '%f\n' | sort > entries.txt; cat entries.txt
   .dotfile
   edited
   gone-there
   grown
   notes
   repo-index
   repo-same
   top-file.txt
   $ awk -f classify.awk entries.txt dry.txt dry.err > verdicts.tsv; echo "rc=$?"; cat verdicts.tsv
   rc=0
   identical	.dotfile
   differs	edited
   only-local	gone-there
   differs	grown
   identical	notes
   git-noise	repo-index
   identical	repo-same
   identical	top-file.txt
   

   Expected: rc=0 and one verdict per entry. rc=2 with rsync stderr: or unmapped line: messages and an empty verdicts.tsv: see Failure modes.

   • identical: no line under the entry; every file in it exists on the source with the same checksum, and every symlink with the same target.
   • git-noise: the only differing files are .git/index and .git/FETCH_HEAD, and every object, ref and work-tree file matched by checksum. The index caches each tracked file's stat data, which differs between two machines even for identical files, and git status rewrites it when it refreshes that data; FETCH_HEAD records the last git fetch. On the loopback tree, one git status on the source side and a fetch there from a second URL produced exactly this. The index is also git's staging area, so whatever the copy had staged goes with it; the objects it names are on the source, because no file under .git/objects differed. Any other differing file under .git leaves the entry as differs.
   • only-local: the entry itself is missing on the source; the copy is the only one.
   • differs: new or changed files inside; either side may hold the newer version.

6. List the deletion candidates and look at what stays.

   $ awk -F'\t' '$1 == "identical" || $1 == "git-noise" { print $2 }' verdicts.tsv | sort > planned.txt; cat planned.txt
   .dotfile
   notes
   repo-index
   repo-same
   top-file.txt
   $ awk -F'\t' '$1 == "only-local" || $1 == "differs"' verdicts.tsv
   differs	edited
   only-local	gone-there
   differs	grown
   

   Expected: planned.txt holds only identical and git-noise entries. Remove from it anything you want to keep anyway. The other entries stay: an only-local entry exists nowhere else, and a differs entry may hold the newer copy of a file.

7. Re-run the dry run right before deleting, and keep only entries that still qualify. The source can change between step 3 and the deletion:

   rsync --dry-run -rlc --itemize-changes --out-format='%i %n' "$LOCAL"/ "$SRC":"$REMOTE"/ > dry2.txt 2> dry2.err; echo "rc=$?"; cat dry2.err
   awk -f classify.awk entries.txt dry2.txt dry2.err > verdicts2.tsv; echo "rc=$?"
   awk -F'\t' '$1 == "identical" || $1 == "git-noise" { print $2 }' verdicts2.tsv | sort | comm -12 planned.txt - > delete.txt
   comm -23 planned.txt delete.txt | sed 's/^/dropped: /'; cat delete.txt
   

   In the loopback run, a line was appended to notes/b.txt on the source between the two dry runs (a harness step, to show what a change does):

   $ rsync --dry-run -rlc --itemize-changes --out-format='%i %n' "$LOCAL"/ "$SRC":"$REMOTE"/ > dry2.txt 2> dry2.err; echo "rc=$?"; cat dry2.err
   rc=0
   $ awk -f classify.awk entries.txt dry2.txt dry2.err > verdicts2.tsv; echo "rc=$?"
   awk -F'\t' '$1 == "identical" || $1 == "git-noise" { print $2 }' verdicts2.tsv | sort | comm -12 planned.txt - > delete.txt
   comm -23 planned.txt delete.txt | sed 's/^/dropped: /'; cat delete.txt
   rc=0
   dropped: notes
   .dotfile
   repo-index
   repo-same
   top-file.txt
   

   Expected: rc=0 twice, and delete.txt lists the planned entries that are still identical or git-noise; dropped: lines name those that are not. When the classifier refuses (exit 2), verdicts2.tsv is left empty (0 bytes in the slashless test), so delete.txt comes out empty.

8. Delete the entries in delete.txt from the copy.

   WarningThis step is irreversible on this machine

   The entries are gone from the copy. Their files matched the source by checksum at step 7, apart from .git/index and FETCH_HEAD in git-noise entries, whose versions in the copy cannot be brought back. Undo pulls everything else back from the source while the source keeps it.

   Size first:

   $ (cd "$LOCAL" && xargs -d '\n' du -shc --) < delete.txt
   4.0K	.dotfile
   132K	repo-index
   128K	repo-same
   4.0K	top-file.txt
   268K	total
   

   Then delete:

   while IFS= read -r e; do
     if [ -z "$e" ] || [ "$e" = . ] || [ "$e" = .. ] || [ "${e#*/}" != "$e" ]; then echo "skip: $e" >&2; continue; fi
     rm -rf -- "${LOCAL:?}/$e" && printf 'removed\t%s\n' "$e"
   done < delete.txt | tee -a removed.log
   

   $ while IFS= read -r e; do
     if [ -z "$e" ] || [ "$e" = . ] || [ "$e" = .. ] || [ "${e#*/}" != "$e" ]; then echo "skip: $e" >&2; continue; fi
     rm -rf -- "${LOCAL:?}/$e" && printf 'removed\t%s\n' "$e"
   done < delete.txt | tee -a removed.log
   removed	.dotfile
   removed	repo-index
   removed	repo-same
   removed	top-file.txt
   

   Expected: one removed line per entry in delete.txt and no skip: line. The loop refuses empty names, ., .. and anything containing /, and ${LOCAL:?} makes the shell refuse the rm when $LOCAL is empty or unset instead of running it against /<entry>.

9. Re-check the copy against the source while you still have access.

   $ find "$LOCAL" -mindepth 1 -maxdepth 1 -printf '%f\n' | sort > entries-after.txt
   rsync --dry-run -rlc --itemize-changes --out-format='%i %n' "$LOCAL"/ "$SRC":"$REMOTE"/ > dry3.txt 2> dry3.err; echo "rc=$?"
   awk -f classify.awk entries-after.txt dry3.txt dry3.err
   rc=0
   differs	edited
   only-local	gone-there
   differs	grown
   differs	notes
   

   Expected: rc=0, and only only-local and differs verdicts, plus any entry you took out of planned.txt on purpose. notes is differs here because of the source-side edit from step 7.

10. Remove the temporary key from the source. This is the one change on the source after step 1; ask its owner first.

    BLOB=$(cut -d' ' -f2 ~/.ssh/<KEY>.pub)
    $RSYNC_RSH "$SRC" "cd ~/.ssh && cp -p authorized_keys authorized_keys.tmpkey-bak && { grep -vF '$BLOB' authorized_keys.tmpkey-bak > authorized_keys; [ \$? -le 1 ]; } && rm authorized_keys.tmpkey-bak; grep -cF '$BLOB' authorized_keys"
    

    Expected: 0, the number of lines still holding the key (the command exits 1 because that last grep matched nothing). This edits authorized_keys only, the file ssh-copy-id appended to; sshd also reads authorized_keys2 by default, which step 1 did not touch. > rewrites the file in place, so it keeps its mode; the backup is deleted only after the filter succeeded. If grep reports an error instead, authorized_keys may be cut short and the backup is still there: cp -p ~/.ssh/authorized_keys.tmpkey-bak ~/.ssh/authorized_keys on the source puts it back (not run). The same command, run through the loopback shell against a fake home, left a file that had held two other keys plus this one with exactly the two others, and a file that had held only this key empty (0 bytes); both kept mode 600, and the backup was gone both times. It needs a POSIX-style login shell on the source and was tested with /bin/sh only. [UNVERIFIED: not run against a real sshd.]

11. Close any shared connection and check that the key no longer logs in.

    ssh -O exit "$SRC"
    ssh -G -F /dev/null -i ~/.ssh/<KEY> -o IdentitiesOnly=yes -o BatchMode=yes -o ControlPath=none "$SRC" true | grep -E '^(identityfile|identitiesonly|batchmode|controlpath) '
    ssh -F /dev/null -i ~/.ssh/<KEY> -o IdentitiesOnly=yes -o BatchMode=yes -o ControlPath=none "$SRC" true; echo "rc=$?"
    

    ssh -O exit closes a master connection if one exists. Without one it fails harmlessly; with a control path that has no master behind it, a local test printed:

    $ ssh -F /dev/null -o ControlPath=/nonexistent/master.sock -O exit "$SRC"; echo "rc=$?"
    Control socket connect(/nonexistent/master.sock): No such file or directory
    rc=255
    

    ssh -G prints the configuration the test login will use and exits without connecting. With -F /dev/null, ssh reads no ~/.ssh/config and no system-wide config, so the temporary key is the only identity:

    $ ssh -G -F /dev/null -i keytest/tmp-key -o IdentitiesOnly=yes -o BatchMode=yes -o ControlPath=none "$SRC" true | grep -E '^(identityfile|identitiesonly|batchmode|controlpath) '
    batchmode yes
    identitiesonly yes
    identityfile keytest/tmp-key
    

    Without -F /dev/null, an IdentityFile line in a config still joins the list, even with IdentitiesOnly=yes:

    $ cat keytest/ssh_config; ssh -G -F keytest/ssh_config -i keytest/tmp-key -o IdentitiesOnly=yes "$SRC" true | grep '^identityfile '
    Host *
        IdentityFile keytest/other-key
    identityfile keytest/tmp-key
    identityfile keytest/other-key
    

    Expected: the test login fails with a Permission denied message and rc=255. Any other error (Could not resolve hostname, Connection refused) means the test never reached authentication; if the source needs settings from ~/.ssh/config, such as Port or ProxyJump, add them as -o options and test again. Only after a Permission denied, delete the local key files: rm ~/.ssh/<KEY> ~/.ssh/<KEY>.pub; unset RSYNC_RSH.

    GotchaAn open master connection logs in without the key

    With ControlMaster and ControlPersist in ~/.ssh/config, a master connection stays open in the background after the rsync runs, and later sessions reuse its network connection instead of opening a new one. A test login over it reuses a connection that was authenticated before the key was removed, so it says nothing about the key. -O exit closes the master, ControlPath=none keeps the test off any master, and BatchMode=yes stops ssh from falling back to a password prompt.

    [UNVERIFIED: the test login was not run against a real sshd. In the original run the multiplexed connection was closed and a fresh login was checked to fail.]