A -> B -> C -> A: benchmarking a kernel cmdline change you have to reboot for

The third boot that afternoon changed nothing, on purpose. Same kernel, 7.2.9-300.fc45. Same command line, byte for byte. Same IOMMU group summary in its meta.txt: one DMA group, 32 DMA-FQ, two identity. It was the baseline boot again, run because the test plan said to, and its bench started about 22 minutes after the first baseline's.

gpt-oss-120b's Vulkan prefill at 64k context came out 6.2 % faster than on the first baseline boot.

The change being tested, iommu=pt, had just scored +14.7 % on that row. The do-nothing reboot alone was worth 6.2 of it. At 32k the change scored +6.9 % and the repeat boot +4.6 %: two-thirds of the result was the box being in a slightly different mood after a power cycle.

TL;DR — A kernel parameter can only be tested across reboots, and reboots move the numbers by themselves: here by up to 6 % on deep-context prefill, same cmdline. Run four boots, A -> B -> C -> A2, and treat A2 vs A as the noise floor for every row. Verify each boot from sysfs, not from a dmesg line, because the kernel printed Default domain type: Translated on a boot where the IOMMU was never initialised and no groups existed. Check what the setting touches before you measure: on this box the GPU was identity-mapped with the IOMMU on in both modes, so iommu=pt could not change its DMA path. Report the cost next to the gain.

Assumes Fedora (or any GRUB/BLS distro with grubby), SSH to the test box, and a benchmark that writes JSON.

Drift: what one boot per side hides#

The box is a GEEKOM A9 Mega (Ryzen AI Max+ 395, 128 GB); the question, whether IOMMU passthrough speeds up LLM prefill. Round one ran four boots on 13 October: translated (stock), passthrough, translated again, passthrough again, five llama-bench reps per row. In round one's directory names, off means iommu=pt off, the stock translated mode; on means passthrough.

$ python3 -I scripts/iommu-ab-compare.py data/iommu-ab/off data/iommu-ab/on data/iommu-ab/off2 data/iommu-ab/on2 | grep -E 'model|---|gptoss \| pp512 \| [36]'

The ± is the spread of five reps inside one boot. It says nothing about the next boot. The off2 vs off column does: 42 % of the "effect" at 64k, two-thirds of it at 32k.

Two pairs did agree on the sign: passthrough was ahead in both, +6.9 % and +1.4 % at 32k (342.9 against 338.0 in pair two). Sign across pairs tells you the direction. It can't give you the size, and the size is what gets quoted.

A single A/B pair, the obvious way to test a boot parameter, would have reported 14.7 % with a straight face.

Map what the parameter touches before you measure#

The second trap was bigger. The hypothesis was that the GPU reads its 120 GiB of GTT memory through IOMMU translation, and passthrough removes that cost. Nobody had asked which devices sit in which IOMMU group, and the kernel publishes it without root:

$ for g in /sys/kernel/iommu_groups/*; do
    for d in "$g"/devices/*; do
      printf '%s %s %s\n' "${g##*/}" "$(cat "$g/type")" "$(lspci -s "${d##*/}")"
    done
  done 2>/dev/null | sort -n | grep -E 'DMA|Display|Neural'
18 DMA c2:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. RTS525A PCI Express Card Reader (rev 01)
22 identity c6:00.0 Display controller: Advanced Micro Devices, Inc. [AMD/ATI] Strix Halo [Radeon Graphics / Radeon 8050S Graphics / Radeon 8060S Graphics] (rev c1)
29 identity c7:00.1 Signal processing controller: Advanced Micro Devices, Inc. [AMD] Strix/Krackan/Strix Halo Neural Processing Unit (rev 11)

That is the passthrough boot: the only translated group left is the SD card reader. The 2>/dev/null is there because group 0 holds MSFT0201:00, a platform device, and lspci -s refuses a name that isn't a PCI address.

The type values come from the kernel's sysfs ABI: DMA and DMA-FQ mean the group's DMA is translated, and identity means "All the DMA transactions from the device in this group are not translated by the iommu. Maximum performance but zero protection."

The stock translated boot has one DMA group, 32 DMA-FQ and two identity. The two identity groups there are 22 and 29: the GPU and the NPU. The kernel had already left both untranslated. iommu=pt moved the other 32 groups (NVMe, NICs, USB) and nothing the benchmark was measuring. Only amd_iommu=off changes the GPU's DMA path. The IOMMU post has the whole correction.

Run the map on both sides first. If the device in your hypothesis has the same type in both modes, any gain you measure needs another explanation.

The boot order: A -> B -> C -> A2#

Round two tested all three modes, one boot each, with the standing config at both ends. The rows: Flash-Next (fn-vk, a Qwen3.8 GGUF) and gpt-oss-120b on llama.cpp Vulkan, gpt-oss-120b on ROCm, and Strata, a separate inference engine, serving a 64,000-token prompt five times. The times are the kernel's first log line per boot, 14 October:

Two hours and 39 minutes, with the job queue parked throughout. A goes last as well as first because it's the config the box keeps: the drift measurement and the cleanup are one boot.

Same rows, same five reps, same binaries on every boot. meta.txt records the llama.cpp build (79e2e74eb) and the engine commit on each side, so a rebuild in the middle would show.

The toggle: strip every managed arg, then add one#

The script that sets a mode, fix-19-iommu-mode-toggle.sh, does the removal first, on every boot entry:

set -euo pipefail
MODE="${1:?pt|translated|off}"
grubby --update-kernel=ALL --remove-args="iommu=pt amd_iommu=off"
case "$MODE" in
  pt)         grubby --update-kernel=ALL --args=iommu=pt ;;
  translated) ;;
  off)        grubby --update-kernel=ALL --args=amd_iommu=off ;;
  *)          echo "usage: $0 pt|translated|off" >&2; exit 2 ;;
esac
grubby --info=DEFAULT | grep -E '^args='
cat /etc/kernel/cmdline
systemctl reboot

Three choices in there carry the weight:

• Remove all, then add one. Adding amd_iommu=off to a cmdline that still carries iommu=pt gives a boot that is both and neither.
• --update-kernel=ALL. DEFAULT changes one entry, and a fallback or freshly installed kernel then boots without the mode.
• Print the result. On Fedora, grubby also writes the args into /etc/kernel/cmdline and GRUB_CMDLINE_LINUX, the same mechanism that kept the GTT limit from the ttm post alive across kernel updates. The cat shows what the next kernel install will inherit.

My root wrapper copies a script file to the box and runs it under sudo -S, with no way to pass arguments. So each mode gets a fixed-argument copy, made with one (GNU) sed:

sed 's/^set -euo pipefail$/&\nset -- off/' harness/fix-19-iommu-mode-toggle.sh > fix-19-off.sh

Then wait for the box against a value taken before the reboot, never a guessed clock time:

T=$(date +%s)
# ... run the toggle ...
until ssh a9 "test \$(date -d \"\$(uptime -s)\" +%s) -gt $T"; do sleep 10; done

That is the loop the series ran with. uptime -s is derived from the wall clock, though, and this box once came up with its RTC three hours off. /proc/sys/kernel/random/boot_id changes on every boot and ignores clocks. The version I'd use now, checked against the live box (it keeps waiting while the ID is unchanged or the host is unreachable, and exits once it differs):

B=$(ssh a9 cat /proc/sys/kernel/random/boot_id)
# ... run the toggle ...
until n=$(ssh -o ConnectTimeout=5 a9 cat /proc/sys/kernel/random/boot_id) && [ "$n" != "$B" ]; do sleep 10; done

Verify each boot from sysfs, not dmesg#

The toggle script's own header says to verify a boot with journalctl -k -b | grep -iE 'Default domain type|AMD-Vi|iommu'. Here is that grep on the amd_iommu=off boot, with the command-line echo filtered out:

$ journalctl -k -b 79060c21363e4a3aaa1b4c3911e5e399 --no-pager | grep -iE 'Default domain type|AMD-Vi|iommu' | grep -iv 'command line'
Oct 14 20:33:35 a9-mega kernel: iommu: Default domain type: Translated
Oct 14 20:33:35 a9-mega kernel: iommu: DMA domain TLB invalidation policy: lazy mode
Oct 14 20:33:35 a9-mega kernel: amdxdna 0000:c7:00.1: [drm] *ERROR* aie2_init: Running without IOMMU not supported

Default domain type: Translated, on a boot where the kernel docs say amd_iommu=off means "do not initialize any AMD IOMMU found in the system". Whatever that line reports, it isn't whether an IOMMU is running. Read on its own, it certifies the wrong mode.

The same grep returns 57 lines on the translated boot and 55 on passthrough, mostly Adding to iommu group. On the off boot it returns three. The line count is a better verifier than the line, and sysfs beats both. The bench script writes this into meta.txt before the first row:

$ sed 's/root=UUID=[^ ]*/root=UUID=…/' data/iommu-ab2/off/meta.txt
2026-10-14T20:34:09+03:00
7.2.9-300.fc45.x86_64
BOOT_IMAGE=(hd0,gpt2)/vmlinuz-7.2.9-300.fc45.x86_64 root=UUID=… ro rootflags=subvol=root rhgb quiet ttm.pages_limit=31457280 ttm.page_pool_size=31457280 amdgpu.lockup_timeout=10000,60000,10000,10000 amd_iommu=off
iommu group types: 
Oct 14 20:33:35 a9-mega kernel: iommu: Default domain type: Translated
79e2e74eb 2026-10-09
61b3fb5dd3f1e8ec09cf7e4e05208bc6d3c46406
start 2026-10-14T20:35:09+03:00 load 0.12 0.07 0.02 1/601 2574
fn-vk rc=0 2026-10-14T20:51:23+03:00
gptoss-vk rc=0 2026-10-14T20:58:28+03:00
gptoss-rocm rc=0 2026-10-14T21:02:19+03:00
strata rc=0 2026-10-14T21:11:57+03:00
npu-flm-validate rc=1 2026-10-14T21:12:03+03:00
end 2026-10-14T21:12:03+03:00

iommu group types: with nothing after it: no groups exist. That empty field, directly above the log line that says otherwise, is the proof the boot is the one you asked for. The same rule covers other knobs: ttm.pages_limit is checked in /sys/module/ttm/parameters/, mitigations=off in /sys/devices/system/cpu/vulnerabilities/.

The per-boot bench: meta.txt and an idle gate#

Every boot gets the same script, iommu-ab2-bench.sh <label>, run in tmux so a dropped SSH session doesn't end it. Before each row it waits for this:

idle() { local n=0 l g; while [ $n -lt 6 ]; do l=$(cut -d' ' -f1 /proc/loadavg); g=$(cat /sys/class/drm/card*/device/gpu_busy_percent 2>/dev/null | head -1)
  if awk "BEGIN{exit !($l<1 && ${g:-0}<5)}"; then n=$((n+1)); else n=0; fi; sleep 10; done; }

Load under 1 and GPU busy under 5 %, six checks in a row, ten seconds apart; one busy check resets the count. The minimum is a minute, and on the first boot that's what it took: meta.txt was written at 19:13:25 and the first row started at 19:14:25.

The gate exists because the queue that normally owns this box once ran a download under a benchmark.

Reading the table: deltas against the drift column#

The compare script takes the side directories in order and prints each one against the first. Put A2 last and its column is your noise floor:

$ python3 -I scripts/iommu-ab-compare.py data/iommu-ab2/pt data/iommu-ab2/translated data/iommu-ab2/off data/iommu-ab2/pt2

Read it one row at a time. A delta counts when it clears two bars: at least twice the row's drift, |pt2 vs pt|, and more than the within-boot spread of the baseline (pt's ±, as a percentage of its mean). One caveat on Strata: its ± pools five runs where run 1 is cold and the rest follow the same run-to-run pattern on every boot, so for Strata compare run 1 with run 1, run 2 with run 2.

• gpt-oss ROCm and Flash-Next prefill at 64k. amd_iommu=off gives +4.2 % and +2.6 % against drifts of 0.2 and 0.1 % and spreads of about 1.2 and 1.7 %. Both count, the second narrowly.
• Strata, run by run. Off vs pt prefill is +4.6 to +5.0 % on every one of the five runs; pt2 vs pt is -0.7 to -1.0 %. Decode is +1.8 to +2.3 % on every run against -0.2 to -0.4 %. Both count.
• gpt-oss ROCm decode. +0.6 % against a 0.2 % drift and a spread under 0.4 %. It clears both bars, and it's a real 0.6 %: too small to change anything.
• Flash-Next decode. +2.3 % and +2.4 % look like the Strata story until the drift column says +1.6 % and +1.7 %, same sign. Neither is twice its drift. Not claimable.
• Translated vs pt. Nothing clears both bars outside the noisy row below. -0.9 % and -0.7 % on the 64k prefill rows clear their tiny drifts but sit inside their spreads. gpt-oss Vulkan at 32k is -2.5 % against a drift of -2.8 %: the repeat of the same setting moved more than the change did.
• gpt-oss Vulkan prefill at 64k. Off is -10.3 %, the only row where turning the IOMMU off clearly made things slower, and translated is -7.5 %. The within-boot spread is ±6.5 on 117 tok/s, about 5.5 %, and one repeat boot can't characterise a row that loud. On paper both deltas clear both bars. I'm setting the row aside anyway: both pt boots high, both other modes low, against the direction off moved everywhere else. I can't tell noise from a different cause here, so the row gets reported and not explained.

That last row is the same row that produced round one's +14.7 %.

Cost next to the gain#

The +5 % had a price, and the bench script went looking for it. On the off boot it ran flm validate from the NPU runtime, which exited 1 with [ERROR] No NPU device found. The kernel had already said why:

$ journalctl -k -b 79060c21363e4a3aaa1b4c3911e5e399 -o cat --no-pager | grep amdxdna
amdxdna 0000:c7:00.1: [drm] *ERROR* aie2_init: Running without IOMMU not supported
amdxdna 0000:c7:00.1: [drm] *ERROR* amdxdna_probe: Hardware init failed, ret -22
amdxdna 0000:c7:00.1: probe with driver amdxdna failed with error -22

Add the parts nobody benchmarks: no DMA isolation for any device and no interrupt remapping. Back on A2, /dev/accel/accel0 returned. The box stays on iommu=pt, with off kept as a documented switch for LLM-only sessions that don't need the NPU.

Put a functional test of whatever the setting could break into the bench script, and record its exit code in meta.txt like any other row.

Without grubby (untested here)#

Everything above ran on Fedora with grubby and BLS entries. On Debian-style GRUB the toggle step edits GRUB_CMDLINE_LINUX and regenerates the config; on systemd-boot or a UKI it edits /etc/kernel/cmdline and rebuilds. I haven't run either through this method; the boot order, the sysfs check and the drift column don't care.

Filed under: linux, kernel, benchmarks, amd, fedora

Read next#

• IOMMU off buys a 128 GB iGPU 5 %, and costs the NPU — the result this method produced, and the round-one explanation it retracted.
• 120 GiB for the GPU, set in the kernel, not the BIOS — the other boot parameter on this box, set with the same grubby --update-kernel=ALL.
• The download that contaminated the benchmark — why the job queue gets parked for the whole series.

Sources#

• Linux kernel parameters — the amd_iommu=off and iommu=pt entries, fetched 16 October 2026. (The pt entry reads "Use passththrough mode by default". Upstream's spelling.)
• Linux ABI, /sys/kernel/iommu_groups/<grp_id>/type — what DMA, DMA-FQ and identity mean, fetched 16 October 2026.
• Benchmark data and scripts from the A9 Mega test repository: harness/fix-19-iommu-mode-toggle.sh, harness/iommu-ab2-bench.sh, scripts/iommu-ab-compare.py, and the per-boot llama-bench JSONL, Strata measure.py output and meta.txt for both rounds (13 and 14 October 2026). Kernel log excerpts read from the A9's journal on 16 October 2026.