120 GiB for the GPU, set in the kernel, not the BIOS

The A9 Mega's setup menu keeps its GPU memory setting under Advanced → GFX Configuration, and it goes from 0.5 G to 96 G. The machine has 128 GB of LPDDR5X soldered to the board, and it was bought to run models whose weights alone are 63 GB. Dedicated Graphics Memory went to 96 G, the top of the list.

This is what Fedora's kernel made of it at boot:

$ journalctl -k -b -17 | grep -E 'Memory: |memory ready'
Oct 09 13:29:55 fedora kernel: Memory: 31977344K/33184580K available (24451K kernel code, 4814K rwdata, 18648K rodata, 5320K init, 4220K bss, 1162832K reserved, 0K cma-reserved)
Oct 09 13:29:58 fedora kernel: amdgpu 0000:c6:00.0:  98304M of VRAM memory ready
Oct 09 13:29:58 fedora kernel: amdgpu 0000:c6:00.0:  15853M of GTT memory ready.

(The timestamps on boot −17 run three hours fast. Part 1 explains why.)

The firmware took 96 GiB off the top before Linux started counting. The kernel found 33,184,580 KiB of RAM, and about 31 GiB of it ended up usable. As far as the OS was concerned, that was the whole computer. On top of the 96 GiB of VRAM, amdgpu set up 15,853 MiB of GTT. That number is half of 31 GiB, not half of 128.

So: 96 GiB for the GPU, 31 for everything else, and a second GPU pool sized from the smaller number. The setting did exactly what it said, and the rest of the machine paid for it.

The fix: BIOS minimum, then two TTM parameters#

First, set Dedicated Graphics Memory to 0.5 G, the smallest option. Second, raise TTM's page limit on the kernel command line. These are the two lines from the box's fix-07-gtt.sh:

PAGES=31457280
grubby --update-kernel=ALL --args="ttm.pages_limit=$PAGES ttm.page_pool_size=$PAGES"

The unit is 4 KiB pages: 31,457,280 × 4,096 = 128,849,018,880 bytes, exactly 120 GiB. On Fedora, grubby also wrote the arguments into /etc/kernel/cmdline and GRUB_CMDLINE_LINUX, so kernels installed later keep them. To undo it, run the same command with --remove-args="ttm.pages_limit ttm.page_pool_size".

The current boot, same grep, plus the GTT size as the driver reports it:

$ journalctl -k -b 0 | grep -E 'Memory: |memory ready'
Oct 10 16:38:59 a9-mega kernel: Memory: 130666280K/133323588K available (24451K kernel code, 4814K rwdata, 18648K rodata, 5320K init, 4220K bss, 2611692K reserved, 0K cma-reserved)
Oct 10 16:39:01 a9-mega kernel: amdgpu 0000:c6:00.0:  512M of VRAM memory ready
Oct 10 16:39:01 a9-mega kernel: amdgpu 0000:c6:00.0:  122880M of GTT memory ready.
$ cat /sys/class/drm/card0/device/mem_info_gtt_total
128849018880

The firmware now reserves 512 MiB, the GPU may map up to 120 GiB of system RAM, and MemTotal reads 124.9 GiB. ROCm sees the same number. This is the stderr of llama.cpp's ROCm build from the benchmark further down:

$ cat raw/llama-rocm-gpt-oss-120b-MXFP4.err
ggml_cuda_init: found 1 ROCm devices (Total VRAM: 122880 MiB):
  Device 0: AMD Radeon 8060S Graphics, gfx1151 (0x1151), VMM: no, Wave Size: 32, VRAM: 122880 MiB

The GPU is gfx1151, with no override. That's a nice change from the Radeon 680M laptop, where ROCm needed a version lie before it would start and then still miscounted the shared pool.

Why the biggest setting gave the GPU less room#

The carve-out happens in firmware. The kernel never gets those pages, and every default that scales with "system memory" scales from what's left. TTM, the kernel's GPU memory manager, sets its own default limit in ttm_device.c like this:

/* Limit the number of pages in the pool to about 50% of the total
 * system memory.
 */
num_pages = ((u64)si.totalram * si.mem_unit) >> PAGE_SHIFT;
num_pages /= 2;

amdgpu then takes its GTT size straight from that limit, in amdgpu_ttm.c:

gtt_size = ttm_tt_pages_limit() << PAGE_SHIFT;

With the 96 G carve-out, the stock ttm.pages_limit was 4,058,544 pages. That is 15,853 MiB, the same number as the GTT line in the log. Here are the two configurations side by side:

The largest setting in the menu left the GPU 9 GiB less to address and the OS 94 GiB less to live in.

The bigger difference is that one of these memory pools is reserved and the other is only a limit. The 96 GiB carve-out belongs to the GPU whether a model is loaded or not. pages_limit is a ceiling on allocation. ttm_tt.c describes it as "Limit for the allocated pages" and compares it against a running count of pages TTM has actually handed out. Until the GPU asks for them, those pages are ordinary RAM. page_pool_size has the same half-of-RAM default, per NUMA node, and caps TTM's pool of write-combined and uncached pages. The fix raises it to the same number. The GTT size itself comes from pages_limit alone.

amdgpu.gttsize, the knob older guides reach for, is on its way out. Its doc comment in amdgpu_drv.c says "This parameter is deprecated and will be removed in the future." If you set it anyway, the driver logs "Configuring gttsize via module parameter is deprecated, please use ttm.pages_limit". Jeff Geerling's write-up shows the same warning, but it spells the parameters amdttm.. This box's stock Fedora kernel has no /sys/module/amdttm, and /sys/module/ttm/parameters/pages_limit reads back 31457280. Geerling also reports that about 108 GB was his ceiling before errors started. 120 GiB here is a limit that hasn't been tested at its edge yet: the most GTT the box's telemetry has caught in use is 90,682 MiB, on its first day of logging. If the OS ever runs short, 29,360,128 pages is 112 GiB.

F3 resets the carve-out to 64 G#

Part 3 goes through this BIOS's hidden menus. One side effect belongs here. F3, Optimized Defaults, resets Dedicated Graphics Memory to 64 G and doesn't mention it. This was the next boot, with the TTM parameters still on the command line:

$ journalctl -k -b -7 | grep -E 'Memory: |memory ready|Capping GTT'
Oct 10 00:41:01 a9-mega kernel: Memory: 65113224K/66739012K available (24451K kernel code, 4814K rwdata, 18648K rodata, 5320K init, 4220K bss, 1580332K reserved, 0K cma-reserved)
Oct 10 00:41:03 a9-mega kernel: amdgpu 0000:c6:00.0:  65536M of VRAM memory ready
Oct 10 00:41:03 a9-mega kernel: amdgpu 0000:c6:00.0: Capping GTT to 63915M to not exceed available system memory
Oct 10 00:41:03 a9-mega kernel: amdgpu 0000:c6:00.0:  63915M of GTT memory ready.

The command line asked for 120 GiB of GTT on a machine that now had 62.4 GiB left. amdgpu's APU branch ("Cap GTT so that it does not exceed total physical RAM") cut it down to all of that RAM: 63,915 MiB, every page the OS had. Nothing failed, and nothing in the log looks like an error. The next boot, at 00:48, was back at 512 MiB. After any F3, set the carve-out to 0.5 G again.

Vulkan vs ROCm with the full 120 GiB#

With the memory fixed, the next question was which engine to use. The harness runs both with the same models and the same flags:

LB_ARGS=(-ngl 99 -fa on -p 512 -n 128 -r "$REPS" -o json)

The two engines came from different places:

• ROCm: Fedora's own package, llama-cpp-b9840-2.fc45, running natively. This is what a Fedora user gets from the repo.
• Vulkan: ggml-org's ghcr.io/ggml-org/llama.cpp:full-vulkan image, build 11434 (5e03bdd87), run under podman. The image brings its own Mesa (Ubuntu's 26.0.8, not the host's 26.2.4).

These numbers are tok/s from the full run on 2026-10-09 at 11:26, three repetitions per cell:

Vulkan wins all eight cells. Most of the gap is in prefill: +62 % on gpt-oss-120b, down to +10 % on llama-2-7b. Generation is closer, +7 to +9 %, except Qwen3-30B-A3B at +29 %. The harness warned at the start of this run that the load average was 5.97. The other three runs that day (11:14, 20:44, 23:37) didn't trip that warning, and every cell in them lands within 3.5 % of this table, in the same order.

There's a caveat. The two builds are 1,594 llama.cpp build numbers apart, so this table can't separate "Vulkan vs ROCm" from "a newer llama.cpp vs an older one". A fair backend test would build both from one commit. This one compares what Fedora ships against what upstream ships, and on this hardware upstream's Vulkan container wins.

Mean package power: the column that measured waiting#

The same summary.json records mean package power for each row. The sensor is amdgpu's power1_average, which covers the whole APU and is sampled once a second:

Taken at face value, Vulkan runs gpt-oss-120b at 2.5× less power, and Qwen3-30B is the one model where it draws more. Neither reading holds up once you check what the mean is taken over.

Each mean covers the whole row, from the row's start to the start of the next one. That includes the container start and model load, not just the inference. The Vulkan gpt-oss-120b row lasted 54 samples, from 11:36:43 to 11:37:38. From sample 7 through sample 45, the package sat at 14.7–18.2 W with the GPU 2–8 % busy. The work started at 11:37:29: one ramp sample at 44.6 W, then 87–109 W to the end. ROCm's row for the same model was 20 samples, 12 of them over half busy. Add up the one-second samples and you get roughly the energy per row: about 1,510 J for ROCm and 1,616 J for Vulkan. The engine that looked 2.5× more frugal used slightly more energy on that row.

To isolate the inference itself, keep only the samples where the GPU was at least 90 % busy:

Under load, Vulkan drew 17–30 W less on every model, Qwen3-30B included, and it finished the work sooner. The hottest Tctl reading points the same way: 63.9 °C for Vulkan on gpt-oss-120b, 82.5 °C for ROCm. Still, these averages rest on three to nine one-second samples per cell, while a single pp512 pass takes under a second. They show a direction, not an energy-per-token figure, and nothing in this run can give you one.

Dedicated Graphics Memory is a reservation the OS can never get back. On this box, the smallest number in that menu is the one that gives the GPU the most memory.

Filed under: amd, linux, kernel, rocm, vulkan, local-llm, benchmarks

Read next#

• Installing Fedora on a computer that hadn't shipped yet: part 1, how this box got its OS before it first ran Linux.
• No BIOS image, so I diffed the NVRAM: part 3, the hidden menus, and the NVRAM byte that most likely holds this setting.

Sources#

• ttm_device.c at v7.2: TTM's default limit, half of the RAM the kernel sees.
• amdgpu_ttm.c at v7.2: GTT size taken from ttm_tt_pages_limit(), the gttsize deprecation warning, and the APU cap at physical RAM.
• ttm_tt.c at v7.2: pages_limit, "Limit for the allocated pages".
• ttm_pool.c at v7.2: page_pool_size and its half-of-node default.
• amdgpu_drv.c at v7.2: the gttsize doc comment marking it deprecated.
• Jeff Geerling, "Increasing the VRAM allocation on AMD AI APUs under Linux": the same fix on a Framework board, with amdttm. names and a ~108 GB ceiling.
• llama.cpp container images: the full-vulkan image used for the Vulkan rows.
• Fedora llama-cpp package: the ROCm build used for the ROCm rows.