LFM2.5 is fast. Gemma 4 finishes the job

Assumes you've pointed a coding agent at a local llama.cpp server and know what MoE and a Q4 GGUF are.

TL;DR — For agent work, use Gemma 4 26B-A4B. LFM2.5-24B-A2B was faster on most short single-file tasks and passed most of them. On the Spring Boot task it wrote no tests in three runs; Gemma wrote eleven in each of its three, and passed all 14 hidden core tests twice. Three runs per cell, so read these as a direction, not a rate.

A friend asked which of these two to run, and LiquidAI's model card answers before you've asked. Its benchmark table puts Gemma-4-26B-A4B-it in a column right next to LFM2.5, and LFM wins most of the rows. LiveClaw: 48.09 against 41.25. SWE-Bench Verified, run through OpenCode: 51.00 against 25.80.

Nearly double, in the same agent client I use. So I put both through the same benchmark the bigger local models get, three runs per task.

The short tasks went the card's way. The long ones did not.

The setup: two small-active MoEs on one 3090#

Both models are mixtures of experts with a small active slice. LFM2.5-24B-A2B is 24B total with 2B active per token, per its card. Gemma 4 26B-A4B is 25.2B total, 3.8B active; I ran the QAT build. LFM2.5 appeared on Hugging Face on 2026-10-12, the unsloth GGUF a day later. Its card says it always thinks and writes its tool calls as Python lists, which llama.cpp has a parser for.

Everything else was held equal:

• unsloth UD-Q4_K_XL GGUFs, 14.9 GB (LFM) and 14.2 GB (Gemma) on disk
• llama.cpp on an RTX 3090 capped at 280 W, 64k context, no speculative decoding
• each card's sampler: LFM temperature 0.1, top-k 50, repeat penalty 1.05; Gemma 1.0, top-p 0.95, top-k 64
• reasoning on for both
• OpenCode 1.18.4, isolated from my own config, one fresh workspace per run
• three runs per model per task, interleaved so a half-finished queue still compared like with like

One caveat on the card: LiquidAI ran its numbers on vLLM against the non-QAT Gemma. This is not a replication of their table. It's what happens on a single consumer GPU with the files people actually download.

The tasks, one line each#

Each task is a prompt plus a scorer that checks the result on disk. Any failed core check makes a run FAIL; most other misses, short of a broken build or destroyed data, make it PARTIAL. The numbering has gaps because this pair skipped the other tasks in the suite.

• t1 — write a Python Markdown table-of-contents CLI with unit tests.
• t2 — a CSV stats CLI over three prompts: write it, add --json, then exit 2 on a missing file.
• t4 — three file chores: normalize photo names without losing a file, pull every ERROR line from the logs into errors.txt then delete the rotated logs, and make a dirty git repo clean without losing the work.
• t5 — seven prompts in one session growing an inventory CLI: total, a renamed command with an alias, CSV export, validation, CSV swapped for NDJSON, then a question about what was abandoned.
• t6 — find the one bug in calc_v37 of a large Python module and fix only that function, with a test.
• t10–t12 — single-file scripts against nasty inputs: a messy CSV (BOM, CRLF, quoted newlines) in Python, a content-based file sorter in bash, a timezone-aware JSONL merge in Node.
• t14 — add an idempotent, atomic, concurrency-safe Transfer to a small Go ledger service, with tests and docs, checked by 15 hidden core tests.
• t16 — in a Spring Boot 3.5 service, fix a "cancelled event comes back" bug and add a registrations API with validation and 404/409 rules, checked by 14 hidden core tests.

Results: PASS/PARTIAL/FAIL and median wall time#

Totals over 30 runs each: Gemma 20 PASS, 4 PARTIAL, 6 FAIL. LFM 16, 7, 7. Close, and the totals hide the row that matters most: two of Gemma's three t16 "non-passes" cleared all 14 hidden core tests.

Short tasks: LFM is up to four times quicker#

On t2, t5 and t6, LFM's median wall time runs from under half of Gemma's down to about a quarter: 51 s against 118 s, 114 s against 473 s, 23 s against 69 s. On t1 they tie. I didn't measure tokens per second for this pair, so I can't split the gap between the smaller active slice and Gemma simply thinking longer at temperature 1.0. On a read-one-file, write-another job, the wall clock is what you feel anyway.

It isn't universal. On the Node JSONL merge (t12), LFM was the slower one, 68 s against 51 s. And on the bash file sorter (t11), each model had one run that put all nine files into other, the classifier equivalent of shrugging. LFM's spent 80 tool calls getting there.

The PARTIALs need reading#

The t4 and t5 rows look like a clean LFM loss and a clean Gemma loss. Opening the workspaces made both less clean.

t4, Gemma. All three Gemma runs ended with an errors.txt holding all three ERROR lines, including the one that lives only in the rotated app.log.1. All three put that file in the workspace root rather than logs/, where the scorer expects it. That's the two PARTIALs.

The FAIL is better. Gemma's first run had a complete errors.txt, deleted the rotated log, then regenerated the file with a fresh script and lost the line it had just saved. So it decided to put the evidence back:

$ mkdir -p logs
echo "ERROR db timeout" > logs/app.log
echo "ERROR old crash" > logs/app.log.1
echo "ERROR debug oops" > logs/debug.log
echo "Keep this" > logs/README-IMPORTANT.md

(no output)

That last line overwrote the directory's runbook, whose entire content was do not delete this runbook. It now reads Keep this. The current app.log lost its two INFO lines too. Gemma then re-ran its grep and delete, and its summary said every other file in logs/ had been kept. Technically the files were kept.

t4, LFM. LFM wrote errors.txt in the right place every time. Two runs missed the line from the rotated log; the third got all three and left the git repo dirty.

t5, LFM. All three are PARTIAL with the note ndjson missing. The scorer checks that with a case-sensitive grep ndjson on the source file. In two of the three runs, LFM had built a working export FILE.ndjson that writes one JSON object per line. It just calls it NDJSON in every comment and help string. The third run had no export command at all. So the honest t5 reading is closer to two passes and a real miss than to 0/3/0. I'm leaving the recorded verdicts alone, since a scorer fix moves every model's rows, not just this one.

None of that changes the direction. It does mean the short-task gap is narrower than the table makes it look.

Multi-file work: Gemma writes tests, LFM doesn't#

The Spring Boot task is where the two models separate.

Gemma passed all 14 hidden core tests in two runs and 13 of 14 in the third, which missed one email validation case. Every run added 11 tests of its own, and every run's tests caught all 7 seeded mutants. The PARTIALs are detail checks, the same "one error entry per field" item the dense Qwen3.8-27B runs also trip on.

LFM wrote zero tests in all three runs. The hidden suite scored them 9/14, 2/14 (it never fixed the seeded bug), and a compile error. The compile error is two public records in one Java file:

public record ApiError(int status, String error, String message, String path, List<FieldError> fieldErrors) {
    public ApiError(int status, String error, String message, String path) {
        this(status, error, message, path, null);
    }
}

/**
 * A single field-level validation error.
 */
public record FieldError(String field, String message) {
}

javac allows one public top-level type per file, named after the file. Any mvn test would have said so before running a single test.

The Go ledger (t14) failed for both, 0/3 each, in instructive ways. The prompt allows adding to the existing exported API. Gemma took that literally in two runs and added methods to the AccountStore interface, which is the one kind of addition that breaks everyone else: the hidden suite's fake store no longer satisfies the interface, so 0 of 15 core tests even compile. Its third run wrote no tests and no Transfer the scorer could find.

LFM's failures were stranger. One run created errors_new.go and new_errors.go, byte-for-byte identical, both declaring ErrSameAccount, and the build stopped at ErrSameAccount redeclared in this block. Another wrote no tests and left test.txt, test2.txt and test3.txt in the module. The best run reached 10 of 15 hidden core tests with a failing test of its own, after 442 tool calls and 2,733 seconds.

Count the tests across both tasks. Five of LFM's six multi-file runs ended without a single test of their own. Gemma wrote tests in five of its six (one Go run's only test failed; LFM's best run also had a failing test), and on Spring Boot its tests caught every mutant. LFM writes the code, then skips the part where it finds out whether the code works.

My own mistake: nine runs that never reached the GPU#

The first LFM pass over t10–t12 recorded nine FAILs, each one 1 to 2 seconds long with zero tool calls. That's not a model failing fast. The benchmark's OpenCode config had no catalog entry for the new model, so OpenCode exited before sending a single request:

Error: {
  "name": "UnknownError",
  "data": {
    "message": "Unexpected server error. Check server logs for details.",
    "ref": "err_2b1527d3"
  }
}

The server logs had nothing to check, since nothing reached the server. I quarantined the nine rows, added the missing entry, and reran the arm; the 7/1/1 in the table is the rerun. A new model key now needs four config entries in that repo, and the fourth is written down.

What I told my friend#

For an agent that edits a multi-file project and should prove its change works, Gemma 4 26B-A4B. It's slower, it once rebuilt a deleted log from memory, and neither model finished the Go ledger, where LFM's best run actually got further. But Gemma writes and runs tests, and on Spring Boot that was the whole difference. For quick single-file jobs where you read the diff yourself, LFM2.5 is fast and usually right.

Neither is the top of what fits on a 3090. On the same Spring Boot prompt, same client, the dense Qwen3.8-27B (IQ4_XS) passed all 14 hidden core tests in three runs out of three at 64k, and three more at 128k.

A card's agent score is a measurement of someone else's setup. Run yours.

Filed under: local-llm, llm, ai-agents, benchmarks

Read next#

• Zig needed context, not time — what 64k versus 128k context does to the same kind of ledger task.
• The deadlock test can't see a half-held lock — what model-written tests catch on the Zig version of this ledger task.

Sources#

• LiquidAI/LFM2.5-24B-A2B model card — parameter counts, "always thinks", Pythonic tool calls, recommended sampler, benchmark table with the Gemma-4-26B-A4B-it column.
• unsloth/LFM2.5-24B-A2B-GGUF — the GGUF used here.
• unsloth/gemma-4-26B-A4B-it-qat-GGUF — the QAT GGUF used here; 25.2B / 3.8B active and the recommended sampler.
• google/gemma-4-26B-A4B-it — upstream non-QAT card, the model in LiquidAI's comparison column.