The model card said think harder. The audits got worse

Assumes you've run a local reasoning model as a coding agent and know what a reasoning-effort setting does.

The Qwen3.8-27B model card lists three reasoning efforts and marks one as the default:

xhigh (default): for complex tasks demanding thorough analysis

A few lines further down it adds a tip aimed at people like me:

In multi-turn agentic tasks, lower reasoning effort does not always reduce overall task completion time. Although it may produce faster per-turn responses, it can also lead to insufficient analysis, more failures, and repeated retries, which may increase total latency and token consumption.

My launchers run every Qwen3.8 tune at medium. I picked that on the scripting tier, where medium passed 9 of 9 for every tune and was faster than xhigh for each of them. Then I read the card while checking the bench's settings against it on 19 October. Sampler, preserved thinking and draft count all matched. Effort was the one gap. It also happened to sit next to the two tasks every local model had just failed in the hard-tier re-baseline, both of them analysis jobs, which is exactly where the tip says lower effort hurts.

So I gave the card its default and ran the two tasks again.

Two tasks the local models kept failing#

Both are operator jobs, not code. The agent gets files and a request, in Pi, at 128k context, with the card's sampler.

• u2, log forensics. Two 2,000-line cluster logs from the Loghub dataset, labels stripped. Flag the lines on-call should be paged for. There are 143 real alerts, all in one file. The other log has none and is full of "failed" and "Warning" lines as a decoy. Recall below 0.8, or precision below one in three, fails the run. A full pass also needs precision of 0.7; between one in three and 0.7 the run is partial.
• u4, security audit. Thirty Java servlets from the OWASP Benchmark, comments and hints removed. Fifteen are exploitable, fifteen are the benchmark's deliberate false alarms. For each, say whether an attacker can really reach the dangerous behaviour. Passing needs 70 percent accuracy and 70 percent recall.

At medium, three runs for each of four 27B tunes: u2 passed 0 of 12, four of them partial on precision. u4 passed 4 of 12.

The first xhigh attempt ran out of time#

The bench gives u2 1,800 seconds and u4 2,700. At xhigh, two of the three qwen38pi runs on u2 ended at 1,802 and 1,803 seconds, still working. A timeout is not a verdict on reasoning effort, so I reran u2 at xhigh with 5,400 seconds for all four tunes. The longest run took 3,674 seconds, and none reached the limit. No u4 run at xhigh came near its limit: the longest took 984 seconds, so those rows stand.

Results: medium against xhigh#

Log forensics: 0 of 12 at both efforts. Security audit: 4 of 12 at medium, 1 of 12 at xhigh.

xhigh did think more. Summing the reasoning in each run's Pi session, the median u2 run went from 46,782 characters at medium to 113,986 at xhigh, and the median u4 run from 31,070 to 69,852. Wall time followed: median u2 run 574 seconds to 1,478, median u4 run 341 to 561.

So it was more than twice the thinking and up to 2.6 times the wait, for fewer passes. The card predicts more failures at lower effort. On these two tasks, more of them came at the higher one.

Same answer, written out at greater length#

Four passes against one, in twelve runs each, is too few to rank the efforts on its own, so I compared the answers themselves. The scorer records one character per servlet, right or wrong, in file order. Nine of the 24 audits, four at medium and five at xhigh, spread across all four tunes, returned the identical string: 9 of 15 real vulnerabilities found, no false alarms, the same six misses in the same six files. At xhigh, 8 of 12 runs landed on exactly 9 of 15.

The misses have a pattern. I checked them file by file in two tunes' earlier single runs. The models judge every servlet by taint: does request input reach the sink? If a variable is overwritten before the dangerous call, they mark it safe. That logic is right for injection, and the benchmark's false alarms are built to trap anyone without it. It misses the weak primitives, where the danger isn't a flow at all. One servlet issues a remember-me cookie from java.util.Random. Another sets setSecure(false). Any request reaches both, and no input needs to reach anything. The prompt asks whether "an attacker can really reach the dangerous behaviour", which covers them.

More effort let the models apply that same rule more carefully. It didn't get them to step outside it. The four medium passes came from runs that broke the pattern, finding 11 to 14 of the 15. At xhigh only one run got that far.

Log forensics failed in a different way and at both efforts. Four medium runs and five xhigh runs flagged between 870 and 917 lines of the decoy log, which has no real alerts in it. The prompt says "I do not want to be paged for noise, and I will check every line you flag." More than twice the reasoning didn't make that sentence land.

The bar is reachable#

A task nobody passes might just have a wrong bar. To rule that out I ran the same prompts, generators and scorers through Claude Code with Opus 5.5, isolated from this machine's configuration files, three runs per task. It passed 6 of 6. On u2 it found 142 or 143 of the 143 alerts and flagged no decoy lines. On u4 it scored 30 of 30 every time. Each run took about three minutes.

That says the tasks can be done in the time given. It doesn't say why the 27B tunes can't. It's a different model in a different harness, and I only used it to check the bar.

The coding tier at xhigh: one model improved#

The audits are operator work, and the card's tip is about agentic tasks in general. The bench's hard tier is the coding side of that: three tasks, three runs each, in OpenCode at 128k context with the card's sampler. A Java ledger service (t13), the same ledger in Go (t14), and a registration endpoint for a Spring Boot app (t16). Medium had its turn in the re-baseline. I ran it again at xhigh, with the time limit raised from 3,600 to 5,400 seconds. Neither limit came into play: the longest run took 1,222 seconds at medium and 1,526 at xhigh.

27 of 36 at medium, 24 of 36 at xhigh, and every model took longer per run. The base model is the one xhigh helped. It passed the Spring task 2 of 3 times, against 0 of 3 at medium, and passed all six ledger runs. The three tunes went the other way on Spring: 4 of 9 at medium, 0 of 9 at xhigh. The Java ledger came out one run lower (12 to 11 of 12) and the Go ledger held at 11. Both xhigh ledger misses were swift15's: an untyped IllegalStateException in Java, and an fmt.Errorf in Go where the task wanted a sentinel error. No run at xhigh failed a hidden core check. Every miss was a detail.

One entry per invalid field#

Most of the Spring misses, at both efforts, are one hidden check. The prompt asks for a 400 whose body carries "an array with one {"field": "<request field name>", "message": "<why>"} entry per invalid field". The hidden test registers with an email of 300 x characters, malformed and too long at once. All 24 Spring runs at both efforts put both @Email and @Size on the email field, so two constraints fire on one field.

The runs that pass this check collapse the errors by field. This is ThinkingCap's handler from its third xhigh run:

Map<String, String> byField = new LinkedHashMap<>();
e.getBindingResult().getFieldErrors()
        .forEach(fieldError -> byField.putIfAbsent(fieldError.getField(), fieldError.getDefaultMessage()));

The runs that fail map Spring's field errors one to one and send two entries for the email. ThinkingCap again, first xhigh run:

List<FieldError> fieldErrors = e.getBindingResult().getFieldErrors().stream()
        .map(f -> new FieldError(f.getField(), f.getDefaultMessage()))
        .toList();

Unlike the blank-key check in part 13, this one is spelled out in the prompt. It's also a common miss at either effort: 8 of the 12 medium Spring runs failed it, and 9 of the 12 at xhigh. The one xhigh Spring miss that wasn't this check is the run quoted above, ThinkingCap's third. Its handler gets the entries right, and its own test is where putIfAbsent bites back. The test sends a 260-character email, which breaks both constraints, and asserts the message must be at most 254 characters. putIfAbsent keeps whichever error the validator reports first, and the order of @Email and @Size isn't fixed. The test passed in the run and in one rerun, and failed in two others, so the row stays partial.

Two passes against four, or none against four in the three tunes, is too few to say xhigh makes Spring worse. The firmer result is the audits' pattern again. A sentence that most runs misread at medium, most runs still misread at xhigh. On Spring, the extra reasoning time bought the base model two passes and cost the three tunes four.

What this doesn't say#

• Three runs per cell, one context size, Pi for the audits and OpenCode for the coding tier. That's enough to see that xhigh didn't fix these tasks. It's not enough to rank efforts in general, and the scripting-tier result still stands.
• One model did improve, by two runs. The base qwen38 went from 6 to 8 of 9 on the coding tier, both gains on Spring. That's a direction, not an effect size.
• Low effort wasn't tried here.
• The card's tip is about agentic tasks in general. It may be right where retries are the failure mode. On the audit, the failure was a fixed way of reading the problem, and more tokens spent inside it bought nothing.

The launchers stay at medium. The card's default is a sensible place to start. Measured here, it cost time on every model, and on net it cost passes: 4 to 1 of 24 on the audits, 27 to 24 of 36 on the coding tier. One model gained two runs.

Filed under: local-llm, llm, ai-agents, benchmarks

Read next#

• The harness decided how long the model thought — the same models reasoning 3.6 times as much depending on the client, with the same pass rate.
• The hidden test asked for more than the prompt — the other time a wall of failures turned out to say something specific.
• It fits in 16 GB. That was the easy question — what these 27B models cost to run on a smaller card.

Sources#

• Qwen/Qwen3.8-27B model card — reasoning_effort levels (xhigh default, medium, low) and the multi-turn agentic tip quoted above.
• Loghub — source of the BGL and Thunderbird 2k log samples behind u2.
• OWASP Benchmark for Java — source of the servlet subset behind u4.
• Benchmark artifacts (local, not published): results rows for the medium, xhigh-1800 s and xhigh-5400 s arms, each run's Pi session (reasoning characters counted from its thinking parts), scorer metrics including the per-servlet hit strings, the queue scripts, the frontier reference runs, 2026-10-19 to 2026-10-20, and the hard-tier rows at medium (2026-10-18) and xhigh (2026-10-20 to 21) with the t16 prompt, its hidden detail_validation_oneEntryPerField test and each run's exception handler, and the 2026-10-22 rescore of those rows (README, formatter and doc-comment checks moved out of the verdict).