Qwen3.8-27B, Muse-Glimmer-30B, Qwen3.6-35B: all tests green, only one accepted

Ran three local models — Qwen3.6-35B-A3B, Muse-Glimmer-30B and Qwen3.8-27B — through one and the same spec: a markdown editor in Rust and GTK4. Judged them not by eye but with a harness: 13 machine gates, 27 contract tests and 14 functional UI checks driven through AT-SPI, with screenshots. Each model got up to 30 rounds, and between rounds it received a report of exactly what had failed.

Hardware — RTX 5080 Laptop with 16 GB, llama.cpp on Vulkan, OpenCode as the agent.

Results

Only one of the three passed acceptance.

The other two tripped over the same thing: the markdown rendering function is supposed to return HTML, and Qwen3.6-35B-A3B produced Pango markup while Muse-Glimmer-30B produced plain text with the markup characters stripped out.

Tests that confirm the mistake

The interesting part is not in the table. All three models wrote tests, as the spec required, and all three had their own tests passing. Here is a real test from Muse-Glimmer-30B:

let out = render_markdown("# Title\nSome text");
assert!(out.contains("Title"));
assert!(!out.contains('#'));

It is green. And the function returns bare text instead of HTML. The model wrote an oracle that confirms its own reading of the task — and the reading was wrong.

Hence the conclusion that justifies the whole exercise: telling a model to "work test-first" is not enough. The model authors both the code and the check for it, so the check has to come from something you did not write together with it. Qwen3.8-27B had its own version of the same act — for nine rounds it kept fn main() {} and wrote library code, never once running the application. The unit tests passed the entire time.

On reasoning effort

Qwen3.8-27B thinks at maximum by default — xhigh. Ran it again on medium, changing nothing else.

Same result — full acceptance, at half the price. And look at the last row: the number of responses barely moved. The whole difference is in how much the model thinks per step, not in the number of steps. xhigh does not find more solutions; it deliberates longer over the same ones.

On speed

20.5 tokens per second — and that is the ceiling of the laptop, not of the model. The card is locked at 80 W against the 175 W it is rated for: on Linux the hp-wmi driver never sends the firmware the unlock command that OMEN Gaming Hub sends on Windows. Built llama.cpp with CUDA instead of Vulkan — zero difference in generation, 20.97 against 20.84. The backend is not to blame; it is short on watts.

Multi-token prediction (MTP) gives +25–37% on short prompts, but on a 59,000-token prompt it came out 12% slower: the draft acceptance rate falls from 70% to 35–47%, and the extra pass stops paying for itself. No good for coding.

Conclusions

Of the three, Qwen3.8-27B is the one to use for coding, and by a clear margin: the only model to reach acceptance, it wrote twice as many tests as the runner-up, and along the way it found a defect in my own verification harness. Run it on medium rather than xhigh — twice as fast for the same result.

A caveat: one run per model and one task, so this is not statistics. Qwen3.6-35B-A3B stalled on its own at round 11; Muse-Glimmer-30B I stopped at round 22, after thirteen rounds without movement.

And the main thing the experiment showed: the difference between "the model wrote code" and "the code works" can only be measured by something the model did not write.