$ grep -r Tag: «llm»

Ran an interesting experiment today. The same spec file as before, for a markdown editor in Rust and GTK. Two different approaches.

1. The whole task done by Opus 5 through Claude Code. To keep the comparison clean, Claude Code spawned a separate subagent with a fresh context.
2. The task done by Opus 5 paired with a locally running Qwen3.6-35B-A3B. Here Opus 5 acts as the orchestrator and the brains, while all the code writing is delegated to Qwen. The orchestrator is a separate subagent with a clean context too, so that both approaches start from the same place.

The spec lists the criteria the app has to meet — interfaces, features, tests — plus validation by external scripts that check whether the thing actually works.

First, I was curious whether two models working together could handle the task at all. Second, I wanted to find out whether there would be any difference in token spend. The results are rather interesting.

Results

Both handled it.

By the formal criterion it is a draw. On tokens the pairing is 1.77× cheaper on output and 2.08× cheaper on cache reads, and that is a real saving: the local model costs nothing against my Claude limits.

But half the cost bought me barely more than a third of the code. The pairing did exactly what I ordered stage by stage — the mandatory core, and not a step further. Opus, entirely unprompted, went on to finish twelve more items the spec marks as optional: syntax highlighting, split view, themes, session restore, watching the file for external changes, HTML export, single-instance mode.

The saving is in limits, not in speed: on the clock the pairing is twice as slow.

One more observation, more about me than about the models. My validation scripts turned out to be weaker than I thought. They check that a window opened, that there are controls, and that clicking them does not crash the app. The main scenario — type some text and save it — is not covered at all. In the paired run Qwen produced code that sailed straight through the scripts, yet edits in the editor never reached the document model: Save wrote an empty file. The orchestrator caught that by reading the code, not by running the scripts, and sent it back to be fixed.

Conclusions

Delegating to a local model is worth it if you need exactly what you specified, and you are willing to slice the task into stages yourself and check every one of them. It saves almost half of your limits.

The orchestrator does have to work from a fresh context. I had a third run where I drove Qwen straight from a long working session — it ate three times more, purely on re-reading the accumulated history on every turn. That swallowed the entire gain.

But you pay for the saving with initiative. The pairing closes the list of requirements; it does not make a product. If you want the latter, not delegating is cheaper.

It's amazing to see how much progress models and llama.cpp have made in just a few months. Before this, I tested the Qwen3.6-27B model. It wouldn't fit in my 16 GB of VRAM, ran quite slowly, and generated at 12–18 tokens per second.

It took about 20 hours to complete my test task: building a Markdown editor in Rust with GTK on Linux.

It was an interesting experiment, but practically useless. Way too slow.

And in the meantime, the Qwen3.6-35B-A3B model was released. It uses a MoE architecture: 35B total parameters, but only ~3B are activated per token (8 routed experts + 1 shared expert out of 256). This makes the model compute much more efficiently while retaining quality on par with much larger models in the same class. llama.cpp has added support for this architecture, along with numerous performance improvements and MTP (Multi-Token Prediction) — an additional "head" trained to predict multiple tokens ahead, alongside the main model. MTP delivers a ~1.4–2.2x generation speedup without sacrificing accuracy, though for MoE models the boost is usually more modest (~1.15–1.25x).

So today I decided to feed my test task to this model. First, I had Claude Code run several tests to find the optimal settings for it.

Saying I'm surprised is an understatement. First, the entire model fits comfortably in VRAM. Second, I'm getting a 256K context window, which I've never been able to use with other models (even smaller ones struggled with this).

Prompt processing is running at roughly 260–400 tokens per second. Generation speed on an empty context was nearly 100 tok/s, dropping to 62–65 tok/s as the context filled up. Still, it's damn fast.

With the 27B model, it took about 10 hours just to reach a state where the app would run, even with a buggy and broken UI. Today, the 35B model got there in 50 minutes!

Energized by the speed, I started telling the model what to fix. In just 4 hours, I got a fully working application with all the flaws and even minor details corrected.

Looking forward to running this model on my real projects now. Who knows, maybe the dream of a capable local coding model isn't so far out of reach after all.

Over the past few months I've tried out a lot of different local models. I have my own kind of benchmark for them — a file with a detailed spec for a notes editor for Linux written in Rust. Syntax highlighting (not hard, it's available out of the box in the system component) and a preview panel that should render formatted text. Overall, it's pretty similar to my Swifty Notes app, but in Rust.

So, Qwen3.6-27B is so far the only model that's made it to the finish line. It worked for 3 days (not around the clock, only when I was at the computer and could guide it). Occasionally the model had issues with tool calls. Sometimes it couldn't edit a file and would just stop — I had to ask it to try again. Overall I'd give this part a 4 out of 5.

The most important thing is that the model didn't go in circles. Rust is a great choice here: if there's an error, the project simply won't compile. Combined with the fact that the spec requires writing tests first and then the code (Test Driven Development), a lot of issues get caught and fixed at the earliest stages. And here the model performed pretty well. It fixed errors fairly quickly, followed the instructions from the spec, and consulted documentation if it suddenly used a nonexistent API or just needed to check what was available. A solid 4 out of 5 here.

The model size is 17.5 GB. I downloaded it through LM Studio but ran it via llama.cpp — surprisingly, the performance there is significantly better, even though LM Studio supposedly uses llama.cpp under the hood. Maybe more recent versions of llama.cpp are just better optimized.

The model doesn't fit entirely into VRAM (I have 16 GB), so it ran partially on the CPU, which heavily impacted speed. This whole thing was eating 15.94 GB of VRAM and another ~12 GB of RAM at peak (when the context was nearly full). I set the context size to 64k — at first that was plenty, but towards the end, as the codebase grew, the model would compact the context fairly often. Still, overall it handled a task of this size.

I'd give the speed a 2 out of 5.

When the model reported that everything was done, I launched the app and saw raw HTML text in the preview panel instead of a rendered view. I had to ask the model to fix it, and after another hour and a half or two I got an acceptable result. That said, it's worth mentioning that this kind of thing happens all the time even with flagship models — large tasks rarely work on the first try and usually require revisions and clarifications.

Bottom line: Claude Code would've done this task in 20–30 minutes, and I'd probably spend another 20–30 minutes finishing things up. Let's round up to 1 hour. With Qwen3.6-27B it took about 20 hours of continuous work in total, roughly 3 full working days. Of course, most of that time you're just sitting there watching the model work. Generation speed on the preliminary tests Claude helped me run was 12–18 tokens per second on my hardware.

The speed difference is huge. The result is acceptable. A lot of the required functionality just doesn't work, but I blame the context size for that — it's hard for models to hold that much code in memory.

Still, I'm happy with the result. It gives me hope that programming with local models on regular hardware is feasible. A year ago even cloud models were making similar mistakes, and local models of this size were simply not suitable for programming at all. Maybe in another year progress will get us to the point where subscriptions won't be needed for small projects. And I really like the trend that open-source free models have already caught up with the flagships from OpenAI and Anthropic.