Since my last post, I'd been habitually trying different local LLMs via Llama.cpp.

While there were some promising candidates, I always ended up in a situation where small models that fit fully within my 16GB of VRAM were not as capable as I'd like. Using mixture-of-experts offloading via Llama.cpp allowed me to use larger models, but at a huge performance penalty. I believe this offload failed due to a PCIe bottleneck as my GPU is connected via thunderbolt to my laptop.

So for some time, I'd settled for Tiel Coder at a quantisation level of IQ3_XXS. It worked well enough for smaller tasks, but I still found myself using larger models via OpenRouter so that things worked reliably. It was also slower than I'd like at a decode speed of around 22 tokens per second.

Custom Inference Engines

It turns out that if you write (or vibe-code) a completely custom inference engine suited to a specific LLM architecture, you can get better performance than Llama.cpp. One such engine that's been increasingly hyped up, is Strata. It's an inference engine well suited for larger mixture-of-experts models being split between GPU VRAM and system RAM.

Something neat about Strata is that it features a GUI application that suggests the optimal models and settings for your hardware. In my case, the model I selected was Qwen 3.8 Flash Next at a quantisation level of IQ3_S. This is a 125B parameter mixture-of-experts model with 6B parameters active for each forward pass.

Ordinarily I would never dream of running such a large model with my current hardware, but Strata made this possible. I didn't even really have to dive deep into configuration settings, as Strata started an auto-tuning process which ended up selecting parameters for me.

In the end I wound up with a shockingly capable model running at decode speeds of around 36 tokens per second and prompt processing speeds of around 500 tokens per second. This did of course fill up almost all of my 16GB of VRAM and 64GB of RAM, but Strata does optionally leave some RAM available for your work applications.

Examples from a Week of Usage

One of the tasks I used my new setup for was creating a small point cloud visualiser for 64-bit terrain data using WebGL. The model worked well, preparing a reasonable plan and reliably executing it in OpenCode.

Another test I ran was converting a repository of OpenAPI 3.0 specs to the latest OpenAPI 3.2 standard. This also went smoothly, as the latest spec differences were retrieved via a web fetch and the changes across dozens of files were carried out.

The final example I tried was to reverse engineer an old Android app I had purchased, but was no longer available on the Google Play Store. I retrieved an archived APK file from a sketchy hosting website, and proceeded direct the model to analyse the contents. This was an old language tutorial app containing walls of text, so the model had no trouble re-assembling this into a simple web application that I can host locally.

What's Next

To me this setup alleviates one of my biggest gripes with cloud LLMs, which was the weird pricing surrounding them. At this point I feel more empowered to attempt long-running unsupervised tasks as my tokens costs are basically just the cost of electricity. I'm especially interested in using my new setup for getting better at reverse engineering and decompiling binaries.