In my opinion, LLM pricing doesn't make sense.

You could pay for a monthly subscription subsidised by providers. This is usually a good deal for the end user, except for the constantly shifting rate limits and terms of service changes. Like having your data suddenly opted-in to LLM training by default.

Paying per-use API rates though something like OpenRouter seems better to me as you're paying something closer to the actual cost of an LLM. I enjoy OpenRouter for the control it provides and for how it has essentially commoditised inference. However, it still has core issue that LLMs are fundamentally unreliable. It feels bad to have paid for something that has no use to you.

The best solution here seems to be self-hosting LLMs. The trouble is that you need reasonably good hardware to run an LLM at usable speeds. Despite this, I wanted to see how far my mid-range GPU from 2023 could get me in terms of actual utility.

Finding a Model Candidate

I should clarify that the way I use LLMs is more limited than your typical "AI" booster. This software should be used to replace keystrokes, not actual design or intent. So I needed something that could well run reasonably fast within a coding harness to carry out straightforward tasks for me.

I'm also constrained by the Radeon RX 7900GRE 16GB GPU that I own. 16GB is not a lot of VRAM by LLM standards.

With the above in mind, I opted to try using Ornith-1.5-9B. It seemed to have a lot of hype surrounding its launch, and I could comfortably run a 9 billion parameter model on my GPU with a reasonable context window. I also opted to use the Q8_0 quantisation of this model to hopefully make up for its low parameter count.

Software Configuration

I use OpenCode as a harness as I enjoy its plan-then-execute type model. It's also nice to not have to configure a bunch of plugins beforehand.

The model was hosted using the Vulkan version of Llama.cpp with the following settings:

version = 1
 
[*]
n-gpu-layers = all
temp = 0.6
top-k = 20
min-p = 0
 
[Ornith-1.5-9B]
model = /home/shaylin/Models/Ornith-1.5-9B/Ornith-1.5-9B-Q8_0.gguf
c = 102400
spec-type = draft-mtp
spec-draft-n-max = 3

In use, I could reliably get decode speeds of around 72 tokens per second, and prompt processing speeds of around 700 tokens per second (useful for dealing with the large system prompts used by coding harnesses).

Examples from a Week of Usage

The first task I tried was to convert a few Python FastAPI endpoints from POST requests with a body, to GET requests with query string parameters. This is a fairly straightforward task, and was pleased to find that the model could easily complete it without much issue.

Another example would be a refactoring I needed to do in a large React TypeScript codebase. This involved replacing many inline form validation with an existing utility function. Surprisingly, this also worked well.

Feeling ambitious, I then asked the model to search this same large React codebase for instances where could use useEffectEvent to our event handling out of our useEffect external system syncing. The model failed miserably at this. It seemed to not even comprehend what useEffectEvent was despite it also having a web search tool at its disposal. I ended up cutting it off when it recommended a plan including the web Event API.

Lastly, I asked the model to drop an unused MySQL table in a Python Django project. It did reasonably well at this where it correctly cleaned up the old models associated with the table, and also removed mention from the documentation. However, I still had to cut it off when it started going off the rails and manually writing a migration (whereas it could have been automatically generated by Django ORM).

What's Next

All of the above examples were in well-established codebases with quality gates such as linting, unit tests and integration tests. So it was really an ideal situation. I've been pleasantly surprised with Ornith despite it being such a small model. A year or two ago, a 9 billion parameter model would never have been this useful for software development.

Moving forward, I'd like to test out a larger mixture of experts model like Qwen3.6 35B. The reason being that Llama.cpp has the ability to configure these models to spill over into system RAM without as much of a performance penalty.