I got tired of paying for cloud inference. I wanted to know one concrete thing: how fast can I get a 27B-parameter model running on consumer-grade hardware, and what does it actually cost in electricity?
The answer ended up being ~200 tokens/second on a dual AMD box, found through 300+ structured experiments. This post is the overview — the hardware, the software stack, real benchmark numbers, and an honest cost breakdown. It's not a tutorial. It's a working system and the research that built it.
The hardware
Three machines, each with a clear job. No cluster, no cloud.
| Machine | Hardware | Role |
|---|---|---|
| AI Server | Dual AMD AI PRO R9700 (gfx1201) | vLLM inference, llama-swap model routing — the workhorse |
| Tower | RTX 3080 Ti 12GB, Ryzen 9 8945HX | Local Ollama (multi-model), ComfyUI image generation |
| Unraid | — | Docker host: Agent Zero, RAG pipeline, monitoring, DNS |
The AI server does the heavy lifting. Two R9700s running a 27B model in a 4-bit quantization (MXFP4), with tensor-parallel across both cards. The tower handles smaller models and image generation on a single consumer GPU. Unraid is the glue — it hosts the Docker containers that tie everything together.
The software stack
What runs where, and where the code lives:
| Component | What it does | Repo |
|---|---|---|
| vLLM | The inference engine on the AMD box | vllm-project/vllm |
| llama-swap | Routes between resident models on the AI server | mostlygeek/llama-swap |
| Ollama | Multi-model local LLMs on the tower | ollama/ollama |
| ComfyUI | Local image generation (Z-Image-Turbo) | comfy-org/comfyui |
The internal projects (Agent Zero, Scaffold RAG, and others) don't have public repos yet — they're the consumers of the inference stack.
The models resident on the AI server right now: Qwen3.8 in both MXFP4 and NVFP4 quantizations, plus a 35B MoE (A3B) for parallel/background work. llama-swap swaps them in and out of VRAM as needed.
The performance
These are real numbers from the experiment tracker, not marketing. The frozen production baseline is a 27B model in MXFP4, tensor-parallel across both R9700s:
| Metric | Value | What it means |
|---|---|---|
| Decode speed | 185.8 tok/s | Sustained token generation |
| TTFT | 84 ms | Time to first token (prompt processing) |
| Prompt processing | 3,841–4,215 tok/s | How fast a long context is ingested |
| GPU power (load) | 146 / 151 W | Per-card, under active decode |
| GPU power (idle) | 47 / 50 W | Per-card, at rest |
The ~200 tok/s headline is the rounded figure; the measured frozen config sits at 185.8. That's the number I optimize against.
A few experiments that shaped the config (each tested, each with a verdict):
| Experiment | Change | Result | Verdict |
|---|---|---|---|
| E01 | Disable fused attention | −22% decode | Fail — keep attention on |
| E05 | Single GPU (TP1) | −57% decode | Fail — TP2 wins everywhere |
| M4 | MTP-8 spec decoding | −47% decode | Fail — dflash-7 stays |
| FP8 | FP8 vs MXFP4 | −61% decode | MXFP4 wins decisively |
| M7 | Disable prefix caching | −7% decode | Fail — prefix caching stays on |
The pattern: most "obvious" optimizations made things worse on this hardware. The hand-written attention kernel, 4-bit quantization, and prefix caching are the three that actually moved the needle.
The cost
This is the part that actually matters. I have measured per-GPU power, and I'll be honest about where I'm estimating.
Measured (from the experiment tracker): - Under load: ~146W + ~151W = ~297W across both GPUs - At idle: ~47W + ~50W = ~97W across both GPUs
Estimated (system total, clearly labeled): adding the CPU, RAM, drives, fans, and ~90% PSU efficiency, I estimate the whole box draws ~400W under load and ~175W at idle. That's the number I use below — if your system draw differs, scale proportionally.
Cost per million tokens (decode at 185.8 tok/s)
| Electricity rate | Per 1M tokens | Per 1M tokens (monthly, ~100M tokens) |
|---|---|---|
| $0.10 / kWh | ~$0.06 | ~$6 |
| $0.16 / kWh | ~$0.10 | ~$10 |
| $0.30 / kWh | ~$0.18 | ~$18 |
How: 1,000 tokens takes 5.38s. At 0.4 kW that's 0.0006 kWh per 1,000 tokens. × rate, × 1,000 for a million.
For comparison, a 27B-class model on a commercial API typically runs $0.50–$3.00 per million tokens. So local inference is roughly 5–30× cheaper at scale — and the marginal cost of a token is basically the electricity it takes to generate it.
The idle cost (the box is always on)
| Electricity rate | Idle draw | Monthly (730 hrs) |
|---|---|---|
| $0.10 / kWh | ~175W | ~$13 |
| $0.16 / kWh | ~175W | ~$20 |
| $0.30 / kWh | ~175W | ~$38 |
The idle cost is the real price of convenience — it's what you pay to keep the model warm. If you only run the box when you need it, the idle cost drops to near zero and the per-token cost is all you pay.
What it's actually used for
This isn't a science project. The stack runs daily drivers: - Agent Zero — an autonomous AI agent that does real work (this very site was built by it) - Scaffold — a multi-source RAG pipeline (files, S3, git, APIs) - ComfyUI — local image generation
The whole point: a 27B model running at ~200 tok/s, on hardware I own, for the cost of the electricity it uses. No per-token billing, no rate limits, no data leaving the house.
What I'd do differently
- Start with the 4-bit quantization. FP8 looked like the obvious choice and was 61% slower. MXFP4 won decisively — I wasted a week finding out.
- Trust the benchmarks over the claims. Most "optimizations" in the vLLM/ROCm world made things worse on this hardware. Measure, don't assume.
- Tensor-parallel from day one. Single-GPU was 57% slower. The two cards working together is the whole win.
- Track power from the start. The cost story only works because I logged wattage on every experiment. That data is what made the electricity math possible.
The full experiment log and config files are available on request.