A Whole New World

Lately I've been using agents and SOTA models for all of my software development work. It wasn't easy for me to give up writing code by hand. But in early May, while working on a new Kubernetes operator for QuestDB Enterprise, everything about agentic coding just seemed to click. I was running multiple concurrent sandboxed sessions in tmux, each with their own set of subagents churning away at a greenfield codebase. And after around a week, I had a working product with multiple layers of tests (unit, integration, and live cloud e2e) across multiple k8s versions and cloud providers. Was the code perfect? Not even close. But I was able to accomplish something in a mere week that would've taken me months in the pre-AI days. I knew then that writing code by hand would soon be a thing of the past. And my old code wasn't perfect anyway.

Using AI exclusively for coding does stoke one of my existential fears as a software developer: having complete and utter dependence on third parties to do my job. What if all of the big model servers were down, like they were a few weeks ago, but for longer than a few hours? Or what if inference providers decide to jack up pricing to a point where it's not affordable for individuals to produce meaningful levels of useful output?

So I wanted to learn more about the local AI scene and run some models on my own GPUs. While the hardware I could afford still wouldn't get me frontier-level intelligence, it would at least allow me to somewhat control my own destiny. And maybe I'd learn a thing or two about the foundational models and technology that have driven these recent developments.

Not My First Rodeo

This actually wasn't my first foray into local models. Around a year or two ago, I experimented with a used RTX 3090 Ti (24GB VRAM) in my daily driver SFFPC (Ryzen 7600X/64GB RAM/4 TB SSD). At that time, many of the inference performance improvements that are commonplace today weren't as widely known. It was more difficult to load useful models into that small amount of VRAM that I had. So I tried a few heavily quantized models on ollama with limited context windows and blindly compared them to the latest Claude/ChatGPT models. My mistake. There was no hope for my 24GB card. And since the SOTA models were cooking so well for me, the GPU sat unused for a while and I eventually sold it. Oops, I probably left at least a couple hundred on the table given today's GPU prices!

In this recent local model expedition, I didn't want to make the same mistake again. I was determined to obtain enough VRAM to build a useful local agentic system. After talking about this with my coworker, he suggested that I take a look at the AMD R9700 AI Pro GPU: a 32GB VRAM card, which allows me to store larger models and KV caches in GPU memory. After a bit of digging around, I found a Discord dedicated to the R9700, with users getting >100 tok/s results using vllm-radiance, a custom vLLM build specifically designed for R9700 cards.

A Renewed Attempt, With New Hardware

So I bought a card, swapped out my old GPU, and loaded up the vllm-radiance docker image with the latest Qwen model, 3.6 at the time. But my speeds weren't great; I was only getting around 40 tok/s. After a lot of experiments and tweaking settings, I got a decent running configuration using Lemonade's llamacpp backend and around 100k context window, driven by Pi. This was the first truly useful local setup that allowed me to work on complex features for my infrastructure projects, mostly in Go.

But I knew that we could go faster. Vllm-radiance was optimized for 2 R9700 cards. And users were reporting well over 100 tok/s. As I saw GPU prices continue to rise across the board, I figured that I should secure a second GPU now instead of later (and prices have indeed risen since then).

Expanding my rig to 2 cards was a bigger challenge. This required a mostly-new PC: a new motherboard, case, and power supply to support the 2 GPUs over PCIe with x8/x8 lane splits. I planned out my build on the venerable PCPartPicker, using the R9700 Discord channel for specific hardware tips. Due to the wild price increases of RAM and persistent storage, I salvaged the NVMe drives and DDR5 from my previous build, saving me probably close to $2k. While I was in "build-mode," I also upgraded to a 12-core Ryzen 9900X.

New rig

And here's the result! I realized that it's difficult to get a decent picture of a tempered glass case (a Lian Li Lancool 216), but hopefully you can enjoy some of the tasteful RGB inside the case.

I'll be posting some benchmarks and experiment results soon. Vllm-radiance just released a new configuration that promises over 200 tok/s on a single card with dflash, so I'm looking forward to giving that a spin.