Why build a whole machine just to run AI?

July 21, 2026

I’ve written before about why I run models locally at all — privacy, control, and the way per-token cloud pricing punishes high-volume work. That post ends with a promissory note: add the GPU plan once the AI host is live. This is me starting to make good on it.

But there’s a gap between the argument in that post and what I’m actually building, and it’s worth being honest about. “Run models locally” is easy to say. In practice, for a while, it meant a modest GPU riding along in a machine that had other jobs — enough to run small, heavily-quantized models, enough to prove the idea. What I’m building now is a different thing entirely: a whole server whose only purpose is inference. Dual sockets, a pile of RAM, a real GPU, four rack units of dedicated hardware.

That’s a big step up from a spare card in a shared box. So the fair question — the one I’d ask if someone told me they were doing this — is: why? You can rent a frontier model by the token, today, with no capital outlay and no maintenance. Why build the machine?

Here’s the reasoning, and I’ll say up front: it isn’t “local is better.” It’s narrower and more interesting than that.

Dabbling has a ceiling

A small GPU sharing a host with other workloads is a great way to learn what local inference feels like. It’s a terrible way to actually depend on it.

The wall shows up in two directions at once. Model size: the good open models — the ones that are genuinely useful rather than merely impressive for their size — don’t fit in a small card’s memory, so you’re stuck quantizing them down until the quality you came for is gone. And concurrency: the moment you want to run more than one job at a time, or keep a model resident and ready instead of loading it on every request, a shared card starts thrashing and the other tenant on the box starts complaining.

You can push through this for a while with clever quantization and patience. But at some point the honest conclusion is that dabbling and depending are different activities, and they want different hardware. If local inference is going to be a service the lab offers itself rather than a party trick, it needs a machine that isn’t apologizing to anything else for the resources it uses.

The workload that pays for the box

The thing that turned this from “would be neat” into “build it” is a specific, boring, recurring job.

There’s a class of work in the lab that runs constantly and cheaply: batch classification and triage — taking a large pile of items, having a model read each one and sort, tag, rank, or summarize it. It’s not hard reasoning. It doesn’t need the smartest model on earth. It needs a good enough model run over a lot of items, on a schedule, forever.

Right now some of that runs on a cloud API, metered per token. Each individual run is cheap. But it recurs, indefinitely, and “cheap × forever” is just a subscription you didn’t mean to sign up for. That’s the exact shape of cost that local inference is good at eating: trade a variable bill that never ends for a fixed one you pay once. Buy the hardware, pay for electricity, and the marginal cost of the next million tokens rounds to zero.

The tipping point isn’t philosophical, it’s arithmetic. Below some monthly volume, cloud wins and you’d be foolish to build. Above it, the machine pays for itself and then keeps paying. I’m building because the always-on, high-volume, doesn’t-need-to-be-brilliant work has crossed that line — and unlike a benchmark you run once to feel good, that workload will still be there next year justifying the electricity.

The thing you can’t rent: room to be wasteful

There’s a softer reason that I think is actually the bigger one, even though it’s harder to put on a spreadsheet.

When every experiment has a meter running, you don’t experiment. You plan the call, you make it, you move on. That’s fine for production. It’s poison for learning. Half of understanding this stuff comes from being able to run the same thing forty times with slightly different settings, leave a model loaded all afternoon poking at it, or kick off a wildly inefficient job overnight just to see what happens — without a running total in the corner of your eye.

A machine you own gives you permission to be wasteful, and being wasteful in a bounded, private sandbox is where the actual knowledge lives. That’s not a cost saving. It’s a different relationship with the tool.

The bet that makes it worth a whole machine: memory

Here’s the part that turns “a GPU in a box” into “a server worth building,” and it’s the piece I’m most curious about.

The obvious way to think about an inference host is: how big a GPU can I afford? Fair — the GPU does the fast work. But a dedicated dual-socket server can carry something a gaming rig never will: an enormous amount of system RAM. And it turns out that a very large pool of ordinary system memory, paired with a capable GPU, opens a door that neither piece opens alone.

The newest wave of open models are mixture-of-experts designs — models with enormous total size but only a fraction of it active on any given token. That structure is unusually forgiving of a setup where most of the model lives in plain system RAM and only the active parts get pulled onto the GPU. The result is slow — nobody’s holding an interactive conversation at these speeds — but “slow” and “impossible” are very different words. Batched work, overnight jobs, the triage pile that doesn’t care whether it finishes at 2am or 6am: those don’t need interactive speed. They need the job to be possible on hardware I own, and a big memory pool is what makes running frontier-class models at home possible at all.

You cannot do that on a spare card in a shared box. That capability is the actual reason for the whole machine — not the GPU alone, but the GPU and a wall of memory together. It’s the difference between “I can run small models locally” and “I can, slowly and cheaply and privately, run models that are genuinely near the frontier.” That’s a bet worth four rack units.

The honest caveat

As always on this site: you do not need any of this. If you send a moderate volume of prompts to a cloud API, keep doing that — it’s cheaper, it’s zero maintenance, and the frontier hosted models are still better than anything that fits in my rack. Building a dedicated inference host only makes sense at a specific intersection: enough recurring volume that owning beats renting, enough privacy-bound or always-on work that the cloud is a poor fit, and enough curiosity that “room to be wasteful” reads as a feature rather than a liability.

I’m at that intersection. And — the caveat this whole site runs on — it’s also just genuinely fun to build a machine whose entire reason for existing is to think.

That’s the why. Next in this series: the what — the actual hardware, piece by piece, and the reasoning behind each choice. Some of those choices were obvious. A couple were not, and one of them I changed my mind on entirely.

← all posts