AI That Never Leaves the Building. A Working Reference Architecture, Not a Brochure.

November 22, 2026

Ask any public sector CTO what stops them from deploying AI at full speed and you will not hear “the models are not good enough” anymore. You will hear a quieter question: where do my data actually go when someone presses Enter?

The answers on offer are almost all contractual. Data processing agreements, residency clauses, sub-processor lists, audit rights on thirty days’ notice. Fine words, and then the payload ships to somebody else’s GPU in somebody else’s country under somebody else’s terms of service.

I have been running the alternative as a daily production system for over a year now. Not a pilot, not a demo rack for a press photo, but the machine that writes my client’s code, answers my voice assistant, reads my mail and files my timesheets. It runs on open-weight models on commodity hardware inside a building I own, and the interesting part is that it did not require a lab or a budget line. This post is the reference architecture, with the honest caveats, because “sovereign AI” has become a brochure word and brochures are exactly what procurement does not need.

The building

Five machines, all off-the-shelf, most of them second-hand or upgraded gaming parts.

Machine CPU GPU RAM Job
Desktop Ryzen 5 5600X RTX 3090 24GB 64GB Main inference lane (262k context, vision), Git hosting, mail bridge
Bonobo Ryzen 9 3900 RTX 3090 24GB 128GB Second inference lane, agent workloads
Axolotl Ryzen 7 5800X RTX 3090 24GB 128GB Third inference lane, interchangeable with Bonobo
Dell 2x Xeon Silver 4210R none 62GB Deployment target, Postgres, Qdrant vector search, embedding service, the whole voice stack
Laptop Intel integrated 16GB The interface, the only write path for code

Three RTX 3090s, 72GB of VRAM total, bought used when the crypto miners gave up. A family laptop with a 4070 runs the household’s speech-to-text and text-to-speech so even the voice layer stays inside. The entire monthly bill for the core loop is a few euros of electricity. No per-token invoice, no seat licenses, no rate limits at 2 a.m.

The models are open-weight Qwen, quantized to GGUF files. The flagship lane runs a 262k-context vision-capable build; two other lanes run the same family at different quantizations so workloads can move between machines. Updating the model means downloading a file once, then running it offline forever. There is no vendor roadmap between me and my infrastructure, and no kill switch on somebody’s dashboard.

What sovereignty actually means in practice

Here is the honest version, because the marketing version is a lie. This is not an air-gapped bunker. Mail arrives over IMAP through my own bridge, client code pushes to a Git host, and the agent can fetch web pages when I decide a task needs them.

Sovereignty is not the absence of network cables. It is a boundary you draw deliberately, can enumerate, and can prove. So here is my enumeration, and it is short enough to fit on a slide, which is the point.

What never leaves the building: every model inference, every document the agent reads, the full mail corpus indexed for search, all client source code the agents work on, every voice recording and its transcript, the vector database, the journals, the task trees. The working set, the part that would make a data protection officer sweat, is processed exclusively on my GPUs, on my switches, on wires I could physically point at.

What leaves, and only when a human decided so: outbound email, chat messages, published code, and deliberate web fetches. Each one is a separate, logged, human-approved event. More on that gate below, because it is the part procurement should actually demand.

If a public sector body wrote this as a procurement requirement, “the inference and the working corpus never cross this line, and every egress is an explicit logged act”, it would be enforceable in a way that “we care about your privacy” never is.

The governance layer is the real product

Everyone talks about models. The part that should matter to a public organisation is what happens when the model wants to do something.

My agents have hands. They can send email, post to chat systems, create tickets, deploy code. And every single one of those actions runs through a human-in-the-loop gate implemented in the code path itself, not in a policy PDF. An agent that composes an outbound message does not send it. It writes a draft row into a pending_messages table, and the message reaches the outside world only when a human opens a review window and presses Send. The tool literally cannot bypass the gate, because the gate is between the draft and the transport.

The same shape covers everything else. Every action is journaled, every durable decision lands in a knowledge base with a date and a source, every task lives in a tree I can walk and audit after the fact. When an auditor asks “what did your AI do on Tuesday”, the answer is a query, not a shrug.

This is what I would put in front of a procurement committee, and it generalises trivially to a ministry. The review gate is not a feature of my cleverness. It is a pattern: draft, queue, human approves, transport. Any organisation can demand it from any vendor, and the moment they demand it, the difference between a toy and a governed system becomes a line in the contract instead of a hope.

The numbers

Metric Cloud API setup This building
Inference latency ~800 ms per call ~200 ms on local inference
Tool round-trip ~150 ms ~20 ms on LAN
Monthly cost tens to hundreds of euros per seat a few euros of electricity
Where the data lives contractual promise these machines, in this room
Model deprecation vendor decides never, unless I decide

The latency column hides a design point. An agent turn is not one model call, it is dozens of tool calls, and every round-trip compounds. Local hardware is not a philosophical preference, it is what makes an agent with 1800+ tools feel instant instead of sluggish. The tool orchestration itself (semantic just-in-time injection of tool schemas, so a local model can command nearly two thousand functions without drowning its context) is a separate post, and the five-times-we-tried-to-build-our-own-agent-engine post-mortem is another.

The failure model, because sovereignty includes staying up

A cloud vendor sells you someone else’s redundancy. A building sells you your own, and mine degrades by design. If the desktop dies, the inference lanes on Bonobo and Axolotl keep serving and background jobs pause. If an inference lane dies, the others absorb its work. If the Dell dies, search pauses but inference and services keep running. No single machine’s death takes the system down, and every failure mode is a known, rehearsed one.

That is the honest trade-off, and I would rather state it than bury it. Sovereign infrastructure has no invisible ops team behind it. You own the uptime, which means you design for it, and a public body has staff who can do exactly that, staff who currently spend their days babysitting someone else’s cloud console instead.

What I would want a procurement to ask for

I am not asking public sector to buy gaming PCs. I am pointing out that the reference architecture already exists, runs real work, and costs less than one SaaS seat per month. So the requirements I would want written into the next AI framework contract are concrete:

  1. Inference and the working corpus stay inside the boundary. Name the boundary.
  2. Every outbound action from an AI system is a draft a named human approves, mechanically, in the transport path.
  3. Every action is journaled and queryable after the fact.
  4. The models are open-weight or otherwise permanently pinned, so no vendor can retire your system under you.
  5. The hardware is commodity and the architecture survives losing any one machine.

None of these are exotic. I run all five, today, on parts I could inventory in an afternoon.

The building does not have to be a building. It can be a ministry’s existing server room, a municipality’s rack, a university’s idle compute. The only thing that has to be true is the sentence in the title. The AI, the data, and the decisions happen inside, and what leaves is a human choice with a timestamp.

That is not a product I am selling. It is a working proof that the requirement is satisfiable, published so that procurement officers can point at it and say: someone already runs this, so why can’t you?

Best Regards, Heikki Kupiainen / Metamatic Systems