Private AI · on-premises & air-gapped
AI that never leaves your network.
Local LLMs and agents running on your own hardware, behind your firewall, for work where data egress is not an option. Proxmox, Docker, PyTorch, Ollama, vLLM — open-source throughout.
01 · stack
The layers, end to end.
A coherent stack — hypervisor to agents — chosen so every layer is replaceable, every model is auditable, and nothing depends on a third-party API.
stack · gpu paths
Two GPU paths, one playbook.
Hardware choice is a budget and supply-chain decision, not a lock-in. The same stack runs on both vendors — the real differences live in three places: passthrough setup, runtime flags, and serving maturity.
- RTX / datacenter cards, passthrough or vGPU
- CUDA 12.8 wheels — no patchwork, no flags
- Best for: smoothest tooling, mixed training + inference
- Radeon 7900 XTX / Instinct MI series via PCIe passthrough
- ROCm builds of Ollama, vLLM, and PyTorch 2.7
- Best for: maximum VRAM per dollar, inference-first boxes
02 · security posture
Six things that are not optional.
A private AI platform is only as good as the boring controls behind it. These are the defaults on every build.
01 · Zero egress
Inference and data stay inside the LAN. Open-weight models are pulled once during provisioning, then the upstream connection is closed. The inference VLAN has no default route to the public internet.
02 · Segmentation
AI workloads live in their own VLAN and firewall zone, separated from user endpoints. MikroTik rules between zones; admin plane on a dedicated management network with its own ACLs.
03 · Provenance & integrity
Every model version is pinned (tag + SHA), checksum-verified before load, and limited to the safetensors format. No pickle, no remote code execution paths, no surprises at deploy time.
04 · Least-privilege agents
Agents start read-only. Tools that write — to disk, to a database, to an external system — require explicit human approval per call. The default is 'ask', not 'act'.
05 · Secrets hygiene
No API keys, tokens, or credentials in prompts, logs, or commit history. A local secret store (HashiCorp Vault or equivalent) with short-lived credentials and documented rotation.
06 · Auditability
Append-only logs of every request, tool call, and action — enough to answer 'what did the agent do, and why' months later. Retention is configurable; integrity is signed.
03 · reference architecture
How the pieces connect.
The same shape every time: users reach the agent gateway through a reverse proxy, the gateway talks to Ollama or vLLM on the GPU VM, and nothing crosses the VLAN boundary.
┌─────────────────────────────────────────────────────────────────┐
│ users (browser / api clients) │
└─────────────────────────────────────────────────────────────────┘
│ tls
▼
┌─────────────────────────────────────────────────────────────────┐
│ reverse proxy · nginx / caddy │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ agent gateway · langgraph / custom loop │
│ · read-only tools by default │
│ · human-in-the-loop gate on every write action │
│ · append-only audit log │
└─────────────────────────────────────────────────────────────────┘
│ │ │
▼ ▼ ▼
┌──────────────────┐ ┌─────────────────────────┐ ┌──────────────┐
│ postgres + vector │ │ ollama / vllm │ │ object store │
│ (rag, audit) │ │ gpu vm · open weights │ │ (docs, logs) │
└──────────────────┘ └─────────────────────────┘ └──────────────┘
│ │ │
└──────────── vlan · ai-internal ──────────────┘
│ no egress
▼
(firewall)04 · engagements
Four shapes the work usually takes.
Each engagement is sized and quoted after diagnosis. Pick the one that matches the problem, or combine them.
05 · when this is the right call
If any of these sound like you, we should talk.
Private AI is the right answer when the public option is the wrong answer. A few honest signals.
Regulated data — health, legal, finance — that cannot leave controlled environments.
Trade secrets or IP inside the prompts: code, models, contracts, customer data, internal strategy.
Offline or low-connectivity sites — research vessels, plants, secure facilities — where the link is intermittent or absent.
Predictable cost at sustained load — flat hardware spend instead of per-token billing that scales with adoption.
/contact
Have data that can’t leave the building?
Tell me what you’re running, what it touches, and what it cannot do. I read every note and reply within two business days.