Private AI for small business: what it costs and what you get

What I mean by "private AI"#

Private AI, in the way I use the term with clients, means running the model on hardware you control — a workstation in your office, a server in a colocation rack, or a rented virtual machine that belongs to you. Your prompts and the model's responses never leave that box. There is no third-party API, no per-token invoice, and no telemetry back to a foundation-model vendor.

That distinction matters more than the model choice. A private 14B model answering from your own PDFs can be a better answer for a small clinic or a law firm than a frontier model phoning home with every transcript. The privacy story is the whole point.

The three tiers I actually deploy#

Private LLM deployments generally land in one of three shapes. The right one depends on call volume, latency budget, and how strict your data-residency requirements are.

Tier 1 — used workstation GPU ($800–1,500)#

A previous-generation NVIDIA card (RTX 3090, 4090, or a 24 GB workstation card from eBay or a refurbished dealer), 64 GB of system RAM, a 1 TB NVMe drive, and a quiet case. Total landed cost in the $800–1,500 range. It will comfortably run a 7B or 14B quantized model at 30–60 tokens per second on a single user, and a 32B at a slower but usable clip with offloading.

For a team of one to five people doing internal search, summarisation, and code assistance, this is the cheapest private AI you can buy. Power draw is around 350–600 W under load, which works out to a few dollars a day on a normal commercial tariff.

Tier 2 — small dedicated server ($3,000–5,000)#

A proper server with a single workstation-class GPU such as an RTX A4000 or A4500, ECC memory, redundant NVMe, and IPMI remote management. This is the tier that fits a clinic or a small professional-services firm with five to twenty users. It will host a 14B or 32B model in a real production setting, keep logs locally, and survive a power blip.

The hardware is more reliable than a workstation, but the real upgrade is operational: you can put it in a rack, monitor it with Uptime Kuma, back it up properly, and patch the OS without walking over to someone's desk.

Tier 3 — rented VPS or bare metal ($40–80 / month)#

If you do not want to buy hardware, a dedicated VPS with a consumer GPU or a bare-metal provider (Hetzner, OVH, Lambda, RunPod, Vast.ai) gets you to the same place on a monthly invoice. Forty to eighty dollars a month puts a 7B–14B model in your VPC, and sixty to a hundred and fifty a month handles a 32B comfortably. The trade-off is data residency — you are trusting a third party with the disk, even if not the model.

For most small businesses this is the entry tier. You start on rented hardware to validate the use case, then buy a box once you know the workload is real.

What you actually get at each tier#

A useful mental model: private models are now genuinely good at the things small businesses ask them to do.

• 7B–14B (Qwen, Mistral, Llama 3.x family): summarisation, drafting, internal Q&A over your docs, classification, extraction, code help. These run on Tier 1 or even on a modern CPU laptop with reasonable speed.
• 30B–70B quantized: noticeably better at multi-step reasoning, longer context, and following complex instructions. Needs Tier 2 hardware or a rented 24 GB+ GPU. This is the sweet spot for most "AI agent" prototypes.
• Frontier (>200B, hosted): still wins at hard reasoning, long-horizon planning, and edge-case code. But you give up the privacy story, and the per-call cost adds up fast at any non-trivial volume.

When private wins#

Private is the right answer when any of these are true:

• Your data is regulated, sensitive, or competitively valuable. Legal, medical, financial, internal HR, customer support transcripts — all of it.
• You want predictable monthly costs. Once the box is paid off, marginal inference is effectively free, no matter how many calls your team makes.
• You need offline or air-gapped operation. Field engineering, defense-adjacent work, remote sites without reliable uplink.
• You are tired of vendor pricing changes. Hosted model pricing and deprecation schedules move on the vendor's timetable, not yours.

Against an equivalent API bill of a few thousand dollars a month, Tier 1 or Tier 2 hardware typically pays for itself inside three to six months — run the arithmetic on your own token volume before committing.

When private is the wrong answer#

It is also worth saying where I do not push this. If your team is small, your call volume is light, your data is not sensitive, and you need a frontier-class model tomorrow — just call the API. The operational overhead of running your own hardware is real, and the cheapest private box is not the cheapest AI.

If you want help thinking through whether private AI makes sense for your shop, the private AI service page walks through what I actually deploy. And if you want a straight answer on whether it pencils out for your case, send me the rough numbers and I will tell you honestly.

---

Frequently asked questions#

How long does it take to set up a private LLM for a small team?#

For a Tier 3 rented setup with Ollama, Open WebUI, and a basic RAG pipeline over your documents, a working pilot is two to three days of work. A proper Tier 2 server with backups, monitoring, and SSO takes closer to two weeks. The slow part is never the model — it is getting your documents into a clean retrieval index.

Is private AI really cheaper than the API?#

It depends on volume. Below a few million tokens a month, the API is cheaper because you have no hardware sunk cost. Above that line, private wins hard. Teams that adopt AI seriously tend to cross that line within a few months, which is why I recommend starting on Tier 3 and graduating to owned hardware once the workload is real.

What models do you actually run?#

Qwen 2.5 and Qwen 3 in 7B, 14B, and 32B sizes, Llama 3.x in 8B and 70B, Mistral, and the occasional specialised model for code or medical text. The open-weights landscape moves fast, so the default gets reviewed quarterly. Everything is served through Ollama or vLLM, so the client never has to think about the model lifecycle — see the private AI service page for the deployment shapes.

---

Sources: Ollama and vLLM documentation; published Hetzner, OVH, RunPod and Vast.ai GPU pricing (mid-2026); community benchmarks on the LMArena and HuggingFace open-model leaderboards. Prices are indicative — verify current rates before budgeting.