The "local AI vs cloud AI" debate has the same problem the "monolith vs microservices" debate had ten years ago: it pretends a design decision is an identity. It is not.
What "local" actually means
Local means the inference runs on hardware you own or control. That includes:
- Self-hosted GPUs in your own data center
- Cloud VMs you operate (still your control plane, not OpenAI's)
- On-device for clients with strict residency requirements
It does not mean "always cheaper," "always faster," "always more private," or "morally superior to API calls." Those framings are marketing, not engineering.
When local wins
- Hard residency constraints. EU data cannot leave the EU. Healthcare data cannot leave a HIPAA-attested environment. Where the law writes the contract, local is non-optional.
- Stable, high-volume, predictable workloads. Once you know your token throughput and it is large and steady, a self-hosted setup with vLLM or TGI beats API per-token pricing decisively.
- Long context with sensitive data. Sending whole repositories or patient histories to a third party is a governance problem even if privacy policies say it is fine.
- Latency-critical inner loops. Co-located inference at the data is often the only way to hit sub-100ms targets.
When cloud wins
- Bursty, unpredictable workloads. Demand swings by an order of magnitude per day. You will overpay for capacity you do not use.
- Frontier model dependency. You need GPT-5-class capability now. Nothing you self-host comes close.
- Tiny team, broad surface. Three engineers cannot operate a multi-GPU inference platform and ship the product features that justify it.
The decision framework
Skip the ideology. Score four dimensions:
- Sensitivity of the data being sent
- Stability of throughput
- Capability gap between best self-hostable and frontier model
- Team capacity to operate inference infrastructure
A weighted score, with the weights set by your business - not by Twitter.
The hybrid case
The most common right answer is not local or cloud. It is a hybrid: most inference local, sensitive-tenant inference local, frontier-capability calls to a cloud API behind a guardrail that strips identifiers, with a fallback path when the cloud is unavailable.
Hybrid is harder to operate. It is usually the correct answer anyway.
The lesson
When someone tells you local always beats cloud, ask them to define "beats." When someone tells you cloud always beats local, ask them to define "always." Architectural commitments that survive contact with production are the ones with decision criteria attached.