When local AI deployment is the right answer

Most businesses should use a cloud AI API. It’s cheaper, faster to implement, and requires zero ops work. We say this to clients often.

There is, however, a narrower set of situations where local deployment is the right answer. This post is about recognizing which situation you’re in.

The default answer is “use an API”

If you’re a general small business — a restaurant, a trades company, an e-commerce shop, an early-stage SaaS — the cloud AI providers (OpenAI, Anthropic, Google) are the correct choice. You pay per request, there’s no infrastructure to manage, and the models are excellent.

We build cloud-API-backed features for most of our clients. It usually takes days of engineering, not months.

When that answer changes

There are a few specific cases where cloud APIs stop being viable:

Regulated industries with data-processor restrictions.

• Healthcare (PHIPA in Ontario, HIPAA in the US). Patient-identifiable information typically cannot be sent to a third-party processor without a signed BAA and a specific legal review. Several of the major cloud AI providers will sign a BAA for enterprise plans, but not on the default product.
• Legal (attorney-client privilege). Sending privileged communications to a third party can, depending on the jurisdiction and the firm’s position on the question, be treated as a waiver. Firms are risk-averse here for good reasons.
• Finance (PCI-DSS, SOC 2, banking regulations). Transaction data, PII, and account information often have controls that preclude a third-party AI API by default.

Government or defense contracts. Data classification and residency rules make this one straightforward.

Contractual restrictions from your customers. We’ve seen B2B SaaS products whose enterprise customers prohibited any third-party subprocessor. If one of your clients has that clause, your AI stack needs to respect it.

What “local deployment” actually involves

Local deployment means running a model on infrastructure you control — either on-prem or in a private cloud tenant (dedicated VPC in AWS, Azure, or GCP).

In practice, the setup is:

1. An open-weight model (Llama 3.1, Mistral, Qwen, or a domain-specific fine-tune).
2. A GPU or small cluster sized for your expected traffic.
3. An inference server (vLLM, TGI, or Ollama) that exposes an HTTP endpoint.
4. A thin API layer that your application calls instead of api.openai.com.
5. Logging, monitoring, and access controls that meet your compliance requirements.

The work isn’t magical, but it isn’t trivial either. For a single-GPU 7B-model deployment, a reasonable estimate is two to four weeks of engineering, plus ongoing ops time measured in hours per month.

What it doesn’t do

Local deployment solves the data-processor question. It does not, by itself, solve any of these:

• Data residency within a specific country (you still need to pick your region).
• Encryption at rest and in transit (you still need to configure it).
• Access controls (still your responsibility).
• Audit logging (still required for compliance).
• Model quality (a smaller local model may be meaningfully worse at some tasks than GPT-4 or Claude).

If someone sells you local deployment as a full compliance solution, push back.

The honest tradeoff

Local deployment is more expensive to build and maintain than a cloud API call. The reason to do it is not cost — it’s control. You own the inference pipeline, the data never leaves your environment, and you can answer “where does this data go?” with a one-line answer.

For regulated industries, that answer is often worth the extra engineering. For most other businesses, it isn’t.

How we approach it

When a client asks about local deployment, we work through a short checklist:

1. Is there a specific regulatory or contractual reason this can’t be a cloud API? If not, we usually recommend the cloud API.
2. What’s the use case? A narrow, well-defined task (classification, extraction, summarization) is much easier to serve locally than an open-ended assistant.
3. What’s the realistic volume? A handful of requests per hour is different from a few requests per second, and the hardware sizing is very different.
4. Who will operate this after launch? If there’s no one, we either run it on a retainer or push back on the decision.

Only after that do we scope the actual build.

If you’re trying to figure out whether local AI applies to your situation, we can walk through it with you.