Azure OpenAI vs Self-Hosted LLMs - Cost, Latency, and Data Residency for EU

In 2026, the LLM market has matured enough that both approaches are production-ready. Azure OpenAI offers OpenAI models with SLA guarantees and Data Zone deployments in Europe. Meanwhile, the self-hosting stack (vLLM, SGLang, TensorRT-LLM) has reached stability levels capable of serving thousands of requests per second on GPU clusters.

The question isn’t “which is better” - it’s “which is better for your scenario”. A company processing 500 requests per day has completely different economics than a platform handling 2 million tokens per hour.

Decision Framework - When Azure OpenAI, When Self-Hosting

Before diving into pricing and benchmarks, here’s a quick framework:

Choose Azure OpenAI when:

• Volume < 50,000 requests/day (dedicated GPU cost is unjustified)
• You need frontier models (GPT-4o, GPT-4.1, o3) - unavailable as open-weight
• Your organisation requires 99.9% SLA without building an MLOps team
• Integration with the Microsoft ecosystem (Entra ID, Azure AI Search, Cosmos DB)
• Capital expenditure on GPU infrastructure is not acceptable

Choose self-hosting when:

• Volume > 200,000 requests/day (economies of scale tip the balance)
• Data cannot leave your infrastructure - not even to Microsoft’s network
• You need models fine-tuned on your own data with full weight control
• Latency < 100ms TTFT is critical (eliminating network overhead)
• You have an MLOps/Platform Engineering team capable of maintaining GPU clusters

Grey zone (50-200K requests/day): Here the decision depends on the ratio of GPU costs to team costs. If you already have a Kubernetes cluster with GPUs (e.g., for training) - self-hosting is a natural extension. If GPU would be a new expenditure - Azure OpenAI will likely win.

Azure OpenAI Service - Architecture, Pricing, and Data Zones

Deployment Types and Their Impact on Data Residency

Azure OpenAI offers three deployment types, each with different location guarantees:

Key for EU organisations: Data Zone Standard guarantees that both inference (prompt processing) and data at-rest remain in European Azure regions. This satisfies GDPR Articles 44-49 without additional data transfer safeguards.

For financial services organisations, Regional Standard with deployment in Sweden Central or Germany West Central provides the strongest guarantees.

Azure OpenAI Pricing - Production Models (Pay-As-You-Go)

Prices per million tokens, Data Zone Standard (EU) deployment:

Provisioned Throughput Units (PTU) - Fixed Cost, Guaranteed Performance

For predictable workloads, Azure offers PTU - reserved dedicated throughput billed per hour, regardless of token count:

• PTU provides guaranteed throughput without throttling
• Cost-effective from ~50M tokens/month (break-even vs. Pay-As-You-Go)
• Requires commitment: minimum 1 month (Monthly Reserved) or 1 year (Annual Reserved with discount)
• Pricing varies by model: GPT-4o ≈ $6/PTU/hour, GPT-4.1 mini ≈ $3.50/PTU/hour

Example calculation: An e-commerce platform with an AI assistant handling 30,000 conversations/day (average 800 tokens input + 300 output per turn, 4 turns per conversation):

• Daily consumption: 30,000 × 4 × (800 + 300) = 132M tokens/day
• Pay-As-You-Go (GPT-4.1 mini): input 96M × $0.40/M + output 36M × $1.60/M = $96/day ≈ $2,880/month
• PTU (estimated 3 units GPT-4.1 mini): 3 × $3.50 × 24h × 30 = ≈$7,560/month

In this scenario, Pay-As-You-Go wins. PTU makes sense at significantly higher volumes or when latency guarantees are critical (eliminating cold starts and queue delays).

Self-Hosting LLMs - Technology Stack and Costs

Reference Architecture on Azure (AKS + GPU)

Self-hosting on Azure means running open-weight models on Azure Kubernetes Service with GPU node pools:

┌───────────────────────────────────────────────────────┐
│  Azure Kubernetes Service (AKS)                       │
│                                                       │
│  ┌──────────────┐  ┌──────────────┐  ┌─────────────┐ │
│  │ Node Pool    │  │ Node Pool    │  │ Node Pool   │ │
│  │ CPU (system) │  │ GPU (infer)  │  │ CPU (tools) │ │
│  │ Standard_D4s │  │ NC A100 v4   │  │ Standard_D8s│ │
│  │ 3 nodes      │  │ 2-4 nodes    │  │ 2 nodes     │ │
│  └──────────────┘  └──────────────┘  └─────────────┘ │
│                                                       │
│  Inference engine: vLLM 0.8+ / SGLang                 │
│  Model store: Azure Blob Storage (Premium)            │
│  Monitoring: Prometheus + GPU metrics exporter        │
│  Load balancer: Azure Application Gateway             │
└───────────────────────────────────────────────────────┘

GPU Costs on Azure - Production Instances

Availability note: In European regions (North Europe, West Europe, Sweden Central), A100 and H100 instances require a prior quota request. Wait time: 1-4 weeks. Plan ahead.

Cost Per Token - Self-Hosting vs Azure OpenAI

The key calculation: how much does it cost to generate 1M tokens on your own infrastructure?

Scenario: Llama 4 Scout 17B on 1× A100 80GB (vLLM, FP16)

• vLLM throughput: ~3,500 tokens/second (output, continuous batching)
• Tokens per hour: 3,500 × 3,600 = 12.6M tokens
• Instance cost: $3.67/hr (PAYG) or $2.34/hr (1Y RI)
• Cost per 1M tokens: $0.29 (PAYG) / $0.19 (RI)

Scenario: Llama 3 70B on 2× A100 80GB (vLLM, 4-bit AWQ)

• vLLM throughput: ~1,800 tokens/second (output, continuous batching)
• Tokens per hour: 6.48M tokens
• Instance cost: $7.35/hr (PAYG)
• Cost per 1M tokens: $1.13 (PAYG) / $0.72 (RI)

Compared to Azure OpenAI GPT-4.1 mini ($1.60/M output): self-hosted Llama 3 70B delivers comparable quality for many tasks at $0.72-1.13/M - a 30-55% saving. But this doesn’t account for operational costs.

Hidden Costs of Self-Hosting - What Goes Into TCO

Raw GPU cost is not the full picture. Complete TCO of self-hosting includes:

At 190M tokens/day (5.7B/month) on Azure OpenAI GPT-4.1 mini:

• Input (60%): 3.42B × $0.40/M = $1,368
• Output (40%): 2.28B × $1.60/M = $3,648
• Total Azure OpenAI: ~$5,016/month

Conclusion: At this volume with GPT-4.1 mini, Azure OpenAI is cheaper than or comparable to self-hosting when TCO is included. Self-hosting wins only when:

1. Volume grows 3-5× (GPU amortises better)
2. You use open-weight models that match GPT-4.1 mini quality for your specific task
3. An MLOps team already exists (marginal cost of managing LLM is low)

Latency - Production Scenario Comparison

Latency Components

End-to-end latency (from sending request to first token - TTFT) consists of:

When Latency Is the Deciding Factor

• Customer-facing chatbots (TTFT < 500ms): Both solutions meet requirements in most cases. Azure OpenAI at P95 may exceed 500ms under load - consider PTU.
• Code autocomplete (TTFT < 200ms): Self-hosting wins decisively. Azure OpenAI Pay-As-You-Go cannot guarantee this level.
• Batch processing (latency irrelevant): Azure OpenAI Batch API with 50% discount - cheapest option.
• Real-time voice/streaming (TTFT < 100ms): Only self-hosting with dedicated GPU without resource contention.

Data Residency - Legal Requirements for EU Organisations

GDPR and Personal Data Processing in LLMs

When prompts contain personal data (names, national IDs, medical data, correspondence), GDPR data transfer provisions apply:

Azure OpenAI - Microsoft guarantees:

• Data Zone Standard: inference and data at-rest in EU regions
• Microsoft as data processor (Data Processing Agreement built into Enterprise agreement)
• Prompts are not used for model training (Zero Data Retention for selected models)
• Audit logs available in Azure Monitor / Diagnostic Settings
• Certifications: ISO 27001, SOC 2 Type II, C5 (BSI), ENS High

Self-hosting - full control:

• Data never leaves your infrastructure (VNet/Private Endpoint)
• You are the data controller - no external processor
• No dependency on cloud provider contractual terms for the AI layer
• Requires your own compliance documentation (DPIA, RoPA)

Regulated Financial Services - Additional Requirements

For organisations regulated by financial authorities (FCA, BaFin, FINMA equivalents), the key considerations are:

1. Cloud outsourcing - requires regulatory notification in line with EBA guidelines on outsourcing arrangements
2. Sub-processor chain - Azure OpenAI introduces Microsoft as a processor and potentially sub-processors (datacenter operators)
3. Right to audit - Microsoft Enterprise Agreements include an audit clause, but practical execution is limited
4. Exit strategy - required by regulators, easier with self-hosting (open-weight model = no vendor lock-in)

Recommendation for financial services: Hybrid architecture - Azure OpenAI for tasks without personal data (generating documentation, analysing publicly available data), self-hosting for processing customer data (complaints classification, correspondence analysis).

Public Sector

Organisations subject to national cybersecurity frameworks and government cloud policies:

• Azure is typically on approved cloud provider lists for EU government use
• Self-hosting on Azure infrastructure in an EU region satisfies location requirements
• Data classification requirement: “standard” data can go to Azure OpenAI (Data Zone EU), “sensitive” data requires self-hosting or on-premise processing

Deployment Scenarios - Three Reference Architectures

Scenario 1: Startup / SME (< 10K requests/day)

Recommendation: Azure OpenAI (Pay-As-You-Go)

┌──────────────┐     ┌──────────────────┐     ┌──────────────────┐
│ Application  │────▶│ Azure API Mgmt   │────▶│ Azure OpenAI     │
│ (App Service)│     │ (rate limiting)  │     │ GPT-4.1 mini     │
└──────────────┘     └──────────────────┘     │ Data Zone EU     │
                                               └──────────────────┘

• Model: GPT-4.1 mini (best quality-to-cost ratio for most tasks)
• Estimated cost: $50-200/month
• Time to deploy: 1-2 days (model deployment + API integration)
• Team required: 0 dedicated FTE (developer integrates API)

Scenario 2: Mid-size / Regulated Sector (10-100K requests/day)

Recommendation: Azure OpenAI with PTU + routing gateway

┌──────────────┐     ┌──────────────────┐     ┌───────────────────────┐
│ Applications │────▶│ AI Gateway       │──┬─▶│ Azure OpenAI PTU      │
│ (multiple    │     │ (routing,        │  │  │ GPT-4o (complex)      │
│  consumers)  │     │  caching,        │  │  └───────────────────────┘
└──────────────┘     │  rate limiting)  │  │  ┌───────────────────────┐
                      └──────────────────┘  └─▶│ Azure OpenAI PAYG     │
                                               │ GPT-4.1 nano (simple) │
                                               └───────────────────────┘

• Model: Mix - GPT-4o/4.1 for complex tasks, GPT-4.1 nano for classification/routing
• Estimated cost: $1,500-5,000/month
• AI Gateway: Azure API Management with LLM policies or open-source (LiteLLM, Portkey)
• Semantic caching: Reduces costs by 20-40% for repetitive queries

Scenario 3: Large Organisation / High Volume (>100K requests/day)

Recommendation: Hybrid architecture - self-hosting + Azure OpenAI as fallback

┌──────────────┐     ┌──────────────────┐     ┌───────────────────────┐
│ Applications │────▶│ AI Gateway       │──┬─▶│ Self-hosted (AKS)     │
│              │     │ (LiteLLM /       │  │  │ Llama 4 Scout 17B     │
└──────────────┘     │  custom router)  │  │  │ vLLM on 2×A100        │
                      └──────────────────┘  │  └───────────────────────┘
                                            │  ┌───────────────────────┐
                                            └─▶│ Azure OpenAI          │
                                               │ GPT-4o (fallback      │
                                               │ + complex tasks)      │
                                               └───────────────────────┘

• Primary: Self-hosted Llama 4 / Qwen 72B for 80% of traffic (simple tasks, sensitive data)
• Fallback: Azure OpenAI GPT-4o for complex tasks requiring deep reasoning
• Routing logic: Classifier (GPT-4.1 nano or rule-based) decides routing
• Estimated cost: $8,000-15,000/month (hybrid) vs. $15,000-30,000 (pure Azure OpenAI)
• Team required: 0.5-1 FTE MLOps/Platform Engineer

Direct Comparison - Decision Table

Migrating Between Approaches - Avoiding Vendor Lock-In

From Azure OpenAI to Self-Hosting

The typical path as volume grows:

1. Phase 1 (0-6 months): Azure OpenAI Pay-As-You-Go - validate use case, collect usage pattern data
2. Phase 2 (6-12 months): Azure OpenAI PTU + evaluate open-weight models on your data
3. Phase 3 (12+ months): Hybrid - self-hosted for 70-80% of traffic, Azure OpenAI for frontier reasoning

Key advice: From day one, use an API abstraction layer (LiteLLM, OpenRouter SDK, or a custom wrapper). The OpenAI Chat Completions format is the de facto standard - vLLM implements it natively. Migrating from Azure OpenAI to self-hosted vLLM is a single URL change in configuration.

API Abstraction - LiteLLM Example

# config.yaml - routing between Azure OpenAI and self-hosted vLLM
model_list:
  - model_name: "production-chat"
    litellm_params:
      model: "azure/gpt-4.1-mini"
      api_base: "https://my-resource.openai.azure.com/"
      api_key: "os.environ/AZURE_API_KEY"
    model_info:
      id: "azure-primary"

  - model_name: "production-chat"
    litellm_params:
      model: "openai/llama-4-scout"
      api_base: "http://vllm-inference.internal:8000/v1"
      api_key: "dummy"  # self-hosted, no auth needed internally
    model_info:
      id: "self-hosted-primary"

router_settings:
  routing_strategy: "cost-based"  # route to cheaper, fail over to more expensive
  num_retries: 2
  timeout: 30

This configuration allows you to:

• Route traffic to self-hosted as the default (cheaper)
• Automatically switch to Azure OpenAI when self-hosted is unavailable
• Change traffic proportions without application code changes

Optimising Azure OpenAI Costs - Practical Techniques

If you’ve chosen Azure OpenAI, these techniques reduce your bill by 30-60%:

1. Semantic Caching

Many queries in business applications are repetitive (FAQ, onboarding, standard procedures). Embedding-level cache with cosine similarity eliminates redundant calls:

# Azure Redis Enterprise with vector search
# or Azure AI Search as cache layer
SIMILARITY_THRESHOLD = 0.95  # > 95% similarity = cache hit

async def get_completion_with_cache(prompt: str):
    embedding = await get_embedding(prompt)
    cached = await redis.ft_search(embedding, threshold=SIMILARITY_THRESHOLD)
    if cached:
        return cached.response  # 0 tokens, 0 cost
    response = await azure_openai.chat.completions.create(...)
    await redis.set(embedding, response)
    return response

Typical result: 20-40% fewer API calls in applications with repetitive traffic.

2. Prompt Compression and Tiered Models

Not every query requires GPT-4o. A tiered model architecture:

• Tier 1 (GPT-4.1 nano, $0.10/M input): Intent classification, routing, simple extractions
• Tier 2 (GPT-4.1 mini, $0.40/M input): Response generation, summaries, translations
• Tier 3 (GPT-4o / o3, $2.50-10/M input): Complex reasoning, document analysis, planning

A Tier 1 router costs pennies but allows routing 60-70% of traffic to cheaper models.

3. Azure OpenAI Batch API

For non-interactive tasks (document processing, report generation, overnight analysis):

• 50% discount vs. standard pricing
• Results available within 24 hours
• Ideal for: document indexing, embedding generation, batch classification

4. Prompt Engineering for Cost

• Minimise system prompts (every token costs on every call)
• Use structured output (JSON mode) - model generates fewer unnecessary tokens
• Limit max_tokens to the actually needed response length
• Set temperature: 0 when creativity isn’t needed (faster generation)

Monitoring and FinOps for LLM Deployments

Regardless of the approach chosen, cost monitoring is critical:

Metrics to Track

Cost Dashboard - What It Should Contain

┌──────────────────────────────────────────────────────┐
│  LLM Cost Dashboard (Azure Workbook / Grafana)       │
│                                                      │
│  ┌─────────────┐  ┌─────────────┐  ┌──────────────┐ │
│  │ Daily cost  │  │ Tokens/day  │  │ Cost/1K req  │ │
│  │ $147 ▲12%   │  │ 45M ▲8%    │  │ $0.032 ▼3%   │ │
│  └─────────────┘  └─────────────┘  └──────────────┘ │
│                                                      │
│  ┌─────────────────────────────────────────────────┐ │
│  │ Cost breakdown by model (stacked area chart)    │ │
│  │ [GPT-4o: 45%] [GPT-4.1 mini: 35%] [nano: 20%] │ │
│  └─────────────────────────────────────────────────┘ │
│                                                      │
│  ┌─────────────────────────────────────────────────┐ │
│  │ Top consumers by application/team               │ │
│  │ 1. Customer Support Bot: $89/day                │ │
│  │ 2. Document Processing: $34/day                 │ │
│  │ 3. Internal Copilot: $24/day                    │ │
│  └─────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────┘

Summary - Decision Matrix

There’s no single right answer. The best organisations start with Azure OpenAI (fast time to business value), measure volume and usage patterns, and after 3-6 months make an informed decision about self-hosting where economics justify it.

The key: from day one, use an API abstraction layer (LiteLLM, standard OpenAI format). This way, migrating between approaches is a configuration change, not an application rewrite.

How We Can Help

At Devopsity we design LLM architectures on Azure, AWS, and GCP - from initial cost estimation to production deployment. Typical engagement:

• Assessment (1-2 days) - use case analysis, model selection, Azure OpenAI vs self-hosting cost calculation
• Proof of Concept (1-2 weeks) - pilot deployment with latency and cost measurement on real data
• Production (2-4 weeks) - full architecture: AI Gateway, monitoring, FinOps, compliance
• Optimisation (ongoing) - semantic caching, tiered models, migration to self-hosting when volume justifies it

Planning to deploy LLMs in production?

Book a free 30-minute technical consultation. We'll discuss your AI architecture and optimal cost model.

Book a call

Frequently Asked Questions

Does Azure OpenAI Service process my data in Europe?

Yes, when using Data Zone Standard or Regional Standard deployments. Data Zone EU guarantees that both inference (prompt processing) and data at-rest remain in European Azure regions (Sweden Central, France Central, Germany West Central). This satisfies GDPR Articles 44-49 without additional transfer safeguards.

How much does Azure OpenAI cost monthly for a typical company?

It depends on volume and model. Typical scenarios: a customer support chatbot (10K conversations/day, GPT-4.1 mini) runs approximately $300-500/month. A platform with bulk document processing (100K+ requests/day) costs $3,000-10,000/month. Self-hosting on 2× A100 GPUs costs $5,000-11,000/month including TCO.

When is self-hosting LLMs cheaper than Azure OpenAI?

The crossover point depends on model and volume. Roughly: above 100-200M tokens per day, self-hosting becomes cost-effective, provided you have an MLOps team (0.3-0.5 FTE) and accept the lack of managed service SLA. At lower volumes, Azure OpenAI wins thanks to its pay-per-use model.

Can I use Azure OpenAI in a regulated financial services environment?

Yes, subject to meeting cloud outsourcing requirements (regulatory notification, DPA, sub-processor chain documentation). We recommend a hybrid architecture: Azure OpenAI for tasks without personal data, self-hosting for processing customer data.

How quickly can I migrate from Azure OpenAI to self-hosting?

If you’ve been using an API abstraction layer (LiteLLM, OpenAI Chat Completions format) from the start - migration is a single URL change in configuration. vLLM implements the OpenAI-compatible API natively. The main cost is time to obtain GPU quota in Azure (1-4 weeks) and AKS cluster configuration.