Running LLM Inference on AWS - Bedrock vs SageMaker vs Self-Hosted on EKS

When to use each option - a decision framework

Before comparing pricing, architecture, and operational overhead, here is the fundamental question: what are you optimising for?

• Bedrock - you want zero infrastructure management, pay only for what you use, and need access to frontier models (Claude Sonnet 4.6/5, Opus 4.6/4.8, Llama 3, Mistral) without managing GPUs
• SageMaker - you need fine-tuned models, custom containers, or dedicated capacity with predictable latency guarantees
• Self-hosted on EKS - you require full control over the inference stack, run open-source models (Llama, Mixtral, Qwen), or need to minimise per-token costs at high volume

In practice, many teams use a combination: Bedrock for prototyping and low-volume production, SageMaker for fine-tuned models, and EKS for high-throughput batch inference. The architecture is not mutually exclusive.

AWS Bedrock - architecture, pricing, and limitations

How Bedrock works

Bedrock is a fully managed service. You send a request (prompt + parameters), AWS routes it to the model provider’s infrastructure, and you receive a response. There are no instances to provision, no GPUs to manage, and no cold starts for on-demand inference.

The request flow:

1. Your application calls the Bedrock InvokeModel or Converse API
2. AWS routes the request to the appropriate model endpoint (within the same region)
3. The model processes the prompt and returns tokens
4. You pay per input token processed and per output token generated

Bedrock pricing breakdown

Bedrock charges per 1,000 tokens (roughly 750 words). Pricing varies significantly by model:

Important for UK organisations: Models marked “EU Geo only” use Bedrock’s Geographic routing - your prompts and responses stay within EU boundaries but may be processed in any EU region (Frankfurt, Ireland, Paris, London). If your compliance requirement is strictly “data must not leave the UK” (rather than “data must stay in EU”), you are limited to models with in-region support. For most UK GDPR purposes, EU Geo routing is acceptable since it satisfies data adequacy requirements.

For Provisioned Throughput, Bedrock offers committed capacity billed per model unit per hour. This makes sense when you have predictable, high-volume workloads and need guaranteed throughput without throttling.

Example cost calculation: a customer support chatbot handling 10,000 conversations/day with an average of 500 input tokens and 200 output tokens per turn, using Claude 3.5 Haiku (available in-region in London):

• Input: 10,000 x 500 / 1,000 x $0.0008 = $4.00/day
• Output: 10,000 x 200 / 1,000 x $0.004 = $8.00/day
• Monthly total: approximately £290 (around $360)

That is remarkably cheap for a production AI system with no infrastructure to manage.

When Bedrock is the right choice

• Inference volume under 50,000 requests/day
• You need frontier model quality (Claude Sonnet 5, Opus 4.8, or Llama 4 via Geo routing) without managing hardware
• UK data residency requirements satisfied by eu-west-2 (London region) availability
• Rapid prototyping where time-to-production matters more than per-token cost
• Teams without ML infrastructure expertise

Limitations

• No fine-tuning for most models - Bedrock supports fine-tuning only for select models (Titan, some Llama variants). For custom models, SageMaker is required.
• Region availability - strict in-region availability in eu-west-2 (London) is limited to Claude Sonnet 4.6, Claude Opus 4.6, Claude 3.5 Haiku, and Llama 3 70B/8B Instruct. Newer models (Sonnet 5, Opus 4.8, Haiku 4.5, Llama 4) are available via EU Geographic routing - your data stays within EU boundaries but may be processed in Frankfurt, Ireland, or Paris rather than strictly in London. For organisations requiring strict single-region processing in the UK, this distinction matters. Always verify current status in the Bedrock region compatibility page.
• Throughput limits - default quotas can throttle high-volume production workloads. Request quota increases proactively.
• No streaming control - limited ability to tune batch sizes, quantisation, or inference parameters beyond what the API exposes.

Official Bedrock pricing documentation: aws.amazon.com/bedrock/pricing

SageMaker Endpoints - architecture, pricing, and when to choose

Inference endpoint types

SageMaker offers three inference patterns:

1. Real-time endpoints - always-on instances serving synchronous requests. Best for low-latency production APIs.
2. Serverless inference - automatically scales to zero, provisions capacity on demand. Good for sporadic traffic but with cold start penalty (10-60 seconds).
3. Asynchronous inference - queues requests and processes them in the background. Ideal for batch jobs and long-running generation tasks.

Pricing model

SageMaker charges per instance-hour for real-time endpoints. You pay for the instance whether it is processing requests or idle.

Example cost calculation: running a fine-tuned Llama 3 8B on ml.g5.xlarge 24/7:

• $1.41/h x 720 hours = $1,015/month
• Monthly total: approximately £810 (around $1,015)

Compare this to Bedrock: if you are processing more than ~250,000 requests/month with average 500-token prompts, SageMaker becomes cheaper than Bedrock on a per-request basis.

When SageMaker wins

• You have fine-tuned a model and need to serve it
• Predictable, sustained traffic (the instance runs efficiently at 60-80% utilisation)
• You need control over batch size, quantisation (GPTQ, AWQ), and serving framework
• Latency SLAs require dedicated capacity without cold starts
• Compliance requires running in a VPC with no shared infrastructure

Autoscaling configuration

SageMaker supports target-tracking autoscaling. A practical configuration:

# SageMaker endpoint autoscaling based on invocations per instance
import boto3

client = boto3.client("application-autoscaling")

# Register the endpoint variant as a scalable target
client.register_scalable_target(
    ServiceNamespace="sagemaker",
    ResourceId="endpoint/llm-production/variant/AllTraffic",
    ScalableDimension="sagemaker:variant:DesiredInstanceCount",
    MinCapacity=1,
    MaxCapacity=8,
)

# Scale based on invocations per instance (target: 100 concurrent)
client.put_scaling_policy(
    PolicyName="llm-invocations-scaling",
    ServiceNamespace="sagemaker",
    ResourceId="endpoint/llm-production/variant/AllTraffic",
    ScalableDimension="sagemaker:variant:DesiredInstanceCount",
    PolicyType="TargetTrackingScaling",
    TargetTrackingScalingPolicyConfiguration={
        "TargetValue": 100.0,
        "PredefinedMetricSpecification": {
            "PredefinedMetricType": "SageMakerVariantInvocationsPerInstance"
        },
        "ScaleOutCooldown": 60,   # seconds before scaling out again
        "ScaleInCooldown": 300,   # seconds before scaling in (conservative)
    },
)

Official SageMaker pricing documentation: aws.amazon.com/sagemaker/pricing

Self-Hosted on EKS - architecture for teams that need full control

Why self-host?

Self-hosting LLM inference on EKS makes sense when:

• You run open-source models (Llama 4, Mixtral, Qwen, Phi) and want to eliminate per-token API costs entirely
• Inference volume is high enough (>100K requests/day) that compute cost per token drops below managed service pricing
• You need custom serving frameworks (vLLM, Text Generation Inference, Triton)
• Multi-model serving on shared GPU infrastructure reduces overall cost
• You already operate EKS and have GPU scheduling expertise

Architecture components

A production self-hosted LLM stack on EKS typically includes:

• GPU node pools - managed node groups or Karpenter-provisioned nodes with NVIDIA GPUs
• Inference server - vLLM, Hugging Face TGI, or NVIDIA Triton serving the model
• Autoscaler - Karpenter for node-level scaling, HPA/KEDA for pod-level scaling
• Load balancer - ALB/NLB with gRPC or HTTP/2 support
• Model storage - S3 + FSx for Lustre or EBS for fast model loading
• Monitoring - Prometheus + Grafana tracking GPU utilisation, queue depth, latency percentiles

GPU instance costs on EC2 (for EKS nodes)

The cost advantage of self-hosting comes from two factors: (1) GPU spot instances at 60-70% discount, and (2) high utilisation through request batching and multi-model serving.

Karpenter NodePool for GPU spot instances

# Karpenter NodePool for LLM inference GPU nodes
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: llm-inference-gpu
spec:
  template:
    metadata:
      labels:
        workload-type: llm-inference
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: gpu-nodes
      requirements:
        - key: "karpenter.k8s.aws/instance-family"
          operator: In
          values: ["g5", "g6"]
        - key: "karpenter.sh/capacity-type"
          operator: In
          values: ["spot", "on-demand"]  # prefer spot, fall back to on-demand
        - key: "kubernetes.io/arch"
          operator: In
          values: ["amd64"]
      taints:
        - key: nvidia.com/gpu
          effect: NoSchedule    # only GPU workloads land here
  limits:
    cpu: "128"
    memory: "512Gi"
    nvidia.com/gpu: "16"        # max 16 GPUs across all nodes
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 5m        # reclaim idle GPU nodes quickly
  weight: 10                    # prefer this pool for GPU pods

vLLM deployment on EKS

# Kubernetes Deployment for vLLM serving Llama 4 Scout 17B
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-llama-8b
  namespace: inference
spec:
  replicas: 2
  selector:
    matchLabels:
      app: vllm-llama-8b
  template:
    metadata:
      labels:
        app: vllm-llama-8b
    spec:
      nodeSelector:
        workload-type: llm-inference
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
      containers:
        - name: vllm
          image: vllm/vllm-openai:latest
          args:
            - "--model"
            - "meta-llama/Llama-4-Scout-17B-Instruct"
            - "--tensor-parallel-size"
            - "1"
            - "--max-model-len"
            - "8192"
            - "--gpu-memory-utilization"
            - "0.9"
          ports:
            - containerPort: 8000
              name: http
          resources:
            limits:
              nvidia.com/gpu: "1"
              memory: "32Gi"
            requests:
              nvidia.com/gpu: "1"
              memory: "24Gi"
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 120   # model loading takes time
            periodSeconds: 10
          env:
            - name: HUGGING_FACE_HUB_TOKEN
              valueFrom:
                secretKeyRef:
                  name: hf-token
                  key: token

Cost model for self-hosted inference

Example: serving Llama 4 Scout 17B on 2x g5.xlarge spot instances (24/7):

• 2 x $0.40/h x 720 hours = $576/month
• Add EKS cluster fee: $73/month
• Add ALB: ~$25/month
• Monthly total: approximately £540 (around $674)

At a typical throughput of 30 tokens/second per GPU, this handles roughly 50,000-80,000 requests/day with average 200-token outputs. The per-request cost is effectively £0.000007 - orders of magnitude cheaper than Bedrock at this scale.

Cost comparison across scenarios

The crossover point where self-hosting becomes cheaper than Bedrock is typically around 10,000-20,000 requests/day, depending on the model and token length. Below that, Bedrock’s zero-infrastructure overhead is hard to beat.

Latency and performance comparison

Bedrock’s latency is acceptable for most interactive applications. For latency-critical paths (real-time trading signals, sub-100ms requirements), self-hosted inference with warm GPU pools is the only viable option.

Decision matrix

Deploying Bedrock in eu-west-2 for UK organisations

For organisations subject to UK GDPR or FCA oversight, data residency is a primary concern. AWS Bedrock offers two routing options relevant to UK deployments:

• In-Region (strict London): Your requests stay entirely within eu-west-2. Available models: Claude Sonnet 4.6, Claude Opus 4.6, Claude 3.5 Haiku, Llama 3 70B/8B, Mistral Large, Amazon Nova. Use when regulations require strict single-region UK processing.
• EU Geographic routing: Requests are processed within EU boundaries (London, Frankfurt, Ireland, Paris) for higher throughput. Available models: Claude Sonnet 5, Opus 4.8, Haiku 4.5, and other latest releases. Acceptable for most UK GDPR purposes since EU adequacy decisions apply.

Key considerations:

• Model availability vs compliance trade-off - the newest, most capable models (Sonnet 5, Opus 4.8) require EU Geo routing. If you strictly need in-region London, Claude Sonnet 4.6 is your best option - same pricing tier, slightly older model.
• VPC endpoints - deploy a Bedrock VPC endpoint to ensure API calls never traverse the public internet
• CloudTrail logging - all Bedrock API calls are logged, supporting audit requirements
• No data retention - Bedrock does not store or train on your prompts/responses (unless you explicitly opt into model evaluation)

Scaleups across Manchester’s growing AI corridor and Edinburgh’s fintech ecosystem face this architecture decision as inference volumes grow beyond prototype stage. The infrastructure choice made at 1,000 requests/day rarely survives 100,000 - planning the migration path early saves significant rework.

Frequently asked questions

Is Bedrock cheaper than running my own GPUs?

At low-to-medium volumes (under ~20,000 requests/day with average-length prompts), yes. Bedrock eliminates the baseline cost of keeping GPU instances running 24/7. Above that threshold, self-hosted inference on spot instances becomes significantly cheaper per token.

Can I use Bedrock and self-hosted together?

Yes - this is a common pattern. Use Bedrock for low-volume, high-quality tasks (complex reasoning, customer-facing chat) and self-hosted inference for high-volume, cost-sensitive tasks (classification, extraction, embeddings). Route requests based on task complexity.

What about SageMaker Serverless Inference for LLMs?

SageMaker Serverless Inference has cold starts of 10-60 seconds for GPU workloads, making it impractical for interactive LLM serving. It works for batch processing or very infrequent inference, but real-time endpoints or Bedrock are better for production chat/API scenarios.

How do I handle GPU spot interruptions on EKS?

Karpenter automatically replaces interrupted spot instances. For inference (stateless), this means a brief capacity dip (30-120 seconds) while a new node provisions. Mitigation strategies: run across multiple instance types and AZs, maintain a small on-demand baseline (1-2 nodes), and use pod disruption budgets to ensure minimum replica count.

What GPU should I start with for a 7B model?

A single g5.xlarge (A10G, 24GB VRAM) comfortably runs Llama 4 Scout 17B or Mistral 7B with AWQ/GPTQ quantisation. For larger models, you need g5.12xlarge (4x A10G, 96GB total) with tensor parallelism, or a single p4d instance. Start small, measure throughput, and scale horizontally.

Summary

• Bedrock is the right default for teams starting with LLM inference. Zero operational overhead, pay-per-token, and available in eu-west-2 for UK data residency. Best for volumes under 50K requests/day.
• SageMaker fills the gap for fine-tuned models and predictable workloads where you need dedicated capacity without managing Kubernetes. Cost-effective at medium volumes with autoscaling.
• Self-hosted on EKS delivers the lowest per-token cost at scale, but requires GPU scheduling expertise, Karpenter configuration, and operational maturity. The 60-80% cost reduction over managed services justifies the investment above 100K requests/day.
• The crossover point between Bedrock and self-hosted is typically £1,000-£1,500/month in Bedrock spend. Below that, the operational simplicity of Bedrock wins. Above it, start planning your EKS GPU infrastructure.
• For retrieval-augmented generation workloads where inference is just one component, see our guide to RAG infrastructure on AWS.

Need help designing your LLM inference architecture or optimising costs for an existing deployment? The Devopsity engineering team helps organisations across the UK move from prototype to production AI infrastructure. Get in touch to discuss your specific requirements.