Agentic AI Infrastructure - GPU Scheduling, Autoscaling, and Cost Control on Kubernetes

In 2026, “AI in production” is no longer a single inference endpoint. Teams are building agentic systems - LangChain, CrewAI, AutoGen, custom frameworks - that coordinate multiple LLM calls, use tools (web search, vector databases, APIs), and autonomously execute complex tasks. Infrastructure for these systems is a fundamentally different challenge from standard model deployment.

The problem is that GPUs are expensive. A p5.48xlarge instance (8x H100) costs ~$98/hour on AWS. Poorly planned autoscaling means either unavailability (agents waiting for GPU) or astronomical bills (GPU sitting idle between tasks). This article shows how to find the balance.

Edinburgh’s School of Informatics - the UK’s longest-established AI research centre - has been exploring multi-agent coordination since the 1990s. The infrastructure challenges they identified then are now production engineering problems for every organisation deploying agentic AI at scale.

Agentic AI Architecture on Kubernetes - Layer Separation

An agentic system consists of three logical layers, each with different infrastructure requirements:

Key insight: The orchestration layer is cheap (CPU) but long-running. The inference layer is expensive (GPU) but short-lived. Optimal design separates them into independent node pools with independent scaling.

GPU Scheduling on Kubernetes - How It Works

GPU Resources in Kubernetes

GPU in Kubernetes is an extended resource. A pod requests GPU via resources.limits:

apiVersion: v1
kind: Pod
spec:
  containers:
  - name: vllm-inference
    image: vllm/vllm-openai:latest
    resources:
      limits:
        nvidia.com/gpu: 1
      requests:
        nvidia.com/gpu: 1
        memory: "32Gi"
        cpu: "4"

Key scheduling rules:

• GPU is not shared between pods (by default) - one pod = one GPU (or more)
• GPU cannot be “partially” allocated (without MIG or time-slicing)
• If no GPU is available on any node - the pod waits in Pending state

NVIDIA Device Plugin and GPU Operator

On EKS, GKE, and AKS you need:

1. NVIDIA Device Plugin - exposes GPU as a Kubernetes resource
2. NVIDIA GPU Operator (optional) - automatically installs drivers and plugin
3. Node labels - GPU type identification on nodes (nvidia.com/gpu.product=NVIDIA-A100-SXM4-80GB)

On EKS with Karpenter (recommended approach):

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: gpu-inference
spec:
  template:
    spec:
      requirements:
        - key: node.kubernetes.io/instance-type
          operator: In
          values: ["g5.xlarge", "g5.2xlarge", "g5.4xlarge"]
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: gpu-nodes
  limits:
    nvidia.com/gpu: 8
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 5m

Multi-Instance GPU (MIG) - Sharing a Single GPU

NVIDIA A100 and H100 support MIG - partitioning one physical GPU into smaller instances. This lets you run several smaller models on one GPU instead of reserving the entire chip for a single pod.

When to use MIG:

• Multiple small models (embedding models, classifiers) on one node
• Dev/test environments where a full GPU is wasteful
• Agents with multiple specialised models (one for classification, another for generation)

When NOT to use MIG:

• Large models (70B+ parameters) that need full VRAM
• Workload requiring full GPU memory bandwidth
• Production inference with low-latency requirements

GPU Time-Slicing - Alternative to MIG

Time-slicing allows multiple pods to share one GPU via time rotation. Simpler than MIG (does not require specific hardware), but with worse isolation:

# NVIDIA Device Plugin ConfigMap - time-slicing
apiVersion: v1
kind: ConfigMap
metadata:
  name: nvidia-device-plugin
data:
  config: |
    version: v1
    sharing:
      timeSlicing:
        resources:
          - name: nvidia.com/gpu
            replicas: 4

This creates 4 “virtual” GPUs from one physical GPU. Each pod gets 1/4 of GPU time. Good for dev/test and lightweight models, but not for production inference sensitive to latency.

Autoscaling for Agentic AI

The Problem: Standard HPA Does Not Work for GPU

Horizontal Pod Autoscaler (HPA) relies on CPU/RAM metrics. For GPU environments this is insufficient:

• GPU utilisation is not exported to the metrics server by default
• Inference response time is a better metric than GPU utilisation
• Agents generate bursty traffic - 0 requests for a minute, then 50 at once

KEDA - Event-Driven Scaling

KEDA (Kubernetes Event-Driven Autoscaling) handles agentic environments better:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: inference-scaler
spec:
  scaleTargetRef:
    name: vllm-deployment
  minReplicaCount: 1
  maxReplicaCount: 4
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus:9090
      metricName: vllm_pending_requests
      query: sum(vllm:num_requests_waiting)
      threshold: "10"
  - type: prometheus
    metadata:
      metricName: vllm_gpu_cache_usage
      query: avg(vllm:gpu_cache_usage_perc)
      threshold: "80"

Metrics for scaling inference:

• vllm:num_requests_waiting - queue length (best metric for scaling up)
• vllm:gpu_cache_usage_perc - KV-cache utilisation (signal that the model is under pressure)
• vllm:avg_generation_throughput_toks_per_s - throughput (signal that scaling down is safe)

Karpenter - Node Pool Scaling for GPU

Karpenter (AWS) or Node Auto-Provisioner (GKE) dynamically adds and removes GPU nodes:

Why Karpenter over Cluster Autoscaler:

• Karpenter makes decisions in seconds (CA: minutes)
• Karpenter selects the optimal instance type from a list (no predefined node groups required)
• Karpenter natively supports Spot with on-demand fallback
• Karpenter consolidates - if two nodes are half-empty, it migrates pods and shuts one down

Key setting: consolidateAfter: 5m means a GPU node will be shut down after 5 minutes of idleness. For agentic environments with bursts every few minutes, consider consolidateAfter: 15m to avoid on/off cycling.

Scale-to-Zero for GPU

The cheapest GPU is a switched-off GPU. For environments that do not require instant response:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
spec:
  minReplicaCount: 0  # scale to zero!
  cooldownPeriod: 300  # 5 minutes after last request
  triggers:
  - type: prometheus
    metadata:
      metricName: inference_requests_total
      query: sum(rate(inference_requests_total[2m]))
      threshold: "1"

Cold start problem: Starting a pod with an LLM model takes 30-120 seconds (loading weights into VRAM). For agents that call inference sporadically, this is an acceptable trade-off. For production APIs requiring <1s response - at least 1 replica must always be active.

GPU Cost Control on Kubernetes

What GPUs Cost in the Cloud

At 730h/month, one g5.xlarge on-demand instance costs ~$737/month. On Spot: ~$255-365/month. The difference is substantial.

For UK organisations running in eu-west-2 (London), pricing is typically 5-10% higher than us-east-1 but eliminates cross-Atlantic latency for UK-based agent users and simplifies data residency under UK GDPR.

Savings Strategies

1. Spot Instances with graceful shutdown

For inference that tolerates 30-second interruptions:

• Karpenter with capacity-type: ["spot", "on-demand"] - prefers Spot, falls back to on-demand
• vLLM/TGI support graceful shutdown - complete current request before termination
• Agent frameworks (LangChain, CrewAI) with built-in retry - if inference returns an error, they retry

2. Right-sizing models

Not every agent needs a GPT-4 class model. Hierarchy:

• Classification/routing: small model (Phi-3-mini, 3.8B) - fits on T4/L4
• Simple generation: medium model (Llama 3.1 8B) - one A10G
• Complex reasoning: large model (Llama 3.1 70B) - requires A100 80GB or multi-GPU

Scaleups graduating from Edinburgh’s Bayes Centre AI Accelerator face exactly this sizing decision as they move from prototype to production. Starting with the smallest model that meets quality requirements, then scaling up only where needed, typically cuts GPU costs by 60-70%.

3. Request batching

vLLM with continuous batching groups requests from multiple agents:

• One vLLM pod handles 10-50 concurrent requests (depending on context length)
• Better GPU utilisation than one request at a time
• Configure --max-num-seqs 32 in vLLM for optimal throughput

4. KV-cache offloading

For agents with long contexts (RAG with large documents):

• Prefix caching in vLLM - repeated queries with the same system prompt do not recompute KV-cache
• Quantised KV-cache (FP8) - 2x less VRAM for cache, minimal quality loss

5. GPU work schedules

If agents primarily work during business hours (9-18):

• Scale-to-zero overnight and weekends
• CronJob with KEDA to spin up pods in the morning
• Savings: 60-70% of GPU costs (10h/day instead of 24h)

Observability for GPU Workload

Metrics You Must Collect

Standard Prometheus + DCGM Exporter (NVIDIA):

• DCGM_FI_DEV_GPU_UTIL - GPU core utilisation (%)
• DCGM_FI_DEV_MEM_COPY_UTIL - GPU memory utilisation (%)
• DCGM_FI_DEV_GPU_TEMP - temperature (throttling >80C)
• DCGM_FI_PROF_PIPE_TENSOR_ACTIVE - Tensor Core utilisation (for LLM inference)

At vLLM/TGI level:

• vllm:num_requests_running - active requests
• vllm:num_requests_waiting - queue (critical metric)
• vllm:gpu_cache_usage_perc - % of KV-cache occupied
• vllm:avg_prompt_throughput_toks_per_s - input token throughput

Alerts (minimum)

• GPU utilisation <10% for 15 minutes - likely waste
• KV-cache >90% - model begins rejecting requests or returning OOM
• Queue >20 requests for 2 minutes - scaling needed
• GPU temperature >85C - throttling risk

Reference Pattern - EKS Cluster for an Agentic System

Architecture for a typical agentic system (e.g. autonomous agent for financial document analysis):

Node pools:

1. System pool (2x m5.large) - CoreDNS, kube-system, monitoring
2. Agent pool (Karpenter, c5.xlarge-c5.4xlarge, Spot) - agent orchestration, tools, vector databases
3. Inference pool (Karpenter, g5.xlarge-g5.2xlarge, Spot with fallback) - vLLM pods with models

Separation:

• Agents (CPU) scale independently from inference (GPU)
• Inference pods have dedicated node pools (do not share nodes with CPU workload)
• Scale-to-zero on inference outside working hours

Estimated cost (8h/day usage, 5 days/week):

• System + Agent pools: ~$200-400/month
• Inference pool (2x g5.xlarge Spot, 8h/day): ~$150-250/month
• Total: ~$350-650/month (vs ~$1,500-2,000 if GPU ran 24/7)

For fintech teams in Leeds and Manchester running regulatory document analysis agents, this pattern delivers the cost predictability that finance directors expect while maintaining the burst capacity that AI workloads demand.

Common Mistakes

1. GPU nodes without taints - ordinary CPU pods land on expensive GPU nodes. Always set taint nvidia.com/gpu=true:NoSchedule on GPU node pools.

2. Missing resource requests on inference pods - the scheduler does not know how much VRAM the model needs. Result: two large models on one GPU = OOM kill. We covered the detailed fix in our vLLM GPU OOM on EKS runbook.

3. Autoscaler targeting GPU utilisation - GPU utilisation at 50% does not mean there is free capacity. KV-cache may be full. Scale based on application metrics (request queue depth).

4. One large model instead of many small ones - a 70B model on 4x A100 costs 10x more than proper routing: classifier on T4 + small model on A10G + large model only when it is genuinely needed.

5. No per-agent budget - without limits, one agent in a loop can generate 100,000 tokens per minute. Implementing rate-limiting at the orchestration layer is essential.

How We Can Help

At Devopsity, we design and deploy Kubernetes infrastructure for AI systems - from single inference endpoints to full agentic platforms. A typical engagement:

• GPU cluster architecture design (node pools, autoscaling, networking) - 2-3 days
• vLLM/TGI deployment with KEDA autoscaling + Karpenter - 3-5 days
• GPU observability and alerting configuration - 1-2 days
• Cost optimisation (Spot, scale-to-zero, model right-sizing) - 2-3 days

If your team is building an agentic system and facing the decision: managed API (Bedrock, OpenAI) vs self-hosted - our comparison of these approaches is here.

Building infrastructure for AI?

Book a free 30-minute call. No pitch - a technical conversation about your architecture.

Book a call

Frequently Asked Questions

Do I need GPU to run AI agents?

Not necessarily. Agent orchestration (LangChain, CrewAI) runs on CPU. You only need GPU for LLM inference. If you use APIs (OpenAI, Bedrock, Vertex AI) - you do not need your own GPUs. Self-hosted models on GPU make sense at high volume (>1M tokens/day) or when data privacy requirements mandate it.

How many GPUs do I need for an agentic system?

Depends on the model and load. One vLLM pod on an A10G (24 GB VRAM) handles a 7-8B parameter model with throughput of ~2000 tokens/s. For 10 concurrent agents with short context - one GPU is sufficient. For 100 agents with long context (RAG) - you need 3-5 GPUs or larger instances.

Spot Instances for inference - is it safe?

Yes, if you have fallback. Karpenter with capacity-type: ["spot", "on-demand"] automatically provisions on-demand nodes when Spot is unavailable. vLLM supports graceful shutdown - completes the current request before termination. Agents with retry logic survive a 30-second interruption.

Karpenter or Cluster Autoscaler for GPU?

Karpenter. Cluster Autoscaler requires predefined node groups and scales in minutes. Karpenter makes decisions in seconds, selects the optimal instance type from a pool, and natively supports consolidation (shutting down empty nodes). For GPU, where idle cost is high - this difference is critical.

How much does agentic AI infrastructure cost?

Minimal MVP (1x g5.xlarge Spot, 8h/day): ~$150/month. Production solution (2-4 GPUs, 24/7, redundancy): ~$1,500-3,000/month. For comparison: Bedrock/OpenAI API at 5M tokens/day costs ~$4,000-8,000/month. Self-hosting break-even threshold: typically >2M tokens/day.