vLLM Inference - Performance Degradation from KV Cache Fragmentation

This runbook covers diagnosing and fixing KV cache fragmentation in vLLM. We discuss GPU scheduling and AI model autoscaling in detail in our article Agentic AI infrastructure - GPU scheduling and autoscaling on Kubernetes. For a comparison of inference costs on self-hosted infrastructure vs managed APIs, see Azure OpenAI vs self-hosted models - cost, latency, and data residency in the EU. If you need help optimising GPU inference - book a consultation.

Symptom

Throughput (tok/s) measured at the vLLM /metrics endpoint drops gradually over time. Prometheus shows a characteristic pattern: stable performance for the first hour, then a linear decline:

# Sprawdź aktualny throughput z metryki vLLM
kubectl exec -n inference deploy/vllm-server -- \
  curl -s localhost:8000/metrics | grep -E "vllm:generation_tokens_total|vllm:prompt_tokens_total"

# Metryki KV cache - kluczowe dla diagnozy fragmentacji
kubectl exec -n inference deploy/vllm-server -- \
  curl -s localhost:8000/metrics | grep -E "vllm:gpu_cache_usage_perc|vllm:num_preemptions_total"

# Sprawdź GPU memory utilization (nvidia-smi)
kubectl exec -n inference deploy/vllm-server -- nvidia-smi --query-gpu=memory.used,memory.total,utilization.gpu --format=csv,noheader

# Porównanie TTFT (Time To First Token) - świeży start vs po 8h
kubectl exec -n inference deploy/vllm-server -- \
  curl -s localhost:8000/metrics | grep "vllm:time_to_first_token"

Prometheus query to detect degradation over time:

# Rate generowania tokenów - porównanie last 1h vs 8h ago
rate(vllm:generation_tokens_total[5m])

# Spadek throughput > 30% vs baseline (alert rule)
(
  avg_over_time(rate(vllm:generation_tokens_total[5m])[1h:5m])
  /
  avg_over_time(rate(vllm:generation_tokens_total[5m])[1h:5m] offset 8h)
) < 0.7

# KV cache utilization - jeśli oscyluje blisko 100% to sygnał fragmentacji
vllm:gpu_cache_usage_perc > 0.95

# Liczba preemptions - rosnąca liczba oznacza brak ciągłych bloków
rate(vllm:num_preemptions_total[5m]) > 0

How to distinguish fragmentation from other causes of throughput degradation:

Root Cause

The problem stems from the PagedAttention mechanism in vLLM and how KV cache memory blocks are allocated:

1. Block fragmentation with variable-length prompts:

vLLM divides GPU memory into fixed-size blocks (default 16 tokens/block). When short requests (100 tokens = 7 blocks) and long requests (32K tokens = 2048 blocks) arrive alternately, after short requests finish, “holes” appear - free blocks scattered between occupied blocks of long requests. New long requests cannot allocate a contiguous sequence of blocks and must wait for preemption.

2. gpu_memory_utilization set too high (>0.92):

With the default setting --gpu-memory-utilization 0.95, vLLM allocates 95% of GPU memory to KV cache. No headroom remains for temporary allocations during block reorganisation. The result: the scheduler increasingly has to preempt (interrupt) active requests to reclaim blocks.

# Typowa konfiguracja powodująca problem
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Meta-Llama-3.1-70B-Instruct \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.95 \     # ZA WYSOKO - brak headroom
  --max-model-len 32768 \
  --disable-log-requests

3. No prefix caching = repeated allocation of identical system prompts:

Every request allocates new blocks for the system prompt, even if it is identical to the previous one. With a 2000-token system prompt and 100 req/s, this means 100x allocation and deallocation of the same blocks every minute - massive fragmentation.

4. Long requests “pin” blocks:

A request with a 32K-token context window occupies ~2048 KV cache blocks for the entire duration of generation (which can take 30-60s). Short requests finishing in the meantime release their blocks, but these free spaces are too small for subsequent long requests. The effect accumulates over hours.

5. No compaction mechanism at runtime:

In vLLM versions <0.8 there is no memory defragmentation mechanism during operation. The only way to “reset” the block layout is to restart the process.

Resolution

A) Enable prefix caching (--enable-prefix-caching):

Prefix caching allows sharing KV cache blocks between requests with the same prefix (system prompt, few-shot examples). This drastically reduces the number of allocations/deallocations:

# Dodaj flagę do komendy startowej vLLM
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Meta-Llama-3.1-70B-Instruct \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.88 \
  --max-model-len 32768 \
  --enable-prefix-caching \
  --disable-log-requests

In Kubernetes Deployment:

# deployment-vllm.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-server
  namespace: inference
spec:
  replicas: 2
  selector:
    matchLabels:
      app: vllm-server
  template:
    metadata:
      labels:
        app: vllm-server
    spec:
      containers:
        - name: vllm
          image: vllm/vllm-openai:v0.8.3
          args:
            - "--model"
            - "meta-llama/Meta-Llama-3.1-70B-Instruct"
            - "--tensor-parallel-size"
            - "4"
            - "--gpu-memory-utilization"
            - "0.88"
            - "--max-model-len"
            - "32768"
            - "--enable-prefix-caching"
            - "--disable-log-requests"
          resources:
            limits:
              nvidia.com/gpu: 4
          ports:
            - containerPort: 8000

B) Lower --gpu-memory-utilization to 0.85-0.88:

Leave 12-15% of GPU memory as headroom. This allows the scheduler to reorganise blocks more effectively without preemption:

# Oblicz optymalną wartość:
# GPU VRAM: 80 GB (A100)
# Model weights: ~35 GB (Llama 3.1 70B w FP16, TP=4 → ~35GB/GPU)
# KV cache headroom: min 10% VRAM = 8 GB wolnego
# Optymalne: (80 - 35 - 8) / 80 ≈ 0.46 → ale vLLM liczy to inaczej
# Praktycznie: 0.85-0.88 daje najlepszy balans throughput vs fragmentacja

# Zweryfikuj ile pamięci faktycznie używa KV cache
kubectl exec -n inference deploy/vllm-server -- \
  curl -s localhost:8000/metrics | grep "vllm:gpu_cache_usage_perc"
# Powinno oscylować między 0.70-0.85, nie powyżej 0.92

C) Scheduled rolling restart (CronJob + PodDisruptionBudget):

Schedule restarts every 6 hours while maintaining availability:

# pdb-vllm.yaml - gwarantuj min 1 pod dostępny
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: vllm-pdb
  namespace: inference
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: vllm-server

---
# cronjob-restart.yaml - rolling restart co 6h
apiVersion: batch/v1
kind: CronJob
metadata:
  name: vllm-rolling-restart
  namespace: inference
spec:
  schedule: "0 */6 * * *"  # co 6 godzin
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: vllm-restart-sa
          containers:
            - name: restart
              image: bitnami/kubectl:1.30
              command:
                - /bin/sh
                - -c
                - |
                  echo "Starting rolling restart of vllm-server..."
                  kubectl rollout restart deployment/vllm-server -n inference
                  kubectl rollout status deployment/vllm-server -n inference --timeout=600s
                  echo "Rolling restart completed."
          restartPolicy: OnFailure

---
# rbac for restart SA
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: vllm-restart-role
  namespace: inference
rules:
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "patch"]
  - apiGroups: ["apps"]
    resources: ["deployments/status"]
    verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: vllm-restart-binding
  namespace: inference
subjects:
  - kind: ServiceAccount
    name: vllm-restart-sa
roleRef:
  kind: Role
  name: vllm-restart-role
  apiGroup: rbac.authorization.k8s.io

D) Routing by prompt length - separate node pools:

Split traffic into two deployments: short context (<4K tokens) and long context (>4K tokens). This eliminates alternating block allocations of vastly different sizes:

# Istio VirtualService - routing po headerze x-prompt-length
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: vllm-routing
  namespace: inference
spec:
  hosts:
    - vllm.inference.svc.cluster.local
  http:
    - match:
        - headers:
            x-prompt-tokens:
              regex: "^[0-3][0-9]{0,3}$"  # 0-3999 tokenów
      route:
        - destination:
            host: vllm-short-context
            port:
              number: 8000
    - route:
        - destination:
            host: vllm-long-context
            port:
              number: 8000

Alternatively - a simple Python proxy that classifies requests before sending them to the appropriate server:

# router.py - przykład prostego token-length routera
import tiktoken
from fastapi import FastAPI, Request
import httpx

app = FastAPI()
enc = tiktoken.get_encoding("cl100k_base")

SHORT_BACKEND = "http://vllm-short-context:8000"
LONG_BACKEND = "http://vllm-long-context:8000"
THRESHOLD_TOKENS = 4096

@app.post("/v1/chat/completions")
async def route_request(request: Request):
    body = await request.json()
    messages = body.get("messages", [])
    total_tokens = sum(len(enc.encode(m.get("content", ""))) for m in messages)

    backend = SHORT_BACKEND if total_tokens < THRESHOLD_TOKENS else LONG_BACKEND

    async with httpx.AsyncClient(timeout=120) as client:
        resp = await client.post(f"{backend}/v1/chat/completions", json=body)
        return resp.json()

E) Upgrade to vLLM 0.8+ with improved memory management:

vLLM 0.8 introduces an improved block allocator with partial online defragmentation. Upgrading significantly alleviates the problem:

# Upgrade obrazu w deployment
kubectl set image deployment/vllm-server -n inference \
  vllm=vllm/vllm-openai:v0.8.3

# Zweryfikuj wersję po starcie
kubectl exec -n inference deploy/vllm-server -- python -c "import vllm; print(vllm.__version__)"

Validation

After deploying the solutions, verify throughput stability for a minimum of 24 hours:

# Prometheus - throughput powinien być stabilny (±10%) przez 24h
# Alert rule: spadek >20% vs baseline
- alert: VLLMThroughputDegradation
  expr: |
    (
      avg_over_time(rate(vllm:generation_tokens_total[5m])[30m:5m])
      /
      avg_over_time(rate(vllm:generation_tokens_total[5m])[30m:5m] offset 6h)
    ) < 0.8
  for: 15m
  labels:
    severity: warning
    team: ml-platform
  annotations:
    summary: "vLLM throughput spadł >20% vs 6h temu"
    description: "Pod  generuje  bazowego throughput. Możliwa fragmentacja KV cache."

# Porównanie before/after - wykonaj po 24h od wdrożenia
# Before: metryki z dnia przed fix (z Prometheus/Grafana)
# After: bieżące metryki

# Sprawdź brak preemptions (kluczowy wskaźnik)
kubectl exec -n inference deploy/vllm-server -- \
  curl -s localhost:8000/metrics | grep "vllm:num_preemptions_total"
# Expected: wartość nie rośnie lub rośnie bardzo wolno (<1/min)

# Sprawdź cache hit ratio (po włączeniu prefix caching)
kubectl exec -n inference deploy/vllm-server -- \
  curl -s localhost:8000/metrics | grep "vllm:prefix_cache_hit"
# Expected: hit ratio > 60% dla workloadów z powtarzalnym system promptem

# Sprawdź stabilność TTFT w czasie
kubectl exec -n inference deploy/vllm-server -- \
  curl -s localhost:8000/metrics | grep "vllm:time_to_first_token"
# p50 powinno być stabilne (nie rosnąć w czasie)

Grafana dashboard query - visualising stability after the fix:

# Panel 1: Throughput (tokens/s) - powinien być płaski
rate(vllm:generation_tokens_total[5m])

# Panel 2: KV cache utilization - powinien oscylować 70-85%, nie 95%+
vllm:gpu_cache_usage_perc

# Panel 3: Preemptions rate - powinien być bliski zera
rate(vllm:num_preemptions_total[5m])

# Panel 4: Prefix cache hit ratio - powinien rosnąć po warm-up
rate(vllm:prefix_cache_hit_total[5m]) / (rate(vllm:prefix_cache_hit_total[5m]) + rate(vllm:prefix_cache_miss_total[5m]))

vLLM losing performance after a few hours?

Book a free 30-minute call. We will analyse your inference cluster configuration, identify the source of fragmentation, and implement a solution ensuring stable 24/7 throughput.

Book a call