Azure Azure OpenAI rate limiting FinOps LLM

Azure OpenAI Service - 429 Rate Limit When Token or Request Quota Is Exceeded

Resolving 429 errors in Azure OpenAI: identifying the bottleneck (TPM vs RPM), implementing retry logic, increasing quota, or switching to PTU.

Jerzy Kopaczewski · 2026-07-29

Azure OpenAI API returns HTTP 429 Too Many Requests when your application exceeds its allocated token quota (TPM) or the number of requests per minute (RPM). The result: users experience errors in chatbots, RAG pipelines, and automated document processing systems. During peak hours this can mean losing tens of percent of production traffic to GPT-4o or GPT-4 models.

This runbook covers diagnosing and resolving rate limiting in Azure OpenAI Service. For a detailed analysis of AI model costs and cloud architecture, see our article Azure OpenAI vs self-hosted models - cost, latency, and data residency in the EU. If you need help optimising costs and performance of AI workloads - book a consultation.

Symptom

The application receives an HTTP 429 response from Azure OpenAI API. Logs show a RateLimitError or 429 Too Many Requests message:

# Sprawdź aktualne zużycie quota dla deploymentu
az cognitiveservices usage list \
  --name my-openai-resource \
  --resource-group rg-ai \
  --query "[?name.value=='OpenAI.Standard.gpt-4o'].{model:name.value, current:currentValue, limit:limit}" -o table

# Sprawdź metryki rate limitingu z ostatnich 30 minut
az monitor metrics list \
  --resource /subscriptions/<sub-id>/resourceGroups/rg-ai/providers/Microsoft.CognitiveServices/accounts/my-openai-resource \
  --metric "AzureOpenAIRequests" \
  --dimension StatusCode \
  --interval PT1M \
  --start-time $(date -u -d '-30 minutes' +%Y-%m-%dT%H:%M:%SZ) \
  --query "value[0].timeseries[?metricValues[?statusCode=='429']]" -o json

# Lista deploymentów z ich limitami
az cognitiveservices account deployment list \
  --name my-openai-resource \
  --resource-group rg-ai \
  --query "[].{model:properties.model.name, version:properties.model.version, capacity:sku.capacity, name:name}" -o table

In the HTTP response from Azure OpenAI, check the rate limit headers:

# Pojedyncze zapytanie testowe z wyświetleniem nagłówków
curl -s -D - https://my-openai-resource.openai.azure.com/openai/deployments/gpt-4o/chat/completions?api-version=2024-10-21 \
  -H "Content-Type: application/json" \
  -H "api-key: $AZURE_OPENAI_KEY" \
  -d '{"messages":[{"role":"user","content":"test"}],"max_tokens":5}' \
  2>&1 | grep -i "x-ratelimit\|retry-after"

# Spodziewane nagłówki:
# x-ratelimit-remaining-tokens: 0        <-- brak dostępnych tokenów
# x-ratelimit-remaining-requests: 0      <-- brak dostępnych zapytań
# retry-after: 12                         <-- czas oczekiwania w sekundach

If x-ratelimit-remaining-tokens or x-ratelimit-remaining-requests is 0 - the deployment has exhausted its quota for the current minute.

Root Cause

Azure OpenAI applies two independent limits, either of which can trigger a 429 error:

TPM (Tokens Per Minute) - the total number of tokens (prompt + completion) processed within a minute. For a GPT-4o deployment, the default quota in the Sweden Central region is e.g. 80K TPM. Long prompts (RAG with document context) exhaust this limit quickly.
RPM (Requests Per Minute) - the number of API calls per minute. The RPM value is automatically calculated as 6x the TPM value / 1000. For 80K TPM = 480 RPM. Many short requests (e.g. sentence classification) can hit RPM before exhausting TPM.

Additional factors contributing to the problem:

Global Standard vs Data Zone deployments - deployment types have different base quotas. Global Standard offers higher default limits, but Data Zone (with regional data residency guarantees) has lower pools, particularly for GPT-4o in European regions. Quota is shared across all deployments of the same model within a subscription and region.
Traffic burst - even if average usage fits within the limit, a sudden traffic spike (e.g. a cron job processing a batch of documents) can exhaust the quota in the first seconds of a minute. Azure does not buffer the overflow - it immediately returns 429.
Token estimation mismatch - the application does not account for tokens from the system prompt, function calling schema, or conversation history. Actual TPM consumption is 2-3x higher than estimated from user messages alone.

Resolution

A) Retry with exponential backoff:

The simplest solution for sporadic 429s. Implementation using the tenacity library:

import openai
from tenacity import retry, wait_exponential, stop_after_attempt, retry_if_exception_type
from openai import AzureOpenAI, RateLimitError

client = AzureOpenAI(
    azure_endpoint="https://my-openai-resource.openai.azure.com",
    api_version="2024-10-21",
    api_key=os.environ["AZURE_OPENAI_KEY"],
)

@retry(
    retry=retry_if_exception_type(RateLimitError),
    wait=wait_exponential(multiplier=1, min=2, max=60),
    stop=stop_after_attempt(6),
    before_sleep=lambda retry_state: print(
        f"429 received, retry {retry_state.attempt_number}/6, "
        f"waiting {retry_state.next_action.sleep}s..."
    ),
)
def call_openai(messages: list, model: str = "gpt-4o") -> str:
    response = client.chat.completions.create(
        model=model,
        messages=messages,
        max_tokens=1024,
    )
    return response.choices[0].message.content


# Alternatywnie - wykorzystaj nagłówek retry-after z odpowiedzi
@retry(
    retry=retry_if_exception_type(RateLimitError),
    wait=lambda retry_state: float(
        getattr(retry_state.outcome.exception(), 'response', None)
        and retry_state.outcome.exception().response.headers.get('retry-after', 5)
        or 5
    ),
    stop=stop_after_attempt(5),
)
def call_openai_with_retry_after(messages: list) -> str:
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=messages,
    )
    return response.choices[0].message.content

B) Increase quota via Azure Portal / CLI:

# Sprawdź aktualne limity quota w regionie
az cognitiveservices model list \
  --location swedencentral \
  --query "[?model.name=='gpt-4o'].{model:model.name, version:model.version, maxCapacity:model.skus[0].capacity.maximum}" -o table

# Sprawdź aktualną capacity deploymentu
az cognitiveservices account deployment show \
  --name my-openai-resource \
  --resource-group rg-ai \
  --deployment-name gpt-4o \
  --query "{capacity:sku.capacity, model:properties.model.name}" -o json

# Zwiększ capacity deploymentu (np. z 80K do 150K TPM)
az cognitiveservices account deployment create \
  --name my-openai-resource \
  --resource-group rg-ai \
  --deployment-name gpt-4o \
  --model-name gpt-4o \
  --model-version "2024-08-06" \
  --model-format OpenAI \
  --sku-capacity 150 \
  --sku-name "GlobalStandard"

# Jeśli region nie ma wystarczającej capacity - sprawdź dostępność w innych regionach
az cognitiveservices model list \
  --location eastus2 \
  --query "[?model.name=='gpt-4o'].{version:model.version, available:model.skus[0].capacity.maximum}" -o table

Note: The sku-capacity value is specified in thousands of TPM. A value of 150 means 150,000 TPM.

C) Load balancing across multiple deployments (LiteLLM):

Distribute traffic across several OpenAI resources in different regions:

# litellm_config.yaml
model_list:
  - model_name: gpt-4o
    litellm_params:
      model: azure/gpt-4o
      api_base: https://openai-swedencentral.openai.azure.com/
      api_key: os.environ/AZURE_OPENAI_KEY_SWEDEN
      api_version: "2024-10-21"
    model_info:
      tpm: 80000
      rpm: 480

  - model_name: gpt-4o
    litellm_params:
      model: azure/gpt-4o
      api_base: https://openai-eastus2.openai.azure.com/
      api_key: os.environ/AZURE_OPENAI_KEY_EASTUS2
      api_version: "2024-10-21"
    model_info:
      tpm: 80000
      rpm: 480

  - model_name: gpt-4o
    litellm_params:
      model: azure/gpt-4o
      api_base: https://openai-francecentral.openai.azure.com/
      api_key: os.environ/AZURE_OPENAI_KEY_FRANCE
      api_version: "2024-10-21"
    model_info:
      tpm: 60000
      rpm: 360

router_settings:
  routing_strategy: "usage-based-routing-v2"
  enable_pre_call_checks: true    # sprawdza TPM/RPM przed wysłaniem
  retry_after: 5
  num_retries: 3
  allowed_fails: 2
  cooldown_time: 30               # sekundy cooldown po 429

# Uruchom LiteLLM proxy
litellm --config litellm_config.yaml --port 4000

# Test - zapytania automatycznie routowane do dostępnego deploymentu
curl http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o","messages":[{"role":"user","content":"test"}]}'

D) Switch to Provisioned Throughput Units (PTU):

For stable, predictable production workloads - PTU guarantees consistent throughput without throttling:

# Sprawdź dostępność PTU w regionie
az cognitiveservices model list \
  --location swedencentral \
  --query "[?model.name=='gpt-4o'].model.skus[?name=='ProvisionedManaged'].{name:name, minCapacity:capacity.minimum, maxCapacity:capacity.maximum}" -o json

# Utwórz deployment typu Provisioned (minimalnie 50 PTU)
az cognitiveservices account deployment create \
  --name my-openai-resource \
  --resource-group rg-ai \
  --deployment-name gpt-4o-ptu \
  --model-name gpt-4o \
  --model-version "2024-08-06" \
  --model-format OpenAI \
  --sku-capacity 50 \
  --sku-name "ProvisionedManaged"

# Monitoruj utilization PTU (powinno być <80% dla rezerwy)
az monitor metrics list \
  --resource /subscriptions/<sub-id>/resourceGroups/rg-ai/providers/Microsoft.CognitiveServices/accounts/my-openai-resource \
  --metric "ProvisionedManagedUtilizationV2" \
  --dimension ModelDeploymentName \
  --interval PT1M \
  --start-time $(date -u -d '-60 minutes' +%Y-%m-%dT%H:%M:%SZ) \
  --query "value[0].timeseries[].data[-5:]" -o table

Cost note: PTU is a reservation commitment (billed hourly/monthly). 50 PTU for GPT-4o costs approximately $2/hour. It pays off when pay-as-you-go usage exceeds roughly $1.5/hour on a given deployment. Details in the calculator: Azure OpenAI Pricing.

Validation

# 1. Wyślij serię zapytań testowych i sprawdź brak 429
for i in $(seq 1 20); do
  STATUS=$(curl -s -o /dev/null -w "%{http_code}" \
    https://my-openai-resource.openai.azure.com/openai/deployments/gpt-4o/chat/completions?api-version=2024-10-21 \
    -H "Content-Type: application/json" \
    -H "api-key: $AZURE_OPENAI_KEY" \
    -d '{"messages":[{"role":"user","content":"Say OK"}],"max_tokens":5}')
  echo "Request $i: HTTP $STATUS"
  sleep 0.5
done
# Expected: wszystkie odpowiedzi HTTP 200

# 2. Sprawdź metryki - brak nowych 429 w ostatnich 5 minutach
az monitor metrics list \
  --resource /subscriptions/<sub-id>/resourceGroups/rg-ai/providers/Microsoft.CognitiveServices/accounts/my-openai-resource \
  --metric "AzureOpenAIRequests" \
  --dimension StatusCode \
  --interval PT1M \
  --start-time $(date -u -d '-5 minutes' +%Y-%m-%dT%H:%M:%SZ) \
  --query "value[0].timeseries[].data[].{time:timeStamp, total:total}" -o table
# Expected: brak wierszy z StatusCode=429

# 3. Skonfiguruj alert na przyszłość
az monitor metrics alert create \
  --name "alert-openai-429-rate-limit" \
  --resource-group rg-ai \
  --scopes /subscriptions/<sub-id>/resourceGroups/rg-ai/providers/Microsoft.CognitiveServices/accounts/my-openai-resource \
  --condition "total AzureOpenAIRequests where StatusCode includes 429 > 10" \
  --window-size 5m \
  --evaluation-frequency 1m \
  --severity 2 \
  --action-group /subscriptions/<sub-id>/resourceGroups/rg-monitoring/providers/Microsoft.Insights/actionGroups/ag-platform-team \
  --description "Azure OpenAI rate limiting - >10 requests z 429 w 5 minut"

Once the fix is deployed, verify in Application Insights that the OpenAI call success rate has returned to >99%. Dashboard:

// Application Insights - KQL query do monitorowania 429
dependencies
| where timestamp > ago(1h)
| where target contains "openai.azure.com"
| summarize
    total=count(),
    failed_429=countif(resultCode == "429"),
    success_rate=round(100.0 * countif(resultCode == "200") / count(), 2)
  by bin(timestamp, 5m)
| order by timestamp desc

An unresolved 429 problem in Azure OpenAI directly impacts user experience: chatbots stop responding, RAG pipelines interrupt processing, and content generation automations halt. In architectures without retry logic, a single traffic spike can cause cascading failures across microservices that depend on the LLM. For production applications with an SLA above 99.9%, we recommend combining solutions B+C (increased quota + load balancing) or D (PTU) - retry alone (solution A) merely masks the problem without eliminating it.

Azure OpenAI throttling your requests?

Book a free 30-minute call. We will analyse your traffic patterns, select the optimal quota strategy, and implement an architecture resilient to rate limiting.