Rate-Limit-Aware AI API Gateways: Shape RPM, TPM, Bursts, and Tenant Fairness Before 429s Hit

A 429 from an LLM provider is not just a retry signal. In production, it is often evidence that your application has already lost control of admission, tenant fairness, latency, or provider-specific quota accounting.

The common fix—exponential backoff—is necessary but incomplete. Backoff reacts after the provider rejects traffic. A rate-limit-aware AI API gateway should shape traffic before requests leave your system: estimate token pressure, reserve quota, isolate tenants, queue the right work, reject the wrong work, and adapt when provider limits change.

This article describes a practical gateway quota governor for teams sending production workloads to multiple LLM providers through a unified API.

The reader problem: 429s are multi-dimensional

Many teams treat rate limits as if they were a single requests-per-minute number. That assumption breaks quickly with LLM APIs.

Facts from current provider documentation:

• OpenAI documents that limits may be enforced over shorter windows than the advertised per-minute limit, so short bursts can fail even when the average minute looks safe.
• Azure OpenAI quota is assigned by subscription, region, model, and deployment type in tokens per minute. Assigning TPM to a deployment also determines enforced inference RPM limits, and RPM-to-TPM ratios vary by model.
• Azure OpenAI also notes that rate-limit token calculations are estimated when the request is received and are not the same as final billing token counts.
• Anthropic documents separate requests-per-minute, input-tokens-per-minute, and output-tokens-per-minute limits. Exceeding limits returns a 429 with a retry-after header.
• Anthropic warns that sharp traffic increases can hit acceleration limits and recommends gradual ramp-up.
• For most Claude models, Anthropic documents that cache-read input tokens do not count toward input-token-per-minute limits, which means prompt caching can change effective headroom.
• Google Gemini API rate limits are tied to project usage tiers, with higher tiers depending on billing setup, cumulative spend, and elapsed time after payment milestones.

The operational lesson is clear: an OpenAI-compatible request shape does not imply OpenAI-compatible quota behavior. A multi-provider gateway needs an internal quota model that is richer than “retry if 429.”

Design goal: make admission control a gateway responsibility

A rate-limit-aware gateway should answer five questions before dispatching a request:

1. Which provider, model, deployment, region, project, or workspace will receive the request?
2. How much request, input-token, output-token, and concurrency capacity might it consume?
3. Which tenant, team, API key, customer, or workload class should be charged against shared capacity?
4. Should the request be admitted now, queued briefly, downgraded, routed elsewhere, or rejected?
5. How should the reservation be reconciled after the provider returns actual usage?

The gateway becomes a quota governor. It does not replace provider limits. It makes provider limits visible, predictable, and fair inside your own system.

Build a normalized quota model

Start by defining internal limiter dimensions that can represent the major providers without forcing them into one misleading bucket.

Recommended limiter dimensions

• RPM: requests per minute.
• Input TPM: prompt, message, tool, and context tokens per minute.
• Output TPM: completion tokens per minute, reserved separately for streaming and long generations.
• Total TPM: useful for providers or deployments that expose combined token pressure.
• Concurrency: active requests, active streams, or in-flight jobs.
• Streaming duration: long-lived streams can occupy connection and output-token headroom even when RPM is low.
• Provider-specific scope: Azure subscription/region/deployment, Anthropic workspace/model class, Google project/tier, or OpenAI organization/project/model group.

Do not hide provider-specific dimensions. Normalize them into a common schema, but preserve enough detail to explain a rejection later.

{
  "provider": "provider_a",
  "model_profile": "fast-chat",
  "provider_scope": {
    "project": "prod",
    "region": "us-east",
    "deployment": "chat-large-01"
  },
  "limits": {
    "rpm": 1200,
    "input_tpm": 800000,
    "output_tpm": 250000,
    "concurrency": 200
  }
}

This internal object should be configured explicitly, not inferred from model names alone. Provider dashboards, account tiers, regional deployments, and workspace settings can all change the effective capacity of the same model family.

Estimate token pressure before dispatch

Provider-side rate limiting often happens before final billing usage is known. Your gateway should make the same kind of conservative estimate before sending traffic.

Preflight reservation inputs

• Serialized prompt and message length.
• Model-specific tokenization and overhead for roles, tools, images, or structured output instructions.
• max_completion_tokens or equivalent output cap.
• Historical completion ratio for this endpoint, tenant, model profile, and request class.
• Expected cache-read tokens if prompt caching is available and measurable.
• Streaming flag and expected stream duration.

A simple reservation rule is often enough to start:

estimated_input_tokens = tokenize(request_messages) + model_overhead
estimated_output_tokens = min(
  max_completion_tokens,
  p95_historical_output_tokens_for_route
)
reserved_total_tokens = estimated_input_tokens + estimated_output_tokens

For unknown routes, use a conservative default. For stable production routes, update estimates continuously from actual usage.

Reserve, then reconcile

Quota reservations should not become permanent charges. Treat them like holds:

1. Quote: estimate input and output pressure.
2. Reserve: deduct from the relevant token buckets before dispatch.
3. Settle: replace the estimate with provider-reported usage when available.
4. Refund or debit: return unused reserved capacity or charge overages to the next window if needed.

This matters most for long-context and streaming calls. If you only check input TPM before dispatch, a stream can start successfully and then run into output-token pressure later. Reserving output headroom separately reduces mid-stream failure and stall risk.

Use hierarchical token buckets for tenant fairness

A single global limiter protects the provider account but does not protect tenants from each other. One long-context batch job can consume shared TPM and cause interactive requests from other teams to fail.

Use hierarchical token buckets:

organization
  └── tenant
      └── team
          └── api_key
              └── model_profile
                  └── provider_deployment

A request must pass each relevant bucket. This lets you enforce several policies at once:

• The organization cannot exceed provider capacity.
• A tenant cannot consume more than its contracted share.
• An API key cannot exceed its intended environment or application limit.
• A batch model profile cannot starve an interactive model profile.
• A provider deployment cannot be overloaded even if another deployment has spare quota.

Fair sharing versus utilization

Recommendation: use weighted fair sharing with controlled burst borrowing.

Strict per-tenant caps are easy to explain but can strand unused capacity. Burst borrowing improves utilization by allowing a tenant to temporarily use idle quota from a shared pool. The trade-off is complexity: dashboards must show what was guaranteed, what was borrowed, and when borrowing was revoked.

A practical rule:

• Give each tenant a guaranteed baseline.
• Allow burst borrowing from unused shared capacity.
• Reclaim borrowed capacity when higher-priority or guaranteed traffic appears.
• Never let borrowed traffic create provider-level 429s for guaranteed traffic.

Separate traffic classes before they contend

Not all requests deserve the same queue behavior. Put traffic into model profiles with separate queues and quota pools.

Queuing improves success rate but increases tail latency. A gateway should make that trade-off explicit. For example, an interactive request might wait up to 300 milliseconds for quota, then fall back or fail. A nightly batch job might wait 20 minutes and still be considered successful.

Normalize 429s into a single error schema

Even with good admission control, provider 429s will still occur. Limits can change, provider estimates can differ from yours, and traffic can arrive in sharper bursts than expected.

Normalize every provider 429 into a gateway error object:

{
  "error": {
    "type": "rate_limited",
    "limiter": "output_tpm",
    "provider": "provider_a",
    "model_profile": "fast-chat",
    "provider_model": "model-x",
    "retry_after_ms": 2400,
    "tenant_id": "tenant_123",
    "api_key_id": "key_456",
    "request_class": "interactive",
    "estimated_input_tokens": 4200,
    "estimated_output_tokens": 800,
    "gateway_decision": "admitted_then_provider_rejected",
    "fallback_allowed": false,
    "trace_id": "trace_abc"
  }
}

The key field is gateway_decision. A 429 after the gateway admitted the request is different from a request the gateway rejected locally before dispatch. The first indicates a limiter calibration problem. The second indicates intentional protection.

Adapt from provider headers, but do not depend on them

Some providers return useful headers such as retry-after or remaining capacity indicators. Use them when available.

Recommendation: provider headers should tune your local governor, not replace it.

Reasons:

• Header availability differs by provider and endpoint.
• Headers may not expose every limiter dimension.
• Retry-after tells you when to try again, not which tenant should get capacity next.
• Provider-side token estimates may differ from your billing or internal accounting.

A robust implementation updates local bucket refill rates and cooldowns based on headers, while still enforcing tenant, API key, traffic class, and provider deployment limits inside the gateway.

Add ramp governors for migrations and scheduled jobs

Many rate-limit incidents happen during planned changes: moving from one model to another, shifting providers, enabling a new agent workflow, or launching a scheduled evaluation run.

Recommendation: treat traffic growth as a controlled rollout.

• Feature-flag model migrations by tenant, route, or percentage of traffic.
• Set per-minute growth ceilings for new provider deployments.
• Warm up traffic gradually over hours instead of switching all traffic instantly.
• Pause rollout when 429 rate, downgrade rate, queue depth, or p95 latency crosses a threshold.
• Keep an emergency rollback route with a compatibility policy, not just a spare model.

Prediction: as provider routing modes, priority tiers, and workspace-level controls become more common, ramp governance will become a standard gateway feature rather than an incident-response script.

Fallback is a policy decision, not just a capacity decision

When one provider returns a 429, routing to another provider may be the right answer. It may also be unsafe.

Fallback can change:

• Output quality and instruction following.
• Context length.
• Tool-call behavior.
• Structured-output reliability.
• Data-retention and residency posture.
• Cost and latency.

The quota governor should ask a compatibility layer whether fallback is allowed for this request class. If not, it should queue or fail with a clear local rate-limit response rather than silently changing semantics.

Expose quota dashboards that explain decisions

A quota system that nobody can understand will be bypassed. Build dashboards around operational questions:

• Which tenants are consuming the most RPM, input TPM, and output TPM?
• Which model profiles are queueing, rejecting, or falling back?
• Which provider scope is the bottleneck: project, region, deployment, workspace, model class, or account tier?
• How often do gateway estimates differ from provider usage?
• What is the retry-after distribution by provider and limiter type?
• How much effective headroom is created by prompt cache reads?
• Which traffic classes are borrowing burst capacity?

For customer-facing or partner-facing products, expose safe controls:

• Per-key rate limits.
• Per-team burst limits.
• Per-customer daily caps.
• Emergency pause for a tenant or key.
• Alerts for 429 spikes, queue growth, and abnormal token pressure.
• Partner API endpoints for reseller quota management.

This turns rate limiting from a mysterious provider error into an auditable part of team API governance.

Implementation checklist

Phase 1: observe and classify

• Log provider, model, deployment, region, workspace, project, tenant, API key, and request class for every call.
• Capture provider 429s with retry-after and raw error metadata.
• Record estimated and actual input/output tokens separately.
• Separate interactive, batch, eval, and background traffic in telemetry.

Phase 2: local admission control

• Create internal limiter objects for RPM, input TPM, output TPM, total TPM, and concurrency.
• Add preflight token estimation.
• Reserve quota before dispatch and reconcile after provider usage arrives.
• Reject locally when a request cannot fit its tenant or provider bucket.

Phase 3: fairness and queues

• Add hierarchical buckets from organization to provider deployment.
• Assign guaranteed tenant shares and controlled burst borrowing.
• Create separate queues by traffic class.
• Set class-specific maximum wait times and fallback rules.

Phase 4: adaptation and operations

• Use provider headers to adjust cooldowns and refill assumptions.
• Add ramp governors for migrations and scheduled jobs.
• Expose quota dashboards and alerts.
• Review estimation error and stranded quota weekly.

Actionable conclusion

If your gateway only retries 429s, it is operating after the failure. A production-grade AI API gateway should prevent most rate-limit failures by deciding who is allowed to send what, when, and against which provider quota.

Start with a normalized limiter model, preflight token reservation, and traffic-class queues. Then add hierarchical tenant fairness, provider-header adaptation, and ramp governors. The result is not just fewer 429s. It is clearer capacity allocation, more predictable latency, safer migrations, and rate-limit behavior your engineering, finance, and customer-support teams can actually explain.

Related reading

• preflight reservation and usage reconciliation
• prompt cache accounting and cache-hit analytics
• compatible fallback routing policies