Reliable LLM API Routing: Timeouts, Retries, and Model Fallbacks Without Semantic Regressions
A practical architecture for classifying LLM API failures, enforcing one latency budget, selecting compatible fallback models, protecting side effects, and validating every accepted response.
A fallback request is not successful merely because another model returned HTTP 200. The replacement may exceed the original latency budget, omit required JSON fields, call a different tool, or produce an answer with materially different semantics. Reliable LLM API routing therefore requires more than an ordered list of models: it requires a contract, a failure classifier, a bounded attempt policy, and validation before acceptance.
The central rule is simple: retry only when the failure is plausibly temporary, and fall back only when the next route can still satisfy the original request contract.
Define the routing contract before choosing models
Start by describing what a successful response must provide. This routing contract should be machine-readable and attached to each workload or request class.
{
"workload": "invoice_extraction",
"modalities": ["text", "image"],
"max_input_tokens": 50000,
"requires_tools": false,
"structured_output": {
"required": true,
"schema_id": "invoice-v3",
"strict": true
},
"allowed_model_classes": ["document-extraction"],
"max_cost_usd": 0.08,
"deadline_ms": 8000
}The contract should cover required modalities, context capacity, tool support, structured-output behavior, acceptable model classes, maximum cost, and the end-to-end deadline. Add application-specific constraints where necessary, such as permitted regions, minimum output length, or a required finish reason.
Recommendation: maintain separate, tested route groups for plain text, schema-constrained output, tool use, vision, and long-context requests. A model that is an acceptable text fallback is not automatically an acceptable fallback for tool calling or image input.
Classify the failure before taking action
Authentication errors, malformed requests, rate limits, and server failures require different responses. Treating every non-success response as retryable wastes capacity and can hide defects.
| Failure class | Examples | Default action |
|---|---|---|
| Permanent request failure | Invalid credentials, malformed parameters, unsupported feature | Stop and return a clear error |
| Route incompatibility | Context too large, unsupported image input, unavailable schema mode | Try only a compatible route |
| Transient transport failure | Connection reset, DNS failure, selected timeout | Retry within the remaining budget |
| Capacity or rate failure | HTTP 429, overloaded service, selected 5xx responses | Honor retry hints or use a healthy fallback |
| Invalid successful response | Malformed JSON, unknown tool, missing required field | Reject, then retry or fall back if policy permits |
| Ambiguous execution | Connection lost after the provider may have accepted the request | Deduplicate before replaying |
Fact: unsuccessful rate-limited requests can still count against provider limits. Aggressive immediate retries may therefore deepen throttling instead of resolving it. Retries also consume additional capacity during an outage, and retry policies at several application layers can multiply the resulting load.
Recommendation: let one layer own model-generation retries. In a typical architecture, the AI API gateway is the right owner because it sees route health, attempt history, latency, and cost. Disable automatic retries in lower-level clients where possible, or count them explicitly in the same attempt budget.
Spend one end-to-end latency budget
Per-attempt timeouts are insufficient. Three attempts with a five-second timeout can turn an intended five-second operation into a fifteen-second response, before backoff and validation are included.
Record an absolute deadline when the request enters the gateway. Before every attempt, calculate the remaining time:
remaining = deadline - current_time
required = connect_allowance + generation_allowance + validation_allowance
if remaining < required:
stop_without_launching_another_attemptFor an eight-second deadline, a reasonable initial allocation might reserve 300 ms for gateway work and final validation, allow up to 4.5 seconds for the primary route, and retain approximately 3.2 seconds for one fallback. These values are an example, not a benchmark. They must be derived from measured latency distributions for the actual providers, models, regions, and output sizes.
Use capped exponential backoff with jitter for transient retries:
delay = random(0, min(cap, base * 2^retry_index))Provider retry hints, such as a retry-after value, should take precedence when they fit within the remaining deadline. Stop after a small number of attempts. A common policy is one primary attempt plus one fallback, with an optional same-route retry only for an early connection failure that could not have generated billable output.
Trade-off: sequential fallback improves availability but increases tail latency. Parallel or hedged requests can reduce latency during slowdowns, but they consume more capacity and may incur charges for multiple successful generations. Hedging should be restricted to latency-critical, side-effect-free workloads with cancellation and cost controls.
Select fallbacks by capability, not rank
A fallback table should encode compatibility rather than a global preference order. Filter candidate routes against the contract before considering health, latency, or price.
candidates = routes
.filter(supports_required_modalities)
.filter(context_limit >= estimated_input_size)
.filter(supports_required_tools)
.filter(supports_requested_schema_mode)
.filter(model_class in allowed_model_classes)
.filter(estimated_cost <= remaining_cost_budget)
.filter(not_temporarily_suppressed)
selected = rank(candidates, health, latency, cost)Structured-output support deserves explicit testing. Even when two routes advertise schema-constrained generation, they may support different JSON Schema subsets or interpret edge cases differently. Tool-capable models can likewise differ in tool selection, argument construction, and parallel-call behavior.
Fact: switching model families can preserve transport availability while changing style, reasoning quality, safety behavior, tokenization, and tool selection. HTTP success is not evidence of semantic equivalence.
Prediction: as model catalogs expand, production routing policies will increasingly use versioned capability profiles and workload-specific acceptance tests instead of static model lists. Treat this as a design direction, not a guarantee about provider behavior.
Validate the response before accepting it
Run every response, including the primary response, through the same acceptance pipeline. Validation should occur before the result is cached, billed internally as successful, or passed to a tool executor.
- Confirm the transport completed and the response envelope can be parsed.
- Check the finish reason and reject truncation when complete output is required.
- Validate structured output against the original schema.
- Verify required fields, enum values, and application invariants.
- Allow only registered tool names and validate arguments against each tool schema.
- Apply workload-specific semantic checks where false acceptance would be costly.
For invoice extraction, semantic checks might require a nonnegative total, a supported currency code, and line-item totals within an explicitly defined tolerance. For classification, require a label from the permitted set. For code generation, parsing or compilation may be appropriate. These checks do not prove quality, but they prevent predictable contract violations from being treated as successes.
Do not silently repair every malformed response. Deterministic normalization, such as removing harmless surrounding whitespace, can be acceptable. Guessing missing financial fields or rewriting tool arguments changes the model's meaning and should trigger rejection or human review.
Separate generation retries from side effects
LLM requests commonly use HTTP POST, which is not inherently idempotent. More importantly, a model response may initiate an external action such as charging a payment method, sending a message, creating a ticket, or modifying infrastructure. Retrying generation and replaying that action are separate decisions.
Assign an operation ID at the application boundary and an attempt ID to every model call. Persist tool execution state against a deterministic key, such as:
execution_key = operation_id + tool_name + canonical_arguments_hashBefore executing a tool, check whether that key is pending, completed, or failed. Return the stored result for a completed execution rather than running it again. For operations whose arguments may legitimately change, require application-level approval or a new operation ID.
An ambiguous timeout requires special handling. If the connection fails after a request was transmitted, the gateway may not know whether generation occurred. A provider-supported idempotency key can help when available. Otherwise, log the outcome as unknown and apply a workload-specific replay policy instead of assuming nothing happened.
Suppress unhealthy routes and expose every attempt
A circuit breaker or temporary health suppression prevents each new request from rediscovering the same failing route. Open the circuit after a defined error-rate or consecutive-failure threshold, then admit limited probes in a half-open state. Tune thresholds by route and failure class so a malformed client request cannot make a healthy model appear unavailable.
Record one request-level event and one event per attempt. Useful fields include operation ID, attempt ID, selected provider and model, failure class, status code, latency, token counts, estimated cost, fallback reason, validation result, circuit state, and final outcome. Redact or hash prompts, outputs, and tool arguments according to their sensitivity and retention requirements.
Useful operational metrics include fallback rate, attempts per completed request, deadline exhaustion rate, validation rejection rate, ambiguous outcomes, cost per accepted response, and latency by final route. A rising HTTP success rate alongside a rising validation rejection rate is a warning that transport availability is masking contract failures.
Production rollout checklist
- Define a versioned routing contract for every workload class.
- Map provider errors into permanent, transient, incompatible, invalid-response, and ambiguous categories.
- Choose one retry owner and cap total attempts.
- Propagate an absolute deadline through gateway, provider client, validation, and tool execution.
- Build capability-tested fallback groups rather than one global model chain.
- Validate schemas, tool calls, finish reasons, and domain invariants.
- Deduplicate side effects with operation and execution keys.
- Add route suppression with bounded half-open probes.
- Log attempt-level latency, tokens, cost, failures, and acceptance results.
- Inject timeouts, 429s, selected 5xx errors, malformed JSON, context overflow, and slow successes in staging.
Begin with a primary route and one compatible fallback for a single low-risk workload. Compare accepted-response quality, latency, and cost before expanding the policy. The objective is not the highest possible fallback rate. It is a bounded system that either returns a response satisfying the original contract or fails clearly before it causes duplicate work or semantic damage.