Build a Responses API Compatibility Layer in an AI API Gateway

Do not implement /v1/responses by translating every request into /v1/chat/completions and hoping the shape is close enough. That adapter may return text, but it can silently lose the parts developers care about: response items, server-side state, tool calls, reasoning continuity, stream lifecycle events, cancellation semantics, and item-level usage attribution.

The practical goal is a compatibility layer that treats the Responses API as a richer protocol. Keep Chat Completions support for existing clients, but build Responses as its own gateway surface with its own state model, stream normalizer, tool-call ledger, capability matrix, and fallback rules.

What is factual, what is policy, and what is prediction?

Facts: OpenAI describes the Responses API as unifying capabilities that were previously split across Chat Completions and Assistants, including support for tools such as web search, file search, and computer use. The API exposes fields such as previous_response_id, streaming, tool selection, and built-in tools. SDK documentation shows previous_response_id can provide conversation continuity, while previous instructions are not automatically carried forward and must be resent when they should still apply. OpenAI’s streaming reference includes distinct response lifecycle and output events rather than only token deltas.

Recommendations: A gateway should preserve these semantics rather than flattening them by default. It should reject or explicitly downgrade requests when a target provider cannot support required behavior.

Prediction: More agent workloads will depend on response-item structure, tool execution traces, and stateful reasoning context. Gateways that model those concepts now will be easier to extend than gateways that treat Responses as a cosmetic endpoint.

Define a separate compatibility contract for Responses

The first implementation mistake is assuming OpenAI-compatible means one universal request and response schema. In practice, /v1/chat/completions and /v1/responses should be separate compatibility contracts.

Keep a shared authentication, billing, quota, and routing layer, but separate the protocol layer:

• Chat Completions surface: messages, choices, deltas, tool calls in chat format, legacy client behavior.
• Responses surface: input items, output items, response IDs, previous response references, richer tool events, lifecycle stream events, reasoning-related fields, and final response state.

This split matters for conformance tests. A provider adapter that passes chat tests may still fail Responses tests because it cannot preserve previous_response_id, item ordering, refusal structure, hosted tool metadata, or streaming event names.

A minimal compatibility contract should answer:

• Which request fields are accepted, rejected, transformed, or ignored?
• Which response item types are preserved?
• Which tool types are supported per provider and model?
• Can the provider maintain conversation state, or must the gateway maintain it?
• What happens when store=false is requested?
• What stream events are guaranteed?
• How are cancellation, timeout, and partial usage recorded?

If you already have an AI API gateway, treat Responses support as a protocol expansion, not a route alias.

Use a canonical response item model

The Responses API returns more than one assistant message. It can represent different output items and events. Your gateway needs an internal canonical model before it maps to any provider.

A practical internal item schema can start like this:

{
  "gateway_response_id": "gw_resp_...",
  "provider_response_id": "resp_...",
  "tenant_id": "ten_123",
  "key_id": "key_456",
  "model_alias": "agent-default",
  "provider": "openai",
  "items": [
    {
      "item_id": "item_1",
      "type": "text",
      "role": "assistant",
      "content": [{ "type": "output_text", "text": "..." }],
      "status": "completed"
    },
    {
      "item_id": "item_2",
      "type": "function_call",
      "call_id": "call_abc",
      "name": "lookup_order",
      "arguments_json": "{\"order_id\":\"123\"}",
      "status": "completed"
    }
  ],
  "usage": {
    "input_tokens": 0,
    "output_tokens": 0,
    "reasoning_tokens": null,
    "tool_units": []
  },
  "status": "completed"
}

Include item types even before every provider can produce them. Useful categories include:

• Text output
• Refusals
• Function calls
• Function outputs submitted by the application
• Reasoning summaries or reasoning-related metadata where available
• File references
• Web search, file search, computer-use, or other hosted tool events
• Final usage and billing metadata

The point is not to expose a proprietary schema to users. The point is to keep the gateway from throwing away information before it can audit, bill, stream, replay, or transform it.

Build a gateway-owned state ledger

previous_response_id is the field that most exposes the difference between stateless chat proxying and Responses compatibility. If a client references a previous response, the gateway must know what that ID means, whether the tenant is allowed to use it, and whether the provider can continue from it.

Create a state ledger keyed by tenant and response ID:

{
  "gateway_response_id": "gw_resp_789",
  "provider_response_id": "resp_provider_789",
  "previous_gateway_response_id": "gw_resp_456",
  "tenant_id": "ten_123",
  "user_id": "user_999",
  "key_id": "key_456",
  "model": "gpt-...",
  "provider": "openai",
  "store_mode": "provider|gateway|none",
  "retention_policy": "standard|zero_retention|custom_30d",
  "instructions_hash": "sha256:...",
  "tool_policy_id": "tools_readonly_v3",
  "created_at": "...",
  "expires_at": "...",
  "deleted_at": null
}

Important rule: do not automatically emulate previous_response_id by replaying full chat history unless the tenant has explicitly allowed that retention and cost behavior. Replay may increase token cost, change privacy posture, and alter model behavior. It is safer to return a clear capability error than to silently send stored conversation content the application did not expect you to retain or reuse.

State-handling modes

• Provider state: The upstream provider stores enough context, and the gateway maps gateway response IDs to provider response IDs.
• Gateway state: The gateway stores necessary prior items and reconstructs context when allowed.
• No state: The request uses store=false or tenant policy forbids retention. previous_response_id should be rejected unless the provider can honor the request without gateway retention and the policy allows it.

Also remember that previous instructions may need to be resent by the client when they should continue to apply. The gateway should not invent hidden instructions to compensate unless that behavior is part of an explicit tenant policy.

Validate tools before dispatch

Responses makes tool use more central. A compatibility layer should handle two broad categories:

• Application tools: Function definitions supplied by the client, executed outside the model provider, with outputs submitted back to the API.
• Hosted provider tools: Web search, file search, computer use, code execution, grounding, or similar tools executed by the provider or gateway-controlled infrastructure.

At ingress, validate tool schemas before routing:

• Reject invalid JSON Schema early.
• Enforce maximum schema size and nesting depth.
• Check tool names for provider compatibility.
• Apply tenant, key, user, and environment scopes.
• Require approval gates for tools that write data, spend money, access sensitive systems, or call external connectors.

For application function calling, require a stable call ID. The model emits a function call with call_id; the application submits the tool output referencing that ID; the gateway records both in the same trace. Without that join key, audit logs and retries become ambiguous.

For hosted tools, reserve budget before dispatch and settle cost afterward. Hosted tools may add charges outside ordinary token accounting, so connect the tool ledger to unified AI API billing rather than hiding those costs inside a generic model-call total.

Normalize streaming as events, not token text

A chat proxy can often get away with forwarding token deltas. A Responses gateway cannot. The stream has lifecycle meaning: a response can start, output items can start and complete, text can arrive in deltas, tool calls can be assembled incrementally, usage can arrive at the end or during the stream, and the response can fail or be cancelled.

Define a gateway event schema, then map each provider stream into it:

event: response_started
data: { "response_id": "gw_resp_123", "status": "in_progress" }

event: output_item_started
data: { "item_id": "item_1", "type": "text" }

event: text_delta
data: { "item_id": "item_1", "delta": "Hello" }

event: tool_call_delta
data: { "item_id": "item_2", "call_id": "call_abc", "arguments_delta": "{\"order" }

event: usage_delta
data: { "output_tokens": 12 }

event: completed
data: { "response_id": "gw_resp_123", "usage": { ... } }

Recommended normalized events:

• response_started
• output_item_started
• output_item_completed
• text_delta
• refusal_delta
• tool_call_delta
• tool_result_received
• usage_delta
• completed
• cancelled
• failed

When the client disconnects, propagate cancellation upstream if the provider supports it. Record the partial response state either way. If the provider later returns final usage through a delayed callback or final chunk, reconcile the ledger. Streaming compatibility is as much about accounting and lifecycle as it is about latency.

Create a provider capability matrix

Multi-model routing is useful only when the gateway understands what can safely be routed. Add Responses-specific capabilities to your model catalog:

{
  "model_alias": "agent-default",
  "routes": [
    {
      "provider": "openai",
      "model": "...",
      "supports_responses": true,
      "supports_previous_response_id": true,
      "supports_store_false": true,
      "supports_builtin_web_search": true,
      "supports_function_calling": true,
      "supports_stream_lifecycle_events": true,
      "supports_reasoning_context_continuity": true,
      "max_tool_schema_bytes": 65536
    },
    {
      "provider": "provider_b",
      "model": "...",
      "supports_responses": false,
      "chat_adapter_available": true,
      "loss_profile": ["no_previous_response_id", "no_hosted_tools", "flattened_stream"]
    }
  ]
}

Fallback should be loss-aware. If the request requires built-in web search and the fallback provider cannot perform it, do not silently answer without search. If the request depends on preserved reasoning context and the fallback route cannot preserve it, return a capability error or a downgrade response that the client explicitly opted into.

A useful request option is:

{
  "model": "agent-default",
  "input": "...",
  "fallback_policy": {
    "allow_lossy": false,
    "allowed_losses": []
  }
}

For less sensitive use cases, tenants can allow specific lossy downgrades:

{
  "fallback_policy": {
    "allow_lossy": true,
    "allowed_losses": ["flattened_stream", "no_reasoning_summary"]
  }
}

The gateway should log the fallback decision either way. That makes later debugging possible when an agent behaves differently after a provider outage or model reroute.

Attribute usage at response and item level

Responses calls can cost more than equivalent chat completions because they may include tool execution, longer context, reasoning tokens, file search, web search, or repeated instructions. A single aggregate token count is not enough for an AI API usage analytics dashboard.

Record usage at two levels:

• Response level: tenant, key, user, model, provider, latency, final status, input tokens, output tokens, reasoning tokens where reported, total cost, and fallback route.
• Item/tool level: tool name, call ID, hosted tool units, file IDs, search query count if available, tool latency, tool cost, and approval policy result.

This lets developers answer concrete questions:

• Did cost increase because of longer state, reasoning effort, tool calls, or fallback?
• Which tenant or API key is generating hosted tool charges?
• Which response failed after a tool call but before final text?
• Which cancelled streams still incurred upstream usage?

Handle zero-retention and deletion as first-class behavior

Server-side state is helpful, but it changes the gateway’s retention obligations. Build policy into the protocol layer instead of treating it as a logging setting.

For every Responses request, resolve:

• Tenant retention policy
• Request-level store preference
• Provider retention compatibility
• Whether gateway replay is allowed
• Whether tool inputs and outputs may be stored
• Expiration and deletion behavior for response state

If retention is disabled, the gateway may still keep minimal operational metadata: timestamps, IDs, status, token counts, cost, and policy decisions. Avoid storing raw prompts, full tool outputs, or reconstructed history unless policy permits it.

Conformance fixtures to add before launch

Do not rely on happy-path manual tests. Add fixtures that verify protocol behavior across direct OpenAI routes, provider-adapted routes, and fallback scenarios.

Minimum test set

• Basic response: text item is returned with stable response ID and usage.
• Multi-turn state: second request references previous_response_id; gateway validates tenant ownership and state mode.
• Repeated instructions: verify that omitted instructions are not silently invented by the gateway.
• Function call round trip: model emits call ID; application submits output; final response joins both records.
• Hosted tool policy: unauthorized built-in tool is blocked before dispatch.
• Streaming order: response start, item start, deltas, item completion, usage, and completion are emitted in valid order.
• Stream cancellation: client disconnect triggers upstream cancellation where supported and records partial usage.
• Fallback rejection: provider without required Responses semantics returns capability error.
• Lossy fallback opt-in: request with allowed losses receives an explicit downgrade marker.
• Zero-retention mode: state replay and gateway-side prompt retention are blocked.

Recommended rollout sequence

1. Expose a beta route. Add /v1/responses without changing existing chat behavior.
2. Implement pass-through first for providers with native Responses support. Preserve IDs, items, streams, usage, and errors.
3. Add the state ledger. Map gateway IDs to provider IDs and enforce tenant ownership.
4. Add canonical items. Store item metadata needed for auditing, billing, and stream reconstruction.
5. Add tool governance. Validate schemas, enforce scopes, and record tool-call joins.
6. Add streaming normalization. Convert provider-specific streams into gateway lifecycle events.
7. Add capability-aware routing. Allow only safe fallbacks by default.
8. Add analytics and billing settlement. Attribute token, reasoning, and tool usage separately.
9. Publish compatibility notes. Tell developers which fields are native, emulated, unsupported, or lossy.

Actionable conclusion

A Responses API compatibility layer should preserve protocol meaning, not merely return plausible text. Build it around five durable objects: a canonical response item model, a conversation-state ledger, a tool-call ledger, a streaming event normalizer, and a provider capability matrix.

The safest default is strict compatibility: if a route cannot preserve required state, tools, reasoning context, stream events, or retention behavior, return a clear capability error. Add opt-in lossy fallback only when developers understand what will be dropped. That approach may feel less convenient than automatic flattening, but it prevents the worst failure mode: an application that appears compatible while silently losing the semantics that made it use the Responses API in the first place.

Related reading

• streaming token accounting and cancellation handling