Migrating to an OpenAI-Compatible API Gateway: Build a Compatibility Contract Before You Flip the Base URL

Changing base_url, api_key, and model is often enough to make a simple chat demo work against an OpenAI-compatible API. It is not enough to prove that a production migration is safe.

The failures usually appear later: streamed tool calls arrive in a different shape, a JSON schema mode is ignored, an embeddings model returns a different vector size, usage fields are missing, retries double-submit a side effect, or a provider-specific reasoning option silently does nothing. The practical goal is not to ask whether an endpoint is “OpenAI compatible” in the abstract. The goal is to define which parts of the OpenAI-shaped contract your applications depend on, test those parts, and route through a gateway only after the contract is explicit.

This guide shows how to migrate a team from provider-specific SDKs or scattered compatible endpoints to one OpenAI-compatible gateway while preserving reliability, usage attribution, and rollback options.

What is fact, recommendation, and prediction in this migration?

Facts: Several providers document OpenAI-compatible paths or SDK usage for parts of their APIs. Google documents Gemini access through OpenAI Python and TypeScript libraries and REST by changing the API key, base URL, and model, while also recommending direct Gemini API usage for applications that are not already using OpenAI libraries. Gemini’s compatibility documentation covers chat completions, streaming, function calling, image understanding, embeddings, reasoning-effort mappings, and provider-specific options through extra request bodies. Together AI documents OpenAI REST and SDK compatibility for multiple modalities, but its matrix also lists unsupported OpenAI-shaped surfaces such as Assistants, Threads, and Runs. Mistral documents a migration path for OpenAI-compatible clients by changing base URL and model name. Groq exposes OpenAI-path chat completions endpoints. vLLM offers an OpenAI-compatible server for completions and chat, while documenting parameter differences. The OpenAI Agents SDK documentation warns that many non-OpenAI providers do not yet support the newer Responses API and that Chat Completions mode is often the safer compatibility target.

Recommendations: Treat compatibility as a tested application contract. Inventory the exact endpoints and features your apps use, create a provider and model capability matrix, write conformance tests before traffic migration, normalize known request and response differences at the gateway boundary, and roll out with per-application keys and rollback profiles.

Prediction: OpenAI-compatible surfaces will remain useful as a lowest-friction integration layer, but provider-native features will continue to diverge. Teams that maintain a compatibility contract will be able to adopt new models faster than teams that rely on informal “drop-in replacement” assumptions.

Step 1: inventory every current AI call

Start with an inventory, not code changes. A migration fails when teams assume all AI calls look like chat completions and discover hidden dependencies only after release.

Create one row per call site. Include scheduled jobs, internal tools, notebooks, background workers, eval harnesses, and customer-facing services.

app: support-assistant
owner: customer-platform
current_provider: provider_a
current_sdk: provider_a_python_sdk
endpoint_shape: chat.completions
model: provider-a-large-2026
features:
  - streaming
  - tool_calls
  - json_schema_output
  - usage_accounting
latency_budget_ms: 8000
retry_policy: retry_429_5xx_no_tool_side_effects
monthly_volume_estimate: 2.4M requests
rollback_contact: oncall-customer-platform

Classify each call by endpoint and feature, not only by model. A single model name can hide very different compatibility requirements depending on how it is used.

Inventory checklist

• Chat: messages, system instructions, temperature, top-p, max tokens, stop sequences.
• Streaming: server-sent events parser, final chunks, usage in stream, cancellation behavior.
• Tools: function schemas, parallel calls, argument JSON, tool result messages, side-effect safety.
• Structured outputs: JSON mode, JSON schema, strict validation, fallback repair logic.
• Vision or multimodal input: image URL, base64, MIME handling, detail parameters.
• Embeddings: model ID, vector dimension, normalization expectations, index compatibility.
• Files and batch: upload APIs, job polling, cancellation, output formats.
• Reasoning controls: reasoning effort, thinking budget, hidden tokens, provider-specific settings.
• Errors: rate limit shape, timeout shape, content policy errors, retryable status codes.
• Usage and billing: prompt tokens, completion tokens, cached tokens, reasoning tokens, cost allocation tags.

The output of this step is a dependency map. It tells you which apps can migrate with a simple OpenAI-compatible API profile and which apps need adapter work.

Step 2: build a compatibility contract table

A compatibility contract is a table that says, for each application feature, what the gateway must guarantee and how you will test it. It should be specific enough for engineering and product teams to make rollout decisions.

This table also prevents overpromising. If a provider supports chat and embeddings but not a files or assistants-like workflow, the contract should say so. “Unsupported” is a valid migration result when it avoids a production surprise.

Step 3: create model profiles instead of scattering model IDs

Do not replace one hard-coded model ID with another hard-coded model ID across every app. Use model profiles.

profile: support-chat-fast
openai_model_alias: support-chat-fast
provider: provider_b
provider_model: provider-b/chat-large-fast
endpoint: chat.completions
features:
  streaming: true
  tools: true
  structured_outputs: schema_validated
  vision: false
  embeddings: false
request_policy:
  drop_unsupported_params: false
  reject_unknown_params: true
  pass_through_extra_body: ["reasoning_effort"]
fallback_profile: support-chat-safe
cost_center_required: true

This profile gives applications a stable name while the gateway owns provider mapping. It also handles providers that use namespaced model IDs rather than a flat model namespace. The app asks for support-chat-fast; the gateway decides whether that currently maps to a Together-style namespaced model, a Gemini-compatible model, a Mistral-compatible model, a Groq chat model, a self-hosted vLLM endpoint, or another approved target.

The trade-off is governance overhead. Profiles must be documented, reviewed, and versioned. The benefit is that migrations, rollbacks, and model replacements do not require every application to redeploy.

Step 4: write conformance tests before migration

Conformance tests are small, repeatable checks that verify your contract against each target profile. They should run before the first rollout and whenever a provider, model, SDK, or gateway adapter changes.

Minimum test suite

• Golden prompt tests: Send deterministic prompts and verify response shape, finish reason, safety behavior, and basic semantic requirements. Do not require exact wording unless the application truly depends on it.
• Streaming parser tests: Confirm your client can parse every chunk, reconstruct final text, handle cancellation, and detect stream completion.
• Tool-call round trips: Force a tool call, parse the arguments, execute a fake tool, return the tool result, and confirm the model continues correctly.
• Tool-call streaming tests: Verify that partial argument deltas can be buffered and reconstructed before tool execution. If not, disable incremental tool execution for that profile.
• JSON schema validation: Test valid output, invalid output, missing fields, extra fields, and refusal or error cases.
• Embedding dimension checks: Confirm vector length, numeric type, and compatibility with the target vector index before reusing an existing index.
• Retry and idempotency tests: Simulate 429, 500, timeout, and partial stream failures. Ensure tool side effects are not repeated accidentally.
• Usage reconciliation: Compare gateway usage records with provider-reported usage fields and your billing ledger expectations.

Keep tests close to production traffic patterns. A single “write a poem” prompt proves almost nothing about a workflow that depends on tools, JSON, embeddings, and usage accounting.

Step 5: normalize quirks at the gateway boundary

An OpenAI-compatible gateway should reduce application code changes, but it should not pretend every provider behaves identically. Use adapters for known differences and make the behavior visible.

Request normalization

• Model aliases: Map stable app-facing profile names to provider-specific model IDs.
• Unsupported parameters: Reject unsupported parameters with a clear error by default. Silent dropping is convenient during demos and dangerous in production.
• Provider-specific options: Allow controlled pass-through fields, such as reasoning or thinking controls, only in documented model profiles.
• Message conversion: Normalize system, developer, user, assistant, and tool messages where the target provider expects a different shape.
• Timeout budgets: Apply one application-level deadline rather than letting SDK defaults accumulate.

Response normalization

• Text and tool choices: Return a consistent shape for assistant text, tool calls, and finish reasons.
• Streaming chunks: Normalize common deltas and document where buffering is required.
• Usage fields: Store provider-native usage plus normalized prompt, completion, and total token counts where available.
• Error shape: Map status codes, retryability, provider error code, and request ID into one error schema.
• Cost metadata: Attach app, team, profile, provider, model, and environment labels for later analysis.

The main trade-off is portability versus provider power. Normalizing to the smallest common surface improves interchangeability. Allowing provider-specific fields preserves advanced capabilities, but every pass-through option becomes part of the profile documentation and test matrix.

Step 6: roll out with per-app keys and rollback profiles

Migration should be reversible without a code redeploy. Use separate API keys for each application, environment, and team. A single shared key makes usage attribution and emergency rollback harder.

A safe rollout sequence looks like this:

1. Development profile: Route only local and staging traffic through the gateway. Fix request-shape and parser issues.
2. Shadow tests: Replay representative requests to the new profile without affecting user-visible output. Compare schema validity, tool behavior, latency class, and usage fields.
3. Small production slice: Move a low percentage of traffic or one internal tenant. Watch errors, retries, user-facing quality signals, and cost.
4. Per-app expansion: Migrate one app at a time. Do not migrate chat, embeddings, batch, and files together unless they share the same risk profile.
5. Rollback profile: Keep a known-good provider/model profile available behind the same app-facing alias or a fast configuration switch.
6. Post-migration lock: Once stable, remove direct provider keys from application environments so traffic cannot bypass gateway controls.

Rollback should be tested like any other path. If a model profile can be switched in the gateway, test that switch during a quiet period and confirm application logs, usage analytics, and billing attribution remain coherent.

Example: replacing scattered endpoints with one gateway contract

Assume a team has three apps:

• A customer support assistant using streaming chat and tools.
• A content classifier requiring strict JSON output.
• A search service using embeddings stored in a vector database.

A risky migration would change all three apps to the same base URL and pick three new model IDs. A safer migration separates the contracts:

• support-chat profile: Requires streaming, tool calls, buffered tool-call deltas, retry classification, and usage logging.
• classifier-json profile: Requires schema validation, refusal handling, and no silent parameter dropping.
• search-embedding profile: Requires a fixed vector dimension and an index migration plan if the dimension changes.

Each profile gets its own conformance tests and rollout. The support assistant might need streaming adapter work. The classifier might pass quickly if schema validation is external to the model. The embedding service might require a new index rather than an in-place model swap. The gateway gives the team one OpenAI-compatible base URL, but the compatibility contract keeps the migration honest.

Migration checklist

• List every AI call site, including background jobs and internal scripts.
• Classify calls by endpoint, feature, model, owner, and rollback path.
• Define app-facing model profiles instead of hard-coding provider model IDs.
• Create a capability matrix for each provider and model profile.
• Reject unsupported parameters unless a profile explicitly allows pass-through.
• Test streaming, tools, structured outputs, embeddings, errors, retries, and usage fields.
• Use per-app and per-environment API keys for attribution and control.
• Run shadow tests before user-visible production traffic.
• Roll out one application or feature class at a time.
• Keep a tested rollback profile available without code redeploys.

Actionable conclusion

An OpenAI-compatible API gateway is most valuable when it becomes a controlled migration layer, not just a different URL. The base URL switch reduces mechanical code changes. The compatibility contract reduces operational risk.

Before you flip production traffic, write down what your applications actually require: streaming behavior, tool semantics, schema guarantees, embedding dimensions, retry rules, usage fields, and error meanings. Convert those requirements into model profiles, adapter rules, and conformance tests. Then roll out with per-app keys, analytics, and rollback profiles.

If the simple chat path works, treat it as a good start. Treat the rest of the migration as engineering work that deserves the same discipline as a database, queue, or payments provider change.

Related reading

• structured output validation across multiple model providers
• model replacement and rollback runbook
• usage analytics and token reconciliation in a gateway