Multi-Tenant RAG Behind an OpenAI-Compatible API Gateway

Customer-facing AI assistants need retrieval-augmented generation, but RAG becomes harder when requests flow through an OpenAI-compatible API gateway instead of one model provider's native stack. The gateway must keep tenant data isolated, preserve citations across model providers, delete indexed content on schedule, and attribute embedding, retrieval, and generation costs to the right customer.

The practical answer is to treat retrieval as a first-class gateway subsystem. Do not hide it inside one provider integration. Keep retrieval separate from generation, give every request a tenant-scoped retrieval context, normalize citations before returning them, and record each billable step in a ledger.

The Reader Problem

A team building an AI assistant for many customers usually starts with a simple flow: upload documents, embed chunks, retrieve the top matches, put those snippets into the prompt, and ask a model to answer. That works until the product needs multiple model providers, customer-level billing, offboarding, and auditability.

The risk is not only inaccurate answers. The larger operational risks are tenant namespace mistakes, unverifiable citations, stale indexes after document deletion, and margins that cannot be explained because retrieval costs disappear into generic infrastructure spend.

This article separates facts, recommendations, and predictions. The facts are implementation capabilities documented by current provider and vector database APIs. The recommendations are architecture choices for a gateway product. The predictions are where this architecture is likely to need flexibility as provider retrieval features keep changing.

Reference Architecture

A gateway-level RAG design should have five components:

• Tenant resolver: maps the incoming API key, workspace, customer account, or Partner API customer to a canonical tenant_id.
• Retrieval profile: defines which corpus to search, which embedding model to use, result count, filters, reranking options, citation requirements, and fallback behavior.
• Retrieval adapter layer: calls native provider retrieval, an external vector database, or a custom search service through one internal interface.
• Prompt assembly and generation adapter: passes retrieved context to the chosen model provider without exposing vector backend details to callers.
• Usage and audit ledger: records embedding, indexing, retrieval, prompt tokens, completion tokens, tenant, model, provider, and trace identifiers.

A minimal request contract can stay provider-neutral:

{
  "tenant_id": "tenant_123",
  "model": "gpt-compatible-or-claude-compatible-model",
  "retrieval_profile": "support_docs_v2",
  "citation_required": true,
  "messages": [
    {"role": "user", "content": "What is our refund policy for annual plans?"}
  ]
}

The response should also be provider-neutral:

{
  "answer": "Annual plans can be refunded within the configured policy window...",
  "citations": [
    {
      "source_id": "doc_789",
      "title": "Billing Policy",
      "url_or_internal_ref": "kb://billing-policy",
      "chunk_id": "chunk_044",
      "offsets": {"page": 3},
      "score": 0.82,
      "retrieval_provider": "vector_db",
      "model_provider": "openai_compatible",
      "provider_payload": {}
    }
  ],
  "retrieval_trace_id": "rt_456",
  "billable_tenant": "tenant_123",
  "embedding_usage": null,
  "retrieval_usage": {"queries": 1, "results": 6},
  "model_usage": {"input_tokens": 1920, "output_tokens": 180}
}

Fact: Provider Retrieval Features Are Not Identical

OpenAI's Vector Stores API supports vector stores that can be created, searched, configured with chunking strategies, associated with file metadata, and deleted. Vector store search supports queries, filters, maximum result counts, ranking options, score thresholds, and query rewriting controls. Those controls give gateway authors useful knobs for latency, relevance, and cost.

OpenAI platform data controls also make lifecycle design important: customer content in vector stores is retained until deleted. If a tenant offboards, or if a temporary project expires, the gateway cannot assume the provider will remove indexed content automatically on the product's business schedule.

Anthropic exposes a different pattern for citations. Applications can provide search result content blocks with source and title metadata, and when citations are enabled the model can attach citation references to generated text. There are practical constraints: search result citation settings are all-or-nothing within a request, search result blocks support text content, and citation granularity depends on how the content is split into blocks.

The implication is direct: a gateway should not expose one provider's retrieval shape as its public contract unless it intends to make that provider the permanent retrieval authority.

Recommendation: Use Retrieval Adapters, Not Retrieval Lock-In

Create an internal retrieval adapter interface. The gateway can support several backends behind it:

• Native provider retrieval: useful when a customer wants the fastest path into one provider's file search or vector store features.
• External vector database: useful when the product must support many model providers with consistent tenant isolation and lifecycle controls.
• Pre-fetched search result blocks: useful when the gateway assembles retrieved text and passes it to a provider that supports explicit citation-aware context.

The adapter should return the same internal structure regardless of backend:

interface RetrievalResult {
  retrievalTraceId: string;
  tenantId: string;
  corpusId: string;
  chunks: Array<{
    sourceId: string;
    title: string;
    text: string;
    urlOrInternalRef?: string;
    chunkId: string;
    offsets?: { page?: number; byteStart?: number; byteEnd?: number; tokenStart?: number; tokenEnd?: number };
    score?: number;
    metadata: Record<string, string | number | boolean>;
    providerPayload?: unknown;
  }>;
  retrievalUsage: {
    provider: string;
    queryCount: number;
    resultCount: number;
    billableUnits?: number;
  };
}

This lets the generation layer receive context without knowing whether it came from OpenAI vector stores, Pinecone, Weaviate, a database full-text search index, or an internal hybrid retriever.

Tenant Isolation Starts Before the Vector Query

Tenant isolation must not depend on prompt instructions. It must be enforced before retrieval, at the storage boundary and query boundary.

For Pinecone-style systems, the documented multitenancy pattern is one namespace per tenant in serverless indexes. Data-plane operations target a namespace, which simplifies tenant isolation and offboarding because deleting the namespace removes that tenant's records. Pinecone also documents trade-offs between namespaces and metadata filtering: filtering inside a large shared namespace can scan more data, cost more, and perform slower than namespace-scoped queries.

For Weaviate-style systems, multi-tenancy stores each tenant on a separate shard, so one tenant's data is not visible to another tenant. Tenant deletion deletes the associated shard. Weaviate also supports tenant states such as active, inactive, and offloaded, which creates a lifecycle option for rarely used tenants.

Implementation Checklist

• Resolve tenant_id from the authenticated gateway identity, not from a user-supplied body field alone.
• Map tenant_id to a vector namespace, shard, or provider vector store identifier through a server-side registry.
• Reject requests where the API key tenant and requested corpus tenant do not match.
• Keep shared public corpora separate from private tenant corpora.
• Use metadata filtering for document type, language, product area, or date range after the tenant boundary has already been selected.
• Log namespace, shard, corpus_id, retrieval_profile, and retrieval_trace_id for auditability.

Reserve cross-tenant search for explicit administrative workflows with separate authorization, separate indexes, or controlled aggregation paths. Do not make cross-tenant search an accidental side effect of metadata filters.

Normalize Citations as Gateway Objects

Citations are a product contract, not just decoration. A customer support assistant, legal drafting tool, or internal knowledge assistant needs to show why an answer was produced and where the supporting text came from.

The gateway should normalize citation data into its own schema:

{
  "source_id": "doc_123",
  "title": "Refund Terms",
  "url_or_internal_ref": "kb://refund-terms",
  "chunk_id": "chunk_006",
  "offsets": {"page": 2, "byte_start": 4410, "byte_end": 5020},
  "score": 0.79,
  "retrieval_provider": "weaviate",
  "model_provider": "anthropic",
  "model_provider_citation_payload": {}
}

Keep the normalized fields stable and allow provider-specific extensions. Some providers will expose richer citation details than others. Some will cite search result blocks. Some will cite uploaded files. Some will not provide the exact offset format your application wants. The gateway should preserve what exists without pretending every provider has identical citation semantics.

Strict Citation Mode

When citation_required is true, define failure behavior up front. A strict mode can require that every factual paragraph include at least one citation, or that the final answer contain citations from retrieved chunks above a minimum score threshold. If the selected model provider cannot satisfy the citation contract, the gateway should fail fast, use a compatible provider, or return a structured refusal.

This is a recommendation, not a universal rule. Strict citation mode improves trust, but it can increase refusals, retries, and fallback complexity. For low-risk creative workflows, citations may be optional. For customer-facing support or regulated internal workflows, citation_required should often be part of the retrieval profile.

Index Lifecycle Is a Product Feature

RAG systems accumulate data. Temporary uploads become permanent by accident. Former customers leave behind embeddings. Product teams change chunking strategies and forget to rebuild old indexes. A gateway should make lifecycle controls explicit.

Recommended lifecycle controls include:

• Temporary corpus expiration: documents uploaded for a short-lived session should have an expiration timestamp and a deletion job.
• Tenant offboarding: deleting a tenant should enqueue deletion of namespaces, shards, provider vector stores, and related file objects.
• Cold tenant handling: where supported, inactive tenants can be marked inactive or offloaded to reduce resource use.
• Reindexing version control: store embedding model, chunking policy, parser version, and indexed_at for each chunk.
• Deletion status exposure: Partner API workflows should show whether document deletion, vector deletion, and provider-side deletion completed.

The important fact is that some vector store content is retained until deleted. The architecture recommendation is to make deletion visible and testable instead of burying it in an asynchronous job with no customer-facing state.

Track Three Cost Ledgers

A single token ledger is not enough for RAG. A gateway needs at least three ledgers:

• Embedding and indexing cost: document parsing, chunking, embedding calls, file storage, index writes, and reindexing.
• Retrieval cost: vector database reads, native vector store search, reranking, query rewriting, and result expansion.
• Generation cost: input tokens from user messages and retrieved context, output tokens, tool calls, retries, and fallbacks.

This is especially important for agencies, SaaS vendors, and internal platform teams that resell or allocate AI costs. Without separate ledgers, RAG margins become hard to explain. A tenant with small generation usage may still be expensive if it constantly uploads documents, reindexes large corpora, or runs broad retrieval queries.

Each ledger event should include tenant_id, customer_id if different, API key id, retrieval_profile, corpus_id, model, provider, trace_id, and billable units. This lets usage analytics answer practical questions: which tenants have expensive retrieval profiles, which corpora are stale, which models produce citation failures, and which customers are generating oversized prompts because retrieval returns too much context.

Failure Modes to Test

A gateway RAG subsystem should have tests for the failure modes that create customer-visible damage:

• Missing citations: citation_required is true, but the provider response contains no usable citation references.
• Stale indexes: a document was updated or deleted, but old chunks still appear in retrieval results.
• Tenant mismatch: the request resolves to tenant A, while the corpus or namespace belongs to tenant B.
• Over-broad retrieval: the profile returns too many chunks, increasing cost and diluting answer quality.
• Chunk size mismatch: chunks are so large that citations are imprecise, or so small that context loses meaning.
• Provider feature mismatch: one model can emit citations in the required shape while another cannot.
• Lifecycle failure: deletion is requested but provider-side storage remains active or unverified.

These tests should run at the gateway contract level, not only inside one provider adapter. The goal is to prove that the public behavior remains stable when the retrieval backend or generation provider changes.

Trade-Offs

Native provider retrieval can reduce application code and accelerate a first version. The trade-off is that storage lifecycle, citation format, query controls, and feature availability may become tied to one provider.

External vector databases add operational surface area. The benefit is stronger portability across OpenAI-compatible models, Anthropic models, and future providers. They also make tenant-scoped namespaces or shards easier to reason about when the gateway is responsible for billing and offboarding.

Fine-grained chunks improve citation precision and auditability. They also increase index size, retrieval volume, and prompt assembly complexity. Coarse chunks are simpler, but they may produce citations that point to a broad page or section rather than the exact supporting passage.

Strict citation-required mode improves user trust. It also forces the gateway to handle models that cannot produce the required citation format, which may mean refusing the request, changing models, or returning an answer with a lower confidence state.

Prediction: Retrieval Will Become More Native, But Gateways Still Need Their Own Contract

Provider-native retrieval features are likely to become more capable. More models will accept retrieved context with structured source metadata. More APIs will expose ranking controls, query rewriting, and citation settings. That does not remove the need for a gateway contract.

The gateway still owns tenant identity, key management, spend limits, usage analytics, Partner API workflows, and customer-facing deletion promises. Provider features can be used behind the adapter layer, but the product should not force every tenant, model, and billing workflow into one provider's retrieval abstraction.

Actionable Conclusion

Build multi-tenant RAG as a gateway subsystem with explicit boundaries. Resolve tenant identity before retrieval. Use tenant-scoped namespaces, shards, or vector stores. Keep retrieval behind adapters. Normalize citations into a gateway-owned schema. Add lifecycle states and deletion verification. Track embedding, retrieval, and generation costs separately.

This architecture keeps RAG grounded without locking the product to one retrieval provider. It also gives teams the operational controls they need when an AI assistant moves from a prototype to a customer-facing system: isolation, citations, portability, lifecycle management, and cost attribution.

Related reading

• trace retrieval, token usage, and tenant analytics in one observability model
• normalize provider-specific outputs behind a stable gateway contract
• connect tenant provisioning and usage metering to Partner API workflows