Оперативная изоляция кэша в шлюзах AI API: почему общие ключи провайдера могут привести к утечке сигналов между арендаторами

When an AI API gateway routes many tenants through one upstream provider account, prompt caching can stop being a purely local cost feature. The cache may be scoped by the provider identity behind the gateway rather than by the tenant key that called the gateway. That creates a practical failure mode: tenant B may receive cache-read signals caused by tenant A's earlier prompt, even though both tenants used separate gateway API keys.

The risk is not that a provider returns another tenant's text as the model answer. The risk is subtler: cache-hit metadata, latency differences, billing attribution, and debugging traces can reveal that a long prefix has been seen before under the same upstream identity. For private system prompts, customer documents, tool definitions, policy packs, or reseller workloads, that signal may be enough to violate an isolation expectation.

What Is Known, What Is Assumed, and What Must Be Tested

Fact: major model providers expose prompt or context caching in different forms. OpenAI prompt caching is automatic for supported models once prompts pass a token threshold, and usage metadata can report cached input tokens. Anthropic supports automatic and explicit prompt caching controls with cache-control breakpoints and TTL choices. Gemini documentation describes context caching and reports cached token usage through usage metadata.

Fact: recent research has examined shared cache state and credential pooling in multi-tenant LLM gateways. One paper reports that tested open-source gateways did not bind customers to upstream credentials by default and that shared credentials exposed cross-customer cache reads in the authors' experiments. Another paper frames shared KV-cache state as a timing side-channel risk in multi-tenant inference.

Recommendation: do not assume that a gateway API key automatically becomes the provider's cache boundary. Treat cache scope as an explicit design decision, the same way you treat logging scope, billing scope, and rate-limit scope.

Prediction: prompt-cache isolation will become a normal procurement and security-review question for AI infrastructure. The cost savings are real, but the boundary is easy to misunderstand when gateways abstract provider accounts away from application teams.

The Failure Mode: Tenant Keys Are Not Always Cache Keys

A typical multi-tenant gateway has at least two credential layers:

• Downstream credentials: the API keys issued by the gateway to tenants, teams, services, or reseller customers.
• Upstream credentials: the provider accounts, projects, workspaces, or API keys the gateway uses to call model providers.

Many gateways centralize upstream credentials because it simplifies purchasing, quota management, failover, and operational support. That is reasonable for some workloads. The mistake is treating this simplification as invisible. If a provider's prompt cache is associated with the upstream account, project, organization, workspace, or key, then many downstream tenants may share one cache scope.

Consider a gateway with two customer tenants:

• Tenant A sends a 6,000-token support-policy prompt containing private procedural content.
• The gateway routes the request through provider key provider-prod-1.
• Five minutes later, tenant B sends a probe that includes the same long prefix, intentionally or accidentally.
• The gateway again routes through provider-prod-1.
• The provider response reports cached input tokens, or the request returns faster than a cold prefix would normally return.

Tenant B may not learn the entire prompt from the response. But tenant B can learn that the prefix existed in the shared cache scope. In some environments, that is already sensitive. A reseller platform, for example, may host competing customers with similar document structures. A malicious or curious tenant could test for specific policy text, contract fragments, internal tool schemas, or customer-specific system instructions.

Why Ordinary Gateway API-Key Management Is Not Enough

Per-tenant API keys inside the gateway are still necessary. They support authentication, authorization, spend limits, usage analytics, revocation, and audit trails. They do not automatically control every upstream side effect.

Prompt caching is one of those side effects. Others include provider rate limits, abuse-monitoring state, batch job namespaces, file storage, fine-tuning resources, and hosted tool state. If the gateway collapses many tenants into one upstream identity, the provider may see one customer where the gateway sees thousands.

The practical question is not "is prompt caching safe?" The practical question is "what identity does the provider use to decide cache reuse, and does that identity match the isolation promise made by the gateway?" If the answer is unclear, route high-sensitivity workloads through a stricter credential mode until testing proves the boundary.

Define Cache Scope Before Enabling Prompt Caching at Scale

A gateway should have a cache-scope policy that is explicit enough for engineers, security reviewers, finance teams, and reseller partners to understand. Start with four dimensions.

1. Tenant Boundary

Decide whether cache entries may be reused across tenants. For most private application traffic, the default should be no. A tenant may mean a company, workspace, reseller customer, internal department, or regulated data boundary. The important part is that the definition is stable and visible in provisioning records.

2. Workload Class

Not every workload needs the same isolation. Public documentation Q&A, shared coding examples, or generic benchmark prompts may tolerate pooled caching. Legal review prompts, customer support transcripts, private RAG context, internal policies, and agent tool instructions usually should not.

A useful minimum set of classes is:

• public_shared: prompts intentionally safe for cross-tenant reuse.
• tenant_private: prompts may be cached only within one tenant boundary.
• customer_private: prompts must be isolated to an end customer or reseller subtenant.
• no_cache: prompts should avoid provider caching when practical, or should be routed to providers and modes compatible with that expectation.

3. Upstream Credential Strategy

The gateway needs to map each cache-sensitive route to an upstream identity strategy. The three common modes are:

• Provider-pooled: many tenants share gateway-managed provider credentials.
• Tenant-bound: each tenant, workspace, or reseller customer uses a dedicated upstream provider credential or provider project.
• Customer BYOK: the tenant supplies its own provider credential, and the gateway acts as a policy, routing, analytics, and billing layer around it.

Provider-pooled mode gives the best purchasing leverage and may improve cache hit rates. Tenant-bound mode gives clearer cache isolation but increases provider account management overhead. BYOK gives customers the clearest provider-level control, but it raises onboarding and support complexity.

4. Prompt Structure

Some teams try to isolate caches by adding a tenant-specific salt or identity block to the prompt. This can work only if it is placed deliberately. Put it too early and it destroys reuse of stable public prefixes. Put it too late and private shared prefixes may still be cacheable across tenants.

A safer pattern is to structure prompts in ordered regions:

region 1: provider-neutral product instructions that are safe to share
region 2: tenant or customer isolation marker
region 3: tenant-private system instructions
region 4: request-specific user content and retrieved context

This preserves reuse for genuinely shared content while preventing private regions from sharing a cache prefix across tenants. The exact placement depends on provider cache semantics, especially whether caching is automatic, breakpoint-based, prefix-based, TTL-based, or controlled through a separate cache resource.

Credential Modes and When to Use Them

Provider-Pooled Mode

Use pooled credentials for low-sensitivity workloads where tenants intentionally share the same prompt surface or where the business value of pooling is greater than the isolation risk. Examples include shared evaluation prompts, public documentation classification, public code examples, and internal platform tests.

Implementation details:

• Mark the route with cache_scope=provider_pool.
• Restrict allowed workload classes to public_shared or similarly low-sensitivity labels.
• Record the provider credential ID in the usage ledger.
• Expose cache-hit metrics by tenant and route so unexpected hits can be investigated.

Tenant-Bound Mode

Use tenant-bound credentials when tenants expect private prompts, private system instructions, or private retrieved context. This mode is usually the right default for business applications running unrelated customers through one gateway.

Implementation details:

• Assign a stable provider credential, provider project, or provider workspace to each tenant boundary.
• Route cacheable requests through that identity only.
• Block fallback to pooled credentials unless the workload is explicitly downgraded to a shared class.
• Include credential provenance in audit events and billing records.

Customer BYOK Mode

Use BYOK for regulated customers, security-conscious enterprises, reseller platforms, and customers that need provider-level auditability. BYOK does not remove the need for gateway controls; it changes the upstream trust boundary.

Implementation details:

• Store customer credentials in a vault with scoped runtime access.
• Validate provider capabilities before enabling prompt caching features.
• Separate credential validation errors from model errors in logs and customer-facing diagnostics.
• Support rotation without changing tenant API keys or application base URLs.

Add a Black-Box Cache-Isolation Test

Policy is not enough. Add a conformance test that verifies the gateway route behaves as expected. The test should use synthetic content, never real customer prompts.

A practical test flow:

1. Create two gateway tenants: tenant_a and tenant_b.
2. Generate a long synthetic prefix above the provider's cache threshold. Include a unique random marker so the test cannot collide with normal traffic.
3. Send the prefix through tenant_a to warm any eligible cache.
4. Send the same prefix through tenant_b on the route that is supposed to be isolated.
5. Inspect provider usage metadata for cached input tokens where available.
6. Optionally compare latency against cold probes, but do not rely on latency alone.
7. Fail the route if tenant_b observes a cache hit and the expected policy is tenant isolation.

Example test record:

{
  "test_name": "prompt_cache_cross_tenant_probe",
  "route": "chat-default",
  "model_alias": "support-large",
  "tenant_a": "isolation-test-a",
  "tenant_b": "isolation-test-b",
  "expected_cache_scope": "tenant",
  "provider": "example-provider",
  "provider_credential_a": "cred_tenant_a",
  "provider_credential_b": "cred_tenant_b",
  "tenant_b_cached_input_tokens": 0,
  "result": "pass"
}

This test can be probabilistic because provider cache TTLs, regional routing, model updates, and cache admission behavior can vary. Run it repeatedly in CI for gateway adapters and on a schedule in staging. For production, use a dedicated synthetic route and strict rate limits so tests cannot pollute real customer analytics.

Metrics to Record Without Storing Raw Prompts

Cache observability should not require prompt hoarding. The gateway can collect enough evidence using usage counters, route metadata, and synthetic fingerprints.

Record these fields for each request where available:

• Tenant ID, workspace ID, reseller customer ID, and gateway API key ID.
• Internal model alias and resolved provider model.
• Provider credential ID or credential pool ID.
• Workload class and expected cache scope.
• Input tokens, output tokens, cached input tokens, and cache-write tokens where available.
• Cache hit ratio by route, tenant, credential, and model.
• Prompt fingerprint for synthetic probes only, or a privacy-reviewed hash of stable prompt templates when allowed.

OpenTelemetry GenAI semantic conventions include token and cache-read attributes that can support this style of instrumentation. The important operational rule is to make cache metrics joinable to tenant and credential decisions. A global cache-hit counter is useful for finance, but it is not enough for isolation debugging.

Billing and Attribution Implications

Prompt caching changes both cost and attribution. If tenant B receives a cache discount because tenant A warmed the provider cache, the gateway has to decide who benefits. In pooled mode, passing the discount to the immediate requester may be acceptable. In tenant-bound mode, cross-tenant discounts should not occur. In reseller mode, the wrong choice can create disputes because one customer's private workload may influence another customer's bill.

A billing ledger should keep provider-reported cached tokens separate from gateway policy decisions. Do not reduce a tenant invoice solely because cached tokens appeared. First confirm that the cache scope is allowed for that route. A conservative ledger can store:

• Provider raw usage counters.
• Gateway-normalized billable tokens.
• Cache policy applied at settlement.
• Reason code for any cache discount, surcharge, or ignored provider discount.

This separation makes later reconciliation easier and prevents provider implementation details from silently rewriting tenant contracts.

Operational Checklist

• Inventory all routes that use provider prompt caching, automatic context caching, or explicit cache-control features.
• Classify each route by workload sensitivity and expected cache scope.
• Map each expected cache scope to an upstream credential mode.
• Block high-sensitivity workloads from provider-pooled credentials by default.
• Add tenant or customer isolation markers only after checking prompt-prefix behavior.
• Record cache metrics by tenant, route, provider credential, model, and model alias.
• Add black-box cache-isolation probes to gateway conformance tests.
• Keep raw customer prompts out of routine cache diagnostics.
• Expose cache-scope options during reseller or Partner API tenant provisioning.
• Document the cost trade-off: stronger isolation can reduce cache reuse and increase credential operations.

Actionable Conclusion

Prompt caching should be treated as a security and attribution boundary, not only as an AI API cost control feature. The gateway must decide whose identity is allowed to warm and read a cache entry: the provider pool, the tenant, the reseller customer, or the end customer using BYOK.

The practical path is straightforward: classify workloads, bind sensitive routes to tenant-bound or customer-owned upstream credentials, instrument cache-read metrics without storing raw prompts, and run synthetic cross-tenant cache probes as part of gateway conformance. Pooled credentials still have a place, but they should be an explicit low-sensitivity mode rather than the hidden default for every cacheable prompt.

Related reading

• provider credential vault design
• prompt-cache control across providers
• safe AI usage analytics without raw prompt logging