LLM Observability in a Multi-Model API Gateway: Traces, Token Ledgers, Tenant Analytics, and Safe Prompt Logging

Aggregate request counts and monthly spend are not enough when a customer asks why one workflow became slower, more expensive, or less reliable yesterday. A multi-model API gateway can answer that question if it treats observability as part of the control plane: every request gets a trace, every model call updates a usage ledger, every tenant and workflow is attributable, and sensitive content is protected by default.

This article describes a practical design for AI usage analytics and LLM observability in a gateway that fronts multiple providers through an OpenAI-compatible API. The pattern is useful even if you do not use any specific vendor: instrument once at the gateway, normalize model telemetry, preserve billing attribution, and capture prompt content only under explicit policy.

The reader problem: “Which tenant, model, prompt, or retrieval path caused the change?”

Most teams eventually face the same debugging gap. Application logs show that a feature failed. Provider dashboards show that token usage increased. Finance sees a bill. None of those views, by themselves, explains the full path from tenant request to model call to retrieval context to retry to billed cost.

The goal is not another dashboard with total tokens. The goal is to answer operational questions such as:

• Which tenant or API key caused a spend spike?
• Did latency increase after a model alias changed?
• Are retries or fallbacks double-counting cost?
• Which prompt version burns the most error budget?
• Did a RAG workflow become expensive because retrieval added too many context tokens?
• Can support debug an incident without reading private user prompts?

Facts, recommendations, and predictions

Facts: OpenTelemetry documents Generative AI semantic conventions and attributes for model operations, including operation names such as chat, generate_content, and text_completion. The same documentation warns that GenAI input and output message attributes may contain sensitive information or PII and may require filtering or truncation. Major model providers also expose usage dashboards, APIs, or exports that can support provider-side reconciliation, although the details differ by provider.

Recommendations: Use OpenTelemetry for provider-neutral traces, but keep gateway-owned business dimensions in your own attributes and ledgers. Do not store raw prompts or outputs by default. Store metadata, hashes, token counts, prompt template IDs, schema names, error classes, and safety labels first. Add content capture only as an opt-in, access-controlled, short-retention debugging feature.

Prediction: LLM observability will become less about isolated provider dashboards and more about cross-provider control planes. Teams will expect one place to investigate latency, cost, quality, policy events, tenant behavior, and billing deltas across models.

Reference architecture: observe the whole request path

A gateway can see the full request lifecycle without requiring every application team to build custom telemetry. A useful trace model starts with one parent span for the incoming customer request and child spans for the steps that affect cost, latency, and quality.

Recommended span structure

• Gateway request span: request accepted, authenticated, authorized, rate-limited, and routed.
• Model call span: provider, model, operation, token usage, response status, and latency.
• Retrieval span: index queried, document IDs or hashed IDs, chunk count, retrieval latency, and context token share.
• Tool call span: tool name, status, latency, error class, and side-effect classification.
• Retry span: retry reason, attempt number, provider status, and incremental cost.
• Fallback span: original model, fallback model, trigger, compatibility policy, and final outcome.
• Guardrail or moderation span: policy invoked, decision, labels, and whether output was blocked or transformed.
• Post-processing span: JSON validation, schema repair, citation checks, or final formatting.

The parent span should carry stable correlation identifiers. The child spans should carry normalized technical attributes. The usage ledger should carry durable billing and analytics records. Avoid forcing all information into metrics labels; high-cardinality values such as tenant IDs, prompt hashes, and document IDs are better stored in traces, logs, or ledger tables and then aggregated into dashboards.

Normalize the metadata captured on every LLM call

Every model request should produce a consistent record, regardless of provider. The exact schema will vary, but a practical minimum looks like this:

{
  "request_id": "req_01J...",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "tenant_id": "tenant_123",
  "team_id": "team_456",
  "app_id": "support_bot",
  "gateway_key_id": "key_789",
  "operation": "chat",
  "provider": "provider_name",
  "model": "provider-model-id",
  "model_alias": "fast-support-chat",
  "prompt_template_id": "refund_policy_v5",
  "prompt_hash": "sha256:...",
  "response_schema": "support_answer_v2",
  "status": "completed",
  "error_class": null,
  "latency_ms": 1842,
  "input_tokens": 2110,
  "output_tokens": 384,
  "cached_input_tokens": 1200,
  "estimated_cost_usd": "0.00492",
  "final_billed_cost_usd": null,
  "finish_reason": "stop",
  "retry_count": 0,
  "fallback_used": false,
  "content_capture_policy": "metadata_only"
}

Keep two ideas separate: telemetry explains what happened, while the usage ledger records what should be charged, reconciled, and reported. They reference each other with request IDs and trace IDs, but they do not have to live in the same storage system.

Build a token and cost ledger, not just counters

Token counters are useful for charts, but they are not enough for billing or incident investigation. A ledger should represent state transitions. Create a row when the gateway accepts a request, then update it as the request progresses.

Useful ledger states

• accepted: authentication and policy checks passed.
• forwarded: the request was sent to a provider.
• streaming: the provider began returning tokens.
• completed: the response finished successfully.
• user_aborted: the client disconnected before completion.
• retried: an additional provider attempt was made.
• fallback_used: a different model or provider was selected after failure or policy match.
• failed: the request ended without a usable response.
• reconciled: provider-side usage or cost data was compared and applied.

This state model helps catch common billing and analytics errors: streamed responses where the client disconnected, retry attempts that were charged by the provider but hidden from the user, fallback paths that counted the wrong model, and cache accounting differences across providers.

Use OpenTelemetry GenAI conventions, then extend carefully

OpenTelemetry GenAI semantic conventions provide a portable vocabulary for model operations. Use those conventions for common attributes such as operation name, provider, model, request parameters, response finish reasons, token usage, and error status where they apply.

However, provider-neutral conventions will not cover every business dimension in a gateway. Add gateway-owned attributes or ledger columns for:

• tenant ID, team ID, reseller customer ID, and app ID;
• gateway API key ID and key scope;
• billing plan, spend limit, and budget policy;
• model alias and routing policy version;
• prompt template ID and prompt version;
• workflow name and workflow step;
• estimated cost, final billed cost, and reconciliation status.

The trade-off is cardinality. These fields are valuable for investigation, but they can make metrics expensive and noisy if used as metric labels everywhere. A practical rule is: low-cardinality aggregates go to metrics; high-cardinality identifiers go to traces, logs, and ledgers.

Design safe prompt and output logging

Full prompt logging makes debugging easier, but it increases privacy, compliance, storage, and insider-risk exposure. The safer default is metadata-first observability.

Default: metadata only

For most production traffic, store:

• prompt template ID and version;
• hashes of normalized prompts and outputs;
• input, output, cached, and context token counts;
• response schema name and validation outcome;
• safety labels and policy decisions;
• error summaries and provider error classes;
• retrieval metadata, not raw documents.

Opt-in: controlled content capture

If you need raw or redacted content for deep debugging, require an explicit policy. Good controls include environment allowlists, tenant consent, sampling, maximum payload length, automatic redaction, short retention windows, encryption, role-based access, audit logs, and a break-glass approval path for sensitive incidents.

Do not treat redaction as perfect. It reduces risk; it does not eliminate it. For regulated or high-sensitivity workloads, consider storing only hashes and replaying issues in a synthetic harness with approved test data.

Add RAG observability as a separate layer

Retrieval-augmented generation can change both quality and cost. Logging only the final model call hides the root cause when the retriever returns too many chunks, stale documents, or irrelevant context.

For each retrieval step, capture:

• index or collection name;
• retrieval strategy and embedding model;
• document IDs or hashed IDs;
• chunk count and total context tokens;
• retrieval latency;
• top score distribution, if available;
• citation coverage;
• whether retrieved context was used in the final answer.

This lets you distinguish “the model got worse” from “the retriever started sending low-quality or excessive context.” It also helps identify workflows where context tokens dominate total cost.

Reconcile gateway usage with provider billing

Gateway estimates are available immediately. Provider-side billing data is usually slower but more authoritative. Use both.

A daily reconciliation job should compare gateway ledger rows with provider usage APIs, cost APIs, dashboard exports, or invoice exports. Group deltas by provider, model, project, and time window. Track differences separately for input tokens, output tokens, cached tokens, request counts, and cost.

Common reconciliation differences

• Streaming disconnects: the gateway may see an aborted client while the provider still bills generated tokens.
• Retries: multiple attempts may be billed even if only one final response is returned.
• Prompt caching: providers may expose cached-token accounting differently.
• Rounding: small per-request differences can become visible at scale.
• Batch or tier discounts: provider invoices may apply pricing that the real-time estimate did not know yet.
• Provider-side changes: model pricing, tokenization behavior, or billing exports can change over time.

When reconciliation finds a delta, avoid silently overwriting your ledger. Store the original estimate, the provider-reconciled value, the reconciliation source, and the reason code if known.

Dashboards that answer operational questions

Start dashboards from reader problems, not vanity metrics. Useful views include:

• cost per tenant, team, app, and workflow;
• cost per successful task, not just cost per request;
• p50, p95, and p99 latency by provider, model, and model alias;
• fallback rate and retry rate by route;
• timeout rate and provider error class trends;
• cache hit ratio and cached-token savings estimate;
• structured-output validation failure rate;
• top prompt versions by error budget burn;
• RAG context token share by workflow;
• guardrail blocks and prompt-injection classifier hits.

For alerting, combine technical and business signals. A sudden tenant spend spike may be more urgent than a small global latency increase. A fallback-rate jump after a model alias change may indicate a compatibility problem. Repeated 401, 429, or 5xx responses may point to key issues, quota exhaustion, or provider instability.

Minimal implementation flow for an OpenAI-compatible proxy

For a /chat/completions proxy, the flow can be simple:

1. Receive the request and assign request_id and trace context.
2. Authenticate the gateway key and resolve tenant, team, app, and policy scope.
3. Create the parent gateway span.
4. Create a ledger row with state accepted.
5. Resolve model alias to provider model and routing policy version.
6. Record metadata: operation, prompt template ID, schema name, prompt hash, and content capture policy.
7. Start the model call span using GenAI semantic attributes where applicable.
8. Forward the request to the selected provider.
9. For streaming, update state when the first chunk arrives and count usage as accurately as the provider response allows.
10. On completion, parse provider usage, finish reason, status, and error class.
11. Update the ledger with tokens, estimated cost, retry/fallback details, and final request state.
12. Emit metrics from the ledger and span data.
13. Run daily reconciliation and store provider-confirmed cost separately from the original estimate.

Rollout checklist

• Define canonical request IDs and trace IDs.
• Adopt OpenTelemetry GenAI attributes for common model telemetry.
• Create a gateway usage ledger with request state transitions.
• Normalize provider, model, model alias, tenant, app, and workflow dimensions.
• Keep high-cardinality investigation data out of metric labels.
• Make raw prompt and output capture disabled by default.
• Add explicit policies for sampling, redaction, retention, and access control.
• Capture retrieval metadata for RAG workflows.
• Build dashboards for cost, latency, reliability, validation, and tenant behavior.
• Reconcile gateway estimates with provider usage and cost exports.
• Alert on spend spikes, latency regressions, fallback jumps, validation failures, and security-relevant events.

Conclusion

A multi-model gateway is the right place to implement LLM observability because it sees requests before they reach any provider and can attach business context that providers do not know. The strongest design is not “log everything.” It is a layered model: provider-neutral traces for execution, a durable token and cost ledger for billing, tenant analytics for governance, RAG metadata for retrieval quality, and privacy-first prompt logging for safe debugging.

Start with metadata, state transitions, and reconciliation. Add content capture only when the policy, retention, and access controls are ready. That sequence gives developers the evidence they need to debug latency, quality, and spend without turning observability into a new data exposure risk.

Related reading

• retry and fallback behavior in LLM API routing
• prompt caching and batch cost-control patterns
• team API key management and spend limits