Streaming Token Accounting in an AI API Gateway: Final Usage, Cancellations, and Partial Responses

Streaming LLM responses are easy to proxy and hard to bill correctly. If an AI API gateway forwards server-sent events to the client but treats the first chunks as the usage record, tenant analytics will drift. The drift usually appears in disputes such as: “the user only saw half the answer,” “the provider billed more than our dashboard shows,” “quota was released too early,” or “a timeout produced tokens but no invoice line.”

The root problem is that streamed calls are not one event. They are a sequence: request accepted, upstream stream opened, bytes delivered, final usage reported, provider stopped, client disconnected, gateway timed out, and billing settled. A reliable gateway should model those states explicitly instead of assuming that a completed HTTP response is the only successful path.

The failure mode: streaming hides the accounting boundary

Non-streamed completions usually return one response object with usage metadata. A gateway can normalize that usage, write a ledger row, update quota, and emit analytics in one pass.

Streaming changes the boundary. The user experience is incremental, but the billing truth may arrive at the end, in a provider-specific final event, in a cumulative delta, through an aggregated SDK response, or later through provider reporting APIs. If the client disconnects before the final usage event, the gateway may have delivered only part of the answer while the provider still generated and billed more tokens.

Fact: OpenAI documents that streaming callers who want usage data should set stream_options with include_usage. OpenAI also provides organization-level usage and cost endpoints, while noting that usage and costs may not always reconcile perfectly for financial purposes.

Fact: Anthropic streaming uses server-sent events such as message_start, content_block_delta, message_delta, and message_stop. Its message_delta usage information is cumulative, so a gateway must not add each usage delta together.

Fact: Gemini and Vertex-style streaming APIs can expose incremental chunks while SDKs may also provide an aggregated response object. For gateways, that aggregated path can be a better source for completed usage than the visible chunks alone.

Use a stream state machine, not a boolean success flag

A streamed request should have a durable usage record before the upstream call starts. That record should move through explicit states. A practical minimum is:

• accepted: the gateway authenticated the key, attributed the tenant, and created an open ledger row.
• first_byte_sent: at least one output event reached the downstream client.
• provider_completed: the upstream provider emitted a normal stop signal or completed response object.
• client_aborted: the downstream socket closed before normal gateway completion.
• provider_error: the upstream provider returned an error after the stream began or before final usage arrived.
• gateway_timeout: the gateway enforced its latency budget and ended the request.
• settled: the gateway converted usage into tenant cost and quota consumption.
• reconciled: later provider usage or cost data confirmed or adjusted the row.

This model prevents a common analytics bug: marking every stream that produced text as “successful and exact.” A stream can be useful to the user, incomplete from the provider, estimated for billing, and pending reconciliation at the same time.

Recommended ledger fields

Keep the request-time row small but explicit:

{
  "request_id": "gw_req_...",
  "tenant_id": "tenant_123",
  "api_key_id": "key_456",
  "provider": "openai|anthropic|gemini|...",
  "provider_request_id": null,
  "model": "provider-model-id",
  "state": "accepted",
  "stream": true,
  "input_tokens": null,
  "output_tokens_billed": null,
  "output_tokens_delivered_estimate": 0,
  "provider_usage_source": null,
  "billing_status": "pending_reconciliation",
  "client_abort_at": null,
  "provider_completed_at": null,
  "settled_at": null,
  "error_class": null
}

The important separation is output_tokens_billed versus output_tokens_delivered_estimate. Users care what reached their application. Finance cares what the provider billed. Those numbers can differ after disconnects, tool-call streams, hidden reasoning tokens, cached tokens, safety stops, or gateway timeouts.

Provider-specific capture rules

A provider-neutral OpenAI-compatible API is useful for application developers, but the gateway adapter still needs provider-specific accounting rules.

OpenAI-compatible streaming

For OpenAI routes, expose a gateway option that enables upstream usage reporting where supported. A common pattern is to accept a gateway-level default such as:

{
  "stream": true,
  "stream_options": {
    "include_usage": true
  }
}

If the downstream caller omits it, the gateway can decide whether to inject it for routes where doing so is compatible. Document this behavior because some clients expect exact wire compatibility and some models or upstreams may not support final usage in the same way.

Recommendation: do not settle tenant cost from early chunks. Keep the ledger row open until the final usage event is captured, the provider response ends without usage, or the stream enters an error or cancellation path.

Anthropic streaming

Anthropic’s cumulative usage requires a different rule. If a gateway sees three message_delta events with output token counts of 10, 25, and 40, the output count is 40, not 75.

let latestUsage = null;

for await (const event of anthropicStream) {
  if (event.type === "message_delta" && event.usage) {
    if (latestUsage && event.usage.output_tokens < latestUsage.output_tokens) {
      emit("cumulative_usage_regressed", requestId);
    }
    latestUsage = event.usage;
  }

  forwardToClient(event);
}

settleFromLatestCumulativeUsage(latestUsage);

Recommendation: record the latest cumulative usage value and emit an observability event if it regresses. A regression may indicate parser bugs, duplicated events, provider changes, or mixed streams.

Gemini and Vertex-style streaming

Gemini supports streaming chunks to reduce perceived latency. In Vertex-style SDKs, streaming can expose both an async stream and an aggregated response object. A gateway should preserve that aggregated path when available.

const streamingResult = await model.generateContentStream(request);

for await (const chunk of streamingResult.stream) {
  forwardChunk(chunk);
  countDeliveredBytesOrText(chunk);
}

const aggregated = await streamingResult.response;
settleFromAggregatedUsage(aggregated);

Recommendation: avoid building all accounting from visible chunks if the SDK gives a completed response record. Chunks are for latency. The final object is often better for billing and analytics.

Handle client disconnects as first-class accounting events

Client disconnects are where many gateways lose money or overcharge customers. A browser tab closes, a mobile network drops, or an application cancels a request. The gateway notices the downstream socket closed, but the upstream provider may still be generating.

The gateway should make an explicit policy choice:

• Cancel upstream immediately: reduces wasted generation and provider cost, but may break workflows where the backend still needs the result after the UI disconnects.
• Continue upstream in the background: can preserve work for server-side consumers, but the user may not see all tokens generated and billed.
• Route-dependent behavior: cancel for interactive chat, continue for job-like workflows, and make the setting visible to tenants.

A practical default for interactive streaming is to cancel upstream when the downstream client disconnects, then mark the ledger row as client_aborted. If final usage arrives during cancellation, settle from that authoritative usage. If not, mark the row estimated or pending_reconciliation rather than pretending it is exact.

downstream.on("close", async () => {
  if (!providerCompleted) {
    ledger.markClientAborted(requestId);
    await upstream.abort().catch(() => {
      ledger.emit("upstream_cancel_failed", requestId);
    });
  }
});

Recommendation: expose transparent billing labels such as final, provider_reconciled, estimated, waived, or pending_reconciliation. This is more defensible than showing every streamed call as immediately exact.

Quota enforcement during a stream

Accurate billing usually depends on final provider usage, but quota enforcement cannot always wait until the end. A tenant with a hard budget should not be allowed to stream indefinitely because exact usage is unavailable mid-flight.

Use two mechanisms together:

1. Preflight reservation: reserve an estimated maximum based on model, requested max tokens, tenant policy, and current balance.
2. Streaming pressure checks: estimate delivered output during the stream and stop if the request crosses a configured safety boundary.

This is a control mechanism, not the final bill. Providers may count cached tokens, reasoning tokens, multimodal tokens, or hidden tokens differently from a gateway’s estimator.

Trade-off: real-time estimates help enforce budgets, but they can diverge from provider-billed tokens. Final settlement should use authoritative provider usage when available, and reconciliation should adjust estimates later.

Observability events that catch accounting bugs

Streaming billing failures are easier to debug when the gateway emits targeted events instead of only generic request logs. Add events such as:

• final_usage_missing: stream ended without authoritative usage.
• cumulative_usage_regressed: cumulative token count moved backward.
• stream_ended_without_stop_event: no normal provider stop marker was observed.
• aborted_after_provider_completion: the provider completed, but the downstream client closed before the gateway finished forwarding.
• settled_from_estimate: tenant ledger used an estimate because final usage was unavailable.
• reconciliation_adjusted_usage: provider reporting later changed the row.

Fact: OpenTelemetry GenAI semantic conventions recommend using provider-returned usage information for streaming responses when available, and warn against reporting usage metrics if token counts cannot be obtained efficiently or accurately.

For AI usage analytics, this means dashboards should support confidence levels. A chart that mixes final, estimated, and reconciled values without labels may look clean but mislead finance and support teams.

Conformance tests for streaming accounting

Do not rely on manual testing with a happy-path chat prompt. Each provider adapter should have conformance tests for the cases that break ledgers:

• Normal stream: final usage arrives, stop event observed, ledger settles as final.
• Tool-call stream: tool call deltas are forwarded, usage is captured, structured metadata does not corrupt token counting.
• Safety or refusal stop: provider stops early, usage still settles correctly.
• Forced client disconnect: downstream closes after partial output; upstream is cancelled or continued according to policy.
• Upstream 5xx after partial output: gateway records partial delivery and does not mark the request as a clean success.
• Gateway timeout before final usage: row becomes estimated or pending reconciliation.
• Missing final event: adapter emits final_usage_missing and avoids exact billing labels.

These tests should assert state transitions, ledger fields, emitted observability events, and downstream behavior. Byte-for-byte stream compatibility is not enough; the accounting side effects are part of the contract.

Practical implementation checklist

• Create the usage ledger row before dispatching the upstream request.
• Store tenant, key, user, model, route, provider, and request identifiers at request time.
• Enable provider final usage reporting where supported, such as OpenAI-compatible stream_options.include_usage.
• For cumulative providers, store the latest usage value instead of summing events.
• Preserve aggregated response objects when SDKs provide them.
• Track delivered output separately from provider-billed usage.
• On disconnect, cancel upstream according to route policy and mark client_aborted.
• Use transparent billing statuses: final, estimated, pending reconciliation, provider reconciled, or waived.
• Emit accounting-specific observability events.
• Reconcile later against provider usage or cost reports when available, while preserving request-time tenant attribution.

What to show tenants

Tenants do not need every internal event, but they do need honest labels. A useful usage table might show:

• Status: final, estimated, or reconciled.
• Request outcome: completed, client aborted, provider error, or gateway timeout.
• Delivered output: approximate text or bytes sent to the client.
• Billed tokens: provider-normalized usage used for cost.
• Adjustment: any later reconciliation delta.

This design reduces support ambiguity. If a user saw only part of a response, the dashboard can explain whether the provider had already completed, whether the gateway cancelled upstream, and whether the charge is final or estimated.

Recommendations versus predictions

Recommendations: treat streamed requests as state machines, wait for authoritative final usage before exact settlement, separate delivered output from billed usage, and label estimated rows honestly. Provider adapters should encode provider-specific usage semantics rather than flattening every stream into a generic byte proxy.

Prediction: streaming accounting will become more important as models expose more hidden work: reasoning tokens, cached-token discounts, multimodal processing, tool-use traces, and safety stops. Gateways that already separate provider-billed usage from client-visible output will adapt more easily than gateways that only count streamed text.

Actionable conclusion

If your gateway supports streaming, audit one path today: force a client disconnect after the first few chunks and inspect the ledger row. If it says “success” with exact-looking token counts, your analytics are probably lying.

The fix is not to abandon streaming. Keep the fast user experience, but make stream completion, cancellation, provider errors, missing final usage, and reconciliation explicit accounting states. That gives product teams responsive output, finance teams defensible costs, and support teams enough evidence to explain partial responses without guessing.

Related reading

• designing the billing ledger behind every model call
• joining traces, token ledgers, and safe usage analytics
• building an OpenAI-compatible gateway compatibility contract