Kontrollplansrevisionsspår för AI API-gateways: Rekonstruera nyckel-, hyresgäst- och utgiftsändringar under incidenter

When AI spend spikes or a key leaks, runtime telemetry is not enough. Security and FinOps teams need to reconstruct control-plane changes: who created or changed an API key, which tenant or project was affected, which upstream credential was involved, whether a spend limit changed before the spike, and what was revoked during containment.

The practical pattern is a normalized, append-only audit-event pipeline for AI API gateways. Runtime traces answer what a model call did. Control-plane audit logs answer who changed access, budgets, routing, keys, service accounts, tenant status, model profiles, credits, and provider credentials.

Facts, recommendations, and predictions

Facts: Some AI providers expose admin or audit surfaces that cover organization configuration, projects, API keys, service accounts, workspaces, members, usage, and cost reporting. Coverage varies by provider, account type, and deployment context. OpenTelemetry also provides general semantic conventions for events and GenAI telemetry attributes, while warning that captured input and output content can contain sensitive data.

Recommendations: Treat the gateway as the canonical audit layer for gateway-specific concepts: virtual keys, tenants, reseller customers, wallets, spend limits, model profiles, Partner API automation, upstream credential pools, and emergency quarantine actions. Correlate provider admin events where available, but mark gaps explicitly instead of pretending every upstream console action is observable.

Predictions: AI control-plane audits will become a standard requirement for enterprise AI gateways, especially where teams resell model access, automate customer provisioning, or manage high-risk service accounts. Provider audit APIs will improve, but gateways will still need their own normalized event model because provider resource concepts will remain different.

Define the investigation questions first

An audit trail should be designed backward from the queries responders will run during an incident. Start with a short list of questions:

• Which keys were created, updated, re-enabled, or revoked before the anomaly?
• Which tenant, customer, project, workspace, wallet, or cost center did each key affect?
• Which actor made the change: human admin, service account, Partner API client, SCIM process, support operator, or scheduled automation?
• Which authentication method was used: SSO session, admin API token, service account, break-glass account, or webhook replay?
• Did spend limits, model access, routing profiles, rate limits, or credit balances change before usage increased?
• Which upstream provider credential, project, service account, or workspace handled the resulting traffic?
• What containment actions happened, and were they complete?

If the gateway already has API key management, billing, team controls, and provider routing, the audit trail should join those systems rather than create another isolated log stream.

Use a normalized audit event schema

Do not store arbitrary JSON blobs and hope analysts can interpret them later. Use a stable schema for every control-plane event, then allow source-specific extensions under a controlled namespace.

Core fields

{
  "event_id": "evt_01J...",
  "occurred_at": "2026-09-09T13:51:22.481Z",
  "ingested_at": "2026-09-09T13:51:23.040Z",
  "source_system": "gateway_admin_api",
  "action": "key.issued",
  "result": "success",
  "risk_level": "high",
  "actor_type": "service_account",
  "actor_id": "sa_partner_provisioner_42",
  "actor_org_id": "org_acme_reseller",
  "auth_method": "partner_api_key",
  "resource_type": "virtual_api_key",
  "resource_id": "vk_123",
  "tenant_id": "tenant_789",
  "customer_id": "cust_456",
  "project_id": "proj_support_bot",
  "workspace_id": null,
  "key_id_hash": "sha256:...",
  "upstream_provider": "provider_a",
  "upstream_credential_id_hash": "sha256:...",
  "request_correlation_id": "req_abc",
  "idempotency_key_hash": "sha256:...",
  "ip_address": "203.0.113.10",
  "user_agent_hash": "sha256:...",
  "policy_before_hash": null,
  "policy_after_hash": "sha256:..."
}

The schema should support both human-readable investigation and machine joins. The most important fields are action, actor, resource, tenant, key fingerprint, upstream credential lineage, correlation ID, and policy snapshot hashes.

Why hashes instead of raw values?

Audit trails need stable join points, not secrets. Store a deterministic hash or fingerprint for virtual keys and upstream credentials. Store hashes of policy snapshots, with the redacted snapshot available only to authorized audit viewers. This allows investigators to prove that a spend policy, routing profile, or key scope changed without exposing raw credentials or sensitive internal configuration in every exported log.

Separate runtime telemetry from control-plane audit logs

AI gateways often collect runtime metrics: request ID, model, provider, token counts, latency, cost estimate, final cost, cache behavior, safety signals, and error status. These records are essential for AI API usage analytics, but they are not the same as control-plane audit events.

A runtime record might say:

{
  "request_id": "req_model_001",
  "tenant_id": "tenant_789",
  "key_id_hash": "sha256:...",
  "model_profile_id": "profile_support_premium",
  "provider": "provider_a",
  "started_at": "2026-09-09T13:56:10Z",
  "estimated_cost_usd": "0.18",
  "settled_cost_usd": "0.22",
  "status": "success"
}

A control-plane audit event says:

{
  "event_id": "evt_profile_002",
  "action": "model_profile.changed",
  "tenant_id": "tenant_789",
  "resource_id": "profile_support_premium",
  "actor_id": "sa_partner_provisioner_42",
  "occurred_at": "2026-09-09T13:54:03Z",
  "policy_before_hash": "sha256:old",
  "policy_after_hash": "sha256:new",
  "risk_level": "high"
}

Both records need a shared join strategy, but they should have different retention, access controls, and payload rules. Control-plane audit logs should not be sampled. Runtime traces may be sampled for observability cost, while usage and billing ledgers should remain complete.

Make gateway-issued virtual keys the join point

In a multi-tenant gateway, the virtual key is the most useful investigation handle. A gateway-issued key can point to tenant, customer, user, service account, model profile, spend limit, routing policy, rate limit, upstream provider credential, and billing wallet.

During an incident, responders should be able to pivot from a key fingerprint to:

• the tenant and customer that owned the key;
• the actor that created or last modified it;
• the scope, model profile, and spend limit active at each point in time;
• the upstream provider credentials used for calls made with that key;
• usage and cost records generated by that key;
• revocation, rotation, quarantine, and restoration events.

This is especially important for platforms that issue customer-scoped keys through Partner API automation. Without a stable virtual-key fingerprint, provisioning events, credit changes, tenant configuration, and runtime usage become separate stories.

Capture first-class Partner API events

Partner APIs create audit risk because they automate actions that used to happen manually in an admin console. They also create commercial disputes: a reseller may ask who topped up credits, when a customer key was issued, or why a tenant exceeded a budget.

At minimum, record these events as first-class audit actions:

• customer.created
• customer.updated
• key.issued
• key.scope_changed
• key.revoked
• credit.topup.requested
• credit.topup.settled
• spend_limit.changed
• rate_limit.changed
• model_profile.changed
• tenant.quarantined
• tenant.unquarantined
• webhook.replayed
• idempotency_key.reused

For money-related events, store exact decimal amounts, currency, previous balance, new balance, external payment reference, and settlement status. For idempotent operations, store the idempotency key hash and the canonical operation ID so duplicate retries are explainable later.

Correlate provider admin logs without depending on them

Provider control planes are not uniform. Some providers expose audit logs and admin APIs for organization settings, projects, keys, users, service accounts, workspaces, usage, and costs. Others require console exports, cloud audit logs, billing reports, or manual reconciliation. Even when APIs exist, their resource names and retention policies differ.

A gateway audit architecture should therefore use capability flags:

{
  "provider": "provider_a",
  "capabilities": {
    "admin_audit_api": true,
    "project_key_inventory": true,
    "service_account_inventory": true,
    "workspace_cost_report": false,
    "credential_last_used_at": true
  }
}

When importing provider events, normalize them into the same audit table but preserve source metadata:

• source_system: provider_admin_api, provider_console_export, cloud_audit_log, billing_report
• source_event_id: original provider event ID where available
• source_resource_path: original project, workspace, key, or service-account path
• provider_capability_gap: true when the gateway inferred context rather than receiving a direct event

The goal is not to claim perfect visibility. The goal is to make coverage and gaps explicit so incident reports can distinguish observed facts from inferred relationships.

Use append-only storage and correction events

Audit events should be immutable. If an event was parsed incorrectly, write a correction event rather than editing the original row. This matters for billing disputes, reseller credit disputes, leaked-key investigations, and post-incident reviews.

A practical storage design uses three layers:

• Hot query store: indexed by tenant, actor, action, key hash, request correlation ID, and occurred_at for fast incident queries.
• Immutable archive: append-only object storage or ledger-style table with retention controls and integrity checks.
• Export stream: SIEM, warehouse, or event bus export for enterprise security workflows.

Export is useful, but do not make the SIEM the only authoritative store. A generic SIEM may not understand gateway-specific concepts such as virtual key, tenant wallet, model profile, reseller customer, or upstream credential pool. Keep the gateway ledger authoritative and export normalized copies.

Build an incident query pack

Incident response should not start with ad hoc SQL. Pre-build saved queries for the most common scenarios.

1. Keys created before a spend spike

SELECT occurred_at, actor_type, actor_id, tenant_id, customer_id,
       resource_id, key_id_hash, auth_method, ip_address
FROM audit_events
WHERE action IN ('key.issued', 'key.reenabled', 'key.scope_changed')
  AND tenant_id = :tenant_id
  AND occurred_at BETWEEN :spike_start - INTERVAL '24 hours' AND :spike_start
ORDER BY occurred_at;

2. Spend-limit changes before abnormal usage

SELECT occurred_at, actor_id, resource_id,
       policy_before_hash, policy_after_hash, request_correlation_id
FROM audit_events
WHERE action = 'spend_limit.changed'
  AND tenant_id = :tenant_id
  AND occurred_at BETWEEN :window_start AND :window_end
ORDER BY occurred_at;

3. Upstream credentials touched during the window

SELECT occurred_at, action, actor_id, upstream_provider,
       upstream_credential_id_hash, source_system
FROM audit_events
WHERE upstream_credential_id_hash IS NOT NULL
  AND occurred_at BETWEEN :window_start AND :window_end
ORDER BY occurred_at;

4. Model-profile promotions to expensive routes

SELECT occurred_at, actor_id, tenant_id, resource_id,
       policy_before_hash, policy_after_hash
FROM audit_events
WHERE action = 'model_profile.changed'
  AND risk_level IN ('medium', 'high', 'critical')
  AND occurred_at BETWEEN :window_start AND :window_end;

5. Containment verification

SELECT occurred_at, action, actor_id, tenant_id, resource_type,
       resource_id, result
FROM audit_events
WHERE tenant_id = :tenant_id
  AND action IN ('key.revoked', 'tenant.quarantined',
                 'upstream_credential.rotated', 'spend_limit.changed')
  AND occurred_at >= :containment_start
ORDER BY occurred_at;

These queries should be available to security, platform, and FinOps teams with role-based access. Not every investigator needs to see policy snapshot details or payment references.

Example investigation timeline

This is a fictional timeline showing the shape of the data, not a real incident:

• 13:48 — spend_limit.changed: tenant monthly limit raised from the previous policy hash to a new policy hash.
• 13:51 — key.issued: new customer-scoped key created by Partner API service account.
• 13:54 — model_profile.changed: tenant routing profile changed to allow a more expensive model class.
• 13:56 — runtime usage begins increasing for the new key fingerprint.
• 14:05 — spend anomaly alert fires for the tenant.
• 14:09 — key.revoked: suspicious virtual key revoked.
• 14:13 — upstream_credential.rotated: provider credential rotated as containment.

This timeline is useful because it connects administrative intent, automated provisioning, routing policy, runtime cost, and containment. It does not require storing raw prompts by default.

Privacy and retention trade-offs

More audit detail improves forensics, but it can increase privacy and compliance risk. Prefer metadata, hashes, redacted policy snapshots, and correlation IDs over raw prompts or full request bodies. If a tenant explicitly enables deeper logging, make that a separate policy with its own audit event, retention period, and access controls.

Recommended retention tiers:

• Control-plane audit events: retain unsampled for the full compliance and dispute window.
• Usage and billing ledger: retain complete financial records for invoice, tax, and chargeback requirements.
• Runtime traces: sample or shorten retention based on sensitivity and observability cost.
• Raw prompts and outputs: off by default for audit; enable only through explicit tenant policy where legally and operationally justified.

High-cardinality fields such as tenant IDs, key fingerprints, request IDs, workflow IDs, and actor IDs are expensive in some observability systems. Do not solve that by sampling control-plane audit events. Instead, put complete audit events in the ledger and export selected fields to high-cost observability tools.

Alert on risk, not every change

Real-time alerts for every admin event create noise. Use risk scoring. Alert immediately when:

• a spend limit is increased materially;
• a key is created by an unusual actor or from an unusual network;
• a tenant is unquarantined;
• a model profile changes to allow more expensive or less restricted models;
• an upstream provider credential is created, exported, rotated, or disabled;
• a Partner API idempotency key is reused with a different payload;
• a webhook replay changes customer credit or key state.

Lower-risk events can go to daily review or SIEM correlation. The point is to surface the control-plane changes most likely to explain downstream abuse, cost spikes, or failed containment.

Implementation checklist

• Define canonical action names for keys, tenants, users, service accounts, spend limits, credits, routing profiles, provider credentials, and quarantine.
• Store immutable event IDs and timestamps for occurrence and ingestion.
• Record actor type, actor ID, authentication method, source system, and request correlation ID.
• Use stable hashes for virtual keys, upstream credentials, idempotency keys, user agents, and sensitive policy content.
• Join audit events to runtime usage through tenant ID, key fingerprint, model profile ID, and request correlation ID.
• Import provider admin events where possible, with capability flags and gap annotations.
• Keep control-plane audit events unsampled.
• Use correction events rather than update-in-place edits.
• Export to SIEM or warehouse, but keep the gateway audit ledger canonical for gateway-specific resources.
• Pre-build incident queries for key creation, spend-limit changes, model-profile changes, Partner API credit events, upstream credential changes, and containment verification.

Actionable conclusion

For AI API incidents, the key question is often not “what prompt was sent?” It is “what changed before this happened?” A gateway audit trail should make that answer fast: new keys, changed spend limits, Partner API automation, routing profile updates, provider credential changes, and containment actions in one timeline.

Start with a normalized schema, make virtual-key fingerprints the main join point, keep events append-only, separate runtime telemetry from control-plane history, and design for uneven provider audit coverage. That gives security, platform, and FinOps teams a defensible record without turning the audit system into a prompt warehouse.

Related reading

• spend anomaly incident runbooks