Unified Batch Jobs Through an AI API Gateway: Durable Queues, Provider Adapters, and Tenant-Level Billing

Batch processing should not be treated as a side door around your AI API gateway. If evals, document enrichment, extraction, moderation sweeps, or embedding jobs leave the synchronous request path, they still need tenant controls, cost attribution, retries, auditability, and usage analytics.

The implementation pattern is to make batch execution a first-class gateway subsystem. The gateway should expose one provider-neutral job contract while adapting to OpenAI, Anthropic, Gemini, and future provider batch APIs behind the scenes.

The reader problem: batch APIs are similar in intent, different in operation

Latency-tolerant workloads are a natural fit for batch execution. The hard part is not deciding whether a job can wait. The hard part is operating batch work consistently across providers.

Verified facts: OpenAI’s Batch API is asynchronous, reads requests from an uploaded file, writes responses to an output file, and currently uses a 24-hour processing window. OpenAI lists statuses such as validating, failed, in_progress, finalizing, completed, expired, cancelling, and cancelled. Anthropic’s Message Batches API processes many Messages requests asynchronously, handles each request independently, requires polling, and returns results after processing ends. Anthropic also recommends meaningful custom_id values because result order is not guaranteed. Gemini’s Batch API exposes long-running operation-style methods such as list, cancel, delete, and update methods, and its cancel operation is described as best-effort.

Those differences matter once you add real business requirements:

• Which tenant, customer, project, or API key owns each item?
• Was budget reserved before the job left the gateway?
• Which completed items are billable if the batch expires or is cancelled?
• How are partial failures retried without duplicating successful work?
• How long can result files be retrieved, and what should the gateway store?
• Can a partner build customer-scoped batch processing without exposing upstream provider credentials?

The answer is not to hide every provider difference. The answer is to normalize the operating contract while preserving provider-native metadata for debugging, reconciliation, and support.

Recommended public API: separate batch jobs from synchronous completions

Recommendation: expose batch jobs as their own API surface, not as a special flag on chat completions. A synchronous request and an asynchronous batch job have different lifecycle, billing, retry, and result retrieval semantics.

A practical gateway contract includes these operations:

• create_job: create a draft job owned by a tenant, project, key, or partner customer.
• append_items or upload_manifest: add individual requests with stable item identifiers.
• submit: validate, reserve budget, select provider, dispatch, and lock the submitted manifest.
• get_status: return normalized job and item counts.
• list_results: page through normalized item results, errors, and usage.
• cancel: request cancellation, without promising immediate termination.
• export_usage: export job-level and item-level cost records for analytics or billing systems.

Example public job object:

{
  "job_id": "job_01j7...",
  "tenant_id": "tenant_acme",
  "customer_id": "cust_123",
  "endpoint": "chat.completions",
  "model": "analysis-large",
  "status": "running",
  "counts": {
    "submitted": 50000,
    "completed": 31240,
    "failed": 180,
    "expired": 0
  },
  "cost": {
    "estimated": "184.20",
    "reserved": "205.00",
    "settled": "117.43",
    "currency": "USD"
  },
  "created_at": "2026-08-19T10:00:00Z",
  "submitted_at": "2026-08-19T10:05:00Z",
  "retrieval_deadline": "2026-09-17T10:00:00Z"
}

The public object should not expose provider file IDs, operation names, or raw upstream errors by default. Those belong in operator-facing metadata.

Use durable job records as the source of truth

A gateway-owned batch layer needs durable state before anything is submitted upstream. Do not rely on provider batch records as your only state store. Provider records are necessary, but they do not know your tenant hierarchy, budget reservations, internal model aliases, partner customers, or analytics requirements.

Minimum database model

A useful schema has three levels:

1. Batch job

batch_jobs
- job_id
- tenant_id
- project_id
- customer_id nullable
- api_key_id
- endpoint
- requested_model
- resolved_provider
- resolved_provider_model
- status
- item_count
- estimated_input_tokens
- estimated_output_tokens
- reserved_amount
- settled_amount
- created_at
- submitted_at
- completed_at
- expires_at
- retrieval_deadline
- cancellation_requested_at

2. Batch item

batch_items
- job_id
- item_id
- custom_id
- idempotency_key
- request_hash
- status
- provider_request_index nullable
- estimated_tokens
- actual_input_tokens nullable
- actual_output_tokens nullable
- settled_amount nullable
- result_pointer nullable
- error_code nullable
- retry_of_item_id nullable
- created_at
- settled_at

3. Provider metadata

batch_provider_metadata
- job_id
- provider
- provider_batch_id nullable
- input_file_id nullable
- output_file_id nullable
- error_file_id nullable
- operation_name nullable
- endpoint
- region nullable
- native_status
- native_request_counts jsonb
- last_polled_at
- raw_error_pointer nullable

Keeping provider metadata separate from the public job contract lets the gateway evolve provider adapters without breaking tenant-facing APIs.

Require stable item identifiers before dispatch

Recommendation: generate a gateway job_id and require a per-item custom_id or idempotency key before dispatch. Never reconcile results by order.

Anthropic explicitly warns that result order is not guaranteed and recommends meaningful custom_id values. Even when a provider appears to preserve order, a gateway should not depend on it. Jobs get chunked, retried, cancelled, partially completed, and re-ingested. Ordering assumptions eventually fail.

A safe item identifier format is descriptive but not sensitive:

tenantA.invoice_extraction.2026-08-19.row_000381

Avoid putting raw emails, names, document titles, or customer secrets in identifiers. Store sensitive correlation data inside your own tenant database, not inside provider-visible IDs.

Normalize statuses without erasing provider detail

Provider batch APIs expose different lifecycles. The gateway should normalize them into a small internal state machine that dashboards, billing, and automation can understand.

Recommended normalized lifecycle:

• draft: job exists but is still editable.
• validating: gateway or provider validation is running.
• queued: accepted but not yet processing.
• running: provider is processing items.
• finalizing: provider has finished compute and is preparing result artifacts.
• completed: all accepted items reached terminal success.
• completed_with_errors: some items succeeded and some failed.
• expired: provider window ended before all work completed.
• cancel_requested: tenant asked to cancel, but final billable work is not settled.
• cancelled: cancellation settled.
• failed: job-level failure prevented useful execution.

Do not collapse native provider errors into generic labels too early. Operators still need access to native statuses, validation errors, request counts, file IDs, and operation names when debugging.

Validate against a capability matrix before submission

Recommendation: run preflight validation before budget reservation and provider dispatch. Batch mode is not just synchronous mode with a delay. Some models, endpoints, request features, regions, and tool configurations may not be supported by a provider’s batch API.

Your internal capability matrix should check:

• Supported endpoint: chat, messages, embeddings, moderation, or generation.
• Model eligibility for batch mode.
• Maximum job size, item count, request size, and uploaded file size.
• Whether streaming is prohibited.
• Tool use and function calling support.
• Structured output or JSON schema support.
• Image, audio, or multimodal input support.
• Region and residency constraints.
• Provider retention and result retrieval windows.
• Batch-specific rate limits and queue limits.
• Cancellation semantics.

A good preflight response is specific:

{
  "error": "batch_capability_not_supported",
  "message": "The selected provider batch adapter does not support streaming responses. Remove stream=true or choose a synchronous endpoint.",
  "field": "items[*].request.stream"
}

This is more useful than accepting the job and failing it after an upstream validation pass.

Reserve tenant budget, then settle actual usage

Batch execution complicates billing because the gateway may lose synchronous access to exact usage until result files are available. The safe pattern is quote, reserve, submit, ingest, settle, and reconcile.

Verified facts: OpenAI states Batch API pricing is offered at a discount compared with synchronous APIs, and expired or cancelled batches may still return completed work that is billable. Anthropic notes that high-throughput batch processing can slightly exceed a workspace spend limit, making gateway-side reservation and post-settlement important.

Recommendation: reserve tenant budget before submission using estimated tokens, selected provider price rules, and a safety margin. After results are ingested, settle actual usage at item level. If the estimate was too high, release the unused reservation. If it was too low, apply the tenant’s configured overage policy.

Practical ledger events:

batch.estimated
batch.reserved
batch.submitted
batch.item.settled
batch.item.refunded
batch.cancel_requested
batch.expired
batch.reconciled

The item-level ledger is essential. If 45,000 items complete and 5,000 expire, the tenant should be billed for completed provider work, not for the original manifest as a single undifferentiated blob.

Build provider adapters as translators, not business logic owners

Each provider adapter should know how to transform the gateway job into the provider’s batch format, submit it, poll or retrieve status, download results, and map native outcomes back to normalized records.

Keep tenant policy outside the adapter. The adapter should not decide whether a customer has enough budget, whether a partner customer is suspended, or whether prompts can be stored. Those are gateway decisions.

Adapter responsibilities

• Render provider-specific request manifests.
• Upload input files or create provider operations.
• Store provider identifiers in metadata.
• Map native status to normalized status.
• Retrieve output and error artifacts.
• Parse item-level results.
• Return native usage records when available.
• Surface retryable versus terminal errors.

Gateway responsibilities

• Authenticate tenant and API key.
• Apply team, project, and customer controls.
• Resolve model aliases and provider routing policy.
• Validate batch capabilities.
• Reserve and settle budget.
• Persist job and item state.
• Enforce retention policy.
• Expose analytics and exports.

This separation makes it easier to add a new provider without rewriting billing, analytics, or tenant governance.

Ingest results idempotently

Result ingestion is where many batch systems accidentally duplicate charges or lose partial work. Treat ingestion as a repeatable process. It should be safe to download the same output file twice, process the same provider operation twice, or replay the same webhook event twice.

Recommendation: use item-level idempotency keys and ledger uniqueness constraints. A result for job_id + custom_id should settle exactly once, even if ingestion is retried.

A robust ingestion flow:

1. Acquire a short-lived lock for the job or result artifact.
2. Fetch provider output and error artifacts.
3. Parse records into normalized item result events.
4. Match each record by custom_id or gateway item ID.
5. Write result metadata and usage in a transaction.
6. Create ledger settlement event only if one does not already exist.
7. Update job counts from item states, not from assumptions.
8. Release unused budget reservation when all terminal states are known.

If webhooks are available, verify signatures and protect against replay. If polling is required, use adaptive polling: poll frequently near expected completion, back off during long-running periods, and stop after terminal settlement.

Retry items, not whole jobs

Recommendation: retry at item level whenever possible. Whole-job retries are simple, but they increase duplicate-work risk and make billing harder.

Classify failures before retrying:

• Validation errors: usually terminal until the request is fixed.
• Provider 5xx errors: often retryable with backoff.
• Quota or rate-limit failures: retry only after capacity is available.
• Safety blocks: do not blindly retry; route to policy handling.
• Expired items: may be retried in a new job if the tenant still wants the work and budget allows.

A retry should create a new item linked to the original:

{
  "item_id": "item_retry_002",
  "retry_of_item_id": "item_001",
  "custom_id": "tenantA.eval.row_901.retry_1"
}

Do not resubmit completed items just because they were part of a job that ended as completed_with_errors or expired.

Decide what to store: raw results, pointers, or hashes

Batch systems are tempting places to accumulate prompts and outputs. That may be useful for exports and debugging, but it increases data-retention responsibility.

Recommendation: make storage policy tenant-configurable. For sensitive workloads, store metadata, hashes, usage, and result pointers rather than raw prompts and outputs. For less sensitive workloads, normalized result storage may be acceptable if retention windows, access controls, and deletion workflows are clear.

Track at least:

• Whether raw input was stored.
• Whether raw output was stored.
• Where provider result artifacts live.
• Provider retrieval deadline.
• Gateway deletion deadline.
• Hash of request and response for audit without content exposure.

Verified fact: Anthropic states batch results are available for 29 days after creation and isolated within the workspace. This kind of provider-specific retrieval window should be reflected in the gateway’s metadata and tenant-facing exports.

Expose analytics that match how teams operate

Batch analytics should exist at both job and item level. A product owner wants to know whether a nightly enrichment completed. A finance admin wants cost by tenant, model, and customer. An engineer wants to know which failure class to retry.

Useful metrics include:

• Submitted, completed, failed, expired, and cancelled item counts.
• Estimated versus settled cost.
• Reserved budget still held.
• Input and output tokens by provider and model.
• Cache-hit indicators where providers expose them.
• Retry count and retry success rate.
• Average time in queued, running, and finalizing states.
• Top validation errors by endpoint and model.
• Partner customer attribution.

For Partner API users, expose batch jobs as customer-scoped resources. That allows agencies and SaaS builders to offer offline AI processing while keeping upstream provider credentials, billing reconciliation, and rate-limit handling inside the gateway.

Trade-offs to make explicit

Gateway abstraction versus provider-specific capability: a unified contract simplifies integration, but it cannot make every provider feature identical. Keep capability errors explicit.

Budget reservation versus estimate accuracy: reservation protects tenants from runaway jobs, but estimates can be wrong. The ledger must support adjustments, refunds, and overage handling.

Polling versus webhooks: polling is simple and reliable, but can waste API calls and delay completion. Webhooks are faster, but require signature verification, replay protection, and monitoring.

Raw result storage versus retention minimization: storing normalized results improves exports and analytics, but increases compliance burden. Sensitive tenants may prefer pointers and hashes.

Large batches versus chunked batches: huge batches may improve provider-side efficiency, but smaller chunks reduce blast radius and make retries easier.

Implementation checklist

• Create a separate batch job API surface.
• Persist job and item records before provider submission.
• Require gateway job IDs and per-item custom IDs.
• Normalize statuses while storing native provider metadata.
• Build a capability matrix for each provider batch adapter.
• Validate manifests before reserving budget.
• Reserve tenant budget before dispatch.
• Settle actual usage at item level after ingestion.
• Make result ingestion idempotent.
• Retry failed items selectively, not whole jobs blindly.
• Track provider retrieval deadlines and gateway retention policy.
• Expose job and item analytics to tenants and partner customers.

Predictions: where this pattern is heading

Prediction: batch execution will become a normal part of AI automation infrastructure, not just a discount mechanism. As teams run more evals, data-cleanup tasks, safety reviews, and enrichment pipelines, they will expect asynchronous workloads to have the same governance as synchronous API calls.

Prediction: provider batch APIs will continue to diverge in useful ways. Some will optimize for files, others for long-running operations, and others for managed datasets or event callbacks. A gateway adapter layer will become more valuable, not less, because the operational contract above the adapters can remain stable.

Actionable conclusion

Do not bolt batch processing onto an AI API gateway as a provider-specific escape hatch. Build it as a durable subsystem with its own job records, item identifiers, status model, provider adapters, budget reservation, idempotent ingestion, and analytics.

The most important design choice is item-level accounting. Once every request inside a batch has a stable identity, the gateway can reconcile unordered results, retry only failed work, bill only completed provider work, and show tenants what happened. That is the difference between sending files to a provider and operating a dependable multi-model API for asynchronous workloads.

Related reading

• quote, reserve, settle, and reconcile AI API billing pattern
• gateway-level usage analytics and token ledger design
• when latency-tolerant workloads fit batch processing