Cut LLM API Costs with Batch Jobs and Prompt Caching: A Practical Playbook

Many teams overpay for LLM APIs because they send every request through the same synchronous path. That is appropriate for chat, coding assistants, support agents, payment flows, and anything waiting on a user. It is wasteful for evaluations, tagging, enrichment, moderation sweeps, embedding backfills, nightly reports, and content pre-processing.

The practical question is not “Which model is cheapest?” It is: which work actually needs an immediate response, and which work can wait? Once you answer that, AI API cost control becomes an engineering workflow: classify traffic, send latency-tolerant jobs to batch processing where supported, structure repeated prompts for caching, and measure the real savings after failures, retries, and operational overhead.

Start with a cost audit by workload, not by model

Before changing architecture, export a sample of recent API usage and group it by workload. A useful audit table should include:

• Endpoint and model: chat completions, responses, embeddings, moderation, or provider-specific endpoints.
• Average input and output tokens: separate long prompts from short classification tasks.
• Prompt shape: stable system instructions, reusable examples, schemas, retrieval context, and dynamic user data.
• Latency requirement: seconds, minutes, hours, or next business day.
• User visibility: whether a person is waiting for the result.
• Retry and failure rate: malformed requests, validation failures, provider timeouts, expired jobs, and duplicate submissions.
• Ownership: project, team, customer, API key, or partner account.
• Business SLA: the latest time the result is still useful.

This audit usually reveals that “LLM traffic” is not one workload. It is a mix of interactive product features, internal automation, reporting, data preparation, and quality evaluation. Treating them as one cost center hides the easiest savings.

Use a three-lane workload classifier

A simple classifier prevents teams from moving the wrong traffic to batch and then being surprised by missed expectations.

Lane 1: real-time interactive requests

Keep these synchronous. They include chat UX, copilots, support agents, human-in-the-loop review, live search or retrieval flows, and tool calls with immediate side effects. If a user is waiting, the value of a cheaper response can be erased by latency.

Recommendation: optimize this lane with model selection, prompt trimming, rate-limit management, caching where applicable, and careful retries. Do not send it to a 24-hour batch queue unless the product explicitly presents it as a background task.

Lane 2: nearline requests that can wait minutes

These jobs do not need to block a page load, but they may still have a same-session or same-hour expectation. Examples include post-upload document analysis, CRM enrichment after form submission, or a report that can notify the user when ready.

Recommendation: place nearline work behind a queue with explicit status states. Depending on provider support and deadline, either run it through small batches or synchronous workers with lower priority. This lane benefits from job IDs, webhooks, and user-visible progress.

Lane 3: offline batch requests that can wait up to 24 hours

This is the main cost-optimization lane. Good candidates include:

• large-scale evaluations;
• dataset labeling;
• catalog or CRM enrichment;
• nightly summarization;
• compliance review queues;
• embedding backfills;
• moderation sweeps;
• periodic report generation;
• content pre-processing before indexing or publishing.

Fact: major providers now offer asynchronous batch APIs for suitable workloads. OpenAI’s Batch API reads requests from an uploaded file, writes results to an output file, and targets processing within 24 hours. OpenAI states that supported Batch API usage is offered at a 50% cost discount compared with synchronous APIs. Anthropic’s Message Batches API is designed for large volumes of Messages requests, asynchronous processing, higher throughput, and 50% lower cost. Google’s Gemini Batch API is designed for large-volume asynchronous requests at 50% of standard cost, with a target turnaround time of 24 hours.

Trade-off: “up to 24 hours” is excellent for backfills and evaluations, but unacceptable for interactive workflows. Batch is a scheduling strategy, not a universal replacement for synchronous inference.

Design the batch path as a job lifecycle

The implementation mistake to avoid is treating batch as a single API call. It is a lifecycle: accept work, validate it, persist it, submit it, poll it, reconcile it, and expose results.

Reference architecture

1. Accept a normalized request: keep the request shape close to your existing OpenAI-compatible API format where possible. Add metadata such as project, team, customer, idempotency key, requested deadline, and cost center.
2. Classify the workload: assign the request to real-time, nearline, or offline batch. This should be policy-based, not hidden inside application code.
3. Create a job ID: return a job identifier immediately for nearline and offline work.
4. Validate compatibility: check whether the selected provider and model support batch for the requested endpoint, modality, file size, tools, response format, and other features.
5. Persist request rows: store normalized JSONL rows or provider-specific payloads. Include a stable row ID for reconciliation.
6. Submit the batch: upload the request file or inline batch payload depending on provider limits and job size.
7. Poll status: track provider states such as validating, in progress, completed, failed, expired, cancelling, and cancelled where applicable.
8. Store output rows: write successful responses, row-level errors, token usage, cached token counts where available, and provider identifiers.
9. Notify consumers: expose a retrieval endpoint, webhook, dashboard notification, or Telegram alert.
10. Reconcile billing: attribute cost to the original project, team, customer, API key, and job ID.

This pattern keeps the application simple. Product teams submit work and receive job states. The gateway or orchestration layer handles provider differences, batch files, retries, and accounting.

Use explicit job states

Define internal states even if each provider uses different names:

• queued: accepted but not submitted;
• validating: provider or gateway is checking the file;
• running: submitted and being processed;
• completed: all available results collected;
• completed_with_errors: some rows failed validation or execution;
• expired: deadline passed before all rows completed;
• cancelled: stopped by user, system, or policy;
• failed: job-level failure that requires intervention.

Fact: OpenAI documents batch statuses including validating, failed, in_progress, completed, expired, cancelling, and cancelled. It also notes that if a batch expires, already completed work is returned and charged while remaining work is cancelled.

Recommendation: never assume batch jobs are all-or-nothing. Build row-level status handling from the start.

Calculate savings after failures and overhead

A simple savings model is enough for most teams:

baseline_cost = synchronous_input_cost + synchronous_output_cost
batch_cost = discounted_batch_input_cost + discounted_batch_output_cost
adjusted_batch_cost = batch_cost + orchestration_cost + storage_cost + rerun_cost
estimated_savings = baseline_cost - adjusted_batch_cost

Then calculate this per workload, not globally. A nightly evaluation suite may save substantially. A nearline workflow with many malformed rows, urgent fallbacks, or repeated reruns may save less than expected.

Track at least these metrics:

• sync versus batch token spend;
• input and output tokens by model;
• batch job count and average rows per job;
• row-level failure rate;
• expired job rate;
• rerun cost;
• fallback-to-sync cost;
• cost by team, project, key, customer, and partner account.

Recommendation: treat automatic synchronous fallback as an exception, not the default. It protects deadlines, but if overused it can erase expected savings. Add a policy such as “fallback only if the business deadline is within two hours and the job has not started.”

Add prompt caching for repeated long prefixes

Batch processing reduces the unit price of eligible work. Prompt caching reduces the effective cost and latency of repeated long prompts when provider behavior supports it.

Fact: OpenAI prompt caching automatically applies to prompts longer than 1,024 tokens on supported models, caches the longest previously computed prefix, and reports cached_tokens in API usage details. OpenAI says prompt caches are typically cleared after 5 to 10 minutes of inactivity and removed within one hour of last use, and that prompt caches are not shared between organizations.

The implementation pattern is straightforward: put stable content first and volatile content last.

Better prompt structure for caching

System instructions
Stable policy text
Stable output schema
Stable examples
Reusable reference context
---
Dynamic record-specific input
Dynamic user or row metadata

For example, a catalog enrichment job might reuse the same taxonomy, output schema, brand rules, and examples across 50,000 products. Each row changes only the product title, description, and attributes. Placing the reusable prefix first gives the provider a better chance to reuse cached computation where supported.

Trade-off: caching is not permanent storage and should not be treated as guaranteed. Cache windows, isolation, minimum prompt length, and reporting differ by provider. Measure cached tokens rather than assuming savings.

Validate provider support before submission

Batch APIs differ. The gateway should validate eligibility before it submits a job.

Facts: OpenAI Batch API does not support streaming and has separate batch rate limits. Anthropic documents batch limitations including a 100,000-request or 256 MB batch size limit, 24-hour expiry, 29-day result availability, rate limits, and the possibility that batches may slightly exceed configured workspace spend limits. Google supports inline batch requests for smaller jobs under 20 MB and JSONL input files for larger batch requests.

Use a compatibility checklist:

• Is the requested model available through that provider’s batch API?
• Is the endpoint supported?
• Does the request require streaming? If yes, reject batch.
• Does it use tools or side effects that must happen immediately?
• Does the batch file exceed provider limits?
• Is the expected result still useful within the provider’s completion window?
• Are outputs available long enough for downstream systems to retrieve them?
• Can the workload tolerate partial completion?

Recommendation: fail validation early with a clear reason. A rejected batch candidate is cheaper than an expired or malformed job that has to be reworked later.

Safeguards for teams, agencies, and partners

Batch systems can quietly spend a lot of money because they process large files in the background. Add controls before broad rollout:

• Per-team batch budgets: separate online and offline spend limits.
• Maximum file size and row count: enforce provider limits and your own operational limits.
• Dead-letter queue: preserve invalid rows with validation errors for review.
• Idempotency keys: prevent duplicate charges from accidental resubmission.
• PII review: batch files may create new data retention and privacy obligations.
• Retention policy: define how long request files, output files, and logs are stored.
• Notification policy: alert owners when jobs fail, expire, or exceed budget.
• Attribution: record project, team, customer, API key, model, provider, job ID, and row ID.

For agencies and resellers, attribution is especially important. If one partner runs enrichment or evaluation jobs for many clients, the system should report cost per client and per job, not only per provider invoice.

How this maps to an AI API gateway

An AI API gateway is a natural place to implement this because it already sits between applications and model providers. The gateway can preserve an OpenAI-compatible API surface for developers while adding cost-aware scheduling behind it.

Useful gateway capabilities include:

• Unified billing: compare synchronous, batch, cached, and fallback spend in one place.
• AI usage analytics: break down usage by model, provider, endpoint, team, project, and API key.
• Team controls: set separate budgets for interactive and offline workloads.
• API-key attribution: identify which service or customer created each job.
• Status notifications: send alerts when batch jobs complete, fail, expire, or approach a deadline.
• Partner API workflows: let agencies or resellers create jobs and retrieve results on behalf of clients while preserving client-level accounting.

Prediction: more teams will manage LLM costs with scheduling policies, not only model substitutions. As batch support matures across providers, the winning architecture will route by urgency, feature compatibility, and accounting requirements before it routes by model price.

Implementation checklist

• Export 30 days of LLM API usage.
• Classify each workload as real-time, nearline, or offline.
• Choose one offline workload with clear ownership and a forgiving deadline.
• Validate provider batch support for the required endpoint and model.
• Define internal job states and row-level statuses.
• Add idempotency keys, job IDs, and per-row IDs.
• Store normalized request and response records with retention controls.
• Submit the first batch behind a feature flag.
• Measure synchronous baseline cost versus adjusted batch cost.
• Restructure repeated long prompts to put stable prefixes first.
• Track cached tokens, failed rows, expired jobs, and fallback spend.
• Expand only after savings and operational behavior are visible in analytics.

Actionable conclusion

Do not start AI API cost control by asking every team to use a cheaper model. Start by separating urgent work from work that can wait. Keep interactive requests synchronous. Move evaluations, enrichment, tagging, backfills, moderation sweeps, and reports to batch when provider support and business deadlines fit. Structure repeated long prompts for caching. Then measure actual savings after failures, reruns, storage, and fallback costs.

The best implementation is boring on purpose: job IDs, validation, row-level statuses, budgets, usage analytics, and clear ownership. That operational layer is what turns provider discounts into reliable savings.

Related reading

• API key attribution and team spend controls
• routing, retries, and fallback policies for LLM APIs
• related Model Gate implementation notes