Zaštitne ograde za multimodalni trošak u AI API pristupniku: mjerne slike, audio, video i generirani mediji prije otpreme

Text-only token accounting is not enough once teams start sending screenshots, scanned PDFs, audio, video, and image-generation requests through the same gateway. A request with a short text prompt can still be expensive because the cost driver is physical media: image dimensions, tile count, audio duration, video seconds, file size, output quality, or generated image size.

The gateway pattern is straightforward: inspect media before dispatch, normalize or reject inputs when policy allows, reserve budget against a provider-specific estimate, then settle the final charge against provider-reported usage. The important design shift is that the ledger must stop treating every request as only prompt_tokens and completion_tokens.

The reader problem: multimodal spend becomes hard to explain

Teams usually discover the problem in one of four ways:

• A support workflow adds screenshots and the monthly bill jumps even though text token volume is flat.
• A mobile app accidentally uploads full-resolution camera images for a classification task that only needed a small preview.
• A meeting summarizer sends long audio or video files without a per-tenant duration cap.
• An image-generation feature exposes high-quality or large-size outputs to every tenant without a budget reservation step.

A gateway that only logs input and output text tokens can under-estimate spend before dispatch, miss tenant-level abuse, and fail to explain why a workflow became expensive. The fix is not just more dashboards. The gateway needs a modality-aware cost-control path before the provider call is made.

Facts to design around

Some provider behaviors are already concrete enough to shape a gateway architecture:

• OpenAI image generation responses for GPT image models expose usage fields such as total tokens, input tokens, output tokens, and input token details split into image tokens and text tokens.
• OpenAI image generation requests include cost-driving parameters such as size, quality, background, output format, and streaming partial image events. A gateway can restrict these settings before forwarding the request.
• OpenAI data-control documentation distinguishes endpoint behavior. For example, image generation with supported GPT image models is listed as Zero Data Retention compatible, while videos API behavior differs and may be blocked for stricter retention modes.
• OpenAI documentation also states that image and file inputs submitted to responses, chat completions, and images may be scanned for CSAM, and matching content can be retained for manual review even under stricter data controls.
• Gemini documentation states that all input and output, including text, image files, and other non-text modalities, is tokenized.
• Gemini provides a count_tokens method that can be called before sending input to check request size.
• Gemini documentation lists multimodal tokenization rules, including small images counted as 258 tokens, larger images tiled, video counted at 263 tokens per second, and audio counted at 32 tokens per second.
• Gemini usage metadata can expose prompt, cached-content, candidate, thoughts, and total token counts, which a gateway can use for reconciliation.
• Anthropic Claude vision documentation recommends multiplying image visual token count by the model’s per-token price to estimate cost.
• Anthropic documentation warns that high-resolution image processing can use substantially more visual tokens and recommends downsampling when extra fidelity is not needed.

Those facts point to the same conclusion: multimodal AI API cost control must be provider-aware and modality-aware. A single generic token field will not preserve enough meaning for policy, billing, or analytics.

Reference architecture: preflight, policy, reservation, dispatch, settlement

A robust multimodal gateway path has five stages:

1. Media inspection: Parse the request, identify media attachments and generation parameters, and measure cost-driving properties.
2. Provider-specific estimation: Convert media properties into estimated upstream units using model and provider rules.
3. Tenant policy enforcement: Apply caps for dimensions, duration, file size, number of attachments, generated output settings, and maximum estimated cost.
4. Budget reservation and dispatch: Reserve enough tenant budget for the worst reasonable estimate, then forward the normalized request.
5. Post-call settlement: Store provider-reported usage, settle the actual charge, and flag estimate differences for analytics.

This flow is useful even when a provider does not expose perfect usage details. The gateway can still record the estimate, the policy decision, the physical media metadata, and the raw provider response for later reconciliation.

Step 1: inspect media before provider dispatch

The preflight inspector should run before the request reaches any provider adapter. It does not need to store raw media by default. In most production systems, storing metadata is enough for cost control and safer for privacy.

Minimum metadata to capture

• request_id, tenant_id, api_key_id, end_user_id_hash if available
• Requested model or internal model alias
• Endpoint type: chat, responses, image generation, transcription, video understanding, file analysis, realtime, batch
• Media MIME type
• Image count, width, height, megapixels, and format
• Audio duration, channel count, sample rate, and format
• Video duration, resolution, frame rate if available, and format
• File count, file size, page count when cheaply available, and extracted text length if extraction is performed
• Image-generation parameters: size, quality, background, output format, number of images, and streaming mode
• Policy profile applied and decision: allow, normalize, require approval, reject

The inspector should fail closed when it cannot identify a high-risk payload. For example, an unknown binary file from a new tenant should not be treated as a tiny text prompt. It should either be rejected, routed to a safer extraction path, or require a stricter reservation.

Step 2: normalize inputs when quality requirements allow it

Media normalization is one of the highest-leverage cost controls because it reduces cost before the provider sees the request. However, it should be tied to workload intent. A document OCR workflow and a broad image-classification workflow should not share the same image policy.

Useful normalization rules

• Downsample routine screenshots: If the task is classification, moderation, routing, or summarization, a lower-resolution image may be enough.
• Cap image count: Reject or require approval when a request includes dozens of images where the product UI expected one or two.
• Reject accidental full-resolution camera uploads: Mobile apps often upload large images by default. Gateway limits can catch this before it becomes a billing problem.
• Trim silence from audio: For transcription and summarization, long silent sections should not consume budget.
• Segment long media: Very long audio or video should be split into governed jobs with separate reservations and resumable processing.
• Default to lower-cost generated-image settings: Put higher quality, larger sizes, or multiple outputs behind an explicit tenant profile.

The trade-off is quality. Downsampling can hurt OCR, UI-agent tasks, medical-image analysis, dense-document analysis, and small-object recognition. Treat normalization as a workload policy, not a universal compression step.

Step 3: estimate provider-specific usage

The gateway should maintain a cost-estimation adapter per provider and modality. The goal is not perfect prediction. The goal is a safe reservation that prevents obvious overspend and produces a useful explanation when a request is blocked.

Example estimate object

{
  "request_id": "req_123",
  "tenant_id": "tenant_a",
  "provider": "example_provider",
  "model": "vision-large",
  "modalities": ["text", "image"],
  "media": {
    "image_count": 3,
    "total_megapixels": 18.4,
    "largest_image": {"width": 4032, "height": 3024}
  },
  "estimated_usage": {
    "text_input_tokens": 420,
    "image_input_tokens": 3100,
    "estimated_total_tokens": 3520
  },
  "estimated_cost_minor": 184,
  "estimate_method": "provider_rules_v2026_08",
  "confidence": "medium"
}

For Gemini-style calls, the gateway can call count_tokens during preflight for expensive, unfamiliar, or near-limit requests. This improves accuracy but adds latency and may require an extra provider call. A practical rule is to use provider-native token counting only when the rough local estimate crosses a threshold, the tenant is new, the media type is unusual, or the request is near a budget limit.

For OpenAI image generation, the gateway can estimate before dispatch based on requested size, quality, output count, and input media, then reconcile against returned usage fields that separate image and text token details. For Anthropic vision, the gateway can store visual-token estimates and multiply by the model’s per-token price, while preserving any provider-reported usage fields for settlement.

Step 4: enforce tenant-level policy knobs

Multimodal limits should be visible and configurable at the tenant, environment, key, and workflow levels. A single global limit is too blunt: a tenant building an audio product needs different caps from a tenant whose support bot only accepts one screenshot.

Recommended policy fields

• max_images_per_request
• max_image_megapixels and max_total_image_megapixels
• max_audio_seconds per request and per hour
• max_video_seconds per request and per day
• allowed_file_mime_types
• max_file_size_mb and max_pages
• allowed_generated_image_sizes
• allowed_generated_image_quality_levels
• max_generated_images_per_request
• max_estimated_cost_per_call
• require_provider_count_tokens_above_cost
• retention_mode_required

Policy errors should explain the physical reason for the block. A generic “budget exceeded” message is not enough for developers to fix the problem.

Better error response

{
  "error": {
    "type": "media_policy_violation",
    "code": "image_megapixel_limit_exceeded",
    "message": "This request includes 4 images totaling 42.1 megapixels. Tenant policy allows 10 megapixels per request for this workflow.",
    "suggested_actions": [
      "Downsample images before upload",
      "Send fewer images per request",
      "Use the document-ocr policy profile if this workload requires high resolution"
    ]
  }
}

This style of error message reduces support work because it tells the developer what changed: number of images, dimensions, duration, output quality, or file size.

Step 5: reserve budget separately from final settlement

Preflight estimates are reservations, not final charges. Generated outputs, reasoning tokens, thinking tokens, and provider-side media tokenization can differ from the estimate. The gateway should reserve against a safe upper bound, then settle based on actual provider usage when available.

Ledger schema for multimodal requests

A useful normalized ledger should include more than text tokens:

• text_input_tokens
• text_output_tokens
• image_input_tokens
• image_output_tokens
• audio_input_seconds
• audio_input_tokens
• video_input_seconds
• video_input_tokens
• file_input_tokens
• cached_input_tokens
• reasoning_or_thinking_tokens
• generated_media_count
• generated_media_size
• provider_reported_total
• estimated_total
• reservation_amount_minor
• settled_amount_minor
• reconciliation_status: exact, estimated, adjusted, provider_missing_usage, failed_before_dispatch
• raw_provider_usage as a versioned JSON blob

Keeping the raw provider usage blob matters because normalized fields will not map perfectly across providers. Provider schemas change, and some details are unique to a specific endpoint. Normalized fields make billing and analytics usable today; raw fields protect future reconciliation and migration work.

Data-retention compatibility belongs in the same decision path

Multimodal cost controls should not be isolated from data controls. Media inputs are often more sensitive than text prompts: screenshots can contain personal data, documents may include regulated information, and audio/video can expose voices, faces, locations, or background details.

The gateway should maintain an endpoint capability matrix that includes retention behavior, moderation or safety-scanning implications, storage defaults, and whether the endpoint is compatible with the tenant’s required retention mode. If a tenant requires a stricter mode and the requested video or image endpoint is not compatible, the gateway should block or reroute before dispatch.

This is a recommendation, not a prediction: cost, policy, and retention decisions should be made in one preflight step because the same media metadata drives all three.

Analytics: show cost by modality, not only by model

Once multimodal fields exist in the ledger, the analytics layer can answer questions that a model-only dashboard cannot:

• Which tenants generate the most image-token spend?
• Which API keys send the longest audio requests?
• Which workflows produce the highest generated-image output cost?
• How often do estimates differ from provider-reported usage?
• Which requests were blocked because of dimensions, duration, or quality settings?
• Which model aliases are receiving media they were not designed to handle?

These views are also useful for abuse detection. A sudden rise in video seconds, audio minutes, or generated image count can indicate a product bug, a leaked key, a retry loop, or a tenant exceeding its intended use case.

Implementation checklist

• Create modality-aware ledger fields before adding more multimodal endpoints.
• Add a media inspector that measures image dimensions, audio/video duration, MIME type, file size, and generated-media parameters.
• Build provider estimation adapters with versioned rules and confidence labels.
• Use provider-native counting selectively for high-risk or near-budget requests.
• Define tenant policy profiles for common workloads such as support screenshots, OCR, transcription, video understanding, and image generation.
• Reserve budget before dispatch and settle after provider usage is available.
• Store raw provider usage alongside normalized fields.
• Return explanatory policy errors that name the physical cost driver.
• Add retention compatibility checks for media endpoints.
• Build modality-level analytics so finance, platform, and product teams can see what is actually driving spend.

Trade-offs to decide explicitly

• Accuracy versus latency: Provider-native token counting improves budget accuracy but adds a preflight call. Use it selectively.
• Cost versus quality: Downsampling and duration caps reduce cost, but can harm high-fidelity workflows.
• Strict limits versus developer flexibility: Hard caps prevent surprise invoices, but approved workloads may need override profiles.
• Normalized billing versus provider detail: Normalized fields simplify invoices and dashboards, but raw usage blobs are still needed for reconciliation.
• Metadata analytics versus privacy: Media metadata is valuable, but raw media storage raises retention and privacy risk. Prefer hashes and derived metadata unless the tenant explicitly enables storage.

Actionable conclusion

If your gateway already has token accounting for text calls, do not bolt multimodal support onto the same two-column usage model. Add a preflight media inspector, provider-specific estimators, tenant-level media policies, budget reservation, and a settlement schema that separates text, image, audio, video, file, cached, reasoning, and generated-media usage.

The result is not just lower spend. It is explainable spend. When a request is blocked or a workflow becomes expensive, the gateway can tell the team why: too many images, too many megapixels, too much audio, too much video, an expensive generated-image setting, or an endpoint that conflicts with the tenant’s retention policy.

Related reading

• budget reservation and settlement ledger pattern
• spend anomaly runbooks for tenant-level abuse and cost spikes
• data-retention-aware routing for sensitive AI workloads