Meter Hosted AI eszközök az átjáróban: webes keresés, fájlkeresés, kódfuttatás és földelés meglepetés számlák nélkül

Hosted AI tools break simple token-based billing. A request that looks cheap at the model layer can become expensive because the model triggered web searches, grounding queries, file-search calls, retrieved-token processing, or a code execution session. If your gateway only records prompt and completion tokens, you will not be able to explain the invoice by tenant, API key, user, workflow, or prompt template.

The fix is not to treat tools as a vague prompt feature. Add a first-class hosted-tool metering layer beside the model-token ledger. That layer should quote possible tool spend before dispatch, enforce per-tenant policy while the request is running, record each billable tool event, and reconcile final charges against provider invoices or usage exports.

The reader problem: token ledgers do not see every billable unit

Many teams start AI API cost control with a straightforward token ledger:

• Estimate input tokens before dispatch.
• Reserve budget for the request.
• Capture final input, output, cached, or reasoning-token usage.
• Settle the request against a tenant balance.

That pattern is still necessary, but hosted tools add other billable dimensions. The gateway may need to meter:

• Per-search or per-grounding-query fees.
• Search or retrieved content tokens billed at model rates.
• File-search storage charges and per-tool-call charges.
• Code execution or container session time.
• Tool schema tokens and tool result tokens that inflate normal model usage.
• Multiple model-driven searches inside one apparent user request.

The difficult part is attribution. A provider invoice might show tool spend, but your customers will ask a more precise question: which tenant, API key, end user, workflow, prompt template, model alias, and tool policy produced it?

Facts to design around

The exact prices change, so the architecture should store pricing snapshots rather than hard-code assumptions. The following provider behaviors are important design inputs:

• Fact: OpenAI lists separate tool pricing for web search, image web search, containers, and file search. Its pricing page describes web search as priced per 1,000 calls, with search content tokens billed at model rates for many modes.
• Fact: OpenAI file search includes both storage pricing and a Responses API per-tool-call price, so a token-only ledger is incomplete.
• Fact: OpenAI container pricing includes hosted shell and code interpreter and is based on container size and session duration, creating a time/session meter separate from tokens.
• Fact: Google Gemini API pricing lists Grounding with Google Search and Grounding with Google Maps as separate paid items after a free monthly allowance for Gemini 3.x models.
• Fact: Google’s Grounding with Google Search documentation says Gemini 3 billing is per search query the model decides to execute, and one prompt can result in multiple billable searches.
• Fact: Anthropic’s Claude pricing documentation says tool-use requests include normal input/output tokens, additional tokens from tool schemas and tool result blocks, and extra usage-based pricing for server-side tools such as web search.
• Fact: Anthropic’s pricing documentation states that the Claude code execution tool is tracked separately from token usage and has minimum execution-time billing behavior.

Recommendation: Assume every hosted tool can introduce at least one non-token billing dimension until your provider adapter proves otherwise. Preserve raw provider metadata for reconciliation, but expose a provider-neutral policy model to product teams.

Architecture: add a hosted-tool usage ledger

Create a separate tool_usage_ledger table or event stream. Do not bury tool charges inside a JSON blob on the model request. Tool events have their own lifecycle: they can be estimated, started, completed, partially observed, reconciled, disputed, refunded, or attributed after a stream disconnects.

Recommended ledger fields

tool_usage_ledger
- id
- tenant_id
- api_key_id
- user_id
- request_id
- trace_id
- workflow_id
- prompt_template_id
- provider
- model
- model_alias
- tool_name
- provider_tool_name
- tool_invocation_id
- provider_invocation_id
- billable_unit_type
- estimated_units
- final_units
- unit_price_snapshot
- estimated_cost
- final_cost
- currency
- pricing_version
- policy_snapshot
- raw_provider_metadata
- reconciliation_status
- created_at
- settled_at

Use billable_unit_type values that match the economic unit, not the product label. Examples:

• search_call
• grounding_query
• retrieved_content_token
• file_search_call
• file_storage_gb_day
• container_session_minute
• code_execution_second
• tool_schema_token
• tool_result_token

Keep the model token ledger and tool ledger linked by request_id and trace_id. The finance view can aggregate both into one customer invoice. The operations view should keep them separate so teams can see when a “cheap model” workflow is expensive because it searches too much or leaves code sessions open.

Pre-dispatch budget reservation

Before sending a request upstream, calculate two estimates:

1. Model estimate: input tokens, likely output tokens, cached-token assumptions, reasoning-token allowance where applicable.
2. Tool estimate: the maximum allowed billable hosted-tool usage under the tenant’s policy.

For hosted tools, exact cost may be unknowable before dispatch because the model may decide how many searches or tool calls to perform. That is why the quote should reserve against policy maxima, not against optimistic predictions.

{
  "model": "research-default",
  "tools": ["web_search", "file_search"],
  "tool_budget_usd": 0.80,
  "allowed_tools": ["web_search"],
  "max_tool_calls": 4,
  "max_search_queries": 3,
  "max_file_search_calls": 0,
  "max_container_minutes": 0,
  "max_retrieved_tokens": 12000,
  "require_approval_over_usd": 0.50,
  "disable_model_initiated_search": false
}

Recommendation: Expose provider-neutral request controls such as tool_budget_usd, allowed_tools, max_tool_calls, require_approval_over_usd, and disable_model_initiated_search. Then map them to provider-specific request parameters, prompt instructions, gateway-side enforcement, or external orchestration.

Trade-off: Strict tool budgets improve cost predictability, but they can reduce answer quality when the model needs one more search, file lookup, or code execution step to complete the task.

Normalize provider-specific tool semantics

A multi-model API gateway should not force application teams to learn every provider’s billing grammar. The gateway adapter should normalize tool usage into shared categories while preserving raw metadata for audit and reconciliation.

Web search and grounding

For search-like tools, normalize at least three concepts:

• Invocation: the model or application requested a search-capable tool.
• Billable query: the provider charged for one or more search queries.
• Retrieved content tokens: search results or grounding content increased token usage.

This distinction matters because a single user prompt can produce multiple provider-side searches. If your customer says, “I sent one request,” your ledger should be able to answer, “Yes, and the model executed three billable search queries inside it.”

File search

File search often has two different cost centers:

• Storage or index lifecycle cost.
• Per-query or per-tool-call cost when the model searches the file corpus.

The gateway should attach storage to the tenant, project, corpus, or vector store, while per-call usage should attach to the original model trace. If you only bill storage at the account level, a noisy tenant can make a shared file-search feature look unprofitable without clear attribution.

Code execution and containers

Code tools create a session-metering problem. You may be billed by session duration, container size, execution time, or minimum time increments. The gateway should record:

• Container or session ID.
• Tenant and user ownership.
• Start time, stop time, and idle timeout.
• Requested size or resource class.
• Linked model request IDs.
• Cleanup status.

Trade-off: Reusing sessions can reduce latency and sometimes cost, but it increases isolation and cleanup risk. For multi-tenant systems, default to tenant-isolated sessions with aggressive idle timeouts unless you have a clear reason to reuse them.

Runtime enforcement: stop runaway tool loops

Pre-dispatch reservation is not enough. Agent loops and model-initiated tool calls can overspend while the request is still active. Add runtime enforcement in the gateway or orchestration layer.

Hard stops to implement

• Per-request maximum billable tool events: for example, no more than five search queries or two file-search calls.
• Per-request maximum tool spend: stop or require approval when the estimated cumulative tool cost reaches the request budget.
• Per-minute tenant burn-rate cap: throttle a tenant if many requests trigger tools at once.
• Duplicate-query suppression: block repeated identical or near-identical search queries inside the same trace unless explicitly allowed.
• Container idle timeout: terminate code sessions that exceed an idle threshold.
• Circuit breakers: temporarily disable expensive tools for a tenant, API key, or workflow when abnormal spend appears.

When a hard stop fires, return a structured error or tool denial that the application can handle. Avoid vague failures such as “provider error.” A useful response says the tool was blocked by policy, identifies the exceeded limit, and includes a safe retry path if one exists.

{
  "error": {
    "type": "tool_budget_exceeded",
    "message": "Web search was stopped because this request reached its max_search_queries limit.",
    "request_id": "req_123",
    "tool_name": "web_search",
    "limit": "max_search_queries",
    "limit_value": 3,
    "retryable": true
  }
}

Streaming and partial failures need independent settlement

Tool events should settle independently from the final model response. A hosted tool may complete and become billable even if the client disconnects before the assistant finishes streaming. Conversely, a provider may begin a tool call, fail, and report no billable usage. Your state machine must handle both.

A practical event sequence looks like this:

1. Create model request trace.
2. Estimate model and tool budgets.
3. Reserve tenant balance for worst-case allowed usage.
4. Dispatch upstream.
5. Record tool_invocation_started events as they appear.
6. Record tool_invocation_completed with final units when known.
7. If the client disconnects, continue consuming provider usage metadata when feasible.
8. Settle completed tool events even if the final assistant response is incomplete.
9. Refund unused reservation.
10. Mark records for invoice reconciliation.

Recommendation: Never wait for a final model usage object before recording hosted-tool events. The final response is one source of truth, not the only one.

Approval gates for expensive or risky tools

Some tools should not run automatically just because the model requested them. Add approval gates for expensive, sensitive, or stateful actions.

Examples:

• Require approval before a request can exceed tool_budget_usd.
• Require approval before enabling code execution for a tenant.
• Require approval before searching a sensitive file corpus.
• Require approval before expanding from web search to maps grounding or image search.

In an interactive product, approval can be a user prompt. In an API workflow, approval can be a policy token, pre-approved workflow configuration, or asynchronous job state. The important point is that approval should be recorded as a ledger-adjacent event with the approver, scope, expiration, and policy snapshot.

Dashboards: separate token spend from tool spend

If all AI spend is shown as one number, teams will optimize the wrong thing. Add dashboard panels that separate:

• Model token spend.
• Web search and grounding spend.
• File-search call spend.
• File storage or index spend.
• Code execution or container session spend.
• Tool-related token overhead.

Then make every panel filterable by tenant, API key, user, model alias, workflow, prompt template, environment, and time range. This is what turns a surprise provider invoice into an actionable investigation.

Useful questions the dashboard should answer:

• Which workflows have the highest hosted-tool cost per successful response?
• Which tenants trigger the most model-initiated searches?
• Which prompt templates cause duplicate queries?
• Which file corpora generate storage cost but little retrieval value?
• Which code sessions remain open longest after the final response?

Implementation checklist

• Create a tool_usage_ledger separate from the token ledger.
• Add provider adapters for search, grounding, file search, and code/container billing units.
• Store raw provider metadata alongside normalized fields.
• Store a price snapshot at request time.
• Quote worst-case tool usage from policy before dispatch.
• Reserve budget for model and hosted-tool spend together.
• Expose provider-neutral controls: allowed tools, maximum tool calls, budget, and approval threshold.
• Record tool events as they occur, not only after the final model response.
• Settle completed tool events even on stream disconnects or partial responses.
• Refund unused reservations.
• Reconcile ledger totals against provider invoices or usage exports.
• Add dashboards that separate token spend from tool spend.
• Add circuit breakers for runaway search, file-search, and code execution loops.

Recommendations, trade-offs, and predictions

Recommendations: Build hosted-tool metering as a first-class system. Quote and reserve budget before dispatch, enforce runtime limits, attach every tool event to the original trace, and reconcile against provider billing data. Give tenants provider-neutral controls, but keep raw provider details for audits.

Trade-offs: Strict budgets make spend predictable but may reduce answer quality. Provider-neutral controls simplify product integration but cannot perfectly hide provider differences. Caching search or file results can reduce cost and latency, but it creates freshness, citation, privacy, and tenant-isolation risks. Code session reuse may be efficient, but it must be balanced against cleanup and data separation.

Predictions: Hosted-tool spend will become a larger share of AI invoices as applications move from single-turn chat to research, coding, retrieval, and agentic workflows. Teams that can attribute tool spend by tenant and workflow will be able to price these features confidently. Teams that only meter tokens will keep discovering expensive behavior after the invoice arrives.

Actionable conclusion

Start by adding one table or event stream: tool_usage_ledger. Populate it for the highest-risk hosted tool first, usually web search, grounding, file search, or code execution. Add request-level limits such as tool_budget_usd, max_search_queries, and max_container_minutes. Link every tool event to the model trace and tenant ledger. Then update dashboards so hosted-tool spend is visible beside token spend, not hidden inside it.

That small architectural separation gives finance, operations, and product teams the same answer to the same question: not just how many tokens were used, but which hosted tools created real cost and why.

Related reading

• model-call billing ledger pattern
• agent tool governance controls
• streaming usage settlement state machine