Prompt caching

Caching depends on the model and provider route. Some providers cache repeated prefixes implicitly; others take explicit cache points. The response's cache read and write usage shows what a request hit.

Explicit cache points

{"type":"text","text":"Reusable instructions for this task.","cache_control":{"type":"ephemeral"}}

Use an eligible native route and its supported content form. The Anthropic and Converse adapters support ephemeral five-minute caching; one-hour cache TTL is refused. Do not add cache controls to every route indiscriminately.

Usage and cost

Read the provider-reported cache read and write usage and the model's cache prices. Cache writes can be priced differently from ordinary input. An inbound Responses prompt_cache_key is ignored and listed in the ignored-fields header.

Keep the same provider route when comparing repeat requests. See Cost and Provider routing.

Compare a cold and warm prefix

Choose a ready provider route and keep it fixed. This example makes exactly two paid calls when run; use it only within your authorized allowance. The first request may create cache entries; read the repeat's cache counters to confirm a hit. Preserve errors instead of automatically retrying them.

import json, os, urllib.request

prefix = "\n".join(
    f"Reference item {i}: code ITEM-{i:04d}; stock {i % 31}."
    for i in range(300)
)
body = {
    "model": "deepseek/deepseek-v4-pro",
    "provider": {"only": ["alibaba-model-studio"], "allow_fallbacks": False},
    "messages": [{"role": "user", "content": prefix + "\nReturn only the stock of item 299."}],
    "max_tokens": 4000,
}
for label in ("cold", "warm"):
    request = urllib.request.Request(
        "https://tokenhub-api.belive.sg/api/v1/chat/completions",
        data=json.dumps(body).encode(),
        headers={"Authorization": "Bearer " + os.environ["TOKENHUB_API_KEY"],
                 "Content-Type": "application/json"},
    )
    with urllib.request.urlopen(request, timeout=120) as response:
        result = json.load(response)
        generation_id = response.headers.get("X-Generation-Id")
    usage = result.get("usage", {})
    details = usage.get("prompt_tokens_details", {})
    print(label, generation_id, "read", details.get("cached_tokens"),
          "write", details.get("cache_write_tokens"))

The named provider must be ready in your catalogue; select a compatible ready route before running, not as a fallback after a failed request. The counters are what the provider reports for your request; they are not minimum cache thresholds or promised savings.

Read each request's generation receipt for settled cost. Cache reads and writes can have different prices; do not multiply all prompt tokens by the uncached rate and then add the cached portion again. Missing cache counters mean unavailable evidence, not a proven miss or free usage.