Prompt Caching

Talos automatically detects repeated prompt prefixes and bills them at a 90% discount. Caching requires no code changes and works on both API surfaces.

Two things behave differently and are worth separating: what you pay, and what our servers recompute.

Pricing

When a message you send byte-for-byte matches a prefix you sent within the last 5 minutes, those tokens are billed as cache read instead of input. New prefixes are registered automatically. Cache state is isolated per organization; your prompts never prime another customer's cache.

Cache writes cost the same as regular input. There is no penalty for a cache miss.

This applies to any repeated message, however short, and is independent of the server-side thresholds below.

Server-side reuse

Our serving stack reuses computation for shared prefixes at block granularity. A repeated system prompt, reference document, or conversation history is reused from where it last diverges, so repeat requests reach the first token faster with no length threshold. Only the newest block of a prefix is recomputed. Reuse depends on how recently the prefix was last used; under load, idle prefixes are evicted first.

The effect grows with prefix length: the longer the shared context, the more a warm request saves over a cold one. We do not guarantee specific latencies.

Your bill is unaffected by server-side reuse. Repeated input is credited at cache-read rates whatever its length. Server-side reuse controls compute on our end only. It never changes what you pay.

Maximizing savings

1. Order your prompt stable-first: system prompt and reference documents before variable user content.
2. Keep prefixes byte-stable. A timestamp or request ID inside the system prompt breaks every hit.
3. Resend conversation history verbatim. Rewriting or truncating earlier turns invalidates the prefix.
4. Stay inside the 5-minute window for session-style workloads.
5. For agents: a long stable system prefix above the reuse threshold also earns faster repeat turns.

Visibility

• Messages API (/v1/messages): each response reports cache_read_input_tokens and cache_creation_input_tokens in usage. Explicit cache_control breakpoints are honored; without them, caching still runs automatically.
• Chat Completions API (/v1/chat/completions): the discount applies automatically. Each response reports usage.prompt_tokens_details.cached_tokens, the number of prompt tokens served from cache, matching the OpenAI field. Totals also appear in the Console usage dashboard.