Prompt Caching

Talos automatically detects repeated prompt prefixes and bills them at a 90% discount. Caching requires no code changes and works on both API surfaces.

Two things behave differently and are worth separating: what you pay, and what our servers recompute.

Pricing

When a message you send byte-for-byte matches a prefix you sent within the last 5 minutes, those tokens are billed as cache read instead of input. New prefixes are registered automatically. Cache state is isolated per organization; your prompts never prime another customer's cache.

Cache writes cost the same as regular input. There is no penalty for a cache miss.

This applies to any repeated message, however short, and is independent of the server-side thresholds below.

Server-side reuse

Our current serving stack reuses computation only for long shared prefixes. As a rule of thumb: reuse begins once the stable part of your prompt (system prompt, reference documents, conversation history) exceeds roughly 4,500 tokens. The most recent few thousand tokens of a prefix are always processed fresh. Reuse also depends on how recently the prefix was last used; under load, idle prefixes are evicted first.

Below the threshold, repeated prompts do not start faster. Above it, repeat requests typically reach the first token sooner, and the effect grows with prefix length. We do not guarantee specific latencies.

This is temporary. The coarse threshold is a property of our current serving stack. We are replacing that layer; reuse will become finer-grained with no changes on your side.

Your bill is unaffected by these thresholds. Repeated input is credited at cache-read rates whatever its length. Server-side reuse controls compute on our end only. It never changes what you pay.

Maximizing savings

1. Order your prompt stable-first: system prompt and reference documents before variable user content.
2. Keep prefixes byte-stable. A timestamp or request ID inside the system prompt breaks every hit.
3. Resend conversation history verbatim. Rewriting or truncating earlier turns invalidates the prefix.
4. Stay inside the 5-minute window for session-style workloads.
5. For agents: a long stable system prefix above the reuse threshold also earns faster repeat turns.

Visibility

• Messages API (/v1/messages): each response reports cache_read_input_tokens and cache_creation_input_tokens in usage. Explicit cache_control breakpoints are honored; without them, caching still runs automatically.
• Chat Completions API (/v1/chat/completions): the discount applies automatically. Totals appear in the Console usage dashboard.