What Is Prompt Caching? How It Works on Anthropic & OpenAI
Prompt caching lets you reuse the expensive part of a prompt a long system message, tool definitions, a retrieved document, or a few-shot block across many requests. Anthropic implements it explicitly: you mark a cache breakpoint with cache_control and get a default 5-minute TTL (with a longer 1-hour option). OpenAI does it automatically for prompts of 1,024+ tokens with no code change. Both discount cached input heavily (~90% Anthropic, ~50% OpenAI), but caching only pays off when the same prefix is reused often enough to earn back the write premium as a rule of thumb, when well over half your input tokens are a stable, reused prefix. Below that, or when the prefix mutates on every call, caching is a net loss.
Published July 17, 2026 · Updated July 17, 2026
What is Prompt Caching?
Prompt caching is a provider-side optimization that stores the processed state of a prompt prefix so that identical prefixes on later requests can be reused instead of recomputed. You pay full price to write the prefix into the cache once, then a steep discount to read it on every subsequent request that matches roughly a 90% discount on cached input tokens at Anthropic and about 50% at OpenAI (per each vendor's official prompt caching docs). The catch is that the match is on an exact token prefix: a single-token change near the start of the prompt invalidates the cache for everything after it.
How Prompt Caching Works
- 1
Identify the stable prefix. Caching works on the beginning of a prompt the part that is byte-for-byte identical across many requests, such as a long system prompt, tool/function schemas, or a retrieved document. Everything after the first difference is not cacheable, so the reusable content has to sit at the front.
- 2
Write the cache on the first request. The first call that sees a given prefix pays to process it in full and store its state. On Anthropic this write can cost more than a normal input token (a premium for the 5-minute cache, and roughly double the base rate for the 1-hour cache, per Anthropic's docs); on OpenAI, caching is automatic and there is no separate write charge.
- 3
Read the cache on matching requests. Subsequent requests whose prefix matches the stored one are billed at the cached-input rate about a 90% discount at Anthropic and roughly 50% at OpenAI and usually return faster because the prefix is not re-processed.
- 4
Watch the TTL. Anthropic's cache has a default 5-minute time-to-live that refreshes each time the entry is hit, with an optional extended 1-hour TTL you opt into per breakpoint. OpenAI evicts entries after a short period of inactivity. If traffic on a prefix is sparser than the TTL, the entry expires and the next call pays the full write cost again.
- 5
Mind the cache key. The match is an exact token prefix. A single changed token a timestamp in the system prompt, a reordered tool definition, a user name injected near the top invalidates the cache from that token onward. Keeping volatile content (dates, per-user data) after the cached prefix is the difference between a high and a near-zero cache hit rate.
- 6
Measure the net effect on the bill. The savings only materialize if reads outnumber writes by enough to cover the write premium. Track your cached vs. uncached input token split and your cost per request over time; in AICosts.ai the drop shows up as lower cost_usd for the same request volume on OpenAI and Anthropic, since it ingests each provider's billing data hourly.
Types of Prompt Caching
Explicit caching (Anthropic)
You opt in per request by adding a cache_control breakpoint to the content you want cached (up to a small number of breakpoints). You control exactly what is cached and whether to use the default 5-minute or extended 1-hour TTL. More control, but you have to structure prompts so the cacheable content comes first and rarely changes.
Automatic caching (OpenAI)
OpenAI caches prompt prefixes automatically for prompts of 1,024 tokens or more, in increments beyond that, with no code changes and no cache-control markers. Cached input is discounted (~50% per OpenAI's docs). You get savings for free, but you have less direct control and typically no explicit TTL to tune.
Extended-TTL caching
Anthropic's optional 1-hour cache trades a higher write cost (roughly double the base input rate) for a longer window. It fits workloads where the same prefix is reused steadily but not every few seconds e.g. an agent that revisits the same large context across a multi-minute task so the 5-minute default would keep expiring.
Application-level caching
Caching you build yourself outside the provider exact-match response caches or semantic/embedding caches (GPTCache-style) that skip the model call entirely for repeat questions. This is different from provider prompt caching: it can eliminate the request rather than discount its prefix, but it risks serving stale or subtly wrong answers if the match is too loose.
Common Use Cases
Long, stable system prompts
A detailed system prompt or policy block that is identical on every request is the canonical caching win put it first, cache it once, and pay the discounted read rate on every call for the life of the TTL.
RAG over large retrieved context
When many requests reference the same large document or knowledge chunk, caching the retrieved context as a shared prefix avoids re-billing tens of thousands of input tokens per question. It only holds if the same context is reused before the TTL expires.
Agent and tool-use loops
Agents re-send the same tool/function definitions and accumulated context on every step. Caching the stable head of that prompt across a multi-step run often with the extended TTL keeps the per-step input cost from compounding.
Few-shot example blocks
A fixed set of few-shot examples reused across classification or extraction calls sits naturally at the front of the prompt and caches cleanly, as long as the examples themselves don't change per request.
High-volume chat with a shared preamble
A chat product where thousands of concurrent conversations share the same instructions and persona keeps the cache warm through sheer request volume, which is when the 5-minute TTL is easiest to sustain and the read discount adds up fastest.
Related terms
Frequently Asked Questions
How much does prompt caching actually save?+
On the cached portion of the prompt, a lot: Anthropic discounts cached input tokens by roughly 90% and OpenAI by about 50%, per each vendor's prompt caching docs. But the savings apply only to the reused prefix, not the whole request, and only after you've paid to write the cache. Your real saving is a blend of your cache hit rate and how much of each prompt is cacheable.
When is prompt caching a net loss?+
When the same prefix isn't reused often enough to earn back the write cost. Two common cases: low prefix reuse (each request has a mostly unique prompt, so you keep paying to write cache entries you never read again), and frequent prefix mutations (a timestamp, user ID, or reordered tool schema near the top changes on every call and invalidates the cache). If your reused, stable prefix isn't well over half your input tokens, caching can cost more than it saves especially with Anthropic's write premium.
How does the 5-minute TTL work on Anthropic?+
Anthropic's cache entries have a default 5-minute time-to-live that resets each time the entry is read, so a steadily used prefix stays warm indefinitely. If more than five minutes pass with no hit, the entry expires and the next request pays the full write cost again. Anthropic also offers an opt-in 1-hour TTL for a higher write cost, which suits prefixes reused steadily but not every few seconds. See Anthropic's prompt caching docs for the exact behavior.
What's the difference between Anthropic and OpenAI prompt caching?+
Anthropic is explicit and OpenAI is automatic. On Anthropic you mark what to cache with a cache_control breakpoint and choose the TTL, giving you control at the cost of some prompt engineering. OpenAI caches prefixes automatically for prompts of 1,024+ tokens with no code change and no markers. Anthropic's read discount (~90%) is steeper than OpenAI's (~50%), but Anthropic charges a write premium that OpenAI does not.
Why did my cache hit rate drop to almost zero?+
Almost always because something near the start of the prompt is changing on every request. The cache key is an exact token prefix, so a single-token delta a current timestamp, a per-user greeting, a tool list serialized in a different order invalidates everything after it. Move all volatile content below the stable, cacheable prefix and the hit rate usually recovers immediately.
Does prompt caching change the model's output?+
No. Caching only reuses the pre-computed processing of an identical prefix; it doesn't alter the tokens the model sees or the response it generates. It's a billing and latency optimization, not a behavioral one the model produces the same output it would have without the cache.
Can AICosts.ai tell me if caching is saving money?+
Indirectly. AICosts.ai never sits in the inference path, so it doesn't implement caching or measure per-request cache hits that lives in the provider's own usage data. What it does is parse the OpenAI and Anthropic invoices you upload and normalize them into daily, per-model spend, so you can compare the bill before and after you turn caching on and confirm the net effect across providers.
Should I use provider caching or build my own cache?+
They solve different problems. Provider prompt caching discounts the input prefix of a request you're still making, and it's safe because the output is unchanged. An application-level exact-match or semantic cache can skip the model call entirely for repeat questions a bigger saving but risks serving stale or subtly wrong answers if the match is too loose. Many teams use both: provider caching for the shared prefix, an app cache for truly identical repeat queries.
Start tracking your AI costs
Unified view across 50+ AI providers with zero impact to your inference path.
Start tracking your AI spend7-day free trial. You upload your billing data. No changes to production.
Related pages
What Is Token-Based Pricing?
Why input and output tokens are billed separately the pricing model that makes caching the input prefix worthwhile.
What Is AI Cost Management?
The broader discipline caching fits into: tracking, attributing, and optimizing AI spend across vendors.
What Is AI FinOps?
Prompt caching is one of the AI-native optimization levers in a FinOps practice.
OpenAI cost tracking integration
How AICosts.ai ingests OpenAI billing so you can see the cost impact of caching on your bill.
Unified AI spend dashboard
One normalized view across 50+ providers to compare cost per request before and after enabling caching.