Prompt caching and context-window budgeting on a Claude knowledge platform

What prompt caching does and when it pays

Prompt caching lets the Claude API reuse the processed form of a prompt prefix across requests instead of reading it each time. You mark a breakpoint; everything before it is cached, and later requests that start with the same bytes pay a reduced rate for that part. It pays when the same prefix is sent repeatedly within a short window.

The economics are simple to state and worth checking against Anthropic's current pricing page rather than memory: a cache write costs more than a normal input token, a cache read costs a fraction of one, and the cache lives for five minutes by default with a one-hour option. A read refreshes the timer. So a prefix that is reused twice within the window has already paid for itself, and one that is reused once a day never will.

We learned the details on an enterprise knowledge platform that came to us as a live proof of concept and left as a product sold to enterprises, over six months and 1,145 billed hours. The prototype answered questions well. It also resent the whole system prompt, the whole tool list and every retrieved document on every turn of every conversation, and the bill tracked that. The fix was two mechanisms working together: caching for the part of the prompt that does not change, and a budget for the part that does.

Where the tokens went on a document-heavy chat product

On a document chat product the input tokens fall into four categories, and only one of them is the user's question. The system prompt and tool definitions are identical on every request. The retrieved passages change with each question but repeat heavily within a conversation. The conversation history grows with every turn. The question itself is a few dozen tokens.

Looking at the prototype's requests that way made the problem obvious. The stable part was large and resent every time, so it should be cached. The history was unbounded, so it needed a ceiling. The retrieved passages were chosen by similarity alone, with no limit on how many, so a broad question could drag in more context than the answer needed. None of this showed up in a demo, where conversations are three turns long and the document set is small.

Designing the prompt so the cacheable prefix is stable

Caching is a prefix match, so the prompt has to be built in stability order: tools first, then the system prompt, then the conversation, with a breakpoint after the last stable block. The API renders tools before the system prompt, so one breakpoint on the final system block caches both. Anything that changes per request must come after that point.

The audit we ran on the prototype's prompt builder is the audit we now run on every project. A timestamp interpolated into the system prompt, a user's name in the preamble, a tool list assembled per request in a different order, a JSON blob serialised with unsorted keys: each one changes a byte near the front of the prompt and silently invalidates everything after it. The API does not warn you. The only signal is a cache_read_input_tokens field in the usage object that stays at zero, which is why we check it in a test after every change to prompt assembly, not once at setup.

The retrieved passages needed their own thinking. Within a conversation the same documents keep coming back, so we ordered the context so that the stable documents sat before the volatile ones and placed a second breakpoint at the end of the shared portion. Up to four breakpoints are allowed per request, and spending them at the real stability boundaries is the whole craft. Anthropic has since added a way to append operator instructions mid-conversation as a system-role message after the cached history, which avoids editing the top-level system prompt; check the current docs for which models accept it.

Two more things bit us. Caches are scoped to a model, so running the same prompt against two models is two cold caches, and a prefix below the model's minimum cacheable length will not cache at all, also without an error. The minimum varies by model, so a prompt that caches on one can silently stop caching on another.

Context-window budgeting: a per-request budget split three ways

A context budget is a fixed token allowance per request, divided between conversation history, retrieved passages and room for the answer, with each part trimmed to its share before the request is sent. It keeps long conversations inside the model's limits and inside a predictable cost, and it makes the trimming a deliberate decision instead of whatever the API truncates.

Current models have very large context windows, which is exactly why a budget is needed. Without one, every turn costs more than the last and the product's margin shrinks the longer a customer uses it. The platform's budget worked like this: the answer gets a reserved share so the model is never squeezed into a two-line reply; history is kept most-recent-first until its share is spent, with the oldest turns dropped; retrieved passages are ranked and admitted until their share is spent. The shares were configurable per workspace, which let the enterprise accounts that paid for deeper context have it without changing code.

The trade-off is real. A tighter history budget means the model forgets what was said twelve turns ago; a tighter retrieval budget means a question that spans many documents gets fewer of them. We tuned the shares against a set of real questions from the client's users, re-run after every change, and judged the answers rather than the token counts.

Streaming and caching together

Streaming and caching solve different problems. Caching shortens the time before the first token arrives, because less of the prompt has to be processed; streaming makes the rest of the wait feel short, because the answer appears as it is generated. The platform did both: chat streamed over Claude and OpenAI, with cache breakpoints on the Claude requests.

One practical note. The usage object with the cache fields arrives at the end of a streamed response, not the start, so the cache check belongs in the completion handler, and the logs should record it per request so a regression shows up as a trend rather than a surprise on the invoice.

Bedrock or the direct API

Prompt caching is available on the Claude API and on Amazon Bedrock, and the mechanics are the same: breakpoints on content blocks, the same stability rules. The differences are in pricing, which Bedrock sets separately from Anthropic, and in cache scope, which is per organisation on Bedrock and per workspace on the direct API.

Choose Bedrock when the client's data agreements, region or AWS spend commitments ask for it; choose the direct API when they do not and you want new API features on the day they ship.

We build for both. The knowledge platform ran on the direct APIs. For clients already standardised on AWS, the same prompt-building code runs behind the Bedrock client with the model identifier changed.

Trade-offs we hit

Cache invalidation when documents change was the first. A document edited and re-embedded changes the bytes of any passage retrieved from it, which invalidates the retrieval breakpoint for every conversation that includes it. The answer was to accept it: documents change less often than questions are asked, and a cold cache after an edit is correct behaviour.

Budget against answer quality was the second, covered above. The third was less obvious: caching rewards a frozen system prompt, and product teams like to iterate on system prompts. We put the prompt under version control with a review step, so a change was a release rather than a tweak, and the cache cold-start after each release was a known cost.

The checklist we use

The short version of this article, in the order we apply it.

• Build the prompt in stability order: tools, system, shared context, history, question.
• One breakpoint after the last stable system block; one at the end of the shared retrieved context.
• Nothing per-request before the first breakpoint: no timestamps, names, IDs or unsorted JSON.
• Serialise tools deterministically and never change the tool list mid-conversation.
• Assert cache_read_input_tokens is above zero on a second identical request, in a test that runs in CI.
• Set the context budget per request and per workspace, with a reserved share for the answer.
• Re-run a fixed set of real questions after every change to the budget or the prompt.

Questions we get about this

Does prompt caching change the answers? No. The cached prefix is the same processed prompt; caching changes what you pay and how long you wait for the first token.

When is caching not worth it? When the start of the prompt differs on every request. There is no shared prefix to reuse, and adding a breakpoint only pays the write premium.

Do I need a context budget if the model's window is large enough? Yes, for cost. The window is a ceiling on what is possible; the budget is a decision about what each request should cost.

Can you add this to an existing product? Usually in a focused engagement. Most of the work is in the prompt builder and the request logging, and it can be done without changing the user-facing product.

What we would do on your project

We start by reading your request logs and your prompt builder, then write down where the tokens go and what a cached, budgeted version would look like. That plan is the discovery sprint, from $2,500, and it is yours to act on with or without us. Book a free architecture call if you would like to talk it through first. The full story of the knowledge platform is in the case study, and the Claude and MCP development service covers the kind of build this article describes.