Every team that ships an LLM feature arrives at caching within a month, because the invoice and the latency both point at it. Most of the questions a support bot or a site assistant gets are the same questions, phrased slightly differently, and paying for a fresh generation each time feels wasteful. It is wasteful. The trouble is that the obvious cache, keyed on the question, serves last week’s answer to this week’s question, and the first time it tells somebody a price that changed on Monday, the feature loses the trust it took months to build.
This is how we think about caching for the chatbots and assistants we build, including the ones described in how to build a RAG chatbot on your own website content.
Three different things called caching
They get confused with each other, and they solve different problems.
Provider-side prompt caching. The model provider keeps the processed form of a prompt prefix, typically the system prompt and any large documents you send every time, so that subsequent requests sharing that prefix are cheaper and faster. You still get a fresh generation. This is almost always worth switching on, it carries no staleness risk because the model still reads the latest user message and retrieved context, and the only design work is ordering your prompt so the stable part comes first and the changing part last.
Retrieval caching. In a retrieval-augmented system, the expensive step before generation is embedding the query and searching the index. Caching the retrieved chunks for a given query is cheap and safe as long as the cache is cleared when the index changes.
Response caching. Storing the final generated answer and returning it without calling the model. This is where the savings are largest and where every staleness problem lives. The rest of this piece is mostly about this layer.
What goes in the cache key
The mistake is keying on the user’s question alone. The answer depends on more than the question, and everything it depends on belongs in the key.
- The normalised question: trimmed, lower-cased, whitespace collapsed. Not stemmed or otherwise altered beyond that.
- The model identifier and the generation parameters. A different model or temperature is a different answer.
- The system prompt version. When you change the instructions, every cached answer produced under the old ones is invalid.
- A hash of the retrieved context. If the answer was grounded in three chunks of your content, the key includes a digest of those chunks. When the content changes, the chunks change, the digest changes, and the stale entry is simply never hit again.
- Any user or session attribute that changed the answer: language, region, plan tier, logged-in state. If the answer is personalised, it is usually not cacheable across users at all.
The context hash is the piece that does the most work. It turns the staleness problem from an invalidation problem, where you have to find and delete the entries that depend on changed content, into a non-problem, because a changed page produces different chunks and therefore a different key. The old entries age out on their own.
Exact-match caching: safe, and lower hit rate than you hope
With the key above, a hit means the same question, the same instructions, the same model and the same grounding content. The answer is as valid as it was the first time. The hit rate is lower than teams expect, because people phrase the same question many ways, but the hits are free and risk-free.
Two ways to raise it without introducing risk. First, normalise harder at the edges: strip trailing punctuation, collapse common greetings, remove the please. Second, cache at the level of the retrieval result rather than the question: two different phrasings that retrieve the same chunks and are asking the same thing can share an answer if you add a cheap classification step to confirm they are the same intent. That second step is where semantic caching begins, and it needs care.
Semantic caching: where business answers go wrong
Semantic caching embeds the incoming question, finds the nearest cached question above a similarity threshold, and returns its answer. It raises the hit rate a lot. It also fails in a specific, predictable way on business content.
Questions that differ in one critical detail embed close together. Does the warranty cover accidental damage and does the warranty cover water damage are near neighbours. What is the delivery charge within the city and what is the delivery charge outside the city are near neighbours. The refund policy for domestic orders and for international orders are near neighbours. A threshold loose enough to catch rephrasings is loose enough to merge these, and the result is a confident, wrong answer that looks like a correct one.
If you use semantic caching on anything with prices, policies, eligibility or dates, do one of two things. Either set the threshold tight enough that it catches only true rephrasings, accept the lower hit rate, and measure the false hits by sampling. Or add a verification step: when a semantic hit is found, send both questions to a small, cheap model and ask whether they are asking the same thing, and only serve the cached answer if it says yes. The verification costs a fraction of a generation and catches the near-neighbour failures.
Never semantic-cache anything that includes a number the user supplied. An order value, a date, a quantity. Those questions are different every time by definition.
Time to live, by what the answer depends on
Even with a context hash, you want a ceiling on how long an entry lives, because not everything that can change is in the retrieved chunks.
Set the lifetime from the content type the answer was grounded in. Answers grounded in evergreen explanatory pages can live for days. Answers grounded in pricing, availability, timings or policy pages should live for hours at most, and should be cleared directly when those pages change. Answers that drew on anything time-sensitive, a current offer, a stock figure, a schedule, should not be cached at all, and the simplest way to enforce that is to tag the source pages and let the tag decide.
Direct invalidation on content change
The context hash handles most staleness, but belt and braces are cheap. When a page in the source content is updated, re-index it, and purge every cache entry whose context included a chunk from that page. That means storing, alongside each cached answer, the identifiers of the pages it was grounded in. A small table, one row per answer per page, and a delete query on publish.
For a WordPress source, that is a hook on save that calls your indexing endpoint and your purge endpoint. For a headless CMS, it is the webhook you already use to rebuild pages. The same event that updates the site updates the assistant.
What must never be cached across users
Anything that includes the user’s own data. A question that contains an order number, an email address, an account detail or a document the user uploaded produces an answer that belongs to that user. Cache it per session if you like, with a short lifetime, and never let it be served to anyone else. The failure mode, one customer seeing another customer’s details, is the kind that ends a project.
Detect it at the key-building stage: if the prompt contains anything matched by your personal data patterns, or if the retrieval touched a user-scoped source, the entry is per-user or not cached at all.
Also think about cache poisoning. If a user can get an answer cached by asking a question, and the answer was shaped by something the user injected into the prompt, the next user with a similar question gets the poisoned answer. Cache only answers produced from your own retrieved content, and never cache an answer whose generation was flagged by your safety checks.
Refusals and errors
Cache refusals briefly. If the assistant correctly declines a category of question, serving that decline from cache is fine for a short period and stops a loop of repeated attempts from costing money. Do not cache errors, timeouts or truncated outputs; a cached failure is a failure that repeats.
Streaming and caching together
A cached answer can be returned instantly, and a user who has seen the assistant type out answers token by token notices when one arrives all at once. Either stream the cached answer at a plausible pace, which feels dishonest once people realise, or simply show it immediately with no typing effect. We prefer immediate, with the same formatting as a streamed answer. Consistency matters less than speed, and speed is the point.
Measuring whether it is working
Four numbers, on a dashboard the team actually looks at:
- Hit rate, split by exact and semantic.
- Cost per answered question, before and after, which is what the finance side cares about.
- Staleness incidents: answers served from cache that a sampled review or a user report found to be out of date. If this is not zero, the lifetimes are too long or the hash is missing something.
- False semantic hits: cached answers served for a question that was not actually the same, found by sampling hits weekly and reading them.
The last two are the ones that protect the feature. A cache that saves money and occasionally lies is worse than no cache.
What is a waste of effort
Caching before switching on provider-side prompt caching. The provider layer is free money with no risk; do it first and measure again.
Keying on the raw question and nothing else.
Semantic caching with a loose threshold on a pricing or policy bot.
Building a bespoke cache store. A key-value store with expiry, which you already run, is enough.
Caching the output of an agent that took actions. The actions happened once; replaying the summary of them is a lie. The agent patterns in how to build AI agents that use your website’s data safely are mostly uncacheable by design.
A sensible order of work
- Switch on provider prompt caching and reorder the prompt so the stable prefix comes first.
- Cache retrieval results, cleared on re-index.
- Add exact-match response caching with the full key: question, model, parameters, prompt version, context hash, user attributes.
- Set lifetimes by source content type, and tag time-sensitive pages as never cached.
- Store page identifiers per cached answer and purge on publish.
- Exclude anything containing user data from the shared cache.
- Only then consider semantic caching, with a tight threshold or a verification step, and sample the hits every week.
The same decisions apply to a WhatsApp assistant, with the added pressure that people expect replies in seconds; see how to build a WhatsApp chatbot grounded in your website content. If your assistant’s bill is the thing stopping it from going live, get in touch.