Controlling LLM Costs in Production

The bill is usually input tokens, not output, which means the biggest savings are in what you send rather than which model you pick. The levers in order of return, the guards that stop a surprise, and the metric that actually matters.

The first invoice for a language model feature in production surprises almost everyone, and the surprise is usually in the wrong direction. Teams expect the cost to be in the answers. It is in the questions, or more precisely in everything attached to the questions.

A retrieval system sends the model a system prompt, the conversation so far, and several retrieved passages, on every single turn. The reply is a paragraph. The input can be twenty times the output, and input is where the money goes.

Once that is clear, the levers sort themselves into an order.

First, know what you are measuring

Cost per token is the number the provider gives you and it is almost useless for decisions.

The numbers that matter are cost per conversation, cost per resolved query, and cost per feature. A support assistant that costs more per conversation than the human it was meant to assist is a failed project regardless of how cheap each token was.

Log tokens in and out on every request, tagged by feature, by user and by conversation. Without that you are reading a monthly total and guessing which part of the product produced it.

The levers, in order of return

Send less context. This is the largest lever by a distance and it is the one teams reach last. If you retrieve twenty passages and send them all, you are paying for fifteen that were not relevant. Rerank and send five. Chunk more carefully so each passage carries less dead weight. We covered both in separate pieces on chunking and reranking, and the cost argument alone justifies them.

Trim the system prompt. Prompts grow. Every incident adds a rule, every edge case adds a paragraph, and six months later the instructions are two thousand tokens sent on every turn. Review it quarterly and cut what is no longer load-bearing.

Limit conversation history. Sending the entire conversation on every turn means the cost of turn ten includes turns one through nine. Summarise older turns, keep the recent ones verbatim, and cap the total.

Route by difficulty. Not every query needs your most capable model. A cheap classification step decides whether a request is simple, and simple requests go to a smaller, cheaper model. Escalate only when the small model is uncertain or the question is clearly complex. For most business assistants the majority of traffic is simple.

Cache. Two different kinds. An exact-match cache returns a stored answer for a question you have seen before, and in most businesses a small number of questions account for a large share of volume. Separately, several providers now discount the repeated prefix of a prompt when it is unchanged between requests, which rewards keeping your system prompt and reference material stable at the start of the input. Structure your prompts to take advantage of it.

Constrain output. Set a maximum length and instruct brevity. Models given no limit produce more than needed, and a support answer does not need four paragraphs.

Batch what is not interactive. Classification, enrichment, summarisation and evaluation runs do not need an instant reply. Several providers offer substantially discounted processing for work submitted in batches with a delayed turnaround. Use it for everything that is not a person waiting.

The guards that prevent the surprise

Cost control is partly optimisation and partly not getting ambushed.

  • A hard spending limit at the provider, set below the point where you would be in trouble, with an alert well before it.
  • Per-user rate limits. One person, or one script, can loop a public assistant indefinitely. Limit requests per user per hour and watch for outliers.
  • A maximum conversation length, after which the assistant ends the session or hands over.
  • Separate keys for development, testing and production, so a test loop cannot consume the production budget and so you can see which is which on the invoice.
  • Retry limits. A retry policy that tries five times on every failure, during a provider outage, is a bill multiplier.
  • Evaluation budgets. Running your evaluation set is right and it costs money every time. Know what a run costs before scheduling it nightly.

Model choice, honestly

The instinct is to pick the cheapest model that works, and the word doing the work in that sentence is works.

A cheaper model that answers wrongly one time in twenty is not cheaper. It costs a wrong answer to a customer, a support ticket, a lost sale, or a reputational problem, none of which appear on the provider’s invoice. Price the cost of a bad answer in your context before comparing per-token rates.

The sensible position is usually a capable model for anything consequential, a cheaper one for routing and simple answers, and measurement to confirm the cheaper one is actually adequate on your evaluation set rather than on a benchmark.

Self-hosting, and when it stops being attractive

Running an open model on your own infrastructure removes the per-token charge and adds everything else: hardware, operations, updates, security, and somebody who is responsible when it degrades at two in the morning.

It makes sense in three situations. Very high volume of simple tasks, where per-token pricing genuinely dominates. Data that cannot leave your infrastructure for regulatory or contractual reasons. Or a latency requirement that a hosted service cannot meet.

Outside those, it is usually a way of converting a predictable bill into an unpredictable engineering commitment. Be honest about whether you have the operations capacity, because a self-hosted model nobody maintains is worse than a hosted one you pay for.

The Indian arithmetic

Two points specific to businesses here.

Pricing is almost always in dollars, which means your cost per query moves with the exchange rate and your finance team should know that.

More importantly, per-query economics matter enormously for consumer-facing products where revenue per user is modest. An assistant that costs a few rupees per conversation is fine for a business with high-value enquiries and ruinous for one serving a large volume of low-value interactions. Work out the cost per conversation against the revenue per user before launch, not after, and let it decide the architecture.

The hidden costs

Four things that appear on invoices and surprise people.

Development and testing traffic that was never separated from production. Evaluation runs on large sets. Retries during outages. And the long tail of conversations where somebody pasted an entire document into the chat and the system dutifully sent it to the model on every subsequent turn.

That last one argues for input limits as well as output limits.

What is a waste of effort

Optimising token counts before fixing retrieval. Sending five relevant passages instead of twenty mediocre ones saves more than any prompt compression trick.

Switching providers to chase a price list. The migration cost usually exceeds the saving, and the saving disappears at the next pricing change.

Fine-tuning to reduce prompt length. Occasionally justified, usually a large project undertaken to save a modest amount. The decision framework is in fine-tuning or RAG.

Building an elaborate cost dashboard before the basic per-feature logging exists.

Choosing the cheapest model and tolerating wrong answers. Covered above, and worth repeating because it is the most common false economy.

A sensible order of work

  1. Log tokens in and out per request, tagged by feature and user, from day one.
  2. Set hard spending limits, alerts, and separate keys for production and everything else.
  3. Add per-user rate limits and a maximum conversation length.
  4. Fix retrieval: rerank, send fewer passages, chunk tighter.
  5. Trim the system prompt and cap conversation history.
  6. Add an exact-match cache for repeated questions, and structure prompts for prefix caching.
  7. Route simple queries to a cheaper model, verified on your evaluation set.
  8. Move every non-interactive job to batch processing.
  9. Review cost per conversation monthly against the value a conversation produces.

The architecture these levers apply to is set out in building a RAG chatbot on your own content, and the safety constraints that must survive cost-cutting in building AI agents that use your data safely.

If your invoice is growing faster than your usage and nobody can say why, get in touch, or see what our AI development work covers.