Cost engineering

How to Cut Your LLM Costs by 70%

Almost every AI system we're asked to review is spending 3–5x more on LLM inference than it needs to. Here are the five techniques we apply — usually in this order — to bring that cost down without touching product quality.

Naïve LLM implementations look the same everywhere: send the whole prompt to the biggest model, wait for the response, return it. It works. It also runs your monthly OpenAI or Anthropic bill 3–5x higher than it needs to be, and slower than it needs to be, and quietly puts a cost ceiling on how much your product can grow.

The engineering to fix this is not exotic. It's five well-known techniques applied in the right order. Here they are, ranked by return-on-effort.

1. Prompt caching (usually the biggest single win)

OpenAI, Anthropic and Google all now offer prompt caching — you pay a significantly reduced rate for prompt tokens that repeat across requests. The catch: the identical prefix has to appear at the start of the prompt.

Most systems get this wrong. They interleave system prompt, retrieved documents, user question and chat history in an order that changes on every request. Fix:

1. Put the truly-static system prompt first.
2. Follow with slow-changing context (user profile, tenant config).
3. Then the changing bits (retrieved chunks, user turn).

2. Small-model routing

Not every request needs the frontier model. A well-designed system routes "easy" requests to a cheaper, faster model and reserves the expensive model for hard ones.

The routing decision itself can be:

• Rule-based — heuristics on request features (length, entity count, presence of specific keywords). Simple, deterministic, cheap.
• Cheap-model triage — a small model classifies the request as easy/hard and only the "hard" ones escalate. Slightly more accurate, adds latency.
• Try-cheap-then-verify — always try the small model first. If an evaluator model marks the answer as uncertain, escalate to the big model.

Which pattern is right depends on the accuracy gap between models on your workload — which is why evals are a prerequisite for any of this. Without them, you can't tell whether the small model actually is worse.

3. Semantic caching

Different from prompt caching. Semantic caching stores previous responses keyed by an embedding of the request. When a new request comes in that's semantically similar to a cached one, you can return the cached answer without calling the LLM at all.

Works brilliantly for:

• Customer support FAQ-style workloads.
• Product search / recommendation queries.
• Documentation lookups.

Works badly for:

• Personalised outputs — the answer depends on who's asking.
• Time-sensitive information.
• Any workflow where cached answers can accumulate stale facts.

The one to watch: cache hit rate. Semantic caching only saves money if enough real user traffic actually hits the cache. On a support workload we recently audited, a well-tuned semantic cache eliminated 42% of LLM calls entirely.

4. Prompt compression

Long prompts are expensive per request. Two techniques help:

Context selection. In a RAG pipeline, don't stuff the top-10 retrieved chunks in. Rerank them, keep the top 3 that actually matter. In practice this often improves quality (less noise for the model to weigh) while cutting input tokens by 60–80%.

Compression models. Tools like LLMLingua compress prompts by removing tokens the LLM doesn't need. Aggressive but well-tested — we see 2–3x token reduction with sub-1% quality drop on many workloads.

5. Batching (for background workloads only)

If your workload is asynchronous — nightly summarisation, bulk classification, catalogue enrichment — use the batch API instead of the real-time one. OpenAI's batch API is typically half price; Anthropic's is similar.

Caveats: 24-hour SLA on turnaround, not suitable for anything user-facing. But for background jobs that don't need immediate answers, it's a straight 50% saving with no engineering downside.

The stacking effect

The reason we get to 70% (and often more) is that these compound multiplicatively:

• Prompt caching: ~60% off input tokens.
• Small-model routing sending 60% of traffic to a 5x-cheaper model.
• Semantic caching removing 30% of calls entirely.
• Prompt compression halving what's left.

The arithmetic on a typical RAG workload with those numbers lands somewhere between 70% and 85% cost reduction.

What you need to measure

None of the above is a "set it and forget it" toggle. Each technique adds complexity and introduces new failure modes. To deploy them safely you need three observability pieces:

1. Per-request cost tracking — every LLM call logged with tokens in/out, model used, cache hit/miss, latency.
2. Quality evals running continuously — so you can see immediately if the small-model routing has crossed a quality threshold.
3. A dashboard — cost per user, cost per feature, cost per customer segment. Once you have the numbers, the optimisation targets become obvious.

The bottom line

LLM cost engineering is currently one of the highest-ROI investments an AI-heavy product can make. A three-week cost-reduction sprint typically pays for itself within a month and improves gross margins for the life of the product.

The techniques are known. What's missing in most teams is the discipline to treat AI inference as a first-class engineering concern rather than a "just use the API" convenience.