Nobody has ever hired me to do LLM cost optimization. They hire me because a feature they already shipped is quietly eating the subscription that was supposed to pay for it, and somebody in a board meeting asked why gross margin dropped twelve points.
The conversation always opens the same way: "should we switch to a cheaper model?" Almost always, no. The model is rarely where the money went.
Where LLM cost optimization actually starts
Before I wrote a line of generation code for Threadovo, I gave myself a rule: no credit may cost more than a cent to serve. The number was arbitrary. Having a number at all was not. It killed three feature ideas before they got built and forced a two-stage approval gate into the video pipeline, because generating four clips so a human could throw away three of them was never going to fit inside a cent.
A cost target you set before you build changes the architecture. A cost target you discover from an invoice only lets you tune constants.
If you already shipped without one, you're in tuning mode. Fine. Here's the order I work in.
Measure cost per finished job, not per request
Per-request cost is the wrong unit and it will lie to you.
An agent that finishes a task in four cheap calls beats one that finishes in a single expensive call plus three retries, but per-request accounting makes the second one look better. Log token usage keyed by a job ID, sum by job, and look at p95 rather than the mean. The mean is dominated by your median user. The p95 is the user who's actually costing you money.
When I did this on a client's document pipeline, about 60% of spend came from documents that failed validation and got reprocessed. The fix was a parsing bug. No model change, no prompt rewrite.
Caching is the only lever with no tradeoff
Everything else on this list trades something away. Caching doesn't, and most teams have it configured wrong or not at all.
The economics barely need analysis. On Anthropic's API, cache reads run about a tenth of the base input price and a cache write costs about 1.25x. Two requests against the same prefix and you're already ahead. Most other providers land somewhere similar.
The catch is that prefix caching is a byte-for-byte match on everything before your breakpoint. All of these silently destroy it:
- A timestamp or a request ID in the system prompt
- Serialising a config object whose key order isn't stable
- Building the tool list by iterating a map instead of a fixed array
- A feature flag that reorders two paragraphs of instructions for 5% of traffic
None of them throw an error. You just pay full price forever, and the code looks correct in review. Read cache_read_input_tokens off your responses and put it on a dashboard. If it's zero across repeated calls that should share a prefix, something upstream is varying, and finding it is usually a twenty-minute job with a diff of two serialised prompts.
Move work off the critical path
Batch endpoints are typically half price for anything nobody is sitting and waiting for.
More work qualifies than people assume: nightly digests, backfills, re-embedding after a chunking change, classification sweeps, and your own eval runs. On Scrivane, course-draft generation and blog images both fall into this bucket. A tenant clicking "draft this module" doesn't need a response in 900ms. They need it to be there when they come back.
So ask whether the user is genuinely blocked. Often the honest answer is that you made it synchronous because that was the easiest thing to write.
Route down, but prove it with evals first
Model routing works. The published numbers are real: send the hard fraction to the strong model and the rest to a small one, and you can hold most of the quality for a fraction of the spend.
The reason most teams never do it is that they can't tell whether quality dropped. That's an evals problem wearing a cost problem's clothes, and it's worth solving first. I wrote about how to test AI features when the output changes every time because you can't safely make any of these tradeoffs without a graded set of real inputs to check against.
Two things I'd push back on before anyone builds a router.
Try the good model at lower reasoning effort on the same tasks first. On current models that often matches an older model at full effort, and it keeps you on one model instead of two codepaths.
Caches are scoped per model. A cascade splits your traffic across two cache namespaces and can end up costing more than the single model it replaced, especially on chatty agent loops with a big shared system prompt.
Cut the context before you cut the model
Agentic workloads burn multiples of what a chat turn does, and most of that is input tokens you're resending every turn.
The waste tends to sit in the same places. Tool definitions that grew to thirty tools when the agent only ever reaches for six. Conversation history replayed in full when a summary of turns one through eight would do. Retrieval handing back twenty chunks because someone set topK once and never revisited it, which is also one of the reasons your RAG pipeline is probably broken. Trimming retrieval usually improves answer quality at the same time it cuts the bill, which makes it the easiest sell in the room.
Put the ceiling in the code
Some workloads have no cheap lever, and it's better to say so than to optimise something that won't move.
Real-time voice is the clearest case. On Intavue the interviewer has to answer inside a conversational beat, so batching is off the table and caching helps far less than it does on text. The only real control is a hard cap on session length enforced in code, plus metering that ends the session when the budget runs out.
That's the part people skip. A forecast in a spreadsheet won't stop anything. What stops it is a counter in your database that refuses the next call, which is most of why I keep arguing for credit-based billing on AI features. Charge in units that track your own cost and a user can't outrun your margin, however enthusiastic they get.
What I'd check first
In order, on a codebase I'd never seen before:
- Is
cache_read_input_tokensabove zero on repeated calls - What does one completed job cost at p95, not per request
- How much of the spend is retries and failed validations
- What runs synchronously that nobody is waiting for
- How many tools and how much history go into every agent turn
The first four are free. Nothing there degrades output, which is why they come before any argument about which model to use.
Your AI feature doesn't have to be cheap. It has to leave enough room that a heavy user is still a good customer instead of a problem you have to price around.