Side Projects

I Measured Every LLM Cost Lever and Most of Them Lost

Eval-driven tuning of a multi-stage LLM pipeline: how I compare non-deterministic arms, which cost levers survived, and why a cache key has to ignore cache breakpoints.

By 24 min readUpdated
LLM EvalsCost OptimizationPrompt CachingAnthropicBootstrap CIBunTypeScriptSide Project

I am building a personalised, interruptible AI podcast app. A worker keeps a knowledge base fresh from the news, and on request a pipeline writes an episode from it. On 2026-10-22 I weighed where the money and the seconds go, using the llm_calls table of the dev database (calls from 2026-10-06 to 10-15) and the end-to-end runs of the previous round. The proxy I call models through is flat-rate, so I priced every call at public API list prices.

RunCost at list priceBiggest lines
Ingest, per dayabout $15.7triage 45 %, story editor 30 %, analysts 19 %
One 5-minute episodeabout $0.74writer 38 %, reviewer 27 %, Editor 20 %

Over that round and the prompt-tuning round nine days before it, I tested the obvious ways to shrink those lines: a cheaper model on a stage, a stage without extended thinking, fewer steps in the planning loop, skipping work that ran twice. Three changes were adopted. One is a latency win of 3.6 s with a confidence interval that excludes zero, measured on a small set. One removes redundant work: 40 % of a stage’s runs were rewrites of a brief with the same input. The third is a correctness fix found by running live episodes, a race that had story 1 written without its continuity notes in 2 of 4. Two model downgrades, two removals of extended thinking and a few smaller ideas lost, and the losses are the part I would not have guessed from reading the prompts.

Most of what follows is the method that let me call a lever a win or a loss. About a week later I used it on a second project, a training platform for the CRPE, the French primary-school teacher exam, with a different bench. Both projects also got an exact-response cache, and both caches hit the same trap in their key.

What the Pipeline Spends

Each stage name below stands for one kind of model call. Triage extracts the verifiable claims from a freshly ingested document. Domain analysts, one per field (tech, security, medicine and so on), turn the same document into claim cards. A story is a cluster of documents about one event, and the story editor writes its brief. The Editor, an agent with a tool loop, plans each episode from a request. Writers produce each segment’s script. The reviewer reads a written section against its claim cards and flags lines, and section.revise rewrites those lines and leaves the rest alone. Before the later segments are written, a small fast model called the Showrunner writes a continuity ledger: which terms each segment defines, which figures it owns.

Ingest is a background cost, so only its dollars matter. An episode is the opposite: a listener waits for it, so seconds count as much as cents.

One more cost sits outside my code. The proxy writes every call’s uncached input to a 1-hour cache, and its own system block keeps most of that from being read back. Priced the way the public API prices cache writes, that adds 22 % to ingest and 48 % to an episode. I can measure it but not change it from my code. Every lever below is compared at list price without it.

Comparing Arms That Are Not Deterministic

A model call gives a different answer each time. Two runs of the same prompt differ, a judge model grading them adds its own variance, and the provider’s speed changes during the day. A before/after comparison has to survive all three.

The proxy’s speed drifts. A base arm I re-ran three hours after its first run was 4.2 s slower with nothing changed. So draft timings are only compared between arms run side by side, in the same window, and never against a number recorded earlier. The commit for the Editor change says it in a few words: “arms run in the same window”.

The unit that varies independently is the request (or the case). Calls inside one request are not independent of each other. For each request I average its samples, take the difference between the two arms, and bootstrap the mean of those differences. The kit does this in a few lines:

// packages/evals/src/kit/stats.ts
/**
 * Percentile bootstrap of the mean: resample the values (one per case, its trials already averaged) with replacement.
 * Cases are the unit that varies independently; trials of one case are correlated and must not be resampled apart.
 */
export function bootstrapMeanCi(
  values: readonly number[],
  options: { iterations?: number; seed?: number; level?: number } = {},
): Interval | null {
  // ... 4000 iterations by default, seeded, 95 % level
}

The comparison script also reports a minimum detectable effect (MDE): the smallest true difference this many cases and trials could detect with 80 % power at a 5 % false-positive rate. It comes from the within-case standard deviation, which is what another sample of the same prompt moves a case by:

// packages/evals/src/kit/stats.ts
// (the doc comment gives the formula as 2.8 · √(σ_w² (1/t_before + 1/t_after) / n))
export function minimumDetectableEffect(
  sigmaW: number,
  cases: number,
  trialsBefore: number,
  trialsAfter = trialsBefore,
) {
  if (cases <= 0 || trialsBefore <= 0 || trialsAfter <= 0) return Number.NaN
  return 2.8 * Math.sqrt((sigmaW ** 2 * (1 / trialsBefore + 1 / trialsAfter)) / cases)
}

The MDE is what lets me write “within the MDE” next to a result instead of “no change”. In the Editor test below, the share of top stories served went from 0.54 to 0.53 with an MDE of 0.048. That says the experiment could not have seen a drop smaller than about five points. None showed up. The observed change was 0.01.

For subjective quality (does this script sound better?), a judge model compares two outputs, and its verdict can depend on which one it reads first. The pairwise scripts run the judge twice with the order swapped, and a preference counts only when both orders agree. A split is a tie. For a decision, the episode-level judge in the end-to-end bench asks once per pair in a seeded random order instead, which halves the calls and leaves no ties, and reports a sign-test p-value over the pairs.

The harness can also judge a run against its own repeat (an A/A comparison). That number is what “no difference” looks like, and for the reviewer, below, it decided the lever.

Every stage prompt has a version (editor@31), bumped on any wording change, so an arm names exactly what it ran. Scenarios about a listener’s history need the database as it was on a given day. The bench clones a template Postgres database per lane and purpose, then takes the clone back in time: documents and episodes after that day are deleted and story counts are recomputed. Each step of a multi-day scenario is drafted by the production planner at 14:00 of its day, after the earlier steps’ plans were stored as heard episodes.

Case count is the rule I broke first. In an earlier tuning round (section 8.2 of my findings doc), section.revise on a smaller model showed “no difference on 5 cases”. Five cases measured nothing, and the later run with 74 captured revisions says so below. The eval README gives the power I should have asked for: for a pairwise preference, about 110 pairs separate a 65:35 split from a coin flip, and 250 separate 60:40.

The Levers

Everything here was measured on cases drawn from the dev database or from captured real calls. Intervals are 95 % bootstrap intervals over cases.

LeverWhat I expectedMeasuredVerdict
Story brief waits for its pending analystsabout a third fewer story-editor calls40 % of runs were redundant rewrites (32 to 46 % each day), replayed offline from three days of callsadopted
Editor starts a subject-less draft from the browse the code already ranfewer model steps6 browse requests x 2 samples: 17.1 to 13.5 s (-3.6 s [-4.7, -2.2]), 3 to 2 steps, Editor input -24 %adopted
Run a first list for subject requests tooeven fasterfaster, but on-subject share -0.015, then -0.042 [-0.102, +0.026] with off-subject stories uprejected
section.revise on haikucheaper, fasteredits not worse 0.84 to 0.71 (-0.133 [-0.203, -0.065]), 3.5 to 2.5 srejected
story_editor on haiku, native structured outputcheaper8 of 25 stories got no brief, p90 6.9 to 26.6 s, judge 0.92 to 0.86rejected
section.review without upfront thinkingfaster3.9 s in both armsrejected
Triage on haiku with thinking off (round of 2026-10-13)half the latencylost recall on medical documents (below)rejected
Prompt-cache breakpoint after the instructions (2026-10-14)about 5k of the writer’s 13k input tokens read back per segment0 of 5,488 read back through today’s proxykept in code, saves nothing yet

The first row is covered in the next section, along with the race fix. The Editor row is in the table because it is the one lever that paid in seconds: a subject-less request spent its first model step on a list call the code can run itself. The rejected rows follow, in the order that taught me the most.

A Cheaper Model on the Rewrite Step

section.revise rewrites only the lines the reviewer flagged, so it is a short call: about 3.5 s on the writer’s model. Moving it to haiku looked like free money. On 74 captured real revisions, run on both arms, the share of edits that were not worse than the original line fell from 0.84 to 0.71. The share of edits that fixed a defect fell from 0.71 to 0.51, and rewritten lines passing the deterministic checks from 0.66 to 0.55. The time saved was 1 s. A listener does not feel 1 s on a segment that is written minutes before it is voiced, and the quality drop is not subtle. The earlier “no difference on 5 cases” was the same comparison with too few cases to see it.

A Reviewer That Disagrees With Itself

The reviewer thinks before it answers, so I tried turning that thinking off to save time. On 85 live reviews, both arms took 3.9 s. The review already spends about 43 tokens on thinking, so there was nothing to remove.

The more useful result was a by-product. The agreement between the two arms on which lines were flagged was 0.54. The base arm’s agreement with its own repeat (the A/A run) was 0.59. I have no interval on either number, but the gap is small next to what a real change would need to show: the reviewer agrees with a variant about as well as with itself. It is noisy, and any change to it has to beat that noise before it counts.

A Cheaper Model on the Story Brief

The story editor writes one structured brief per story, and it is the second largest ingest line. This stage had already failed on haiku once. In the stage-model matrix of section 8.2, 3 of 4 cases failed schema validation after 3 attempts, and I kept sonnet. The new test gave haiku native structured output on 25 real stories, and 8 got no brief. Its thinking tokens ran into the output cap: output went from 730 to 2,041 tokens, and latency at the 90th percentile from 6.9 s to 26.6 s. The judge score fell from 0.92 to 0.86.

The cause is one I had met before. Haiku’s thinking counts against the output cap, so a cap sized for the answer truncates the answer. In the prompt-tuning round of 2026-10-13, 3 of 24 background-selection calls and 18 of 100 analyst calls had been truncated that way, and the fix there was to size each stage’s cap for the thinking: 2,000 for background selection, 8,000 for the fast analysts (raised to 16,000 later), 16,000 for triage.

Thinking Off on the Biggest Line

Triage is 45 % of ingest, so in the 2026-10-13 round I tested the lever that halves its time: thinking off on haiku. On the curated set it scored as well as the version with thinking and halved latency. Curated sets are small and hand-picked, so I replayed real traffic. In the replay, claims per document on real medical preprints dropped by 29 % (5.08 to 3.62), against 12 % on the curated set. A separate paired run on 28 medical documents, thinking on against off, pointed the same way: claims per document fell from 6.46 to 4.79, and the share supported by their source from 0.92 to 0.82. Triage runs at ingest, so 12 to 13 s against 4 to 5 s is never felt, and the cost is about three times the output tokens. The largest line of my bill keeps its thinking, with its own 16,000-token cap.

A Faster Planner That Planned the Wrong Stories

The Editor’s browse change saved 3.6 s because a request without a subject always spent its first model step on one prescribed list call. Running a first list for requests with a subject too, from the topics and then from the Producer’s search terms, saved more time. It also planned stories that only touch the subject: on-subject share fell by 0.015, then by 0.042 with an interval of [-0.102, +0.026], and off-subject stories went up. That interval includes zero and also includes a 10-point loss, which is reason enough to stop. Requests with a subject keep their own searches.

The Breakpoint That Saved Nothing

On 2026-10-14 I added an Anthropic prompt-cache breakpoint after every Claude call’s instructions (commit c444b40). The proxy caches each request only at its end, so no cache entry ever ended where the instructions end, and a later call with the same instructions but a different prompt read nothing: 0 cache-read tokens on all 27 base writer calls of one run. With the breakpoint, I expected the 5.5k-token writer instructions to be read back, which would cover about 5k of the roughly 13k input tokens of each segment after the first.

The live probe, two calls per mode with a different prompt on the second, gave 0 of 5,488 tokens read back for the writer’s streaming call. The structured call got 669 of 5,682, which is only the tools prefix. The proxy inserts a system block of its own, derived from the first user message, ahead of my instructions, so the prefix differs on every call. In this pipeline, the only calls that read the instructions back are a tool loop’s own steps, which share a first user message. Direct calls without the proxy read 5,135 of 5,170 tokens back, so the breakpoint stays in the code, with that limit in its comment, until the proxy stops adding its block. I did not change the conversation’s shape to dodge the proxy, because the model would then see a different conversation.

Redundant Work and a Race

Waiting for the Analysts

A story brief was written as soon as a document’s triage was stored. Then the document’s domain analysts landed (p50 10 s, p90 2 min after triage), which changed the brief’s input, and a 5-minute rule deferred the rewrite. So 40 % of the story editor’s runs, between 32 and 46 % depending on the day, rewrote a brief about 5 minutes after the first with the same members. The story editor was about 30 % of ingest spend.

The fix is a pure function that says whether a brief should wait:

// packages/enrichment/src/stories/analyst-wait.ts
export const ANALYST_WAIT_WINDOW_SECONDS = 600

export function pendingAnalystWait(members: readonly MemberAnalystWait[], nowMs: number): AnalystWait | null {
  const windowMs = ANALYST_WAIT_WINDOW_SECONDS * 1000
  const pending: AnalystWait['pending'] = []
  for (const member of members) {
    if (nowMs - member.triagedAtMs >= windowMs) continue
    const missing = member.wanted.filter((analyst) => !member.stored.includes(analyst))
    if (missing.length === 0) continue
    pending.push({ documentId: member.documentId, analysts: missing })
    // ...
  }
  // ...
}

A member’s analysts are pending while it was triaged less than 10 minutes ago and one of the analysts its gate lets through has no stored analysis. The wait ends with the window, so an analyst that never lands (refused, deferred, failed for good) delays a brief by 10 minutes at most. Urgent refreshes never wait. The final input of each brief is the same as before.

Replaying three days of recorded calls offline predicted about a third fewer story-editor calls, and what I have seen in production since it shipped looks very good.

Story 1 Written Before the Ledger

I found this one by watching live episodes. The cost data never showed it. episode.start sends the cold open’s writer and then runs the Showrunner, which takes about 6.6 s, while the cold open takes 4 to 5 s to write. The next writer was queued as soon as the cold open was saved, so story 1 was often written without the ledger. Live, that happened in 2 of the 4 normal episodes, the latest included, with the Showrunner finishing 2.5 s after story 1 started.

The end-to-end harness never saw it because it runs the Showrunner first. Now every writer from the first ledger-reading segment on waits until continuity is settled: the ledger saved, the Showrunner failed or skipped, or a bounded wait elapsed. The cost is 0 to 3 s before story 1 starts.

Two Caches, Two Different Jobs

An exact-response cache stores the answer to a call and returns it when the same call comes again. No tokens are spent and the answer is the same. It helps when the same call is repeated by design: replaying an eval run, retrying a failed job. A provider prompt cache (Anthropic’s cache_control breakpoints) stores the model’s processing of a shared prompt prefix and discounts those input tokens on later calls. The model still generates a fresh answer. It helps when many different calls share a long start.

The Exact Cache in the Eval Harness

Every eval call in the podcast app goes through an on-disk cache keyed by a hash of the model id and the complete call parameters (prompt, tools, settings, response format). A re-run with unchanged prompts and models costs nothing.

When I added the prompt-cache breakpoint to the platform, the key started to change. The breakpoint is a field on a message or a content part, and the key hashed the full prompt. Every recorded call of every lane would have missed. A breakpoint changes what a call is billed, not what the model answers, so it must not be part of the call’s identity. The fix strips it before hashing:

// packages/evals/src/kit/cache.ts
const hasCacheControl = (options: ProviderOptions | undefined): boolean =>
  options?.anthropic?.cacheControl !== undefined || options?.anthropic?.cache_control !== undefined

/**
 * A prompt-cache breakpoint (`providerOptions.anthropic.cacheControl` on a message or a content part) changes what the
 * call is billed, not its answer: the key leaves it out, so a request keys as it did before the breakpoint existed.
 */
function keyedPrompt(prompt: unknown): unknown {
  return Array.isArray(prompt) ? prompt.map(stripCacheControl) : prompt
}

An item without a breakpoint is returned untouched, so nothing else about the key moves, and other provider options still count. A unit test checks both: a request with the breakpoint on the system message or on a content part keys like the same request without it. As a replay check, I ran one suite in cache-only mode with the breakpoint on, and 4 of 4 cases replayed with 0 live calls (commit 9dc901d).

The rule I took from it: a cache key holds everything that can change the answer and nothing that only changes the price. It is easy to break in both directions. Leave out a field that matters and you replay a wrong answer. Leave in one that does not and you pay again for every call you already paid for, and you may not notice until the bill arrives. Retries are a third case. A schema failure is retried with the same request, and a cached failure would be replayed as final while production would draw again. So from the second attempt on, the attempt number joins the key, and the first attempt hashes as before, which keeps recorded calls valid.

The Exact Cache in the CRPE Pipeline

The CRPE platform’s first real build had 799 calls that repeated an input already answered, about 4.6M input and 1.3M output tokens, which the commit message puts at a fifth of the spend. Two causes are named there: an exam part extracted again after a failed solve, and a solve on the most expensive model paid for twice after its blind solve hit a rate limit.

The cache is a table whose primary key is the whole identity of a call:

-- packages/db/drizzle/0008_llm_response_cache.sql
CREATE TABLE `llm_response` (
	-- ...
	`output` text NOT NULL,
	-- ...
	PRIMARY KEY(`prompt_id`, `prompt_version`, `model`, `input_hash`)
);

The key raises the same question as in the eval harness. The prompt’s version hashes its template, so rewording a prompt misses on purpose. The shared prefix named for the prompt cache is deliberately left out of that hash, which is the eval harness’s rule again: a billing hint does not change the call’s identity. The model is in the key, so a fallback model’s answer is stored under the model that gave it and a rerun replays it. A stored answer is validated against the output schema again on replay, and the call goes to the model if that fails. A replay is logged as a cache hit with zero tokens. One prompt opts out, because its generation retries on purpose, and --force asks the model again and stores the new answer.

The same commit adds the second kind of cache. A prompt can name the start of its text that other calls share. That start is sent to Anthropic as its own text part with a breakpoint, and the extraction stage runs the first part of each exam subject before its siblings so they read the start from the cache. The call log now records cache reads and writes as the provider reports them. The first real builds with it look very good.

The Same Method on a Second Project

Eight days later I wrote a bench for the CRPE pipeline’s lesson and exercise stages (commit 8d2d2ab, results dated 2026-10-30). It runs the real workflows in memory on a copy of the real database and routes any prompt or role to any provider:model@level. Every arm is logged and priced. All are judged the same way, by blind pairwise judges in both orders against a baseline, with a noise floor. The sample is 34 notions, the syllabus topics a lesson covers.

The noise floor is the most useful line of its README. Production Opus lessons compared against bench Opus lessons, the same model on the same notions, win only 40 % of the time with a Gemini judge and 46 % with an Opus judge. A candidate winning 45 % of the time is a tie. Read against that floor, the cheaper authors were not close:

Lesson authorVerifiedCost per lessonWin rate against Opus
Opus (baseline)34/34$0.080floor 40 to 46 %
Sonnet 5.5 at medium effort28/34$0.0354 %
Haiku 5.532/34$0.0133 %
Opus at low effort33/34$0.04625 % (Gemini judge), 15 % (Opus judge)

The cheaper authors mostly pass the verifier (Haiku 32/34, Sonnet 28/34) but write lessons with about 2.5 times more substantive errors and much less depth, so the verifier gate does not protect that quality. The author stays Opus. The savings came from elsewhere, from stages where the measurement said a smaller model was equivalent:

ChangeMeasured effect
Verify role on Gemini 3.8 Flash (Haiku as fallback)lesson 0.080to0.080 to 0.052, judged equal to baseline: 2 repairs instead of 8, 0 of 33 false flags on clean lessons
New check role for blind checks, on Haiku 5.5 (Flash as fallback)0.0028to0.0028 to 0.0003 per item at the same accuracy (89/90 true keys accepted, 89/90 wrong keys caught)
Exercises written in one call per notion, with new authoring rules0.038to0.038 to 0.0215 per verified exercise, author input 728k to 234k tokens across the 34 notions, p50 57 s to 33 s per notion

The exercise row needs a caveat. Batching alone bought the cost and speed, and with the old rules it only tied the baseline as a set (48 % win rate). The new rules moved the preference. Against the same baseline, one call per exercise with the new rules won 62 %, and batch plus the new rules won 82 % with an Opus judge and 74 % with a Flash judge. At equal rules, batch beat single 61 % as a set, for the variety of writing four exercises together. Per-item rubric quality for the winner is 3.47 out of 5, against 3.56 for the baseline, so I bought variety and cost with a little polish per item.

The bench cost about $100 of API spend, mostly the judges. About 2,700 calls to one Gemini model, together with production, exhausted its gateway quota and put it in a cooldown of about 6.6 days. The same day a model role gained comma-separated fallbacks (MODEL_VERIFY=a,b): when the first model fails after its retries, the call asks the next one, and each attempt is logged with the model that made it.

What I Would Set Up on Day One

  1. A call log from the first call: stage, model, version, tokens in and out, cache reads and writes, latency, and whether it was a replay. The cost table at the top of this article comes from that log, and I could not have written this article without it.
  2. Prompt and model versions that go into every log row and every cache key. Without them, a cost number cannot be tied to the code that produced it.
  3. An exact-response cache for the eval harness, with a key that leaves out billing-only fields, so a re-run of an unchanged suite makes no live call.
  4. A paired comparison script with a bootstrap interval, an MDE and an A/A mode, before the first A/B. An interval that includes zero and a loss of 10 points is a different result from “no change”.
  5. Arms run side by side, always. Latency numbers from different hours do not compare.
  6. A rule that any lever on a model tier gets tested on real traffic as well as on the curated set. Thinking off on triage cost 12 % of claims per document on the curated set and 29 % on real preprint documents.
  7. A minimum case count per decision. The haiku revise result went from “no difference” to a clear loss when the case count went from 5 to 74.

What Is Still Open

The reviewer’s agreement figures (0.54 and 0.59) have no intervals. The Editor’s browse win was measured on 6 browse requests with 2 samples each, out of 24 requests in the bench, so the interval is honest about a small set and says little about the distribution of real requests.

Three ideas were measured and left alone, as my call. Raising the analyst gates saves 34 % of analyst calls and drops the reading of 17 % of the analysed documents in planned stories. Releasing playback earlier, simulated on 72 episodes, gives first audio 5.5 to 8 s sooner with 0 to 6 % short stalls, and it needs a new progress signal. One source family is about 10 % of ingest for 2 of 79 planned stories. None of the three has a clean answer yet.

The prompt-cache breakpoint is the one I want to revisit. Direct calls read the prefix back, and through the proxy the saving is zero. The next step is to run a real day of episodes on a direct API key and price the difference.

100%