Four Counterintuitive Lessons From Months of LLM Experiments: The Cheap Model Is the Expensive One

Four Counterintuitive Lessons From Months of LLM Experiments: The Cheap Model Is the Expensive One

We spent months running model comparisons across pvl-dual-rail, the AutoGEO content pipeline, and the Studio image workflow, and accumulated a set of invoices that are not pretty but are very instructive. This piece collects four conclusions that run against intuition — and they share one thing in common: decisions made by looking only at the price per million tokens are wrong almost every time.

The conclusion up front: the cost of an LLM is not "unit price × volume." It is "unit price × volume × number of retries ÷ probability of getting it right the first time." When the denominator is ignored, every cost-saving decision is losing money.

1. Route different tasks to different models — don't run everything on one

The easiest change to make, and the one with the most direct payoff, is to stop handling all work with a single model.

Our pipeline contains at least four kinds of task with completely different characteristics:

Sending mechanical work to a flagship model means paying for reasoning capability you don't need. Sending open-ended reasoning to a small model means paying for output that "looks finished but isn't usable" — and the latter is far more expensive, because the error is only discovered downstream.

In practice the answer is routing: dispatch requests to different model tiers based on task type, input length, and cost of failure. That is exactly why we open-sourced `pvl-dual-rail` — it handles precisely the question of which requests travel on which rail.

The test to apply: ask "if this task is done wrong, how long until anyone notices, and what does the correction cost?" If the answer is "immediately, and one re-run fixes it," use the cheap model. If the answer is "the client tells us a few days later," use the expensive one.

2. Cheap is not always optimal — count the retries

This is the most counterintuitive point. Suppose you have two models:

• Model A: low unit price, but roughly a 55% chance of getting this class of task right the first time
• Model B: 5 times the unit price of A, with roughly a 92% first-pass success rate

On unit price alone, A is 5 times cheaper. But once retries are included, A needs an average of 1.8 attempts to produce one usable result while B needs only 1.09 — the real cost gap narrows from 5× to under 3×. And that is before the genuinely large expenses:

• Human review time. Close to half of A's output needs a person to read it, send it back, and re-issue instructions. That time is far more expensive than tokens.
• Debugging and tracing. Low-quality output is usually not "obviously broken" but "looks right, wrong in the details." That kind of error takes time to catch.
• Downstream contamination. If bad content reaches the knowledge base or a published page, cleanup costs dozens of times the original generation cost.

Here is a case we actually hit: after switching one extraction task to a cheaper model, the token bill dropped by 60% — but the hours spent that month on manual correction made overall delivery slower. The invoice looked better and the cost went up.

The test to apply: cheap models suit tasks where being wrong doesn't matter, re-running is cheap, and errors are immediately visible. If even one of those three conditions is missing, run the numbers again.

3. Use the expensive model and get it right the first time

This follows naturally from point 2, but deserves its own section, because it involves a cost that is easy to overlook: the price of rebuilding context.

When you use a cheap model on a complex task, fail, and retry, you are not only paying the token cost of the second attempt. You also:

• Resend the full context again (with a long prompt, this is the dominant expense by itself)
• Absorb the accumulated latency of multiple rounds
• Possibly roll back already-completed steps in a multi-stage pipeline

Getting it right once saves you re-running the whole chain. This is especially pronounced on long-context tasks — when the prompt alone is tens of thousands of tokens, the cost of "one retry" approaches the cost of re-running the entire task.

There is one more cost that is hard to quantify but entirely real: the confidence cost. Once a team knows that roughly half of a pipeline's output needs a second look, people stop trusting it by default, so every single item gets checked by hand — and the value of the automation disappears. Using a reliable model to push the pass rate above 90% is what lets human checking downgrade from "inspect everything" to "spot-check," and the benefit of that shift far exceeds the price difference between models.

The test to apply: if a task is going into an automated pipeline where nobody reviews each item, use the best model you can afford. The precondition for automation is trustworthiness, not cheapness.

4. A max_tokens set too low means you paid and received nothing

This is the easiest mistake to make and the most painful, because its failure mode is the most concealed.

When output is truncated by `max_tokens`, you have already paid the full input token cost plus every output token generated before the cut. What you receive is an incomplete fragment — JSON missing its closing brace, an article stopping mid-sentence, half a table. The recovery rate in this situation is zero: full price paid, nothing usable received.

Worse, it frequently does not raise an error. The API returns 200, the content looks like a success, and the problem surfaces only when downstream parsing fails — or worse, when parsing succeeds but content is missing.

What we do:

1. Measure first, then set. Run the actual task 20–30 times, record the distribution of output lengths, take P95 and add 30–50% headroom, instead of picking a round number by feel.
2. Monitor `finish_reason`. It is the most direct signal available. If the proportion of `length` (truncated) exceeds 1–2%, the ceiling is set too tight.
3. Leave extra headroom for structured output. For output with closing structure such as JSON or tables, truncation is far more destructive than it is for plain prose — truncated prose still leaves a usable first half, whereas truncated JSON is a total write-off.
4. Set it per task, not from a global default. 200 tokens is plenty for a classification task; 200 tokens for long-form generation is guaranteed failure.
5. Treat truncation as an error, not a normal response. On detecting `finish_reason: length`, retry or escalate the model — never let a half-finished payload flow downstream.

The test to apply: `max_tokens` is not a cost-saving dial, it is a safety valve. If you want to save money, work on prompt trimming, model selection, and caching — not on choking the output ceiling. That saves nothing; it only converts money you have already spent into waste.

The four points in one sentence

All four lessons point to the same principle: the thing to optimize is the total cost per usable result, not the price per million tokens.

In practice that means three things: grade your tasks, choose models by grade, and set parameters from measured data. This is itself part of our AI Operational Excellence methodology — for an AI capability to keep running, its cost structure has to be understood, measured, and governed, rather than guessed at from a monthly invoice.

The models get replaced every few months. This way of judging them does not expire.