The harness decides the bill, not the ceiling

Two configurations, one benchmark, one task set. The first resolved 60.32% of tasks at $4.66 each. The second resolved 52.38% at $19.94. The same model ran in both. The only thing that changed was the harness around it.

That pair is the whole problem in miniature. One of those two numbers is a solid measurement. The other is a sampling artefact. Most teams act on the wrong one.

Every team integrating agents makes four decisions in the first fortnight. Which model. Which harness. Who serves the weights. Whether to fine-tune. Seven studies published between May and October 2026 are finally controlled enough to say what each of those decisions actually buys.

Three of those four decisions are cheap to revisit. One is permanent. Most teams make it by accident.

This is written for the engineer with a client, a deadline and a 40-task internal eval. The research has an uncomfortable message about that eval. I will get to it at the end.

What is actually in the box

A harness is everything wrapped around the weights. The model contributes exactly one thing, which is what it is capable of. Every other layer was configured by you or by your vendor. Each of those layers decides a different part of what you will experience as model quality.

Read it from the outside in. The outer ring never reaches the model as tokens and still decides outcomes, which is why Claude Code compacts in a fifth to a third of its trials while mini-SWE-agent never compacts and overflows instead. The context ring is where the money goes, because everything inside it is re-sent on every step. The serving stack is the layer almost nobody audits. The core is the only layer you cannot configure.

One connection is worth stating early, because it runs against intuition. Fine-tuning does not only change the core. The trajectories you train on carry the conventions of the two rings above it down into the weights, which is how a portable base model turns into one that works under a single scaffold. That is the subject of the section on the decision you cannot undo.

The rest of this piece works through the four layers in the order they cost you money.

What the harness does not buy

Start with the ceiling, because the honest finding belongs first.

MedicalHarness ran five open-weight models under five harnesses across 107 executable tasks and three seeds, changing nothing but the harness inside each comparison. Wang, Zhao and Ding then decomposed the variance in outcomes. The model accounts for 58%. The harness accounts for 12%, with its interaction with the model worth another 12%. Run-to-run variation across seeds accounts for the remaining 18%.

The authors conclude that no single harness is best across models and tasks. Harness and interaction together come to about a quarter of the variance, a real effect at a modest scale.

A second group arrived at the same place from a different direction. Liu and Han ran five models through three production harnesses on SWE-bench Verified. They also reran identical configurations, which is the step most comparisons skip, to calibrate how much a score moves when nothing changes at all. On the full 447-task pool, the heaviest harness and the lightest came out equivalent within 5 points on both models they ran there. Those two were Claude Code and mini-SWE-agent. On a 45-task hard subset, swapping the harness flipped 13% of tasks. Rerunning the same harness flipped 13% as well. The tasks a harness wins in one run are not the tasks it wins in the next.

One harness effect in their data does clear the noise. That effect is a loss. OpenCode trailed by up to 9 points. On one model, half of that gap sat in runs its own output cap had cut short.

So the harness will not raise your ceiling. Hold that, because the next two sections are about things it decides completely.

Whether the run finishes

This is the finding I had not seen stated anywhere. It changed how I would instrument a deployment.

The MedicalHarness authors separated two questions that a single score collapses into one. Does the episode end with a scorable answer at all? And among the questions every harness answered, who got them right?

On clinical calculation and long-context retrieval, the answer-rate spread across harnesses was the larger of the two in 9 of the 10 model-environment pairs. Scores on the shared questions differed by only 0.02 to 0.06 in eight of those pairs, while answer rates differed by as much as 0.54.

Put that in deployment terms. Changing your harness will barely move whether the model is right. It can move whether the model produces anything at all by more than half your run volume. A single aggregate score cannot tell those two effects apart. A completion problem therefore arrives on your desk looking like a model-quality problem. You then go shopping for a better model to fix a loop that is terminating early.

The same paper names an anti-pattern that early-stage teams reach for constantly. Hiding tools behind a search interface, done to save context, can cost tokens or accuracy when the models call those tools by name. The general mechanism is worth more than the specific case. A harness component changes outcomes only when it fires and when its interface fits the model.

Report answer rate beside score. It costs nothing to log and it separates two failure modes you would otherwise spend a week confusing.

The bill, and where it is set

Liu and Han called their paper "What Does a Harness Buy? Tokens, Mostly." Their accounting is the single most useful thing in this literature for a small team.

Holding the model, the tasks and the price list fixed, cost per task differed by up to 3x across harnesses. That gap does not accumulate evenly through the run. It is set at the first call.

What differs is the preamble each harness carries into every call: its system prompt, its tool schemas, its message wrapper. That preamble measures 16,581 tokens in Claude Code, 7,025 in OpenCode, and 829 in mini-SWE-agent. It is paid again on every step. On GLM-5.3-Flash the first call therefore carries 17.1k tokens under Claude Code against 1.4k under mini-SWE-agent. The slopes after that first call sit within a factor of one and a half of each other.

Their summary deserves to go on a card next to your terminal. The first call, the growth per step and the step count are the three numbers a harness's bill turns on.

That accounting explains the figure this piece opened with. The $19.94 and the $4.66 are not two different models or two different task sets. They are one preamble multiplied by one step count, twice.

Two further details change real invoices. Between 96% and 99% of input tokens in every arm were cache hits, because each step re-sends the previous transcript. The cache-hit price therefore decides how much of your bill the transcript is. Zhipu charges 0.287 times the miss price for a hit. Claude Code's heavier transcript therefore surfaces as nearly twice mini-SWE-agent's cost. DeepSeek charges 0.033 times. There the cached transcript falls to under half the bill while output grows into an equally large term. Repricing each model's tokens under the other vendor's list did not reorder the three harnesses.

The second detail is that the vendor's own harness bought nothing. DeepSeek's harness on DeepSeek-V4-Flash passed 21 of 45 tasks, matching Claude Code and finishing one task behind mini-SWE-agent, at a slightly higher cost per trial. Home field sat inside the range on score and at the top of it on bill. Li and colleagues found the same across a wider grid, where the native Codex and Claude Code pairings were not reliably best for their own models.

The decision you cannot undo

Peng and colleagues at Microsoft Research, Columbia and UIUC ran the ablation this field needed. They held the training data completely fixed at 12,000 resolved trajectories from a MiniMax-M2.5 teacher, with the same fine-tuning recipe throughout. The only variable was which harness collected those trajectories.

The authors put it without hedging: harness lock-in is severe in single-harness training. They attribute the collapse to three specific conventions a harness imprints on its trajectories: tool-call format, observation structure and turn conventions. None of those is a capability. All three are dialect.

That matrix is arresting in isolation. It becomes a different kind of warning beside the control a second group published.

Thangarajah, Chen and Hassan, at Huawei Canada and Queen's University, evaluated open models across four CLI scaffolds and reported the untrained base models in the same table.

The authors state the conclusion directly: the divergence is fine-tuning-induced.

The base models were comparatively portable. Training on one scaffold's trajectories is what created the dependency, which makes harness lock-in something you manufacture during data collection.

The ecosystem is arranged to manufacture it by default. The paper's opening observation is that trajectory datasets used to fine-tune open models are collected almost exclusively under OpenHands. Reach for public agent trajectories to tune a cheaper model and you have chosen OpenHands, whether or not you noticed the choice.

The failure modes are also worse than a dented score. SWE-Lego-Qwen3-32B reached only 8.4% on OpenCode because its tool-call output format is incompatible with OpenCode's parser, which left most trajectories with zero parseable actions. In the Orchard study, Scale-SWE produced invalid outputs under every harness except its native one, yielding no measurable resolve rate at all.

The failure that lives in the weights

One more study shows the mechanism at a scale small enough to hold in your head. A note on that scale first. This one is a single author working with two models of 661.6M and 1,109M parameters, so read it as a worked demonstration of a mechanism.

The two models share decoder, tokenizer and special-token layout. On the series' lenient tool-use metric they score 0.660 and 0.650, which reads as a tie. A verbatim-reproduction check on real training examples separates them completely. The smaller model emits well-formed tool calls on 6 of 6 examples. The larger one does so on none of the samples tested, at every checkpoint.

A first-token probe localised it. The larger model's probability on the tool-call trigger token had collapsed to between 0.0001 and 0.00001. Probes across 38 archived checkpoints showed the cause. The model's own later web-heavy training phase had erased a prior it previously held.

Two things to carry away. The keyword harness scored that model at 0.660 while it was emitting zero valid tool calls, which is a benchmark failing open. And the repair was cheap. A targeted recipe of 2,202 steps and roughly 3.3 GPU-hours moved well-formed emission from 0.100 to 0.959 across 269 corpus rows, using about three orders of magnitude fewer tokens than the dedicated training phase that had failed.

Who serves the weights

This layer never appears on a model card. It is also the cheapest one to check.

Moonshot publishes a vendor verifier that runs identical weights through every provider serving them. Read on 13 October 2026, kimi-k2-0905-preview returned 100.00% tool-call schema accuracy at five providers, then 76.00% under stock vLLM, 73.13% under SGLang, and 71.96% at the lowest-scoring host. Identical weights, with nearly 30 points of schema validity decided by whoever runs the inference.

A second axis sits beside that one. Groq returned 100.00% schema accuracy on the same model with 69.52% trigger similarity, meaning it formats perfectly and decides to call tools differently. Those are independent failures. One aggregate number hides whichever of them you are suffering from.

The obvious caveat belongs in the open. Moonshot is both the referee and the reference model here. The direction of the finding has support from vLLM's own post-mortem on the same weights. The remedy costs you an afternoon in either case.

Harnesses decay, and your eval cannot see it

Five major open-source harnesses are shipping more than two releases a day. Ben Sghaier and colleagues held a model fixed on local vLLM and ran 35 sequential Qwen Code CLI releases against 50 stratified SWE-bench Verified tasks.

Resolve rate fluctuated around a stable mean of 30.5% with no significant trend across release order, at a Spearman rho of 0.208 and a p-value of 0.231. Across those 35 releases, real engineering effort produced no measurable gain in what the agent resolved. Consumption moved a great deal.

The first nine releases averaged roughly 391,000 tokens per task. The latest consumed nearly 668,000, an increase of more than 70% in a few months against a flat resolve rate. Average tool calls per task swung between 6.9 and 14.3. Your harness bill can grow by three quarters while the thing you are buying stays exactly where it was.

Now the part that applies to your own testing. Liu and Han used their rerun data to compute what a harness comparison can actually resolve. At the discordance they measured, 45 tasks catch a 13-point gap only half the time and cannot detect a smaller one at 80% power, while 447 tasks resolve about 5 points. Many published harness gains sit below the first of those numbers.

If your internal eval holds 40 tasks, it cannot see the difference your team is arguing about on Slack. Keep testing anyway. Stop treating a 3-point move as a result. Spend the saved effort on quantities a small sample can actually measure: cost per completed task, answer rate and tool-call validity.

The order that matters

Four checks, in the order I would run them for a client with two weeks and no evaluation infrastructure.

Price the preamble before you price the model. Pull the first call out of your candidate harness and count its tokens. That number, multiplied by your expected step count, is most of your bill. It is available to you in an afternoon, long before any benchmark result is.

Probe your exact serving path. One batch of tool-call prompts against the precise provider and engine that will serve your chosen weights. Record schema validity and trigger behaviour as two separate numbers.

Log answer rate beside score, permanently. The harness-to-harness spread lives there. A loop terminating early is the most common thing mistaken for a weak model.

Decide about fine-tuning with your eyes open. Fine-tune on one harness's trajectories and you have chosen that harness for the life of the model. Orchard's multi-harness training narrows the worst case to a 19.3-point drop, which its authors describe as improved and not solved. Portability is bought during data collection. Afterwards it is gone.

One closing observation, which is really a depreciation schedule. Prompts, scaffolding workarounds and model-specific tunings expire with each model generation, and some can be distilled into the weights outright. Permissions, isolation, action logs, answer-rate instrumentation and portable agent definitions keep their value as models turn over beneath them. Spend your scarce engineering hours on the half of the stack that does not expire.