Routing pays where the answer is checkable

RouteNLP reports a 58% inference-cost reduction over an 8-week pilot at an enterprise customer-service division, holding 91% response acceptance at roughly 5,000 queries a day. Accepted to the ACL 2026 Industry Track, out of the University of Hong Kong with Stellaris AI, it belongs to the small set of routing papers with a real deployment attached. The headline discount is the least useful number in the paper. The number that changes how you scope the work is the spread underneath it, which runs from 40% to 85% across six tasks on one portfolio, one router, one set of thresholds.

The architecture is four tiers spanning roughly 800x in cost, from a fine-tuned DistilBERT at $0.01 per 1K tokens up to GPT-4-Turbo at $8.00, with Mistral-7B and a quantized Mixtral-8x7B in between. A DistilBERT classifier predicts the cheapest tier that clears a task-specific quality bar, adding 4.2ms per query. Each tier computes a token-level uncertainty score and escalates when it exceeds a conformally calibrated threshold. The authors’ own claim to novelty is the loop around all of it: escalation failures get clustered, turned into distillation data, folded back into the cheap tiers, after which the router retrains and the thresholds recalibrate.

Read the deployment before the discount

The fine print moves several of the numbers into a different category. The pilot covers customer service only. Every finance and legal claim rests on benchmark simulation, which the authors list as their first limitation. The deployment was a shadow run with partial traffic routing and no A/B test, which they flag as limiting causal attribution.

The acceptance figure has a protocol worth knowing. Two senior agents audited 500 random pilot responses and found 91% acceptable, 6.4% marginal, and 2.6% unacceptable against a 1.8% baseline, at a Cohen’s kappa of 0.79. The 58% saving cost 0.8 percentage points of additional unacceptable responses. The enterprise partner judged that trade worth taking.

One number deserves relegating. The latency result that will get quoted, p99 falling from 1,847ms to 387ms, comes from an M/M/c queueing simulation under matched production load. The pilot’s own measured outcome is a 23% drop in average first-response time. That is the number to carry into a capacity plan.

The discount tracks what a cheap model can absorb

The obvious reading of that cliff is structured extraction against open generation. The paper carries the variable that spoils it. Legal risk assessment has expert agreement at kappa 0.83, which puts it alongside the extraction tasks on measurability. It still lands at the bottom of the savings table at 40%.

Difficulty and checkability are separate axes. The size of the discount is set by how much of your difficulty distribution the cheap tiers can absorb, which is a capability question about small models. The confidence you can place in that discount is set by whether a cheap oracle exists, which is a measurement question. On the three extraction tasks both answers are favourable. The system banks 78 to 85% and proves it with F1 against annotation. On the two generation tasks it took three domain experts over 200 samples per task to learn what the system had bought: 74.5% of routed outputs match or exceed frontier quality, with 8 to 9% of all queries substantially degraded, which the authors call a deployment risk requiring mitigation.

Set that beside MAP’s finding that the cheapest reliable verifier caps agent autonomy and, in my reading, the same asset prices two different things. A verifier decides how far you can automate. It also decides how cheaply you can serve and how much of the saving you can defend afterwards.

The failure analysis supports the capability half. Multi-step reasoning accounts for 42% of quality failures. The paper’s stated cause is that the router reads the query text without the surrounding document context. That limitation lands on exactly the work a finance desk would route. Ask for the covenant terms in a credit agreement and the query is a dozen words, while the difficulty sits in a document the router never reads. Domain-specific knowledge accounts for 31%, typically rare instruments or recent regulation. The remaining 27% are syntactically simple queries that need nuanced generation.

What the conformal threshold actually buys

In the model-risk reviews I have sat through, a hand-set escalation threshold is what sinks a routing proposal. RouteNLP sets its thresholds with conformal risk control on 500 calibration examples per task and tier at an alpha of 0.05, which yields a distribution-free coverage guarantee. That is a defensible object to put in front of a validation function. The bill is rarely quoted. Each threshold is task-specific and tier-specific. Six tasks across three escalation boundaries puts the calibration set near 9,000 labelled examples before the router serves anything. The oracle returns as a setup cost.

The authors then limit their own claim, the most credible move in the paper. Three caveats come from them rather than from me. The guarantee is marginal over the joint distribution and says nothing about any individual query. It assumes exchangeability, which distribution shift violates. Calibration set size drives tightness. At 500 samples the 95% Wilson interval on the observed 4.2% violation rate runs to 6.6%, which already straddles the 5% target. Under a domain shift the violation rate hits 8.1% and misses outright. Weekly recalibration held the pilot at 4.8%. The authors are clear that weekly will not survive a more dynamic environment. Their own recommendation is to treat conformal thresholds as initialization supported by production monitoring.

Reading it as a per-query promise repeats the error of treating a 99% VaR as a statement about tomorrow. The book breaches on 5% of days by construction. Which day is not in the contract. The guarantee runs on the same machinery this blog has already met in a risk setting. The analogy gives out in one place. A VaR limit gets checked against observable daily P&L, while routed generation quality is visible only through a sampled human audit, which is why the 8 to 9% substantial-degradation rate needed three experts to surface at all.

What I would build from this

The closed loop is the transferable piece. Clustering escalation failures and distilling those clusters into the cheap tiers beat random distillation at identical data volume by more than two to one, for the price of clustering logs you already write, at a one-time cost the authors put near $2,400 at their scale.

They are training data for the tier below.

The break-even matters more than the headline. Savings hold at 58% across a 200x cost span and 41% across 100x. The apparatus stops paying near 25x. Few real portfolios span 800x. Mid-pilot this one stopped spanning it too, once the team swapped the frontier tier for GPT-4o-mini on classification queries and compressed the range to roughly 200x. The routing distribution adjusted with no retraining. That compression is the part I would watch, since every improvement in cheap models narrows the spread that makes routing worth building.

Two cautions on the comparison table. RouteNLP’s cost ratio of 0.159 beats RouteLLM at 0.246. That figure comes from extending its binary routing head to four classes, an adaptation performed by the RouteNLP authors. Faithful two-tier variants ran 2.1 to 3.4 times more expensive, which makes this a comparison against an adapted baseline. The frontier tier is also GPT-4-Turbo, which dates the absolute quality retention even where the mechanism generalizes.

The paper is unusually clear about when to walk away. The list is short: single-task deployments, workloads dominated by hard queries, anything under 100 queries a day, and any process where an unacceptable rate above 3% would force per-query human review. An endurance team does not fit the soft compound to every stint. You spend it on the laps that repay it.

The habit I would take is smaller than the framework. Before pricing a model, price the check. Where a cheap oracle exists, route aggressively and audit continuously with the check you already own. Where none exists, the saving is equally real and observable only through a sampled panel. Budget that panel as a permanent line item on day one.

Price the oracle before you price the model. A cheap check is what lets you bank a routing discount and defend it later. Where no cheap check exists, the audit becomes a standing cost of the saving.