Measuring agents in production: 20 teams, 86 systems, one ceiling

The first systematic study of production agents arrives with an unusual author list. MAP, Measuring Agents in Production, accepted to ICML 2026, spans 25 authors across UC Berkeley, Stanford, UIUC, IBM Research, and Intesa Sanpaolo, the Italian banking group, with Zaharia, Stoica, Gonzalez, and Song on the Berkeley side. The team interviewed 20 deployment teams at length and filtered a 306-practitioner survey down to 86 systems in production or pilot, across 26 domains, collected April to November 2025. The headline is already circulating: production agents are simple. 68% hand control to a human within 10 steps. 70% of case-study teams prompt off-the-shelf models. 74% lean on human evaluation. The number everyone will quote is the what. The finding that pays is the why. The why is verification.

Read the denominators first

Before the findings, the fine print, because the abstract compresses it away. The survey drew 306 responses, filtered to 86 deployed systems. Individual questions then drew fewer answers still, since every question was optional. The step-count result rests on N=60. The prompt-construction result rests on N=53. The evaluation result, the paper’s most-quoted number, rests on 23 of 31 responses. The 70% figure for off-the-shelf models comes from a different sample entirely, 14 of the 20 case studies.

The authors are candid about what this buys. Recruitment ran through the Berkeley RDI summit, an AI Alliance meetup, a Berkeley MOOC, and professional networks, with case-study teams concentrated in the Americas. Their own framing asks readers to treat the results as qualitative evidence about how practitioners approach deployment, with an explicit warning against reading the percentages as “fixed prevalence estimates.” That is the right way to hold every number below. The patterns are consistent across two independent data sources and 26 domains. The third decimal place is noise.

Simple on purpose

The constraint findings all point one direction. 16 of 20 case studies run structured workflows, with open-ended autonomous planning the rare exception. 47% of surveyed systems hand control back within 5 steps. 17 of 20 production teams built custom in-house scaffolds with direct API calls, with 2 teams explicitly migrating off frameworks to get to production. 14 of 20 case studies do no weight tuning at all, with one lone RL user in scientific discovery. A single team allows unconstrained exploration, inside a sandbox, behind CI/CD verification. The open-ended agent playground belongs to research towns like Smallville. Production looks like plumbing.

The concrete picture comes from the paper’s insurance case. The agent runs a fixed sequence of coverage lookup, medical necessity review, and risk identification. It completes each subtask autonomously. The high-level objectives never move. That is an existing business process with an agent poured into it, which is what most of these deployments are.

Two details complicate the simplicity story in useful ways. First, the latency budget is generous, 66% of systems tolerate response times of minutes or longer, because the baseline is human turnaround measured in hours or days. Teams are buying capability with time, holding autonomy constant. Second, 59% of surveyed systems coordinate multiple models, partly for cost routing and modality, partly for a less flattering reason. Interviewed teams keep legacy models running alongside upgrades because scaffolds, prompts, and evals lock onto specific model behaviour. The paper puts it flatly: “newer or more capable models do not guarantee improved agent performance.” Princeton’s reliability study at the same conference measured the same decoupling across 15 models. Model upgrades are a migration project now. Plan them like one.

The verifier sets the ceiling

Here is the observation that organises the whole dataset. The paper makes it almost in passing. Coding agents are named as “a rare case where verification occurs through compilation and test suites,” which lets them iterate fast. The insurance agent gets its feedback “through delayed real consequences such as financial losses or patient approval delays.” Teams in regulated domains reported months of expert data collection and labeling just to build a test set. 75% of case-study teams run no formal benchmark at all, falling back on A/B tests, user feedback, and production monitoring. 26% of surveyed teams say no meaningful baseline for comparison even exists.

Line the findings up against that gap and they stop looking like immaturity. Agents cluster in productivity work, 80% of practitioners cite it, because a stopwatch is a verifier anyone can afford. Autonomy stops at 10 steps because each unverified step compounds risk the team cannot measure. The interviews name the proximate drivers of those caps, problem complexity, planning non-determinism, and latency. In my reading all three are verification problems wearing different coats. Humans dominate evaluation because in most domains a human expert is the only oracle available at any price. Simplicity is not a phase that better models will grow the field out of. It is a rational response to missing verifiers. A 1,340-respondent LangChain survey from June points the same direction from the other side, 89% of teams run observability while barely half run offline evals. Watching is cheap. Judging is the scarce input.

The reading also explains what the paper’s fashionable counterexamples miss. Architectures like the Six Sigma Agent, which runs every atomic task n times across diverse models and votes on the outputs, multiply exactly the thing production teams cannot pay for, unverified model calls. My read is that this is why nothing resembling them shows up in the paper’s 20 architecturally described case studies. And it makes the governance numbers legible. 3% of practitioners rank governance as their top priority while 69% of deployed agents retrieve confidential data, with security achieved “implicitly” through the same constraints that manage reliability, read-only modes, sandboxes, wrapper APIs, internal-only users. The constraint architecture is load-bearing for two different failure classes at once, which is precisely the double duty that worried me in July when a filtered attack rerouted through a channel the filter never covered.

The bottom line

For a desk or a deployment team, MAP is a pricing document. Before scoping any agent, price the domain’s verifier. Where a deterministic oracle exists, compilers, test suites, balance checks and schema validators of the kind the synthetic-company series is built on, you can afford autonomy, iteration speed, and ambitious scope. Where the oracle is a human expert with a queue, the honest design is the one MAP’s teams converged on independently, short sequences, fixed objectives, approval gates, internal users first. For bootstrapping an oracle where none exists, the paper records a convergent recipe, golden question-answer sets built with domain experts, expanded from real user interactions under expert review. The study’s participants sit at the frontier already. Among teams that measured agents against non-agentic alternatives, 83% still prefer the agent. They are running the frontier at the width their verifiers can certify.

The research agenda hiding in this is the one this blog keeps arriving at from other directions. The scarce asset is cheap, trustworthy verification, in domains where the feedback today arrives as a financial loss with a lag. Build the oracle and the autonomy follows.

MAP’s production agents are simple because nothing more ambitious can be verified. The ceiling on agent autonomy is set by the cheapest reliable verifier in the domain. Raising the ceiling means building the verifier first.