Welcome to Smallville: the memory stream behind 25 LLM agents

Drop into Smallville and it looks like The Sims with better dialogue. Its 25 characters wake up, cook breakfast, walk to work, and gossip at the cafe. One of them decides to throw a Valentine’s Day party. Nobody wrote that party into a script. Park and coauthors built the town, handed each agent a running diary of everything it experienced, wired that diary to a gpt-3.5-turbo brain, and let the clock run for two game days. The architecture underneath is the part worth stealing. The believability headline stapled to it carries more fine print than the number admits.

Almost every agent-memory design in circulation now traces back to this build, at least in my reading of what has shipped since. So it is worth taking apart one piece at a time: what the memory stream does, what the evaluation actually proved, and where a quant desk should stay wary before wiring any of this into a market or consumer simulation.

How Smallville actually runs

Forget the role-play. The engine is a retrieval loop small enough to reimplement in an afternoon. Each agent owns a memory stream, an append-only log of natural-language observations with timestamps. The behaviour you see is just whatever comes back when you pull the right slice of that log at the right moment. The whole design problem collapses into one question: which memories surface now?

Three scores answer it. Each is a one-liner.

• Recency is an exponential decay over sandbox hours since a memory was last touched, with a decay factor of 0.995. This morning beats last Tuesday. The gap widens smoothly.
• Importance is scored once, at write time, by asking the model to rate the memory’s poignancy from 1 to 10. Tidying a room comes back a 2. Asking your crush on a date comes back an 8.
• Relevance is cosine similarity between the memory’s embedding and the situation the agent currently faces.

Normalise all three to [0,1], sum them with equal weight, and take the top-ranked records that fit the context window. That final sum is the entire retrieval function. No learned ranker sits behind it.

Two mechanisms sit on top of the loop. This is where the design gets clever. Reflection triggers when the accumulated importance of recent events crosses 150, which lands roughly two or three times a game day. The agent feeds the model its 100 most recent records and asks for the three most salient questions it could answer about them. It retrieves against those questions, then asks the model for five higher-level insights, each citing the specific records it rests on. Those insights get written back into the stream as memories in their own right. They then feed the next reflection. Run that for two days and you build a tree. Raw events sit at the bottom. Self-knowledge the agent never observed sits at the top.

Planning runs the other direction, top-down. The agent drafts a rough daily agenda from its own summary and yesterday, then recursively splits that agenda into hourly and then 5-to-15-minute chunks, each stamped with its own location, start time, and duration. Plans live in the stream too. A single retrieval hands the agent its observations, its reflections, and its commitments at once. It can tear up the plan mid-day when something happens.

Here is reflection paying off, straight from the paper’s controlled test. Ask Maria what to get Wolfgang for his birthday. With reflection switched off, she shrugs, because raw memory only tells her they have met a lot. With reflection on, she has already synthesised that he is into mathematical music composition. She suggests books or software for it. Raw observations record that two agents met often. A reflection records what the meetings were about. That jump is the entire value proposition of the middle layer.

The believability scoreboard has fine print

Now the headline. 100 human judges ranked five conditions on believability. Those ranks became TrueSkill ratings, the Elo generalisation Xbox Live uses to match players. Four conditions were the architecture with pieces torn out. The fifth was a human writing the character by hand.

The ordering reads as a clean result. Strip a component, lose believability, in the order the designers would predict, with the drops statistically real at every agent-to-agent step. That is a genuine ablation finding about which parts carry response quality.

Then read the fine print. Two design facts decide how far the chart travels. Neither rides along with the numbers.

First, the four ablations share one run. Every condition answered from the memories the full architecture accumulated over its two-day run. The stripped variants never lived their own two days. The authors did this on purpose. Re-simulating each one would fork the world into incomparable states. They flag the measured gaps as a conservative floor. So the chart grades interview-time response quality on a shared history. It does not show how a memory-blind agent would have muddled through the actual two days.

Second, the human line. This is the one everybody misreads. Each crowdworker was handed one agent, told to watch a replay of that agent’s whole sandbox life, and told to inspect its memory stream before writing a single answer. They had the record. The authors say plainly that the condition was built as a competency floor, with a skilled human doing their best left out of scope.

I have watched enough vendor demos to know how a 29.89-beats-22.95 line gets used in the wild. It becomes “our agents are more human than humans.” What the study earned is narrower and more useful. Memory, planning, and reflection each measurably improve how an agent answers questions about its own life, graded against a floor whose humans had already read that life. Keep the mechanism. Treat the slogan as marketing.

Two days in, nobody scripted the party

The set piece is the open run. 25 agents, two full game days, zero intervention after the world spins up. The team dropped two facts into the town as diffusion probes and watched them travel.

News about Sam’s mayoral run spread from one agent to eight. Word of Isabella’s party reached 13. The social graph nearly quadrupled in density. All of it emerged from agents bumping into each other, talking, and writing the conversation back into their streams. No diffusion model in the loop. If you have ever hand-tuned a contagion parameter and prayed, watching spread fall out of dialogue is the moment this approach grabs you.

My favourite number is the party. Five of twelve invited agents showed. That reads like a bug until you interview the seven who skipped. Three had real conflicts, including a painter too deep in prep for a show to care about Valentine’s Day. The other four said they were interested and then just never made a plan. The paper files this under coordination. The sharper point is that the flaking is endogenous. No rule told those four to drift. Whether a five-of-twelve turnout matches a real party is untested, since there is no human attendance data to compare against. Read it as behaviour that is internally consistent and not yet validated.

Where the agents glitch

The paper is candid about the failure modes. They are the most instructive part for anyone planning to build on this. Three of them, each at a different layer of the stack.

The first layer is retrieval under load. Agents whiff on memories they provably hold. Asked about the election, Rajiv says he has not followed it, though his stream carries the news of Sam’s run. Worse, they surface half a memory. Tom knows he is meant to discuss the election with Isabella at the party, cannot retrieve that the party exists, and answers that he is not sure there is a party but knows exactly what he will do there. As the map fills in, some agents wander to odd venues, drifting from the cafe to a freshly discovered bar at lunch. The paper reports no degradation curve. Treat this as a real wobble in some agents it never quantified.

The second layer is the world model. Agents misjudge the physics and etiquette of a place. The dorm bathroom holds one person. Its name suggests otherwise, since most dorm bathrooms hold several. Some agents assume this one does too and walk in on each other. The authors think the fix belongs in richer location descriptions.

The third layer is the base model. This one has no analogue in classical simulation. Agents embellish. They rarely fabricate a whole event and did not claim experiences they never had. Isabella confirmed Sam’s candidacy and tacked on that he would announce tomorrow, which nobody said. The tell is Yuriko describing her neighbour Adam Smith as the economist who wrote The Wealth of Nations. Pretraining bled into the town through a name collision. The authors also flag possible effects of instruction tuning, which seemed to push agents toward politeness and cooperation. Isabella fielded party ideas that clashed with her own taste, such as a Shakespearean reading, and rarely pushed back. Her stated interests drifted toward whatever others proposed. For anyone planning to use agents as synthetic customers, an evaluator that will not say no is the failure that matters most, because the no is the informative answer.

One operational number belongs here too. Running 25 agents for two game days cost thousands of dollars in token credits and took multiple days of wall-clock time. The authors name prompt hacking, memory hacking, and hallucination as open robustness risks. This blog put a price on the memory layer in June, when agent memory systems were profiled end to end and energy per correct answer swung by a factor of 47 across implementations. Park and coauthors found the design. The bill showed up later.

Believable is not valid

Here is the distinction that decides whether Smallville belongs anywhere near a trading floor or a marketing plan.

The study measures believability, defined as how human judges rank the plausibility of an agent’s answers under the protocol above. It never claims the agents reproduce what real people would do. Those are two different properties with two different validation bars. This study cleared one of the two.

Finance has run this same play for 30 years. The Santa Fe artificial stock market is the reference case. Its real finding is a fork. When its adaptive traders update their forecasts slowly, the market converges to rational expectations. Technical trading, bubbles, and crashes never appear. When they update at a realistic pace, heterogeneous beliefs survive. The market self-organises into a complex regime. Technical trading turns profitable. The price series grows the GARCH volatility signature of a real tape. The phenomena are conditional on one knob, the adaptation rate. That conditionality is the result. It is a statement about mechanism, well short of forecasting any actual market.

Generative agents inherit that seat with one real upgrade. Their agents leave natural-language traces a human can read, where a Santa Fe trader ran an explicit condition-forecast rule you could open up and count. Both are inspectable. The difference is that a generative agent’s stated reason and its output are loosely coupled, riding on the prompt, on whatever retrieval surfaced, and on the model’s own priors. That is the same shape this blog hit from the opposite side in July, when synthetic financial data cleared every fidelity check while wrecking the behavioural patterns detection systems actually read. Surface fidelity can sit right on top of structural failure.

Two ports out of the sandbox

Nothing in the memory stream is native to games. Two ports look buildable today. Both are my extrapolations from the paper’s verified findings, labelled as such.

The desk port. The Santa Fe lineage points the way. Swap its condition-forecast rules for generative agents and you get a market model whose traders gossip. The diffusion result licenses the experiment I would run first. Seed a simulated desk network with a rumour, the candidacy probe in market clothes, and measure which communication topologies turn it into a price move. The glitch list then writes the model-risk checklist for you. The Adam Smith collision is the sharp item on it. The base model has already read the history of every crisis you might want the simulation to discover fresh. A run that “predicts” 2008-style contagion may just be reciting 2008. So hold out invented tickers and invented shocks that no pretraining corpus can contain. Report cost and wall-clock per simulated agent-day. Then test whether one planted conversation can lodge a durable false memory in an agent that trades.

The boardroom port. For a startup, the party is the demo that transfers. Smallville opened an intent-to-conversion gap with nobody scripting it. Every waitlist learns that same gap the hard way. So run the launch before the launch. Seed a panel of agents with your target customer’s history, drop the announcement in as a diffusion probe, and watch whether word of mouth carries it past the first hop. Run the same war-game for the board meeting, for the regulator, and for the competitor response. Seed one agent per role. The sycophancy glitch binds hardest on this port, because an agent that cannot say no inflates every willingness-to-pay signal it touches. Treat each run as a hypothesis generator. The deliverable is a sharper question list for the real customers you still have to call.

The bottom line

Treat this paper as the birth of an architecture. The memory stream, three-score retrieval, recursive reflection, and top-down planning are the durable contribution. The ablation cleanly isolates each part’s effect on response quality. It has propagated far enough that reading the original still pays, three years on.

The believability result is real and small. Remove memory, planning, or reflection and interview answers degrade in a predictable order, measured against a floor whose human graders had already watched the replay and read the stream. 25 agents produced diffusion, friendship, and one half-empty party, all from retrieval over natural-language memory. None of it establishes what an agent would do when real money is on the line. Both ports above respect that limit. Ask the simulation for mechanisms and for sharper questions. The numbers you act on still have to come from the world.

Generative agents shipped a durable memory architecture and a narrow result about response believability, scored against a baseline that had read the same memory stream. Behavioural correspondence is still an open validation problem. That is the line between a great demo and a model you can trust.