Synthetic financial data: real enough to pass, wrong enough to break

Synthetic data keeps passing the wrong exams. The generator reproduces every marginal distribution, the downstream classifier scores respectably, the return series wears its fat tails and volatility clusters like a native. Then someone connects the data to the system it was meant to feed, a fraud engine or a backtest. The structure that system depends on is not there. Two papers this year land that verdict from opposite ends of finance. The failure they document is the same: the evaluations everyone runs do not test the properties anyone uses.

I have sat through more than one vendor pitch where the fidelity slide was a wall of green checkmarks, marginals matched, correlations matched, AUROC within a few points of real. None of the checkmarks answered the question the desk was actually asking (can I tune a control on this data and trust it in production). Both papers give that question a number. The numbers are bad.

The exam and the street

Bhavana Sajja’s benchmark introduces the missing dimension by name: behavioral fidelity, a third layer under the two everyone already measures. Layer one is statistical fidelity, marginals and correlations. Layer two is downstream utility, train on synthetic, test on real, report the AUROC. Layer three asks whether the generated data preserves the temporal, sequential, and graph patterns that operational fraud systems key on: the burst of transactions in sixty seconds, the device shared by a ring of fresh accounts, the velocity rule that fires when a card runs hot. Four generators go in, CTGAN, TVAE, GaussianCopula, and autoregressive TabularARGN, scored against a degradation ratio where 1.0 means indistinguishable from resampling the real data.

All four generators fail. The sharper finding is that the layers do not predict each other. CTGAN posts the second-best downstream AUROC in the study, 0.798 against a real-data baseline of 0.903, while turning in the single worst graph-motif score, 99.7 times the noise floor. GaussianCopula inverts it, worst AUROC at 0.523 while scoring better on ring structure at 81.6. A validation report built from layers one and two carries no information about the layer a fraud system consumes.

The behavioral numbers themselves deserve a slow read. On the temporal and velocity patterns, composite degradation runs from 24.4 times the noise floor for TVAE up to 39.0 for GaussianCopula. The best case carries a caveat. TVAE hits 24.4 only after a correction. Its unconditional sampler had collapsed the fraud rate from 3.5 percent to roughly 0.03 percent, a second silent failure at the level of the class prior rather than behavior: draws from the latent space decode overwhelmingly to the majority class, fraud all but vanishes, the composite swells to 45.4. The standard SDV interface hands you that collapse with no error raised. The fix, conditional sampling at generation time, is one line. The detector is cheaper still: a fraud-rate check.

The paper proves the deepest failures are structural rather than fixable. A row-independent generator, which is what CTGAN, TVAE, and GaussianCopula all are, mathematically cannot reproduce a shared-device ring, since it has no mechanism for coordinating attribute values across rows. It likewise cannot produce the positive gap-to-gap autocorrelation that is the fingerprint of a fraud burst, no matter the architecture or the training budget. TabularARGN conditions across features inside a row, which is why it lands roughly five times better on rings, 17.2 against 81.6 to 99.7. Across rows it is as blind as the others, 36.3 on the temporal composite. One anomaly is worth pausing on: after correction, TVAE’s autocorrelation degradation is 5.9 against 40.5, 60.9, and 75.1 for the rest, credited to the continuous latent space partially preserving the tendency of short gaps to follow short gaps. Exceptions that size are usually where the next paper starts. Rule-based simulation sits at the opposite pole. PaySim encodes the bursts in hand-coded agent rules, preserving behavior by construction at the price of never learning a fraud pattern nobody wrote down.

One design choice makes every bar a generous reading. Row-independent generators produce no entity identifiers. The benchmark therefore assigns synthetic rows to pseudo-entities drawn from the real entity-size distribution, handing each generator a grouping it could never have produced. Real deployment would need entity structure generated jointly with the rows. The reported ratios are floors under the true degradation.

The operational translation belongs in every model-risk memo that touches synthetic data. A canonical velocity rule, more than three transactions in an hour, fires at an absolute rate 0.36 lower in CTGAN’s synthetic fraud than in real fraud. Tune the rule’s threshold on that data and it will be calibrated for a world where fraud is politer than it is. That control fails silently, in production, in the permissive direction.

When the backtest is the exam

Zhang and coauthors reach the same verdict from the market side. GAN-generated return series routinely pass the visual and statistical inspections, fat tails present, volatility clustering present, and then detonate under a trading backtest. On twenty years of the Shanghai Composite, a plain 60-day momentum strategy on standard GAN output produces an annualized return of 2,467 percent at 992 percent volatility with a 110 percent maximum drawdown. The numbers announce their own absurdity. The quiet detail is that the Sharpe ratio does not: at 2.49 it sits above the real-data 2.18. The single most quoted risk-adjusted metric would have waved the collapsed generator through.

Their fix, the Stylized Facts Alignment GAN, is philosophically the interesting part. Rather than hoping distribution matching implies structure, SFAG makes the structure the loss: four differentiable penalties on the tail index, the autocorrelation of squared returns, the leverage effect, and cross-scale volatility correlation, trained jointly with the adversarial objective. Backtests on the result behave like markets, 27.8 percent annualized against the real 33.1, drawdowns in single digits.

Bring the same skepticism to the SFAG column. Even the structure-preserving generator errs calm, which means a strategy sized on its world arrives at the real one over-leveraged. The paper reads the higher Sharpe as robustness. A risk manager should read it as the residual gap. On volatility, drawdown, and tails the deviation is far smaller than the GAN’s and points the other way, toward calm. The Sharpe is the one metric where both err the same way. SFAG’s miss is wider: 2.97 sits 0.79 above the real 2.18, where even the collapsed GAN overshot by 0.31. The future-work list doubles as a diagnosis: tail asymmetry, volatility-of-volatility, and regime persistence are not yet in the loss. A generator preserves the structure someone has named. The evidence base is one index, one strategy, ten synthetic paths.

The direction of silent failure

Put the two papers side by side and the pattern that emerges is the one worth remembering. Synthetic fraud fires fewer alarms than real fraud. Synthetic markets draw down less than real markets, once the generator is good enough not to explode. The errors lean, consistently, toward a world with less risk in it. Every control tuned on that world inherits the lean. This is the same species of quiet failure this blog keeps meeting in different clothes: the backtest that passes on data containing nothing tradable, the implementation detail that flatters the strategy until deployment. The remedy is also the same. Match the test to the use. If the data will tune velocity rules, measure velocity-rule trigger rates. If it will feed a backtest, run the backtest and read the drawdown and tail rows before the Sharpe. Where a generator must model event streams instead of rows, the bar is higher still, a lesson the order-book diffusion literature learned early against point-process baselines. The two working answers share a shape. PaySim’s rules and SFAG’s penalties both write the behavior straight into the objective, which is the only place it has yet survived generation.

The hard-won version, from years of watching validation reports: a wall of green tells you which questions were easy to ask. Both of these papers built the hard question instead. It flipped the answer.

Synthetic data does not fail loudly, it fails in the calm direction, and every threshold tuned on it inherits the optimism. Test the property you will consume.