Spurious predictability: falsify your backtest before you trust it

A backtest is the easiest thing in finance to fake. Most of the faking is accidental. Try a hundred model variants and keep the one with the best t-statistic. You have manufactured significance out of nothing. Under a no-signal null, the largest in-sample t-statistic a search can find grows with the square root of the log of the number of specifications tried. Sotirios Nikolopoulos puts a probability on the trap: a single model clears the 1.96 bar 5.3 percent of the time, the nominal rate, while a hundred-model search on pure noise clears it 99.9 percent of the time. The number is not a bug in anyone’s code. It is what selection does to a maximum.

His fix, a falsification audit, is the most useful idea I have read on backtest hygiene in a while. Run the entire workflow against data you know to be random, then keep only the workflows that fail to find anything. This blog’s companion piece on double machine learning asked which drivers of a market move are causal. That question only earns its keep once you have ruled out the duller explanation, that the pattern was an artifact of the search that found it. Falsification is upstream of causation.

The cascade that manufactures alpha

The paper’s sharpest framing is a cascade. It begins with leakage, the small timing sins that let a feature see information it could not have had at the moment of decision. Normalization computed over the full sample, a universe filtered to the names that survived, labels that overlap across cross-validation folds: each lowers the effective noise floor and prints an artificial signal. Selection then does its work. An optimizer scanning many specifications reports the one that maximizes the absolute t-statistic. The Winner’s Curse tends to make the retained effect larger than whatever is really there. What arrives at the end is dressed as structural alpha. It is often compensation for a risk exposure the model never named.

The count that bites is the effective number of independent specifications rather than the raw number you ran. Effective multiplicity is the participation ratio of your candidates’ correlation matrix, a quantity that falls toward one as models collapse into each other.

That discount comes with a trapdoor, the paper’s most original finding. An XGBoost tuning family with a measured effective multiplicity of 2.93, from a nominal grid of four hundred configurations, still clears the in-sample significance bar 88 percent of the time on pure noise. The correlation measure said the search placed three bets while the in-sample winners inflated as if it had been far wider. Nikolopoulos names the mechanism spurious orthogonality. Thresholding near-identical scores into discrete buy and sell states decorrelates the realized trading signals. In his mechanism study, prediction scores pinned at a multiplicity of 1.24 produce thresholded trading signals with a multiplicity as high as 3.34. Low measured correlation among your models is no guarantee of a small search.

Put the whole workflow on trial

The audit’s move is to test the pipeline instead of the prediction. Nikolopoulos builds reference classes with no conditional-mean predictability, a martingale-difference null, plus microstructure placebos such as a pure bid-ask bounce. Then he runs the complete workflow on them, feature construction through portfolio mapping, under strict walk-forward validation that never lets an evaluation see its own future. A workflow that still produces significant walk-forward performance on random data has told on itself. The significance is coming from leakage or selection or a timing error rather than from anything in the market. Finding walk-forward edge on a bid-ask-bounce placebo is the starkest version, since the money is being made on an artifact no one can trade.

For the workflows that survive, the paper quantifies how much the search inflated them. The absolute magnitude gap is the plain version, the in-sample winner’s t-statistic minus its walk-forward counterpart, with the walk-forward side bounded under the null. The Backtest Inflation Factor is the ratio of the two, with a neutral reading at one. The gap is the measure to carry between studies, since the ratio leans on a stabilizing constant and only ranks models inside a single study. Both turn the vague worry that a result is too good into a number you can put in a research note.

A quant will reach for the standard defense here. White’s Reality Check, Hansen’s test of superior predictive ability, and Harvey, Liu, and Zhu’s threshold all inflate the critical value to account for the size of the search. Those corrections adjust a p-value after the fact and assume a known model count, a number an adaptive search leaves implicit or unbounded outright. The haircut trims for luck. It cannot see a feature that peeked at the future. The paper treats the two as stages of one lifecycle. The audit is a pre-commitment filter that runs while the workflow is still being built. A Reality-Check correction belongs at the end, when a surviving result is reported to the outside world.

What the audit cannot see

The audit is a screen rather than a certificate. Its power depends on the signal being tested. Against a genuine and reasonably active structure it rejects the null cleanly. Against a real signal that is weak and rare, the paper is candid that power collapses toward zero. Passing the audit means your workflow did not manufacture the result. It does not prove the result is large enough to trade.

The empirical section runs three real workflows through the audit. Two pass. A deep-learning model on the S&P 500 posts an alpha of 0.39 percent against buy-and-hold at a p-value of 0.931, which is another way of saying nothing. The audit explains why: the regime-dependent trading pattern reappears when the identical pipeline is run on a pure GARCH null with no predictable mean. The activity was a threshold artifact. The other two survive. A cross-sectional strategy with a significant gross return keeps a significant intercept after a Fama-French adjustment, an inflation factor of a mild 1.16. Its edge is not merely repackaged factor exposure. A volatility model evaluated under random cross-validation lands within a rounding error of its strict walk-forward result, an inflation ratio of 1.006. The usual leakage worry does not bite here. An audit that only ever cried foul would be useless, whereas this one clears the genuine findings while catching the artifact.

The operational takeaway is small and non-negotiable: add a null pass to the pipeline and run it before the memo goes out. A label shuffle is the same control that collapsed the apparent gain in this blog’s market-feedback retrieval study. A workflow tuned long enough against a single fixed null learns to pass it without learning anything real. It is the motorsport version of tuning the car for the dyno instead of the track. The paper runs five null environments at once, from white noise to a bid-ask bounce to GARCH clustering, each aimed at a different failure channel. Their parameters come from published distributions instead of fixed values. The draws used in the final gate are privately seeded, a target the workflow cannot read. Nikolopoulos packages the diagnostics as a reference implementation called QuantAudit, slated for public release when the work is published. The same discipline sits behind careful backtest accounting. A null pass is cheap to run. Running it credibly takes more than a coin flip.

The bottom line

The uncomfortable truth in this paper is that a good-looking backtest is weak evidence by default, because the process that produced it is optimized to look good. A finding is worth believing once it survives contact with data that cannot possibly contain it, and worth trading only after that. That is a higher bar than significance. It is the right one for anyone whose next step is to put capital behind the number.

A backtest is not evidence that a signal is real, it is evidence that a search ran. The only result worth trading is the one that still shows up after you feed the same workflow data you know to be noise.