← All lessons

Data snooping

Try enough ideas and the best one will look brilliant by luck. · 8 min

Test one idea and a good result is evidence. Test a thousand and keep the best, and a good result is expected, even if every idea is worthless. That is data snooping, and it is the most common way traders fool themselves.

Predict first

A thousand random rules on two years of a market with no edge. How many show a Sharpe ratio above 1, a level many traders call good?

A thousand ideas, no edge

Random rules (moving-average crosses, breakouts, dip-buying) with random settings, tested on a market where none of them can work.

Running the numbers…

The best of many is biased

The Sharpe ratio is the average return divided by its volatility, annualised. A strategy with no edge has a true Sharpe of zero, but its measured Sharpe over a short history scatters around zero. Pick the maximum of that scatter and you have picked luck.

expected best of N useless ideas ≈ σ × [(1 − γ)·Φ⁻¹(1 − 1/N) + γ·Φ⁻¹(1 − 1/(N·e))]

σ is the spread of the measured Sharpe ratios, γ ≈ 0.577 (the Euler–Mascheroni constant) and Φ⁻¹ the inverse normal. For our thousand ideas, σ = 0.50, which predicts a best of about 1.62. The actual best was 1.42: exactly what luck alone delivers. The ten best averaged 1.31 when chosen and -0.25 afterwards.

Worked example

N = 100 ideas, Sharpe spread σ = 0.5:

  • Φ⁻¹(1 − 1/100) = Φ⁻¹(0.99) = 2.326
  • Φ⁻¹(1 − 1/(100e)) = Φ⁻¹(0.9963) = 2.68
  • 0.5 × (0.423 × 2.326 + 0.577 × 2.68) = 0.5 × 2.53 = 1.27

A best-of-100 Sharpe of 1.27 on this data is not evidence of anything.

Common mistake

Reporting the winner without the count. “This rule has a Sharpe of 1.4” means nothing until you know how many rules were tried to find it, including the ones tried in your head, on other charts, last year.

You tried 200 parameter settings and the best has a Sharpe of 1.3 on three years. What should you conclude?
What happens to the expected best Sharpe of useless ideas when you try more of them?

Closing round

Pick the honest result

Four results from the same no-edge market, reported the way a strategy seller might. Only one is a fair estimate of what trading it would earn.

Simulated data from the course’s market engine, not real prices. On this market the true Sharpe of every rule is zero.

Building a fresh case…

Further reading: Aronson, Evidence-Based Technical Analysis, chapter 6 (data-mining bias); López de Prado, Advances in Financial Machine Learning (Pro).