← All lessons

Multiple testing

Raise the bar for every extra idea you test. · 8 min

ProOpen to everyone while Pro is in preview.

A test at the 5% level lets one useless idea in twenty through. Test twenty useless ideas and you should expect one “discovery”; the chance of at least one is 64.2%. The fix is to raise the bar with the number of tests.

Predict first

You test 50 ideas and want only a 5% chance of any false discovery. What level must each one pass?

expected false discoveries = n × α
chance of at least one = 1 − (1 − α)ⁿ
Bonferroni: test each at α ÷ n · Šidák: 1 − (1 − α)^(1/n)
Benjamini–Hochberg: sort p-values; keep those with p(k) ≤ (k ÷ n)·q

The price of testing many ideas

Every extra test is another ticket in the lottery of luck. These are the formulas that set the bar.

Useless ideas expected to pass
1.0
Chance at least one useless idea passes
64.2%
Bonferroni level per test
0.250%
Šidák level per test
0.256%
Sharpe needed, one test
0.95
Sharpe needed, 20 tests (Bonferroni)
1.62

A mixed batch

Of these 20 ideas, 2 are real (a true edge with t ≈ 3) and the rest are noise. Which pass?

RuleReal ideas foundUseless ideas passed
Each at 5%1 / 23
Bonferroni1 / 21
Benjamini–Hochberg1 / 21

Strict or forgiving?

Bonferroni guarantees few false discoveries but misses many real ones. Benjamini–Hochberg controls the share of discoveries that are false instead, so it finds more real ideas at the cost of letting a few false ones in. The batch in the lab shows the trade-off. The deflated Sharpe ratio applies the same idea to strategies: it measures a Sharpe against the best that the number of trials would produce by luck.

Worked example

20 ideas tested at 5%, none of them real:

  • Expected false discoveries = 20 × 0.05 = 1
  • Chance none passes = 0.95²⁰ = 0.358
  • Chance at least one passes = 1 − 0.358 = 64.2%

Common mistake

Counting only the final test. Every variant you looked at counts: other stops, other markets, other years, other indicators. If you cannot count them, assume many and demand fresh data.

You test 10 independent useless ideas at 5%. What is the chance at least one passes?
With 100 tests and Bonferroni at 5%, which p-value passes?

Closing round

Pick the honest result

Four results from the same no-edge market, reported the way a strategy seller might. Only one is a fair estimate of what trading it would earn.

Simulated data from the course’s market engine, not real prices. On this market the true Sharpe of every rule is zero.

Building a fresh case…

Further reading: Aronson, Evidence-Based Technical Analysis, chapter 6; López de Prado, Advances in Financial Machine Learning, chapter 14 (Pro).