Confidence intervals
How sure a result is depends on how many cases stand behind it. · 8 min
ProOpen to everyone while Pro is in preview.
A win rate measured on 49 cases is an estimate. A different ten years of the same market would give a different number. A confidence interval says how far the true value could plausibly be from the estimate.
Predict first
Hammers won 28 of 49 times (57.1%). Does the 95% interval for their true win rate include 50%?
Two intervals you need
The labs use the Wilson interval for win rates, a refinement of the first line that behaves well with small samples and rates near 0% or 100%. For averages, the t multiplier is a little larger than z for small samples.
How sure is a win rate?
Move the sample size and see the interval around the same win rate shrink. The interval is where the true rate plausibly sits.
60.0% of 40: true win rate between 44.6% and 73.7%
The interval includes 50%: this record could come from a coin flip.
Trades needed
To pin a win rate near 60% to ±5 points at 95% you need about 369 trades.
Precision grows with the square root of the sample. Halving the width of an interval takes four times the data. To pin the hammers’ average return to ±0.5% with their spread of 3.2%, you would need about 155 independent cases, more than three times what ten years produced.
Rare patterns are worse. Ten years of the no-edge market hold just 5 morning stars. Set the base-rate lab to forty years and there are still only 27, too few for a verdict, and pinning their average to ±0.5% would take about 127. At that pace, that is close to 190 years of data.
More data, narrower interval
Hammers on one, three, ten and thirty years of the same market, with an edge planted after each signal (or none). Watch the 95% interval of the edge.
Running the numbers…
Worked example
28 wins in 49: p = 0.5714.
1.96 × √(0.571 × 0.429 ÷ 49) = 13.9 points
So roughly 43.3% to 71.0%; Wilson gives 43.3% to 70.0%.
Common mistake
“60% winners over 20 trades proves an edge.” Twelve of 20 has a 95% interval from about 39% to 78%. A strategy with a true 45% win rate produces a run like that about one time in eight. Twenty trades say almost nothing; a hundred say a little.
Closing round
Edge or noise?
A fresh market and a pattern, with or without a hidden edge planted after each signal. Read the study and call it.
Building a fresh case…
Further reading: Aronson, Evidence-Based Technical Analysis, chapter 4 (sampling distributions).