Control groups
A pattern only matters if it beats a random entry in the same data. · 8 min
ProOpen to everyone while Pro is in preview.
A drug is tested against a placebo, not against nothing. A pattern is tested against random entries: days picked by lottery from the same market and period, held for the same number of days.
Predict first
Hammers rose 57.1% of the time and averaged +0.35% over five days. If you drew 49 random days from the same ten years, how often would their average do at least as well, in either direction?
How the control works
- Compute the forward return for every eligible day: the base rate.
- Draw as many random days as there were signals, and average their returns. Repeat 1,000 times.
- Place the pattern’s average among those 1,000. Out in the tail: the pattern did something random entries rarely do. In the crowd: no evidence.
Base-rate lab
The pattern against 1,000 random samples of entry days from the same history. If the pattern mattered, its average would sit out in the tail of the random ones.
Running the numbers…
Switch the planted edge on. The lab then pushes price up after every signal, and the pattern leaves the crowd. With +2% planted, the hammers’ edge becomes +1.91% (interval +0.96% to +2.85%). The test is not blind: it finds an edge when one exists.
Worked example
edge = +0.35% − +0.03% = +0.32%
Its 95% interval runs from −0.59% to +1.23%, and 50% of random samples did as well. The coach’s reading: “49 cases, and the interval of the edge crosses zero: no evidence of an edge.”
Common mistake
Comparing with zero instead of with random entries. In a market that rose 10% a year, almost any long signal shows a positive average. That is the market’s drift, not the pattern. Subtract the base rate first. The reverse also holds: in a falling market a good pattern can show a negative average and still beat random entries.
A control must match everything but the idea
“Gaps fill” sounds testable: after an opening gap, does price trade back to the prior close? On the no-edge market, 756 gaps of at least a quarter ATR filled the same day 41.8% of the time. Ordinary days reach a level that far from their open only 38.3% of the time, so gaps seem to fill more. But gap days are more volatile than ordinary days, and volatile days reach every level more often. The fair control is the same gap days, the same distance the other way: 44.8%. Against it, the “edge” disappears.
Do gaps fill?
Every opening gap of at least a quarter ATR, sized by the ATR of the days before it. A gap fills when price trades back to the prior close.
Running the numbers…
Closing round
Edge or noise?
A fresh market and a pattern, with or without a hidden edge planted after each signal. Read the study and call it.
Building a fresh case…
Further reading: Aronson, Evidence-Based Technical Analysis, chapters 4 and 5 (hypothesis tests).