In-sample and out-of-sample – trading math lesson

ProOpen to everyone while Pro is in preview.

The data you tune on is in-sample. Its results are inflated by the tuning. A holdout, set aside before any work and used once, is out-of-sample: the only fair estimate of what the tuned rule will do.

Predict first

We tune 25 moving-average pairs on six years of the no-edge market and pick the best. On the next four years, will it be above or below zero?

Tune on six years, test on four

Every pair of moving averages, scored by Sharpe ratio (after 5 basis points of cost per trade) on the first six years. Then flip to the last four, which the tuning never saw.

Market

Running the numbers…

Switch the lab to the trending market. There the best pair (MA 20/100) scored 1.02 in-sample and 0.88 out-of-sample: lower, as always, but still positive, because trends in that market persist. Out-of-sample does not prove an edge; it is where a real one stays standing.

A robust result looks like a plateau

Look at the in-sample map. If the best cell is an isolated peak surrounded by poor neighbours, it is probably noise. If a broad region of settings works, the idea is more likely real, and the choice of exact setting matters less.

Worked example

Ten years of daily data, about 2,520 days. A 60/40 split:

In-sample: days 1 to 1,512 (six years). Tune here, as much as you like.
Out-of-sample: days 1,513 to 2,520 (four years). Run the final rule once.
Expect the out-of-sample Sharpe to be lower. A drop of more than half is a classic sign of overfitting.

Common mistake

Peeking at the holdout and going back to tune. Once a holdout result changes a decision, it is in-sample. Each look spends some of its honesty; repeated looks spend all of it.

Your rule scores 1.5 in-sample and 0.4 out-of-sample. What is the most likely reading?Retune on the holdout to fix itMuch of the in-sample result was fitted noiseThe holdout period was unlucky

How should the split be made for market data?Test on the years that look most like todayAt random, day by dayBy time: tune on earlier years, test on later ones

Closing round

Pick the honest result

Four results from the same no-edge market, reported the way a strategy seller might. Only one is a fair estimate of what trading it would earn.

Simulated data from the course’s market engine, not real prices. On this market the true Sharpe of every rule is zero.

Building a fresh case…

Further reading: Davey, Building Winning Algorithmic Trading Systems (in- and out-of-sample testing).