Free tools

Overfitting lab

This lab builds a price series that has no edge by construction, tries many moving-average crossover settings on the first 70% of it, and keeps the best. Then it runs that same setting on the remaining 30%, which the search never saw. The more settings it tries, the better the best in-sample result looks, and the less that result says about anything outside the sample.

Settings triedDrawn at random from 878 fast/slow pairs (fast 2–50, slow 10–200).

SeedSets both the series and which settings are drawn.

The simulated series.

SIMULATED data: a random walk indexed from 100, in which every bar’s return is drawn with an average of zero. No real instrument is shown.

Best setting in-sample—

Its in-sample return—

Same setting out-of-sample—

Median in-sample, all tried—

Median out-of-sample, all tried—

In-sample returns of every setting tried.

Best in-sample return as more settings are tried.

Past performance — actual or hypothetical — is not indicative of future results.

These results are based on simulated or hypothetical performance results that have certain inherent limitations. Unlike the results shown in an actual performance record, these results do not represent actual trading. Also, because these trades have not actually been executed, these results may have under- or over-compensated for the impact, if any, of certain market factors, such as lack of liquidity. Simulated or hypothetical trading programs in general are also subject to the fact that they are designed with the benefit of hindsight. No representation is being made that any account will or is likely to achieve profits or losses similar to those shown.

Read our full Disclaimers, including risk and hypothetical performance

How it works

The series has 1,500 bars. Each bar multiplies the previous price by one plus a random return drawn from a normal distribution with a mean of zero, so no rule can have an advantage on it except by chance. The first 1,050 bars are the in-sample part; the last 450 are out-of-sample.

Each setting is a pair of simple moving averages. At every close it holds the series when the fast average is above the slow one and is out of the market otherwise; while it holds, it earns the next bar’s return. It never takes a short position, and there are no costs. Every setting is scored from bar 200, once the longest average exists.

Why the best in-sample result rises

Every setting’s in-sample return is partly a draw of luck. Picking the largest of many draws picks the luckiest one, and the largest of 500 draws is almost always larger than the largest of 10. That is why the curve climbs as more settings are tried, while the out-of-sample result of the winner stays scattered around the same place as everyone else’s.

Why out-of-sample and walk-forward testing exist

Testing on data the search never touched is the simplest check on this effect. Walk-forward testing repeats it: choose settings on one window, test them on the next, move both forward, and judge only the stitched-together test windows. Neither proves a rule works; they show how much of an in-sample result survived contact with data it was not fitted to. Analess’s backtesting tools are designed around out-of-sample, walk-forward and Monte Carlo checks; the features page describes them and the roadmap shows their timing.