What a backtest can prove, and what it can't

Put two sentences side by side. “This rule made 40 trades on this data under these costs.” “This rule works.” Only the first is what a backtest says. Knowing what a backtest can prove, and where that stops, is most of the skill. By the end you will be able to separate what a backtest establishes from what it cannot, and read any result with its assumptions, sample and alternatives in view.

What a backtest actually establishes

A clean backtest establishes two things. First, the rule was specified precisely enough to replay: every entry, exit, size and filter was written down, with nothing left to judgement. That is real value, because a vague idea cannot be tested at all. Second, under stated data, costs and fills, the rule produced this particular trade list. It proves the rule was defined, not that it works.

A backtest shows a replayable rule and the trade list it produced under stated assumptions; it cannot show future behaviour.

What it cannot establish

A backtest cannot establish that the same behaviour will recur. Markets change, and a pattern present in one stretch of history may be absent from the next. Nor can it establish that the rule carries information rather than fitting noise: a random rule can produce an attractive trade list by chance, especially if many rules were tried.

The results are also not what you would have earned. They are hypothetical: the fills were assumed, the costs were modelled, and the rule was often chosen with hindsight, after the data already existed. No account traded them.

Overfitting looks like success

“At what point did you trust your strategy?” is a question asked in trading forums, and overfitting is one reason it is hard to answer. The more you adjust a rule after looking at the results, the better the curve gets and the less it means. Each tweak can fit the rule more closely to the particular noise of the data you tuned it on. In the synthetic example below, a simple rule was adjusted five times and then twenty times, each time keeping the change that made its past look smoothest. The most adjusted version looks best in-sample and goes flat on the 250 bars it never saw. Bailey, Borwein, López de Prado and Zhu (2017) show how the chance of this grows with the number of variants tried. The overfitting picture walks through it in one chart.

In this synthetic example, every round of adjustment improved the past curve, and the most adjusted rule went flat on new data.

Past performance — actual or hypothetical — is not indicative of future results.

These results are based on simulated or hypothetical performance results that have certain inherent limitations. Unlike the results shown in an actual performance record, these results do not represent actual trading. Also, because these trades have not actually been executed, these results may have under- or over-compensated for the impact, if any, of certain market factors, such as lack of liquidity. Simulated or hypothetical trading programs in general are also subject to the fact that they are designed with the benefit of hindsight. No representation is being made that any account will or is likely to achieve profits or losses similar to those shown.

Read our full Disclaimers, including risk and hypothetical performance

Assumptions are part of the result

The data source, the period, the costs, the fill rule and the universe all shape the outcome. Change one and the result changes. In the second synthetic example, the same rule ran five times on one universe of thirty series, with one assumption changed each time. Doubling the costs removed most of the result. Starting two years later and including the series that were removed along the way, the survivorship-free universe, each cut it noticeably. Even filling one bar later shifted it. A result reported without its assumptions is incomplete.

One rule, five sets of assumptions: in this synthetic test, every change moved the result, and doubling costs moved it most.

Past performance — actual or hypothetical — is not indicative of future results.

Read our full Disclaimers, including risk and hypothetical performance

How to read one honestly

Arnott, Harvey and Markowitz (2019) set out a research protocol for backtests; four checks from that spirit do most of the work.

Trade count. Forty trades is a modest sample, and a handful of trades can carry the whole result.
Range. Look at the spread of outcomes and the worst stretch, not only the total.
Null comparison. Compare with a meaningless baseline, such as random entries with the same number of trades and holding times. A result that a random rule matches shows no evidence of information.
Out-of-sample check. Judge the rule on data it was never tuned on.

Before you look, write down what you would need to see to drop the idea. A kill criterion set in advance is harder to bend to fit the result.

Your first honest read

Take one backtest result, in any tool, and find where it records how the run was configured. List every assumption: the data range, warm-up bars, costs, fill policy, the engine version and the random seed if there is one. Anything you cannot find is an assumption you do not control. Then run a random-entry null with the same trade count and compare the two. If they look alike, the honest conclusion is that this test found nothing, which is a useful result in itself.

Sources

Arnott, R., Harvey, C. R. and Markowitz, H., A backtesting protocol in the era of machine learning, The Journal of Financial Data Science, 2019.
Bailey, D. H., Borwein, J., López de Prado, M. and Zhu, Q. J., The probability of backtest overfitting, author manuscript; journal publication 2017.
Community question on trusting an overfit strategy, r/algotrading, 19 August 2026.
Backtesting guide, Analess product documentation, 25 October 2026.

Analess is an analysis tool, not investment advice. Backtests are hypothetical and do not show what any account earned or will earn. Nothing here is a recommendation to buy, sell, or hold anything.