Backtesting & Validation
Szymon Kopyciński · 23 September 2026
1. Backtesting and validation
Backtest
A simulation of how a strategy would have performed on historical data.
Backtests are sensitive to biases and unrealistic assumptions, several of which are described below.
In-sample
Data used to develop or fit a strategy.
Out-of-sample
Data withheld from strategy development and later used to evaluate performance.
Train set
Data used to fit a model.
Validation set
Data used during model development to compare models or tune parameters.
Test set
Data reserved for final evaluation.
Repeatedly inspecting the test set eventually turns it into another validation set.
Walk-forward testing
Evaluating a strategy by repeatedly training on past data and testing on the following unseen period.
This mirrors how a model is used in live trading.
Overfitting
When a model learns noise or quirks in historical data rather than genuine relationships.
An overfit strategy performs well in-sample and poorly out-of-sample.
Look-ahead bias
Using information that would not have been available at the time a historical decision was made.
This makes a backtest look unrealistically strong.
Survivorship bias
Bias caused by analysing only assets that survived to the present.
For example, testing a strategy on today's index constituents while ignoring companies that failed or were removed from the index.
Data snooping
Repeatedly searching the same dataset until something that appears profitable emerges by chance.
Multiple testing
Testing many hypotheses increases the probability that some will appear significant purely by luck.
Transaction costs
Costs associated with trading. They include:
- spread;
- exchange fees;
- commissions;
- slippage;
- market impact;
- financing costs;
- taxes.
Execution assumptions
Assumptions a backtest makes about how hypothetical trades would have executed.
Optimistic fill assumptions are one of the most common reasons backtests overstate performance.
2. Common research and trading traps
Paper profit
A profit suggested by a model, backtest or unrealised position that may not translate into actual money.
Backtest overfitting
Optimising a strategy on historical data so extensively that it captures noise rather than a repeatable edge.
Leakage
Information entering a model that would not have been available in live trading.
Look-ahead bias is one form of leakage.
Unrealistic fills
A backtest assuming executions that would not have occurred in reality.
For example, assuming every passive order at the best bid receives an immediate full fill.
Mid-price execution
A simplifying assumption in which a backtest treats trades as happening at the mid.
In most markets you cannot trade at the mid: aggressive orders pay the spread, and passive orders fill only when someone trades against them.
Ignoring costs
Testing a strategy without accounting for spreads, fees, slippage and market impact.
Regime change
A meaningful change in market behaviour.
Relationships that held in one market environment may disappear in another.
Non-stationarity
The broader problem that financial markets change over time.
Participants, regulations and technology change, and strategies themselves alter markets.
Part of Glossary Index · Previous: Statistics & QR · Next: Derivatives: Futures, Options & Greeks
NEFS Quant · About · Blog · News & Events · People · Privacy policy