# Backtesting & Validation

Szymon Kopyciński · 23 September 2026



## 1. Backtesting and validation

### Backtest

A simulation of how a strategy would have performed on historical data.

Backtests are sensitive to biases and unrealistic assumptions, several of which are described below.

### In-sample

Data used to develop or fit a strategy.

### Out-of-sample

Data withheld from strategy development and later used to evaluate performance.

### Train set

Data used to fit a model.

### Validation set

Data used during model development to compare models or tune parameters.

### Test set

Data reserved for final evaluation.

Repeatedly inspecting the test set eventually turns it into another validation set.

### Walk-forward testing

Evaluating a strategy by repeatedly training on past data and testing on the following unseen period.

This mirrors how a model is used in live trading.

### Overfitting

When a model learns noise or quirks in historical data rather than genuine relationships.

An overfit strategy performs well in-sample and poorly out-of-sample.

### Look-ahead bias

Using information that would not have been available at the time a historical decision was made.

This makes a backtest look unrealistically strong.

### Survivorship bias

Bias caused by analysing only assets that survived to the present.

For example, testing a strategy on today's index constituents while ignoring companies that failed or were removed from the index.

### Data snooping

Repeatedly searching the same dataset until something that appears profitable emerges by chance.

### Multiple testing

Testing many hypotheses increases the probability that some will appear significant purely by luck.

### Transaction costs

Costs associated with trading. They include:

- spread;
- exchange fees;
- commissions;
- slippage;
- market impact;
- financing costs;
- taxes.

### Execution assumptions

Assumptions a backtest makes about how hypothetical trades would have executed.

Optimistic fill assumptions are one of the most common reasons backtests overstate performance.

## 2. Common research and trading traps

### Paper profit

A profit suggested by a model, backtest or unrealised position that may not translate into actual money.

### Backtest overfitting

Optimising a strategy on historical data so extensively that it captures noise rather than a repeatable edge.

### Leakage

Information entering a model that would not have been available in live trading.

Look-ahead bias is one form of leakage.

### Unrealistic fills

A backtest assuming executions that would not have occurred in reality.

For example, assuming every passive order at the best bid receives an immediate full fill.

### Mid-price execution

A simplifying assumption in which a backtest treats trades as happening at the mid.

In most markets you cannot trade at the mid: aggressive orders pay the spread, and passive orders fill only when someone trades against them.

### Ignoring costs

Testing a strategy without accounting for spreads, fees, slippage and market impact.

### Regime change

A meaningful change in market behaviour.

Relationships that held in one market environment may disappear in another.

### Non-stationarity

The broader problem that financial markets change over time.

Participants, regulations and technology change, and strategies themselves alter markets.

Part of [Glossary Index](/blog/glossary-index) · Previous: [Statistics & QR](/blog/glossary-index/glossary-stats-and-qr) · Next: [Derivatives: Futures, Options & Greeks](/blog/glossary-index/glossary-derivs)