The bias catalog: hunting the ways a backtest lies

IntermediateDuration ~28 min video + 30 min hands-onTools Python 3 + pandas

Why this lesson

Every backtest you will ever build starts out lying to you. Not because you are dishonest, but because the ways a test can flatter itself are subtle, mechanical, and invisible unless you go looking. The single most useful habit an intermediate systematic trader can build is to treat a green equity curve as a suspect, not a result. This lesson gives you the three charges to test it against — look-ahead bias, survivorship bias, and data-snooping bias — and the specific evidence that convicts each one. Ernest Chan’s Quantitative Trading (ch.3) is the cleanest short treatment of these three; QuantStart’s bias articles are the free companion.

Watch

Survivorship bias, made concrete. Darwinex uses the WWII returning-bombers analogy, then a genuinely clean worked example on the Dow.

Watch for: The Dow 30 swapping Exxon Mobil and Pfizer out for Salesforce and Amgen in 2019–2020, then Salesforce's price history — how backtesting today's index members over history smuggles in only the winners.

Now the mechanism behind overfitting, in three short parts. Part 1 is the curve-fitting intuition; part 2 names the two real causes.

Watch for: A 2-parameter line (y = mx + c) generalizes to new data better than a 5-parameter polynomial that fits the history almost perfectly — more parameters fitting the past does not mean a better forecast.

Watch for: Event overfitting: a EUR/USD parameter set that happened to be open through a Non-Farm Payrolls print captures a huge one-off move by luck and ranks top of the optimization, while a genuinely-edged set that got unlucky near news ranks low.

The killer pair — the same live-traded system, optimized first at 3 parameters, then at 5. Watch the out-of-sample fit collapse as the degrees of freedom rise.

Watch for: 3 parameters, 22 FX pairs, 36 combinations: the in-sample-vs-walk-forward scatter has an R-squared of 0.14, and the best in-sample pick still ranks 3rd out-of-sample — a healthy outcome.

Watch for: Same system at 5 parameters (576 combinations): R-squared collapses from 0.14 to 0.08, and the two top-ranked in-sample cells have only 38–41 trades each and rank near the bottom out-of-sample.

One more, for the data-snooping instinct: random systems that look like winners once you run enough of them.

Watch for: Purely random 50/50 entry systems, regenerated in bulk — scaled from 1 to 10 to 30 at once. As the count of tested combinations rises, the odds that at least one meaningless system looks like a strong trend climb sharply.

The explainer

Look-ahead bias is using information in a decision that would not have been available at the moment the decision was made. The textbook version is the one you will plant and fix below: a signal computed from today’s closing price that then trades at today’s open. In live trading the close has not happened yet when the open prints, so the backtest is trading on tomorrow’s newspaper. Look-ahead creeps in quietly — through indicators that peek at the current bar before it completes, through a max() or min() taken over the whole series, through a fill assumed at a price the market only reached later in the bar. The tell is almost always too good: near-perfect entries, a suspiciously smooth equity curve, a win rate that no honest edge produces. The fix is a discipline: every value a decision touches must be knowable strictly before the decision’s timestamp. In pandas this usually means a .shift(1) on your signal so that a bar acts only on information from bars that have already closed.

Survivorship bias is testing on a universe that was selected because it survived. The Darwinex example is exact: backtest the 30 companies in the Dow today, run it back a decade, and you have quietly excluded every company that was dropped for underperforming and included Salesforce precisely because it rose from about $29 to over $227. Index membership is reconstituted upward — losers fall out, winners come in — so “today’s members over history” is a bet on hindsight, not a strategy. It shows up wherever your data source only keeps the things that still exist: delisted stocks, closed funds, discontinued CFD instruments, brokers that went bust. The fix is to reconstruct the universe as it was at each point in time (point-in-time membership), or to accept honestly that you are measuring the survivors and discount the result accordingly.

Data-snooping bias (also called selection bias) is the one that will cost you the most money, because it feels like hard work. Every time you tweak a parameter, try another indicator, or re-run the optimizer, you are drawing another ticket in a lottery — and with enough tickets, pure noise produces a winner. The random-systems demo makes this physical: fifty-fifty coin-flip strategies with zero edge, run thirty at a time, and one of them will show a beautiful trending equity curve every single time. That is not an edge; it is the best of thirty random walks. This is where in-sample and out-of-sample earn their keep. You fit on the in-sample data and you judge on data the fitting never touched. The degrees-of-freedom demo shows why the judgment matters: the same live system had an in-sample-versus-out-of-sample R-squared of 0.14 at three parameters and 0.08 at five. Adding parameters did not add forecasting power — it added freedom to fit noise, and the top-ranked five-parameter cells turned out to rest on 38 to 41 trades of pure luck. Darwinex separates two flavors worth naming: noise overfitting (too many parameters chasing random wiggles) and event overfitting (one lucky one-off move, like being open through a Non-Farm Payrolls print, inflating a parameter set’s rank).

The through-line across all three: a backtest is an argument, and your job is the prosecution. Chan’s rule of thumb is that the more impressive the result, the harder you should look for the leak. Assume guilt, then try to acquit.

Do it

You will plant a look-ahead bug, watch it print fantasy returns, then fix it and see the fantasy evaporate.

1. Generate a synthetic price series so the result is reproducible:

import numpy as np
import pandas as pd

rng = np.random.default_rng(42)
n = 1500
# A random walk with a tiny drift — NO real edge exists in this data.
rets = rng.normal(0.0002, 0.01, n)
close = 100 * np.exp(np.cumsum(rets))
# Open = previous close plus a small overnight gap.
open_ = np.empty(n)
open_[0] = 100
open_[1:] = close[:-1] * (1 + rng.normal(0, 0.002, n - 1))

df = pd.DataFrame({"open": open_, "close": close})

2. Write the buggy backtest. The signal is a simple momentum rule: go long when today’s close is above the 20-bar moving average of the close. The bug: it acts on today’s signal at today’s open.

df["sma20"] = df["close"].rolling(20).mean()
df["signal"] = (df["close"] > df["sma20"]).astype(int)   # uses TODAY's close

# BUG: today's signal trades today's open-to-close move.
df["ret_open_close"] = df["close"] / df["open"] - 1
df["strat_buggy"] = df["signal"] * df["ret_open_close"]

equity_buggy = (1 + df["strat_buggy"].fillna(0)).cumprod()
print("Buggy final equity:", round(equity_buggy.iloc[-1], 3))

3. Find the bug by asking the one question that convicts look-ahead: at the moment I place the open trade, do I actually know this signal? No — the close that defines the signal has not printed yet. Fix it by shifting the signal so a bar only acts on information that closed before it:

df["signal_fixed"] = df["signal"].shift(1)   # act on YESTERDAY's completed signal
df["strat_fixed"] = df["signal_fixed"] * df["ret_open_close"]
equity_fixed = (1 + df["strat_fixed"].fillna(0)).cumprod()
print("Fixed final equity:", round(equity_fixed.iloc[-1], 3))

4. Compare. On data with no real edge, the buggy version will show a meaningfully higher, smoother curve than the fixed version, which should hover near a flat random walk. Print both and write one sentence: the entire difference between the two curves is a bug, not an edge.

5. Stretch: wrap steps 2–4 in a loop over 200 different rng seeds and count how often the buggy strategy “beats” buy-and-hold versus the fixed one. You are measuring how reliably look-ahead manufactures a fake edge from nothing.

Terms introduced

Terms introduced in this lesson

• Look-ahead bias
• Survivorship bias
• Data-snooping bias
• In-sample vs out-of-sample
• Noise vs event overfitting

Each term links to the glossary, where it is defined by the way you just used it.

Check

Check yourself

1. A backtest computes its buy/sell signal from the day's closing price and then books the fill at that same day's open. Which bias is this?

   Look-ahead biasNo bias — this is normalSurvivorship biasData-snooping bias

2. You backtest a strategy on the 30 stocks that are in the Dow Jones index today, running the test back ten years. What is the problem?

   Ten years is not enough historyThe test smuggles in only companies that survived and thrived into today's index — survivorship biasNothing, as long as costs are includedThe sample is too small to be significant

3. In the Darwinex degrees-of-freedom demo, the same system's in-sample-vs-out-of-sample R-squared fell from 0.14 to 0.08 when the number of optimized parameters went from 3 to 5. What does this show?

   More free parameters let the fit chase noise, so the out-of-sample relationship weakensFive parameters is the ideal countR-squared is an unreliable metricMore parameters always improve out-of-sample forecasting

4. Which mindset does this lesson ask you to adopt toward any green backtest?

   Treat it as guilty until proven otherwiseTrust it if it uses more than five years of dataTrust it if the equity curve is smoothTrust it if the Sharpe ratio is above 1

Check answers

You can move on when you can… inspect any backtest and name which of the three biases each part of it is exposed to, explain why a .shift(1) is the difference between a signal you know and a signal you don’t, and articulate why running the optimizer more times makes a green result less trustworthy, not more.

Mark this lesson complete

Go deeper

• Ernest Chan, Quantitative Trading (2nd ed.), ch.3 — the cleanest short treatment of look-ahead, survivorship, and data-snooping bias for beginners.
• QuantStart, “Successful Backtesting of Algorithmic Trading Strategies” — free articles on each bias with worked examples: https://www.quantstart.com/articles/
• David Aronson, Evidence-Based Technical Analysis, ch.6 “Data-Mining Bias: The Fool’s Gold of Objective TA” — the book-length case for why mined rules mostly evaporate out-of-sample.