Overfitting statistics and pre-registration

IntermediateDuration ~20 min video + 40 min hands-onTools Python 3 + NumPy + SciPy

Why this lesson

Everything in Level 2 has been circling one enemy: the beautiful result that came from testing many things and keeping the luckiest. This is the module that gives you the statistics to quantify that luck and the discipline to prevent it. You will meet multiple testing and its correction, the deflated Sharpe ratio; the probability of backtest overfitting (PBO); and the single most powerful habit in the whole course — pre-registration, writing down your hypothesis, metric, sample size, and threshold before you touch the data. This module is unusually free-rich: the Bailey and López de Prado papers are open on SSRN, and “Pseudo-Mathematics and Financial Charlatanism” is the readable version. The one caveat, stated plainly below, is that the embedded video is a prerequisite, not the whole answer.

Watch

This QuantPy video is a genuine prerequisite: it shows why a single Sharpe ratio is untrustworthy before you even get to the multiple-testing problem. One honesty note up front, because the title oversells it.

Watch for: Despite the title, this covers the PROBABILISTIC Sharpe Ratio (PSR) — correcting one Sharpe estimate's confidence for skew, kurtosis, and small sample size — NOT the deflated Sharpe. Watch the Gaussian-mixture demo where two portfolios share an identical true Sharpe but wildly different tails, and the 8-observation example where a fat-tailed, negatively-skewed 'hidden blowup' portfolio shows an inflated Sharpe of ~4 versus a true 1.5. Treat this as the foundation; the deflation step for multiple testing is carried by the explainer below.

The explainer

Start with the intuition the QuantPy video plants: even a single Sharpe ratio, honestly computed, is only an estimate with a confidence interval — and when returns are skewed or fat-tailed (which trading returns always are), that estimate can be badly inflated. The probabilistic Sharpe ratio (PSR) answers “given this much data and this shape of returns, how confident am I that the true Sharpe is above some benchmark?” It corrects one number for non-normality and sample size. Necessary, but not sufficient — because it says nothing about the fact that you tried a hundred strategies and kept the best.

That “kept the best” step is multiple testing, and it is the core problem of this module. Here is the mechanism, and you saw it live in the 2.5 random-systems demo: if you run one coin-flip strategy, its Sharpe is centered on zero. If you run a hundred independent coin-flip strategies and report only the highest Sharpe, that maximum is comfortably positive — not because any of them has an edge, but because the maximum of many noisy draws is large. The more configurations you test, the higher the “best” result you will find from pure noise. So a Sharpe of 1.5 selected from 500 trials is a completely different animal from a Sharpe of 1.5 you committed to in advance. The first might be noise; the second is evidence.

The deflated Sharpe ratio (DSR), from Bailey and López de Prado, is the correction. It takes your observed Sharpe and asks: given that I ran N trials with this much variance in outcomes, what is the probability this result exceeds what the best of N pure-noise strategies would have produced? Concretely, it computes an expected-maximum-Sharpe benchmark from the number of trials (the more trials, the higher the noise benchmark you must clear), then runs the PSR machinery against that benchmark instead of against zero. A raw Sharpe that looked impressive can deflate to a probability barely above a coin flip once you account for how many times you rolled the dice. The lesson: the number of trials is part of the result. A Sharpe reported without its trial count is not interpretable.

The probability of backtest overfitting (PBO) attacks the same problem from a different angle, via combinatorially symmetric cross-validation. You split your history into many pieces, form all the ways of designating half as in-sample and half as out-of-sample, and in each split you pick the configuration that ranked best in-sample and check where it lands out-of-sample. PBO is the fraction of splits in which your in-sample winner underperforms the median configuration out-of-sample. A high PBO means your selection process is essentially picking overfit configurations — the thing that looked best in-sample is, more often than not, mediocre or worse on unseen data. It is a direct, empirical measure of how much your optimization is fooling you.

All three tools share one message: the process that produced a result is part of the result. Which is why the practical antidote is not another statistic but a commitment device — pre-registration. Borrowed from clinical trials, where it exists precisely to stop researchers from testing many endpoints and reporting the one that “worked,” a trading pre-registration is a short document you write before testing that fixes four things: the hypothesis (the specific claim, e.g. “London Breakout has positive net-of-cost expectancy on EUR/USD”), the metric (exactly how you will judge it — net expectancy, or deflated Sharpe, defined before you look), the sample size (how much data and how many trades you need for the verdict to mean anything, from the 2.3 power analysis), and the threshold (the pass/fail line, decided in advance). Crucially it also names a dumb baseline — a deliberately naive alternative like random entries with the same sizing — that your strategy must beat, not merely be profitable against. Pre-registration works because it converts the multiple-testing problem into a single test: you get one honest shot at the out-of-sample verdict, and you agreed to the rules before you could see which way to bend them. Everything you build in the Level 2 capstone runs through a pre-registration exactly like the one you will write below.

Do it

Part A — write and fill a pre-registration template. Copy this into a file named prereg_london_breakout.md and fill every field before running a single backtest. This is the exact document your capstone will use.

# Pre-registration — London Breakout on EUR/USD

- Date written:
- Hypothesis (one falsifiable sentence):
    e.g. "London Breakout (open of London session, N-pip range,
    stop = range, target = 1.5x range) has positive net-of-cost
    expectancy on EUR/USD 2015–2024."
- Data & sample: instrument, timeframe, date range, expected
    number of trades (justify from the 2.3 power analysis).
- Cost model: spread by session, commission, swap, slippage
    estimate (from the Level 1 cost model — non-negotiable).
- Metric: the ONE number that decides it (e.g. net expectancy in R,
    or deflated Sharpe with the trial count recorded).
- Threshold (pass/fail line, decided now):
- Dumb baseline it must beat: e.g. random entries, same sizing,
    same session, same cost model.
- Iteration budget: max number of parameter variants I will test:
- Pre-committed decision: if it fails, I report the failure. I do
    NOT keep tweaking until it passes.

Part B — compute a deflated Sharpe for N trials. The formula, in words: first estimate the noise benchmark — the Sharpe you’d expect from the best of N trials given how much the trial Sharpes vary — then evaluate the probability your observed Sharpe beats that benchmark, correcting for skew and kurtosis and sample length.

import numpy as np
from scipy.stats import norm

def expected_max_sharpe(sr_variance, n_trials):
    """Expected max Sharpe from N noise trials (Bailey & Lopez de Prado).
    sr_variance = variance of the per-observation Sharpe across trials."""
    gamma = 0.5772156649                       # Euler-Mascheroni constant
    z1 = norm.ppf(1 - 1.0 / n_trials)
    z2 = norm.ppf(1 - 1.0 / (n_trials * np.e))
    return np.sqrt(sr_variance) * ((1 - gamma) * z1 + gamma * z2)

def deflated_sharpe(sr_hat, sr0, T, skew, kurt):
    """Probability the observed Sharpe (sr_hat) beats the noise
    benchmark (sr0). All Sharpes are per-observation, not annualized."""
    num = (sr_hat - sr0) * np.sqrt(T - 1)
    den = np.sqrt(1 - skew * sr_hat + ((kurt - 1) / 4) * sr_hat**2)
    return norm.cdf(num / den)

# --- Example: an "impressive" result from a wide search ---
sr_hat = 0.12      # per-observation Sharpe of the selected best strategy
T      = 1000      # number of return observations
n      = 200       # trials you ran to find this "best"
var_sr = 0.01**2   # spread of per-trial Sharpes (from your search)

sr0 = expected_max_sharpe(var_sr, n)
dsr = deflated_sharpe(sr_hat, sr0, T, skew=-0.5, kurt=6.0)
print(f"Noise benchmark (expected max Sharpe of {n} trials): {sr0:.3f}")
print(f"Deflated Sharpe probability: {dsr:.1%}")

Part C — watch it deflate. Rerun Part B with n = 1, then n = 50, then n = 500, holding everything else fixed. The noise benchmark sr0 climbs with the trial count and the deflated probability falls — the same observed Sharpe becomes less and less believable purely because you searched harder to find it. Write one sentence explaining, to a skeptic, why reporting a Sharpe without its trial count is meaningless.

Terms introduced

Terms introduced in this lesson

• Multiple testing
• Probabilistic Sharpe ratio
• Deflated Sharpe ratio
• Probability of backtest overfitting (PBO)
• Pre-registration
• Dumb baseline

Each term links to the glossary, where it is defined by the way you just used it.

Check

Check yourself

1. You try 100 strategy variants and keep the best Sharpe ratio. Why can't you take that Sharpe at face value?

   Selecting the maximum of many noisy trials inflates it — even pure-noise strategies produce a flattering "best" by chanceBecause you did not annualize itSharpe ratios are always overstated by exactly 50%A Sharpe ratio is meaningless below 1.0

2. What does the deflated Sharpe ratio correct for that a plain Sharpe ratio ignores?

   The number of trials you ran and the return distribution's skew and kurtosis, i.e. selection under multiple testingThe choice of brokerTransaction costs and slippageThe size of your account

3. The QuantPy video on the Probabilistic Sharpe Ratio is a prerequisite for this lesson, but note that it covers…

   Position sizingThe deflated Sharpe ratio directlyWalk-forward analysisThe probabilistic Sharpe ratio — correcting a single Sharpe estimate for non-normality and small samples, not the multiple-testing correction itself

4. What is the probability of backtest overfitting (PBO)?

   The estimated probability that the configuration you selected as best in-sample underperforms the median out-of-sampleThe chance a strategy loses money on any single dayThe probability that your broker is a B-bookThe fraction of trades that hit their stop

5. What makes a pre-registration effective as an antidote to overfitting?

   It requires approval from a regulatorIt guarantees the strategy will be profitableIt fixes the hypothesis, metric, sample size, and pass/fail threshold in writing BEFORE any testing, and includes a dumb baseline to beatIt is written up after the test, as documentation

Check answers

You can move on when you can… explain why the number of trials belongs in any reported Sharpe, describe what the deflated Sharpe and PBO each measure, distinguish the probabilistic Sharpe (single-estimate correction) from the deflated Sharpe (multiple-testing correction), and write a pre-registration with all four fields plus a dumb baseline — and mean it when you commit to reporting a failure as a failure.

Mark this lesson complete

Go deeper

• Bailey & López de Prado, “The Deflated Sharpe Ratio” (2014) — free on SSRN: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2460551
• Bailey, Borwein, López de Prado & Zhu, “The Probability of Backtest Overfitting” (2015) — the PBO paper, free on SSRN: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2326253
• “Pseudo-Mathematics and Financial Charlatanism” (2014) — the readable polemic version, free from the AMS: https://www.ams.org/notices/201405/rnoti-p458.pdf
• Stefan Jansen, Machine Learning for Algorithmic Trading, ch.6 (multiple-testing control) and ch.16 (backtest overfitting control).