Walk-forward: the honest way to test an optimized system

IntermediateDuration ~40 min video + 40 min hands-onTools Python 3 + NumPy

Why this lesson

A single-shot backtest optimizes on all the data and then judges itself on the same data — which is like grading your own exam with the answer key open. Walk-forward analysis is the discipline that fixes this: you fit the parameters on one slice of history, test them on the next slice that the fitting never saw, then roll the whole window forward and repeat. The number that comes out — walk-forward efficiency — is the closest honest estimate you can get of what the system would have done live. This is the richest module in Level 2, and it is where a mediocre process quietly manufactures fake edges and a good one refuses to. Robert Pardo’s The Evaluation and Optimization of Trading Strategies (ch.11) is the canonical treatment; it is a paid book and named as such. The videos below are the free companions.

Watch

Start with Pardo himself — the man who invented the method, explaining why backtests fail live and what walk-forward actually is. One dense segment of a long interview.

Segment: 14:20–40:00 — WFA definition, windows, and common mistakeswatch full video

Watch for: The curve-fitter gut-check ('you just got a curve fitter program'); his flat refusal of anchored windows ('I would never use an anchored window… it defeats the whole purpose'); and the robustness bar — wanting more than 3, ideally 15–20, profitable and evenly-distributed walk-forward windows, plus wildly divergent optimized parameters as a curve-fit tell.

Then the definition, cleanly. Why a single optimization window can’t win: too long and it averages across regimes, too short and it loses significance.

Watch for: The regime-compromise problem — a long window's 'best' parameters are a compromise that performs adequately in no single regime, while a short window tracks current conditions but sacrifices statistical significance. There is no free lunch with a single stage.

The cardinal rule, stated precisely.

Watch for: Only the walk-forward (out-of-sample) result should inform the go / no-go decision — never the in-sample number. A losing walk-forward means either no edge or a botched optimization; it is never grounds to fall back on the in-sample result.

Now the quantitative pieces — how to size the windows, and the pitfalls that quietly ruin a walk-forward.

Watch for: The ratio table by degrees of freedom: 1 parameter → 2:1, 2 → 3:1, 3 → 4:1. And the subtle counting error — correlated partial fills from scaling in/out must not each count as an independent trade for sample-size purposes.

Watch for: Hard cap of 3 optimized parameters (prefer 2); parameter steps must differ meaningfully (20, 30, … 100 — not 1, 2, 3); and the fatal disguised-anchored-window mistake of letting the optimization start date stay fixed while only the end advances.

Watch for: Out-of-sample significance depends on the *cumulative* walk-forward range across all stages, not any single stage — so you can push the optimization:walk-forward ratio higher (4:1) to buy back in-sample significance. A concrete config: 10 years, 6 stages, 4:1 ratio, each stage shifted 25% forward.

Davey’s four-way test taxonomy — where walk-forward sits relative to the weaker alternatives — with a hand-run worked example.

Watch for: The four testing methods scored on believability and out-of-sample data; the 2-parameter momentum system hand-optimized over rolling 5-year windows (2007–2011 → trade 2012, etc.), netting ~$30k walk-forward versus ~$60k for the naive full-period fit — and why the smaller number is the *good* sign.

A coded demonstration of the whole thing (vectorbt) — useful for the mechanics, but flagged cost-naive.

Watch for: A moving-average pair picked on total return (32/43-day) loses -4.6% on unseen 2022 data — clean, honest overfitting on real numbers. Then a proper rolling-split walk-forward. Caveat: close-only price with zero cost model — treat as a mechanics tutorial, not a validated result.

The explainer

The mechanics first. You take your history and cut it into an in-sample (optimization) window followed by an out-of-sample (walk-forward) window. You run the optimizer on the in-sample window, pick the parameters, then apply those frozen parameters to the out-of-sample window and record the result. Then you slide both windows forward — a new in-sample block, a new out-of-sample block that again the optimizer never touched — and repeat, stitching the out-of-sample pieces together into one continuous walk-forward equity curve. That stitched curve is the only performance you are allowed to believe. This is the point Darwinex and Pardo both hammer: only the out-of-sample result informs the decision. The in-sample number is the flattering self-graded exam; it does not count.

Anchored versus rolling windows. An anchored window keeps the start date fixed and only extends the end, so the in-sample block grows without bound. A rolling window keeps a fixed length and slides both ends forward. Pardo will not use anchored windows, and his reason is precise: as the years pass, an anchored window’s oldest data — from a market regime that may no longer exist — keeps diluting the fit, so the optimization degrades and stops tracking current conditions. Worse, the disguised version of this mistake (letting the optimization start stay fixed while only the end advances, as the best-practice video warns) collapses your final pre-live optimization into a single full-period fit and quietly destroys the entire point of the exercise. Prefer rolling windows.

Sizing the windows. How much history goes to optimization versus validation is not arbitrary; it scales with your degrees of freedom — the count of parameters you optimize. In-sample statistical significance depends on sample size divided by degrees of freedom, while out-of-sample significance depends on sample size alone. So more free parameters demand a bigger in-sample window just to hold significance constant: roughly 2:1 at one parameter, 3:1 at two, 4:1 at three. And keep the parameter count low — the best-practice guidance is a hard cap of three, ideally two, with step sizes that differ meaningfully (20, 30, 40 rather than 1, 2, 3) so extra combinations add information instead of just overfitting surface.

Walk-forward efficiency (WFE) is the headline number: the out-of-sample performance as a fraction of the in-sample performance. If the system made 60 units in-sample and 30 out-of-sample, its WFE is roughly 50%. A beginner reads that as failure. It is not — it is honesty. The out-of-sample number is the one closer to what live trading delivers, and Davey’s hand-run example makes the point vividly: the walk-forward equity netted about half the naive full-period fit, and that gap is the size of the lie you would otherwise have told yourself. What you do want to see is Pardo’s robustness bar: many walk-forward windows (more than three, ideally 15–20), profitable and evenly distributed rather than carried by one lucky stretch, with optimized parameters that stay in a stable neighborhood from window to window. Wildly divergent parameters between windows are a curve-fit tell.

Finally, parameter sensitivity and the iteration budget — the concepts that tie this back to the bias catalog. A robust system’s performance changes smoothly as you nudge a parameter; if a tiny change in a lookback flips the result from great to terrible, you have found a fragile peak, not an edge. And every time you look at an out-of-sample segment, tweak something, and look again, you spend down its credibility — you are selecting on it, which leaks its specifics into your choices. That is why a mined segment gets burned: after enough passes it is no longer out-of-sample in any meaningful sense, and its verdict is worthless. You get a finite number of honest looks at any piece of data. Spend them deliberately, and write down your iteration budget before you start — the maximum number of variants you will test — because an unbounded search is just data-snooping with extra steps.

Do it

Hand-run a walk-forward on synthetic data and report the WFE. Using synthetic data with no real edge is the point: a good walk-forward should refuse to find gold that isn’t there.

1. Generate reproducible synthetic daily returns:

import numpy as np

def make_prices(seed, n=2600):
    rng = np.random.default_rng(seed)
    rets = rng.normal(0.0001, 0.01, n)   # tiny drift, no real edge
    return 100 * np.exp(np.cumsum(rets))

prices = make_prices(seed=1)

2. Define a one-parameter strategy (a moving-average length) and a function that returns its total return on a slice:

def strat_return(prices, ma_len):
    ma = np.convolve(prices, np.ones(ma_len) / ma_len, mode="valid")
    px = prices[ma_len - 1:]
    signal = (px[:-1] > ma[:-1]).astype(float)   # long when above MA
    daily = px[1:] / px[:-1] - 1
    return float(np.sum(signal * daily))         # summed simple returns

3. Optimize on an in-sample slice, then score the frozen choice on the next out-of-sample slice. Roll forward through the whole history:

candidates = range(10, 105, 5)   # MA lengths to try (one parameter)
in_len, out_len = 800, 200       # 4:1 ratio for a single parameter

is_total, oos_total = 0.0, 0.0
start = 0
while start + in_len + out_len <= len(prices):
    in_slice  = prices[start : start + in_len]
    out_slice = prices[start + in_len : start + in_len + out_len]
    best = max(candidates, key=lambda m: strat_return(in_slice, m))  # fit
    is_total  += strat_return(in_slice, best)   # in-sample of chosen param
    oos_total += strat_return(out_slice, best)  # OUT-of-sample, frozen param
    start += out_len                            # roll forward

wfe = oos_total / is_total if is_total else float("nan")
print(f"In-sample total:  {is_total:+.3f}")
print(f"Out-of-sample:    {oos_total:+.3f}")
print(f"Walk-forward efficiency (WFE): {wfe:.1%}")

4. Read the result. The in-sample total will look positive — the optimizer always finds something that worked in the past. The out-of-sample total should be near zero or negative, giving a low or negative WFE. That is the correct, honest answer for edge-free data: there was nothing to find, and the walk-forward said so.

5. Stretch: loop over 30 seeds and collect the WFE each time. Notice how rarely a genuinely-high WFE appears by chance — and that when it does, it does not repeat on the next seed. That non-repetition is exactly what “no real edge” looks like, and exactly what your capstone must be able to recognize.

Terms introduced

Terms introduced in this lesson

• Walk-forward analysis
• Walk-forward efficiency
• Anchored vs rolling window
• Parameter sensitivity
• Iteration budget

Each term links to the glossary, where it is defined by the way you just used it.

Check

Check yourself

1. In walk-forward analysis, which result is allowed to inform your go / no-go decision?

   The average of the in-sample and out-of-sample resultsThe in-sample optimization resultWhichever of the two is higherThe out-of-sample (walk-forward) result only

2. Robert Pardo, who invented walk-forward analysis, says he would never use an anchored window because…

   Rolling windows always overfitIt is too slow to computeA fixed start date means the optimization degrades over the years and defeats the purpose of walk-forwardAnchored windows need too much data

3. The Darwinex ratio guidance says the in-sample (optimization) window should grow relative to the out-of-sample window as you add parameters — roughly 2:1 at one parameter, 3:1 at two, 4:1 at three. Why?

   Because 4:1 is always the correct ratioBecause out-of-sample data is more expensive to collectBecause in-sample significance scales with sample size over degrees of freedom, so more free parameters need a bigger in-sample window to hold significance constantBecause more parameters make the strategy trade less often

4. A walk-forward efficiency (WFE) well below 100% — say the out-of-sample return is roughly half the in-sample return — is best read as…

   Proof the strategy is overfit and must be discardedA failure that means the strategy is worthlessNormal and honest — it is closer to what live trading would actually deliver than the flattering in-sample numberA red flag that you made a coding error

5. Why does a data segment get "burned" after you have mined it repeatedly?

   Each pass you make selecting what worked on that segment leaks its specifics into your choices, so it can no longer serve as a fair out-of-sample testThe data file becomes corruptedThe market regime changes automaticallyBrokers delete old data

Check answers

You can move on when you can… describe the roll-forward loop from memory, explain why only the out-of-sample curve counts and why an anchored window degrades, size in-sample versus out-of-sample windows from the degrees-of-freedom ratio, and say plainly why a data segment gets burned once you have mined it.

Mark this lesson complete

Go deeper

• Robert Pardo, The Evaluation and Optimization of Trading Strategies (2nd ed.), ch.11 — the canonical, book-length treatment of walk-forward analysis. Paid, and worth it for this module.
• Better System Trader ep.60 (Pardo) and ep.5 (Davey) — the full interviews: https://bettersystemtrader.com
• QuantStart, walk-forward and optimization articles — the free written companion: https://www.quantstart.com/articles/