The improvement engine

ExpertDuration 35 minTools A version-control repo (git), A pre-registration template, A change log

Why this lesson

You will want to improve a live strategy. That instinct, ungoverned, is the single fastest route back to the overfitting you spent Level 2 learning to avoid — except now you are overfitting to live noise, one well-meaning tweak at a time, and the account is real. The improvement engine is the discipline that lets you keep improving without lying to yourself: a version-control loop with hard rules about what may change, how it is tested, and when it is allowed to take over. Frame it plainly — this is the loop that stops your improvement loop from becoming an overfitting engine.

Watch

There is no free video that teaches champion/challenger for retail systems directly — a genuine gap, so the lesson leads with the explainer. The closest practitioner antecedent is Davey’s development-and-portfolio discipline:

Segment: 14:10–24:00 — managing variables and avoiding over-optimizationwatch full video

Watch for: Feasibility-testing on a small slice, the 'monkey test' for whether a component has real merit, and Occam's razor against over-parameterized systems — the mindset the improvement engine formalizes into rules.

The explainer

Seven rules turn “I had an idea” into a change you can trust.

1. The champion is frozen. At any moment one version — the champion — is live and trading, unchanged. You do not retune it, patch it, or “just adjust” it while it runs. Freezing the champion is what makes attribution possible: the live equity curve means something precisely because nothing about the thing producing it is moving.

2. Challengers change exactly one variable. A challenger is a candidate successor that differs from the champion in one thing — one filter added, one parameter moved, one exit changed. Only then, if it wins, do you know what won. Change three things at once and a good result is uninterpretable: you have a better number and no idea which change earned it, which means you cannot trust it to generalise. One variable per challenger is the non-negotiable core.

3. Pre-registration. Before you test a challenger, you write down — and commit to your repo — the hypothesis, the single change, the metric, the sample, and the pass/fail threshold. This is the same pre-registration discipline from Level 2’s overfitting module, now applied to live improvement. Pre-registration is what stops you from running a test, seeing the result, and then deciding what “success” meant. The verdict is bound before the data is seen.

4. Out-of-sample only. A challenger is judged on data — or live-forward performance — the champion has never been fitted to. Testing an improvement on the same history that suggested it is circular; it will always look good. The only test that carries information is one the change has not already seen.

5. The iteration budget: a mined segment is burned. This is the subtlest rule and the one people break. Every time you test an idea against a slice of out-of-sample data, you spend a little of that slice’s validating power. Test twenty ideas against it and the twenty-first “pass” is likely just the best of twenty coin-flips — the segment has effectively become in-sample through repeated contact. So out-of-sample data has an iteration budget: a finite number of looks before it is burned and can no longer honestly validate anything. You budget your challengers accordingly, and you hold back genuinely untouched data for the decisions that matter. Davey’s feasibility-first, few-variables approach is the same instinct: spend your degrees of freedom sparingly, because you cannot get them back.

6. Symmetric demotion. Promotion and demotion use the same bar. If a challenger must beat the champion by a pre-registered margin on out-of-sample data to be promoted, then a champion that fails that same bar going forward must be demoted — retired or benched — with equal willingness. Asymmetry here is how portfolios rot: traders promote eagerly on good news and cling stubbornly through bad, so dead strategies accumulate. The bar is symmetric or it is not a bar.

7. One promotion at a time. Change one live thing, then wait long enough to attribute the result before changing the next. Promote two changes together and you are back in rule 2’s trap at the portfolio level — a shift in results with no clean cause.

Underpinning all seven is the attribution ledger: a running, version-stamped record of every champion, every challenger, every pre-registration, every promotion and demotion, and — crucially — counterfactual logging: what the version you didn’t promote (or the champion you did keep) would have done over the same period. Counterfactuals are what let you learn from decisions you got right and wrong, because you recorded the road not taken. Without the ledger, “the system is up 4%” is a number with no lineage. With it, every point on the curve traces to a specific, pre-registered, one-variable decision you can defend to a skeptic — which is exactly what the Level 3 capstone asks for.

The engine is deliberately slow. That is the feature. An improvement loop that can move fast is an overfitting loop; the rules exist to make each change expensive enough that only real ones are worth making.

Do it

Part 1 — write a challenger pre-registration. For a hypothetical change (“add a session filter so the strategy only trades the London–NY overlap”), fill this in and commit it before testing:

challenger-preregistration.yaml

challenger_id: CH-2026-07-14-session-filter
champion_id: CHAMP-004
hypothesis: >
  The champion's losers cluster in thin Asian-session liquidity. Restricting
  entries to the London-NY overlap (12:00-16:00 UTC) should cut cost-heavy
  losing trades without removing the edge.
single_change: "Add entry filter: only enter when 12:00 <= UTC hour < 16:00."   # exactly one variable
metric: "Net expectancy (R) after full costs"
secondary_metrics: ["trade count", "max drawdown (Monte Carlo 95th pct)"]
test_data: "Out-of-sample slice 2025-01 to 2025-12 — NOT used to design this filter"
iteration_budget_note: "3rd of 5 permitted looks at this OOS slice; 2 remain after this."
pass_threshold: ">= +0.05 R improvement in net expectancy AND no worse 95th-pct drawdown"
demotion_rule: "Symmetric — if promoted then underperforms champion by the same margin OOS-forward, demote."
decision: PENDING          # filled in ONLY after the test, never edited before

Part 2 — write the change-log / attribution-ledger entry recording the outcome:

## Ledger entry — CH-2026-07-14-session-filter

- Champion at test: CHAMP-004 (net expectancy +0.18 R over OOS slice)
- Challenger result: net expectancy +0.24 R (+0.06 R), drawdown envelope unchanged
- Threshold (+0.05 R, no worse drawdown): PASSED
- Decision: PROMOTE -> new champion CHAMP-005 on 2026-07-21 (one promotion at a time)
- Counterfactual to keep logging: CHAMP-004 continues in shadow; record what it
  WOULD have earned for 60 trades so we can confirm the promotion was real, not luck.
- OOS budget: 3 of 5 looks used on this slice; 2 remain.

Terms introduced

Terms introduced in this lesson

• Champion/challenger
• Promotion/demotion
• Attribution ledger
• Counterfactual logging

Each term links to the glossary, where it is defined by the way you just used it.

Check

Check yourself

1. The core risk the improvement engine is designed to prevent is…

   Trading too few instrumentsRunning out of historical dataPaying too much commissionYour ongoing 'improvements' silently overfitting the strategy to recent noise until it dies live

2. In champion/challenger, the champion is…

   The most complex versionRetuned every time a challenger is testedWhichever version had the best backtestFrozen — it keeps trading unchanged while challengers are tested against it, and is only replaced on a pre-registered win

3. Why does each challenger change only ONE variable at a time?

   To keep the code shorterSo that if it improves (or fails), you know exactly which change caused it — attribution is impossible if you change several things at onceTo save computing timeBecause brokers only allow one change per month

4. What does 'a mined segment is burned' mean for the iteration budget?

   You should delete old dataLosing trades destroy your capitalYou can reuse the same test data indefinitelyOnce you've tested many ideas against a slice of out-of-sample data, it has effectively become in-sample — its power to validate the next idea is spent

Check answers

You can move on when you can… freeze a champion, write a one-variable challenger pre-registration with a pass/fail threshold fixed in advance, test it only on unburned out-of-sample data within an iteration budget, apply symmetric demotion and one promotion at a time, and keep an attribution ledger with counterfactual logging — and explain why an unconstrained improvement loop is just an overfitting engine.

Mark this lesson complete

Go deeper

• Bailey et al., “Pseudo-Mathematics and Financial Charlatanism” — why repeated testing burns data and inflates the best-of-many result; the theory under the iteration budget.
• Kevin Davey, Building Winning Algorithmic Trading Systems — the practitioner’s add/remove and feasibility discipline the engine formalizes.
• Robert Pardo, The Evaluation and Optimization of Trading Strategies, chapter 11 — out-of-sample and walk-forward discipline, the validation half of every challenger test.