Multiple testing

Running many strategy trials and reporting the best outcome, which inflates apparent performance because the maximum of many noisy draws is large even when no trial has an edge. A metric such as a Sharpe ratio is therefore uninterpretable without knowing how many trials produced it; the number of trials is part of the result.

First used in Lesson 2.8 · Overfitting statistics and pre-registration — the lesson that makes this term real.