Data-snooping bias

Also called selection bias: the inflation of apparent performance that comes from testing many variants and keeping the best one. With enough trials, pure noise produces a winner, so every extra tweak or optimizer re-run makes a green result less trustworthy, not more. It is controlled by holding out untouched out-of-sample data and by budgeting the number of trials in advance.

First used in Lesson 2.5 · The bias catalog: hunting the ways a backtest lies — the lesson that makes this term real.