In-sample vs out-of-sample

In-sample data is the history a model is fitted or optimized on; out-of-sample data is history the fitting never touched, used to judge the result fairly. Only the out-of-sample performance should inform a go/no-go decision, because in-sample numbers are effectively a self-graded exam with the answer key open.

First used in Lesson 2.5 · The bias catalog: hunting the ways a backtest lies — the lesson that makes this term real.