Calibration: the difference between a model and a mascot
By George Boyle · Updated 2026-08-21 · The Sport Stack
A model is calibrated when things it calls 70% likely happen about 70% of the time. Calibration is measured on held-out data the model never saw during fitting, usually with a Brier score, and it is necessary but not sufficient for profit — the market is extremely well calibrated too, and matching it earns exactly the vig.
Calibration is a testable promise
When a model says a team wins 60% of the time, that single game tells you nothing — either outcome is consistent with the claim. But collect every 60% claim the model ever made, and about 60% of them should have come true. That property is calibration, and unlike “accuracy,” it is directly testable on any archived set of probabilistic predictions.
The standard scores are the Brier score for probability claims (mean squared error between stated probability and the 0/1 outcome — lower is better) and mean absolute error for quantity projections like team runs. Both only mean something on predictions that were locked before the games were played.
Lookahead is how models cheat
The most common way a sports model lies is lookahead: grading itself on predictions that quietly used information from after the fact — final lineups it would not have known, closing lines instead of the lines available at bet time, or a “backtest” refit on the very games it grades. The symptom is a backtest that prints money and a live record that does not.
The defenses are procedural, not mathematical: archive predictions at the moment they are made, grade them only against what was knowable then, and pre-register a system’s rules before the first real bet, including the conditions under which it gets retired. A system that stops performing is retired rather than re-tuned, and the graded losses stay on the public ledger.
A well-calibrated model can still not beat the market
Calibration is necessary, not sufficient. The market’s own probabilities are extremely well calibrated, and matching them earns exactly the vig, negative. We label surfaces honestly for this reason: some of our models are research context that has not beaten the market’s Brier score, and they say so on the page rather than pretending context is edge.
Written by George Boyle, who builds The Sport Stack — the models, the public ledger and these explainers. Corrections and questions: hello@thesportstack.io. Who runs this.