Calibration
When a forecast says 70%, does it happen 70% of the time? First the market (no-vig close against results), then the model, with the market's forecast on the same games beside it.
How to read these numbers
Here the null is the diagonal: stated = actual. Market is Kalshi's last pre-tip price wherever a game has one (2025-26 on), else the sportsbook close; the hypotheses stay on the sportsbook prices they were registered on. ± is one standard error (1 SE). Colour marks only gaps of at least 2 SE from the null: green above, red below; everything else is left plain. Shaded columns are the ones to read first.Glossary
- Lean
- The side the model gives ≥ 50% (as recorded on the row).
- Value side
- The side where the model's probability beats the market's no-vig probability. Graded only through the pre-registered hypotheses; the blanket value bet is retired (see the note under the hypotheses).
- Model %
- The picked side's mean model win probability.
- Market %
- The picked side's mean no-vig market probability (q): the price with the bookmaker's margin removed.
- Win %
- How often the picked side actually won.
- Break-even
- The win rate the posted price needs to pay back; above q by about the hold.
- ROI
- Profit per 1-unit bet.
- Null
- The ROI expected if the market is right (q × payout − 1). vs null = ROI − null.
- z
- (ROI − null) ÷ SE. Across many cells, about one in twenty shows |z| ≥ 2 by chance.
- EV
- Win % − break-even; its null is q − break-even.
- Fav ROI
- 1u on the market favourite of the same games at the same price: the same-row baseline a pick rule has to beat (a tie at 50% goes to home).
- Log loss, Brier
- Proper scores of the probabilities; lower is better. Compared with the close on the same games.
- Native
- Written before tip and frozen: the forward test.
- Reconstructed
- Scored afterwards with that season left out of the fit and graded at the close: hindsight, never forward evidence.
Market: no-vig close vs actual
ESPN BET 1209 games
Each point is one probability bin: across the x-axis what was stated, up the y-axis how often it happened. On the dashed diagonal = calibrated. The bar is ±2 SE, where a correct forecast would usually land; a bar that misses the diagonal is a real miss, not noise. Hover a point for its numbers; the table below has them all.
By price rung and side — actual vs implied, gap ± 1 SE
| Price rung | Home side | Away side | Both sides |
|---|---|---|---|
| ≤ -250 | 79.0% vs 79.2% -0.2 ± 2.0 · n=396 | 78.2% vs 78.0% +0.3 ± 2.9 · n=202 | 78.8% vs 78.8% -0.1 ± 1.6 · n=598 |
| -249 to -175 | 61.8% vs 64.6% -2.9 ± 4.1 · n=136 | 65.9% vs 64.5% +1.3 ± 4.3 · n=126 | 63.7% vs 64.6% -0.8 ± 3.0 · n=262 |
| -174 to -130 | 63.3% vs 57.5% +5.8 ± 4.4 · n=128 | 53.0% vs 57.2% -4.2 ± 4.6 · n=115 | 58.4% vs 57.4% +1.1 ± 3.2 · n=243 |
| -129 to -100 | 44.3% vs 51.1% -6.8 ± 5.6 · n=79 | 65.3% vs 51.1% +14.3 ± 5.8 · n=75 | 54.5% vs 51.1% +3.5 ± 4.0 · n=154 |
| +100 to +129 | 41.8% vs 44.9% -3.1 ± 5.0 · n=98 | 41.2% vs 45.2% -3.9 ± 5.1 · n=97 | 41.5% vs 45.1% -3.5 ± 3.6 · n=195 |
| +130 to +174 | 40.5% vs 38.6% +1.9 ± 4.6 · n=111 | 34.9% vs 39.0% -4.1 ± 4.3 · n=129 | 37.5% vs 38.8% -1.3 ± 3.1 · n=240 |
| +175 to +249 | 33.3% vs 31.9% +1.4 ± 4.5 · n=108 | 32.9% vs 31.4% +1.4 ± 3.8 · n=146 | 33.1% vs 31.6% +1.4 ± 2.9 · n=254 |
| ≥ +250 | 17.6% vs 19.4% -1.8 ± 3.2 · n=153 | 19.7% vs 18.6% +1.2 ± 2.2 · n=319 | 19.1% vs 18.9% +0.2 ± 1.8 · n=472 |
1209 games (0 native, 1209 reconstructed; seasons 2024-25). Two observations per game, one per side; ± is one SE under correct prices, so a gap under ~2± is indistinguishable from fair. No both-sides total: the sides sum to 1 and one wins, so it is 50% by construction. Favourites asks once per game.
Kalshi 1209 games
Dashed diagonal = calibrated; bars are ±2 SE. Hover a point for its numbers.
By price rung and side — actual vs implied, gap ± 1 SE
| Price rung | Home side | Away side | Both sides |
|---|---|---|---|
| ≤ -250 | 79.9% vs 80.3% -0.3 ± 2.0 · n=399 | 81.4% vs 80.0% +1.3 ± 2.8 · n=204 | 80.4% vs 80.2% +0.2 ± 1.6 · n=603 |
| -249 to -175 | 68.8% vs 64.7% +4.1 ± 4.3 · n=125 | 62.0% vs 64.8% -2.8 ± 4.3 · n=121 | 65.4% vs 64.7% +0.7 ± 3.0 · n=246 |
| -174 to -130 | 61.1% vs 57.4% +3.7 ± 4.3 · n=131 | 55.8% vs 57.2% -1.4 ± 4.5 · n=120 | 58.6% vs 57.3% +1.3 ± 3.1 · n=251 |
| -129 to -100 | 38.8% vs 50.6% -11.9 ± 5.6 · n=80 | 55.2% vs 51.1% +4.1 ± 5.4 · n=87 | 47.3% vs 50.9% -3.6 ± 3.9 · n=167 |
| +100 to +129 | 49.2% vs 44.5% +4.6 ± 4.5 · n=120 | 42.9% vs 44.2% -1.3 ± 4.5 · n=119 | 46.0% vs 44.4% +1.7 ± 3.2 · n=239 |
| +130 to +174 | 37.0% vs 37.6% -0.5 ± 4.7 · n=108 | 39.2% vs 37.6% +1.6 ± 4.4 · n=120 | 38.2% vs 37.6% +0.6 ± 3.2 · n=228 |
| +175 to +249 | 36.5% vs 30.7% +5.8 ± 5.0 · n=85 | 25.0% vs 30.4% -5.4 ± 4.2 · n=120 | 29.8% vs 30.5% -0.8 ± 3.2 · n=205 |
| ≥ +250 | 14.3% vs 17.5% -3.2 ± 3.0 · n=161 | 17.6% vs 17.3% +0.3 ± 2.1 · n=318 | 16.5% vs 17.4% -0.9 ± 1.7 · n=479 |
1209 games (0 native, 1209 reconstructed; seasons 2025-26). Two observations per game, one per side; ± is one SE under correct prices, so a gap under ~2± is indistinguishable from fair. No both-sides total: the sides sum to 1 and one wins, so it is 50% by construction. Favourites asks once per game.
Model vs market, same games
Native · forward
No graded model rows with a close yet.
Reconstructed · hindsightESPN BET close · 2024-25log loss model − market +0.0118 ± 0.0050 · n=1209
0.5940 vs 0.5821 · n=1209
Each point is one probability bin: across the x-axis what was stated, up the y-axis how often it happened. On the dashed diagonal = calibrated. The bar is ±2 SE, where a correct forecast would usually land; a bar that misses the diagonal is a real miss, not noise. Hover a point for its numbers; the table below has them all. Both forecasts are for the same games; each is binned on its own probability. Negative model − market on the scores means the model did better; an interval spanning 0 has not separated them.
Model bins in numbers — with the market's mean on the same games
| Model P(home) | n | Model mean | Market mean (same games) | Actual | Actual − model (± 1 SE) |
|---|---|---|---|---|---|
| 0.0–0.2 | 76 | 15.0% | 17.1% | 10.5% | -4.5 ± 4.1 |
| 0.2–0.3 | 103 | 25.6% | 27.9% | 26.2% | +0.6 ± 4.3 |
| 0.3–0.4 | 153 | 35.1% | 35.8% | 37.3% | +2.2 ± 3.9 |
| 0.4–0.5 | 153 | 45.6% | 45.4% | 45.1% | -0.5 ± 4.0 |
| 0.5–0.6 | 152 | 55.1% | 54.5% | 54.6% | -0.5 ± 4.0 |
| 0.6–0.7 | 209 | 64.9% | 64.1% | 64.6% | -0.3 ± 3.3 |
| 0.7–0.8 | 187 | 75.0% | 72.3% | 74.9% | -0.1 ± 3.2 |
| 0.8–1.0 | 176 | 86.4% | 84.4% | 81.2% | -5.1 ± 2.6 |
Reconstructed · hindsightKalshi close · 2025-26log loss model − market +0.0159 ± 0.0055 · n=1209
0.5837 vs 0.5677 · n=1209
Dashed diagonal = calibrated; bars are ±2 SE. Hover a point for its numbers. Both forecasts are for the same games; each is binned on its own probability. Negative model − market on the scores means the model did better; an interval spanning 0 has not separated them.
Model bins in numbers — with the market's mean on the same games
| Model P(home) | n | Model mean | Market mean (same games) | Actual | Actual − model (± 1 SE) |
|---|---|---|---|---|---|
| 0.0–0.2 | 70 | 15.8% | 15.1% | 14.3% | -1.5 ± 4.3 |
| 0.2–0.3 | 95 | 25.6% | 24.4% | 21.1% | -4.5 ± 4.5 |
| 0.3–0.4 | 136 | 35.2% | 34.4% | 33.8% | -1.4 ± 4.1 |
| 0.4–0.5 | 172 | 45.1% | 45.6% | 45.3% | +0.2 ± 3.8 |
| 0.5–0.6 | 204 | 54.9% | 53.6% | 53.9% | -1.0 ± 3.5 |
| 0.6–0.7 | 190 | 65.0% | 64.3% | 65.8% | +0.7 ± 3.5 |
| 0.7–0.8 | 178 | 75.0% | 75.2% | 78.7% | +3.7 ± 3.2 |
| 0.8–1.0 | 164 | 85.7% | 85.7% | 85.4% | -0.4 ± 2.7 |
Bet grading by price band (ROI, EV and their nulls) is on the Ledger.