Three methods, the same 758,807 held-out ranked matches. We publish the receipts — especially the ones most tools never measure.
When a tool says 60%, how often does that team actually win?
Every winrate site will happily show you percentages. Almost none will tell you whether those percentages mean anything. Below: our production model against the method winrate-table sites imply (average the heroes' winrates), scored on the same holdout.
Its stated confidence barely moves reality: whatever it claims, the answer is ~52–54%. Averaged winrate tables aren't a smaller model — they're a broken thermometer.
Perfect calibration = the diagonal. Dot size = number of matches in that prediction bucket.
| Method | Calibration error (ECE) ↓ | Brier ↓ | Log loss ↓ | Accuracy ↑ |
|---|---|---|---|---|
| batru model (production transformer) Full-draft conditional model — the exact ONNX serving batru.gg right now. | 0.0086 | 0.2462 | 0.6854 | 54.9% |
| Naive winrate aggregation Per-hero solo winrates averaged per team, difference calibrated by a logistic fit on train — the strongest honest version of what winrate-table sites imply. | 0.0414 | 0.2523 | 0.6980 | 51.7% |
| Coin flip p=0.5 for every match. | 0.0319 | 0.2500 | 0.6931 | 53.2% |
Note the oddity: naive averaging scores worse than the 50-50 baseline on every metric — summing winrate tables doesn't just add little, it actively misleads. (The 50-50 baseline's 53.2% accuracy is Radiant's side advantage, free for any method that knows it.) And no, ~55% draft-only accuracy is not a typo — Dota is a balanced game; whoever quotes much higher numbers for pre-game prediction is leaking post-game information. Our edge isn't a magic hit rate: it's that the probability we print is one you can act on.
Full context on why we lead with calibration instead of accuracy: methodology →
Every week a freshly trained challenger is scored against the reigning production model on a new held-out set, and this ledger records the outcome — promotions and the weeks the champion held. The rows below are read straight from the append-only decision log our promotion gate writes (dota2-win-predictor-data/models/eval_history.jsonl); nothing is hand-picked.
| Date | Decision | Challenger ECE ↓ | Champion ECE | Challenger BCE ↓ | Acc | Holdout |
|---|---|---|---|---|---|---|
| 2026-07-18 | champion held | 0.0056 | 0.0058 | 0.6825 | 55.9% | 2,214,079 |
| 2026-06-28 | champion held | 0.0069 | 0.0058 | 0.6836 | 55.6% | 1,306,180 |
| 2026-06-22 | promoted | 0.0086 | 0.0153 | 0.6854 | 54.9% | 758,807 |
| 2026-06-18 | promoted | 0.0037 | — | 0.6847 | 55.2% | 561,855 |
| 2026-06-12 | promoted | 0.0025 | 0.0104 | 0.6841 | 55.3% | 507,361 |
| 2026-06-12 | champion held | 0.5287 | 0.0104 | 8.5223 | 47.1% | 501,210 |
| 2026-06-11 | champion held | 0.0166 | 0.0107 | 0.6865 | 54.9% | 450,515 |
| 2026-06-05 | champion held | — | — | 0.6826 | 55.9% | 236,687 |
| 2026-06-05 | promoted | — | — | 0.6824 | 55.9% | 233,639 |
| 2026-06-04 | promoted | — | — | 0.6808 | 56.4% | 229,671 |
| 2026-06-04 | promoted | — | — | 0.6818 | 56.0% | 211,822 |
| 2026-06-03 | champion held | — | — | 0.6810 | 56.3% | 178,811 |
| 2026-06-03 | champion held | — | — | 0.6915 | 52.8% | 157,039 |
| 2026-05-31 | promoted | — | — | 0.6785 | 56.3% | 32,471 |
| 2026-05-30 | promoted | — | — | 0.6841 | 55.4% | 13,416 |
A "champion held" week means the new checkpoint failed to beat production on calibration + loss and was discarded — publishing those weeks is the point: a ledger that only showed wins wouldn't be evidence. Accuracy hovers where a balanced game pins it (see the scoreboard note above); the promotion gate decides on BCE with an accuracy-regression guard, not on chasing headline accuracy.
Current production model promoted Jun 22, 2026 — this date only moves when a new checkpoint actually beats the champion on the same held-out set, never on a routine re-run.