Three methods, the same 3,084,959 held-out ranked matches. We publish the receipts — especially the ones most tools never measure.
When a tool says 60%, how often does that team actually win?
Every winrate site will happily show you percentages. Almost none will tell you whether those percentages mean anything. Below: our production model against the method winrate-table sites imply (average the heroes' winrates), scored on the same holdout.
Its stated confidence barely moves reality: whatever it claims, the answer is ~51–57%. Averaged winrate tables aren't a smaller model — they're a broken thermometer.
Perfect calibration = the diagonal. Dot size = number of matches in that prediction bucket.
| Method | Calibration error (ECE) ↓ | Brier ↓ | Log loss ↓ | Accuracy ↑ |
|---|---|---|---|---|
| batru model (production transformer) Full-draft conditional model — the exact ONNX serving batru.gg right now. | 0.0020 | 0.2389 | 0.6705 | 58.5% |
| Naive winrate aggregation Per-hero solo winrates averaged per team, difference calibrated by a logistic fit on train — the strongest honest version of what winrate-table sites imply. | 0.0324 | 0.2510 | 0.6953 | 52.3% |
| Coin flip p=0.5 for every match. | 0.0293 | 0.2500 | 0.6931 | 52.9% |
Note the oddity: naive averaging scores worse than the 50-50 baseline on every metric — summing winrate tables doesn't just add little, it actively misleads. (The 50-50 baseline's 52.9% accuracy is Radiant's side advantage, free for any method that knows it.) And no, ~58% draft-only accuracy is not a typo — Dota is a balanced game; whoever quotes much higher numbers for pre-game prediction is leaking post-game information. Our edge isn't a magic hit rate: it's that the probability we print is one you can act on.
Full context on why we lead with calibration instead of accuracy: methodology →
On a rolling cadence a freshly trained challenger is scored against the reigning production model on a new held-out set, and this ledger records the outcome — promotions and the rounds the champion held. The rows below are read straight from the append-only decision log our promotion gate writes (dota2-win-predictor-data/models/eval_history.jsonl); nothing is hand-picked.
| Date | Decision | Challenger ECE ↓ | Champion ECE | Challenger BCE ↓ | Acc | Holdout |
|---|---|---|---|---|---|---|
| 2026-11-08 | champion held | 0.0024 | 0.0019 | 0.6687 | 58.8% | 3,084,959 |
| 2026-11-07 | promoted | 0.0019 | 0.0016 | 0.6687 | 58.8% | 3,084,959 |
| 2026-10-05 | promoted | 0.0038 | 0.0056 | 0.6695 | 58.7% | 3,579,956 |
| 2026-10-04 | champion held | 0.0103 | 0.0063 | 0.6849 | 55.2% | 5,096,199 |
| 2026-09-07 | champion held | 0.0028 | 0.0047 | 0.6914 | 53.0% | 4,262,661 |
| 2026-09-06 | champion held | 0.0131 | 0.0046 | 0.6842 | 55.4% | 4,267,447 |
| 2026-09-05 | champion held | 0.0013 | 0.0043 | 0.6839 | 55.5% | 4,247,889 |
| 2026-08-08 | champion held | 0.0122 | 0.0079 | 0.6831 | 55.8% | 2,908,024 |
| 2026-08-04 | champion held | 0.0096 | 0.0079 | 0.6828 | 56.0% | 2,723,374 |
| 2026-08-01 | champion held | 0.0110 | 0.0075 | 0.6825 | 56.0% | 2,609,575 |
| 2026-07-28 | champion held | 0.0173 | 0.0070 | 0.6819 | 56.2% | 2,421,469 |
| 2026-07-28 | champion held | 0.0258 | 0.0071 | 0.6843 | 56.0% | 2,389,255 |
| 2026-07-25 | champion held | 0.0123 | 0.0064 | 0.6825 | 56.0% | 2,316,771 |
| 2026-07-18 | champion held | 0.0056 | 0.0058 | 0.6825 | 55.9% | 2,214,079 |
| 2026-06-28 | champion held | 0.0069 | 0.0058 | 0.6836 | 55.6% | 1,306,180 |
| 2026-06-22 | promoted | 0.0086 | 0.0153 | 0.6854 | 54.9% | 758,807 |
| 2026-06-18 | promoted | 0.0037 | — | 0.6847 | 55.2% | 561,855 |
| 2026-06-12 | promoted | 0.0025 | 0.0104 | 0.6841 | 55.3% | 507,361 |
| 2026-06-12 | champion held | 0.5287 | 0.0104 | 8.5223 | 47.1% | 501,210 |
| 2026-06-11 | champion held | 0.0166 | 0.0107 | 0.6865 | 54.9% | 450,515 |
| 2026-06-05 | champion held | — | — | 0.6826 | 55.9% | 236,687 |
| 2026-06-05 | promoted | — | — | 0.6824 | 55.9% | 233,639 |
| 2026-06-04 | promoted | — | — | 0.6808 | 56.4% | 229,671 |
| 2026-06-04 | promoted | — | — | 0.6818 | 56.0% | 211,822 |
| 2026-06-03 | champion held | — | — | 0.6810 | 56.3% | 178,811 |
| 2026-06-03 | champion held | — | — | 0.6915 | 52.8% | 157,039 |
| 2026-05-31 | promoted | — | — | 0.6785 | 56.3% | 32,471 |
| 2026-05-30 | promoted | — | — | 0.6841 | 55.4% | 13,416 |
A "champion held" week means the new checkpoint failed to beat production on calibration + loss and was discarded — publishing those weeks is the point: a ledger that only showed wins wouldn't be evidence. Accuracy hovers where a balanced game pins it (see the scoreboard note above); the promotion gate decides on BCE with an accuracy-regression guard, not on chasing headline accuracy.
Current production model promoted Nov 7, 2026 — this date only moves when a new checkpoint actually beats the champion on the same held-out set, never on a routine re-run.