How the models work, what they were trained on, and how accurate they actually are.
No black boxes. No "trust us". Real numbers from the most recent production evaluation.
Data refreshes daily from Steam Web API, OpenDota, and Stratz. Each new model checkpoint is evaluated against the held-out set and only promoted to production if it beats the current champion on BCE without regressing on accuracy. The promotion log is public via our internal eval history.
A small Transformer encoder over the 10 hero slots. Each slot is an embedding of (hero_id + team_side); the encoder produces a CLS token that goes through an MLP into a single sigmoid for Radiant win probability.
All three models are exported to ONNX and run on CPU. Inference for a full BP lookahead tree (~100 candidate evaluations) takes under 200 ms.
| Model | Accuracy | BCE (lower = better) | Brier | ECE (calibration) | Holdout size |
|---|---|---|---|---|---|
| Standard | 54.93% | 0.685 | 0.246 | 0.0086 | 758,807 |
| Plus | 55.28% | 0.685 | 0.246 | 0.0134 | 3,345 pro |
| Pro | 54.52% | 0.685 | 0.246 | 0.0255 | 1,825 pro |
How to read these: A binary classifier that always predicts 50/50 gets 50.00% accuracy and BCE 0.693. Our models sit ~5 pp above chance and below the naive BCE -- modest but meaningful. The standout number is ECE 0.0086 on the Standard model: when we say 60%, the actual outcome is 60% (averaged over many such predictions). That's the property serious users care about.
The production model serving these numbers was promoted Jun 22, 2026. This date only moves when a new checkpoint actually beats the champion — never on a routine re-evaluation.
Want to see the model called against actual matches you can verify? The /live page always shows the model's predictions for the most recent finished pro matches with the actual result alongside. Predictions are computed pre-game from the draft only -- the model has not seen the result.
No cherry-picking. The list is the raw OpenDota proMatches feed in reverse-chronological order. If we're wrong, you'll see it.
Stats sites don't run experiments, so they never have null results to report. We do, and we think publishing what didn't work is part of being a model you can trust.
A frequent request: "predictions for my bracket." So we ran the controlled experiment (July 2026, GPU A/B): same 9.66M-match dataset, same architecture, same epochs, same 570K-match holdout -- the only variable was giving the model each match's rank bracket.
| Model | BCE | Accuracy | ECE |
|---|---|---|---|
| Baseline (rank-blind) | 0.6786 | 0.5691 | 0.0106 |
| Rank-conditioned | 0.6780 | 0.5697 | 0.0095 |
| Same model, rank withheld | 0.6781 | 0.5696 | 0.0095 |
Accuracy moved +0.0006 -- noise. The ECE gain looks real until you see the giveaway: the conditioned model scores identically with rank withheld, so the improvement can't be coming from rank information. Root cause: matchmaking balances both teams within every bracket, so every bracket sits near the same base rate and the draft signal dominates.
Nuance worth spelling out: hero win rates genuinely differ by bracket -- that's why our bracket tier lists exist. What doesn't change is the draft-level prediction once the ten heroes are known. That's also why there is no rank selector on the predictor: shipping a control that barely moves the number would be over-promising.
The benchmark doesn't compare us against a strawman. The "aggregate" method is the strongest honest version of what winrate-table sites imply: per-hero win rates from the training split, combined and then calibrated by a logistic regression fitted on training data -- every generosity we could give it. It still comes out at ECE 0.0414 vs our 0.0086 on the same 758,807-match holdout, and it loses to a literal coin flip on calibration. Being generous to the baseline is what makes the comparison credible.
Pre-game accuracy in Dota is capped by design: the better the matchmaker, the closer every game is to 50/50 before it starts. Chasing headline accuracy on that track tells you nothing about whether a probability is honest. Our favorite cautionary tale from the community: a model that beat Valve's own predictor on accuracy, AUC and Brier -- and still lost on calibration (ECE 0.014 vs 0.003). A probability you can act on is one where "60%" empirically means 60%. That's the number we optimize, audit on every promotion, and publish.