← Dota 2

Draft Prediction Benchmark

Three methods, the same 3,084,959 held-out ranked matches. We publish the receipts — especially the ones most tools never measure.

The question that matters

When a tool says 60%, how often does that team actually win?

Every winrate site will happily show you percentages. Almost none will tell you whether those percentages mean anything. Below: our production model against the method winrate-table sites imply (average the heroes' winrates), scored on the same holdout.

batru model
  • says 60–65% → wins 62.6%
  • says ≥65% → wins 69.8%
Naive winrate averaging
  • says 60–65% → wins 54.5%
  • says ≥65% → wins 56.0%

Its stated confidence barely moves reality: whatever it claims, the answer is ~51–57%. Averaged winrate tables aren't a smaller model — they're a broken thermometer.

Reliability diagram

Perfect calibration = the diagonal. Dot size = number of matches in that prediction bucket.

3,084,959 hold-out matches · production model v_20261107_1723

Full scoreboard

MethodCalibration error (ECE) ↓Brier ↓Log loss ↓Accuracy ↑
batru model (production transformer)
Full-draft conditional model — the exact ONNX serving batru.gg right now.
0.00200.23890.670558.5%
Naive winrate aggregation
Per-hero solo winrates averaged per team, difference calibrated by a logistic fit on train — the strongest honest version of what winrate-table sites imply.
0.03240.25100.695352.3%
Coin flip
p=0.5 for every match.
0.02930.25000.693152.9%

Note the oddity: naive averaging scores worse than the 50-50 baseline on every metric — summing winrate tables doesn't just add little, it actively misleads. (The 50-50 baseline's 52.9% accuracy is Radiant's side advantage, free for any method that knows it.) And no, ~58% draft-only accuracy is not a typo — Dota is a balanced game; whoever quotes much higher numbers for pre-game prediction is leaking post-game information. Our edge isn't a magic hit rate: it's that the probability we print is one you can act on.

Methodology (reproducible)

  • Data: 3,084,959 ranked matches, all skill brackets, held out from training (never seen by any method's fitting step).
  • batru model: the exact production ONNX serving this site — not a lab special.
  • Naive winrate averaging: each hero's solo winrate from the training split; prediction = difference of team averages. We even fit it a logistic calibration layer on training data — a courtesy real winrate-table sites don't have. It still misses, because hero winrates ignore who they're with and against.
  • ECE: expected calibration error, 10 equal-width bins — identical convention to our internal model-promotion gate, unchanged since day one.
  • Confidence bands: per prediction, confidence = max(p, 1−p); we report the empirical hit rate inside each band.

Full context on why we lead with calibration instead of accuracy: methodology →

Model history — every evaluation, public

On a rolling cadence a freshly trained challenger is scored against the reigning production model on a new held-out set, and this ledger records the outcome — promotions and the rounds the champion held. The rows below are read straight from the append-only decision log our promotion gate writes (dota2-win-predictor-data/models/eval_history.jsonl); nothing is hand-picked.

DateDecisionChallenger ECE ↓Champion ECEChallenger BCE ↓AccHoldout
2026-11-08champion held0.00240.00190.668758.8%3,084,959
2026-11-07promoted0.00190.00160.668758.8%3,084,959
2026-10-05promoted0.00380.00560.669558.7%3,579,956
2026-10-04champion held0.01030.00630.684955.2%5,096,199
2026-09-07champion held0.00280.00470.691453.0%4,262,661
2026-09-06champion held0.01310.00460.684255.4%4,267,447
2026-09-05champion held0.00130.00430.683955.5%4,247,889
2026-08-08champion held0.01220.00790.683155.8%2,908,024
2026-08-04champion held0.00960.00790.682856.0%2,723,374
2026-08-01champion held0.01100.00750.682556.0%2,609,575
2026-07-28champion held0.01730.00700.681956.2%2,421,469
2026-07-28champion held0.02580.00710.684356.0%2,389,255
2026-07-25champion held0.01230.00640.682556.0%2,316,771
2026-07-18champion held0.00560.00580.682555.9%2,214,079
2026-06-28champion held0.00690.00580.683655.6%1,306,180
2026-06-22promoted0.00860.01530.685454.9%758,807
2026-06-18promoted0.0037—0.684755.2%561,855
2026-06-12promoted0.00250.01040.684155.3%507,361
2026-06-12champion held0.52870.01048.522347.1%501,210
2026-06-11champion held0.01660.01070.686554.9%450,515
2026-06-05champion held——0.682655.9%236,687
2026-06-05promoted——0.682455.9%233,639
2026-06-04promoted——0.680856.4%229,671
2026-06-04promoted——0.681856.0%211,822
2026-06-03champion held——0.681056.3%178,811
2026-06-03champion held——0.691552.8%157,039
2026-05-31promoted——0.678556.3%32,471
2026-05-30promoted——0.684155.4%13,416

A "champion held" week means the new checkpoint failed to beat production on calibration + loss and was discarded — publishing those weeks is the point: a ledger that only showed wins wouldn't be evidence. Accuracy hovers where a balanced game pins it (see the scoreboard note above); the promotion gate decides on BCE with an accuracy-regression guard, not on chasing headline accuracy.

Current production model promoted Nov 7, 2026 — this date only moves when a new checkpoint actually beats the champion on the same held-out set, never on a routine re-run.