Method

How B.O.B. is scored

B.O.B. doesn't run its own tests. It collects published results from the benchmark makers and turns them into one fair-as-possible number. Here is exactly how.

Last rebuilt 15 Oct 2026

  1. 1. Collect. Results come from the Epoch AI Benchmarking Hub (open data, CC BY 4.0), the Artificial Analysis Intelligence Index, and 11 real-use sources (listed below), from blind human votes to creators' own benchmarks. Prices come from OpenRouter. Everything refreshes daily.
  2. 2. Match names. Sources name models differently (“Claude Opus 5.5 (max with fallback)”, “claude-opus-5-5_max”, “anthropic/claude-opus-5.5”). We strip effort settings, providers and dated snapshot tags so they become one model. Each model gets its best reported setting on each benchmark (for example its max-effort run), and the model page shows which setting was used.
  3. 3. Only current models. The field is models released since 1 Jan 2026. A benchmark only counts if at least 5 of those models have a result on it.
  4. 4. Put every benchmark on the same scale. Benchmarks use different units (percent, Elo, index points), so each result becomes a percentile: what share of the other tracked models did this one beat? The best model on a benchmark gets 100 and the worst gets 0. Ties share the average.
  5. 5. Average by category, then overall. Benchmarks are grouped into six categories: Real use (hard to game), Reasoning & knowledge, Math, Coding, Agents & real work, Independent indexes. A model's category score is the average of its percentiles in that category. B.O.B. is the weighted average of its category scores, so a category with many benchmarks (like coding) can't drown out the others. Real use counts double; every other category counts once.
  6. 6. No ranking on thin data. A model is ranked only after it has results on at least 5 benchmarks across at least 3 categories. Newer models wait in “Waiting for more results” until then.

Real use: the check against benchmark-gaming

A model can be trained to ace the well-known tests and still disappoint in real work. People call this “benchmaxxing”. Tests that are hard to train for go in their own category and count double: blind votes from real people, brand-new tasks that appeared after a model was trained, private question sets that are never published, and benchmarks that independent creators run themselves.

Reality check. For every model with at least two real-use results, we compare its average on the lab tests (reasoning, math, coding, agents) with its real-use score. If the lab score is 15 or more points higher, it gets a Benchmaxxed? flag. If real use is clearly higher, it gets Holds up. A flag is a reason to test the model yourself, not proof.

Field reports. Some of the best evidence is a creator building real apps with a model on video. Those tests usually cover a handful of models, not a full leaderboard, so they don't change the score. We show them on the model's page, linked and dated.

Known limits

  • A percentile shows rank, not how big the gap is. A tiny lead and a huge lead can both score 100.
  • Models tested on fewer benchmarks have noisier scores. The benchmark count is shown next to every model.
  • Some results are reported by the labs themselves and collected by Epoch AI. Follow each link to check.
  • The Indexes category (AA, Epoch ECI) partly overlaps with the other benchmarks. It counts as one category, not two.
  • Real-use tests have biases too. Human votes can reward a friendly style, and some creator tests cover only a few models. That is why there are several of them.
  • Different sources test at different times. A brand-new model may be missing from the slower real-use tests for a few weeks.

All 46 benchmarks in B.O.B.

Real use (hard to game)

Reasoning & knowledge

Math

Coding

Agents & real work

Independent indexes

See a mistake? Tell us in the comments on any VTalk video. Back to the leaderboard.