Metrics
Results · Full-precision values
I (Intelligence) and C (Calibration) use a 0–100 scale; ECE, TVD, and RPS use 0–1. Higher I/C and lower ECE/TVD/RPS are better. Accuracy covers every question in each row; other metrics use the subsets below.
Scoring scopes
| Benchmark | I/C | ECE | TVD | RPS |
|---|---|---|---|---|
| CMDB-1500 | Excludes 100 multi-select items: 1,400 overall, 1,100 text, 300 images | Choice/binary: 1,234 overall, 934 text, 300 images | 67 probability-reference items: 47 text, 20 images | 166 ratings, all text |
| JevBench Public 231 | 139 Choice, 74 Noul, 18 Score | Choice / Noul / Score: 67 / 74 / 18 | 7 reference-distribution items | 18 ratings |
| JevBench Hard 111 | 67 Choice, 38 Noul, 6 Score | Choice / Noul / Score: 67 / 38 / 6 | Same 7 items | 6 ratings |
| Decision Bench | Choice only | All questions in each row | One-hot gold labels | N/A |
Hard 111 is included in Public 231; both use the same hard-choice items for Choice ECE and TVD. JevBench's three ECE values correspond to Choice / Noul (binary) / Score (rating). CMDB images are native inputs; Decision Bench image-origin items are text-rendered.
CMDB multi-select accuracy uses exact-set match. Its step probabilities are conditional on previous selections, not independent inclusion probabilities.
Definitions
- ECE: 10 equal-width confidence bins. Choice uses top-label confidence; JevBench Noul uses
p(yes); Score uses the predicted rating's confidence and exact-label correctness. The main CMDB ECE pools choice/binary top-label predictions. - TVD: mean total variation distance from the reference distribution. Decision Bench uses one-hot gold labels, so TVD equals
1 − p(gold). - RPS: ranked probability score for ordinal ratings against reference labels. N/A means no rating questions.
I/C formulas
Local adaptations of JevBench v1.5 and its equal-types amendment, evaluated on the frozen Public 231 / Hard 111 subsets. These are self-evaluations, not official leaderboard scores.
I averages available type components with equal weights: chance-corrected Choice accuracy, Noul confidence-threshold accuracy (p(yes) ≥ 0.8 for yes and ≤ 0.2 for no), and chance-corrected expected-rating error. Public 231 renormalizes tier weights over available easy/standard/hard tiers; Hard 111 uses the hard tier only. CMDB has no tier weighting; its image subset is Choice-only.
Decision Bench treats all items as Choice, with no tier weights:
I = 100 × (accuracy − mean(1 / option_count)) / (1 − mean(1 / option_count))
C averages the available type components:
| Type | Component |
|---|---|
| Choice | Mean of max(0, 100 × (1 − 2 × ECE)) and 100 × (1 − TVD) |
| Noul | max(0, 100 × (1 − 2 × ECE)) |
| Score | Mean of max(0, 100 × (1 − 2 × ECE)) and 100 × (1 − RPS) |
CMDB uses all-choice ECE within each scope. Decision Bench uses Choice-only C with one-hot gold TVD. I/C therefore require matching benchmark and scoring scopes; the website's cross-model comparison uses a separate shared text-probability cohort.
Additional typed ECE, Brier, NLL, and multi-select F1 values are in evaluation_results.json.