# Metrics

[Results](README.md#results) · [Full-precision values](evaluation_results.json)

I (Intelligence) and C (Calibration) use a 0–100 scale; ECE, TVD, and RPS use 0–1. Higher I/C and lower ECE/TVD/RPS are better. Accuracy covers every question in each row; other metrics use the subsets below.

## Scoring scopes

| Benchmark | I/C | ECE | TVD | RPS |
|---|---|---|---|---|
| CMDB-1500 | Excludes 100 multi-select items: 1,400 overall, 1,100 text, 300 images | Choice/binary: 1,234 overall, 934 text, 300 images | 67 probability-reference items: 47 text, 20 images | 166 ratings, all text |
| JevBench Public 231 | 139 Choice, 74 Noul, 18 Score | Choice / Noul / Score: 67 / 74 / 18 | 7 reference-distribution items | 18 ratings |
| JevBench Hard 111 | 67 Choice, 38 Noul, 6 Score | Choice / Noul / Score: 67 / 38 / 6 | Same 7 items | 6 ratings |
| Decision Bench | Choice only | All questions in each row | One-hot gold labels | N/A |

Hard 111 is included in Public 231; both use the same hard-choice items for Choice ECE and TVD. JevBench's three ECE values correspond to **Choice / Noul (binary) / Score (rating)**. CMDB images are native inputs; Decision Bench image-origin items are text-rendered.

CMDB multi-select accuracy uses exact-set match. Its step probabilities are conditional on previous selections, not independent inclusion probabilities.

## Definitions

- **ECE:** 10 equal-width confidence bins. Choice uses top-label confidence; JevBench Noul uses `p(yes)`; Score uses the predicted rating's confidence and exact-label correctness. The main CMDB ECE pools choice/binary top-label predictions.
- **TVD:** mean total variation distance from the reference distribution. Decision Bench uses one-hot gold labels, so TVD equals `1 − p(gold)`.
- **RPS:** ranked probability score for ordinal ratings against reference labels. N/A means no rating questions.

## I/C formulas

Local adaptations of [JevBench v1.5](https://github.com/fstandhartinger/jevbench/blob/bb05a335bc809e61b20c0f745d25499a82b326fc/docs/METHOD-v1.5.md) and its [equal-types amendment](https://github.com/fstandhartinger/jevbench/blob/bb05a335bc809e61b20c0f745d25499a82b326fc/docs/METHOD-v1.5-ADDENDUM-HEADLINE-A-EQUAL-TYPES.md), evaluated on the frozen Public 231 / Hard 111 subsets. These are self-evaluations, not official leaderboard scores.

**I** averages available type components with equal weights: chance-corrected Choice accuracy, Noul confidence-threshold accuracy (`p(yes) ≥ 0.8` for yes and `≤ 0.2` for no), and chance-corrected expected-rating error. Public 231 renormalizes tier weights over available easy/standard/hard tiers; Hard 111 uses the hard tier only. CMDB has no tier weighting; its image subset is Choice-only.

Decision Bench treats all items as Choice, with no tier weights:

`I = 100 × (accuracy − mean(1 / option_count)) / (1 − mean(1 / option_count))`

**C** averages the available type components:

| Type | Component |
|---|---|
| Choice | Mean of `max(0, 100 × (1 − 2 × ECE))` and `100 × (1 − TVD)` |
| Noul | `max(0, 100 × (1 − 2 × ECE))` |
| Score | Mean of `max(0, 100 × (1 − 2 × ECE))` and `100 × (1 − RPS)` |

CMDB uses all-choice ECE within each scope. Decision Bench uses Choice-only C with one-hot gold TVD. I/C therefore require matching benchmark and scoring scopes; the website's cross-model comparison uses a separate shared text-probability cohort.

Additional typed ECE, Brier, NLL, and multi-select F1 values are in [evaluation_results.json](evaluation_results.json).