Hi Hugging Face team,
We would like to submit UncertaintyGym for official benchmark indexing and Evaluation Hub integration.
Overview:
UncertaintyGym evaluates an LLM’s meta-cognitive calibration—measuring whether models explicitly express uncertainty (“I don’t know”), request missing context, or reject false premises instead of hallucinating.
Taxonomy:
- Category A (Solvable): Standard factual queries (control set).
- Category B (Under-specified): Queries lacking necessary context that require disambiguation.
- Category C (False Premise): Queries with impossible premises that require rejection.
- Category D (Inherently Unknowable): Unsolved conjectures, lost history, and future events requiring explicit declaration of unknowability.
Technical Specifications:
- Repository: Muse-Ltd/UncertaintyGym · Datasets at Hugging Face
- Configuration: eval.yaml and lighteval_task.py included for direct LightEval / LM-Evaluation-Harness integration.
- Primary Metric: Meta Cognitive Calibration Score (MCS) and Unanswerable Hallucination Rate.
- License: Apache-2.0
Initial Baseline:
- LiquidAI / LFM2.5-2.6B: 45.0% MCS (Cat A: 100%, Cat B: 20%, Cat C: 0%, Cat D: 60%, Hallucination Rate: 73.3%)
Please let us know if any additional metadata is required for the benchmark badge.
Thank you.
Our Team has fixed some bugs with the Dataset benchmark
Hey! Great work on the benchmark, it looks super interesting. Unfortunately we try to restict the amount of benchmarks on the hub right now to focus on benchmarks already used in model releases and reports before opening the feature to all builders.
We will be sure to let you know when we open the feature to every benchmarks!
Cheers
This benchmark is exactly what’s needed.
I approached epistemic calibration structurally, not via training:
- Meta.md = single verified source of truth. Model loads only that.
- RTTC = live events arrive in separate channel with binary provenance: did it come through the channel? Yes/No.
- If No, model is forced to say “I don’t know, no data in channel” instead of hallucinating.
In my offline Android tests, this reduced hallucinations because uncertainty became verifiable, not probabilistic.
I would love to run UncertaintyGym against this provenance-based method. Does your benchmark test for source provenance or just confidence scores?
Implementation: masterscorpio1974-boop (MASTER S) · GitHub
Thanks! UncertaintyGym evaluates behavioral calibration specifically whether the model explicitly declares uncertainty (“I don’t know” / requests context / rejects false premises) when information is missing or contradictory, rather than relying solely on probabilistic confidence.
It would be awesome to see how your provenance-based architecture scores on it! Let me know if you run the benchmark.