Hmm… This doesn’t necessarily mean the forum is the wrong place, but in the successful cases I could verify, the process seems to have looked roughly like this:
I do not see an obvious indication that the CTGroundBench request itself is malformed.
The main ambiguity seems to be the intake process. Hugging Face’s current Evaluation Results documentation describes Benchmark registration as a work-in-progress feature, and its final registration step is simply:
“(Beta) Get in touch so we can add it to the allow-list.”
It does not currently specify a particular forum category, GitHub repository, email address, or support form.
However, in recent cases where I could follow the public record all the way to a Benchmark being added, the clearest operational conversations happened in the Community tab of the benchmark dataset repository itself.
The recurring pieces were approximately:
- A valid root-level
eval.yaml.
- An
evaluation_framework already present in Hugging Face’s framework registry, or a small PR to add it.
- A discussion in the benchmark dataset’s own Community tab.
- Evaluation-result PRs opened against the corresponding public model repositories.
- Enough metadata to distinguish the evaluation harness and run conditions.
- A Hugging Face team member confirming shortlist or official-list inclusion.
The order was not identical in every case, so I would treat this as an observed working pattern rather than a documented mandatory sequence.
A low-friction route for CTGroundBench
I would keep this forum thread, but also open a short discussion under the CTGroundBench Community tab and link back here.
That dataset-local discussion could summarize the current state and ask for three narrow confirmations:
- whether the dataset is already in the Benchmark shortlist or awaiting review;
- whether any changes are needed to the
eval.yaml or Inspect scorer configuration;
- whether model-repository
.eval_results/*.yaml PRs should be opened now or after allow-listing.
Most of the preparatory work already appears to be present:
| Component seen in successful examples |
CTGroundBench’s public state |
Root-level eval.yaml |
Present and reported as passing push-time validation |
| Registered evaluation framework |
inspect-ai is already in the official framework registry |
| Defined task IDs |
Three native leaderboard tasks are documented |
| Public task and metric documentation |
Present in the dataset card |
| Released model outputs |
Fourteen prediction files from seven models are described |
| Auditable result provenance |
Summary metrics and SHA-256 checksums are provided |
| Dataset-local allow-list discussion |
Not visible at the time I checked |
| Result PRs in public model repositories |
Not mentioned in this request |
Because inspect-ai is already registered, this does not appear to need the additional framework-registration step that some custom benchmarks required.
If useful, one or two result PRs for models with canonical public Hugging Face repositories could also serve as a small end-to-end sanity check. There is probably no need to migrate every result before confirming the expected workflow.
A model-side result file follows the mechanism documented under Adding Evaluation Results: the YAML lives in the model repository’s .eval_results/ directory and points back to the benchmark dataset and task ID.
Useful metadata would include:
- the exact dataset revision;
- the native task ID;
- evaluation date;
- score scale and direction;
- a link to the released predictions or traces;
- whether the run was zero-shot or few-shot;
- tools or no tools;
- the evaluation harness or scorer version.
The successful examples suggest that the notes field is best kept short and human-readable, while longer provenance can go in source.
Public success cases I found
ResearchClawBench
The most complete public example I found is the ResearchClawBench allow-list discussion.
The initial request included:
- a root-level
eval.yaml;
- evaluation results stored in the benchmark repository;
- a Dataset Viewer-compatible manifest;
- complete metadata;
- a PR registering its custom evaluation framework.
A Hugging Face team member then clarified that the result YAML files should be submitted directly to the corresponding model repositories, rather than stored only in the benchmark dataset repository.
They also requested that repeated results for the same model identify the evaluation harness in notes.
The benchmark authors then:
- removed the benchmark-side result YAML files;
- opened result PRs against six public model repositories;
- added concise descriptions of the harness, tool use, code execution, workspace, and completed-task count;
- simplified the
notes fields after review.
After those changes, the Hugging Face team member explicitly confirmed that the benchmark had been added to the list.
The authors later asked whether future coordination should remain in the same dataset Community discussion or move to email. The response was that the Community discussion worked well and should continue to be used.
A concrete model-side submission can be seen in the Kimi-K2.6 ResearchClawBench result PR.
CHI-Bench
The CHI-Bench dataset discussion shows a slightly different ordering.
A Hugging Face team member stated that CHI-Bench had already been added to the shortlist, and that the remaining step was to open result-YAML PRs against the evaluated model repositories.
They also recommended identifying the harness in notes because the same models had been evaluated using multiple harnesses.
The authors then opened PRs for several public models. One example is the DeepSeek-V4-Pro CHI-Bench result PR, which identifies:
- the benchmark;
- the evaluation framework;
- the agent harness;
- the evaluation protocol;
- the task scores;
- the paper as the source.
This case is useful because it shows that shortlist inclusion may happen before the model-result PRs are complete.
WildClawBench
The WildClawBench dataset discussion shows the custom-framework branch.
WildClawBench already had:
- an
eval.yaml;
- model-repository result PRs.
However, its custom evaluation_framework identifier was not yet registered. A Hugging Face team member asked for a small PR to add that identifier to huggingface.js.
After the framework PR was merged, the team confirmed that WildClawBench had been added to the official list.
Interestingly, they then asked for additional model results, so broad model coverage was not treated as a strict prerequisite that had to be completed before allow-listing.
Common elements, without assuming a fixed order
Across these cases, the repeated public elements were:
| Element |
ResearchClawBench |
CHI-Bench |
WildClawBench |
| Dataset-local Community discussion |
Yes |
Yes |
Yes |
Root eval.yaml |
Yes |
Yes |
Yes |
| Framework registered |
Custom framework added |
Existing framework |
Custom framework added |
| Model-repository result PRs |
Before final addition |
After shortlist |
Some before; more requested later |
| Harness/run metadata reviewed |
Yes |
Yes |
Partly visible |
| Explicit HF confirmation |
Added to list |
Added to shortlist |
Added to official list |
This is why I would not describe the process as one strict pipeline. The public examples instead suggest a set of components that are reviewed and connected in somewhat different orders.
Two CTGroundBench-specific branches worth clarifying
1. Task 2 mapping and scorer reproducibility
CTGroundBench Task 2 is somewhat more structured than a direct one-row-in, one-label-out benchmark:
- models predict on 3,020 unique normalized observations;
- predictions are mapped through
normalized_observation;
- scoring is performed against 3,517 instance-level references;
- the leaderboard metric includes interval intersection-over-union and region-count accuracy.
The dataset card already explains this mapping and preserves the released predictions and summary metrics.
The narrow question for the Hub side is not necessarily whether this design is valid, but which level of reproducibility is expected for allow-listing.
There appear to be two related but distinct Hub workflows:
- Aggregating an already-computed scalar result from a model repository’s
.eval_results/*.yaml.
- Running or verifying an evaluation through Inspect and Hugging Face Jobs, producing a
verifyToken.
The official documentation says that an inspect-ai benchmark defines field_spec, solvers, and scorers. It is less explicit about whether every community-submitted score must be reproducible end-to-end through that configuration before the benchmark can be allow-listed.
So a useful question for the dataset-local discussion would be:
For Task 2, is the documented mapping plus the released prediction and metric artifacts sufficient for community-provided scalar results, or should the complete 3,020-to-3,517 mapping and scoring procedure be executable directly through the Inspect configuration?
That asks for the expected contract without assuming that the existing implementation is wrong.
2. Gated dataset access
CTGroundBench is publicly visible but requires an access request in order to preserve the upstream ReXGroundingCT conditions.
Gating alone does not appear to automatically prevent Benchmark registration. Both Humanity’s Last Exam and GPQA currently have the Benchmark badge while also requiring users to accept access conditions.
Those examples do not prove that every gating arrangement is equivalent, but they suggest that the useful question is narrower:
Does the CTGroundBench access configuration require any additional permission or token setup for Hub-side validation, Inspect runs, or future verified evaluations?
That separates dataset licensing and access policy from the allow-list request itself.
Model-repository results and API-only models
The Hub’s native leaderboard mechanism is decentralized: evaluation results are stored in the evaluated model repository, while the benchmark dataset aggregates them.
This creates a practical distinction between:
- models with a canonical public Hugging Face repository;
- hosted or API-only models without a corresponding model repository where a result PR can live.
In the ResearchClawBench discussion, the Hugging Face team member explicitly connected model-repository PRs with the ability to display open models on the leaderboard.
For CTGroundBench, a reasonable split may therefore be:
- use
.eval_results PRs for evaluated models that have public Hugging Face repositories;
- continue preserving API-only model predictions and metrics in the benchmark dataset’s own released results;
- ask whether there is any supported native representation for API-only results before trying to force them into an unrelated repository.
The existing released result files remain useful independently of the Hub-native leaderboard because they preserve the exact predictions, summary metrics, and checksums used for the benchmark report.
Why I would not infer much from unanswered requests
There are other forum posts and GitHub issues asking for Benchmark allow-listing or clarification of the correct contact route.
For example, hub-docs issue #2311 asks almost exactly where an inspect-ai benchmark author should request allow-list access after following the documented setup. At the time I checked, the issue was still open, unassigned, and unlabeled.
That supports the observation that the public routing is not clearly documented.
It does not tell us why any particular request has not received a response. Publicly indistinguishable possibilities include:
- ordinary review latency;
- the relevant maintainer not having seen the request;
- an internal shortlist already existing;
- a missing technical prerequisite;
- a request being routed through another channel.
For that reason, the successful dataset-local discussions seem more informative than trying to reverse-engineer failure causes from unanswered posts.
If both the forum thread and dataset Community discussion remain unanswered, the Hub also now has a general user-menu feedback channel. I would use that as a routing check rather than duplicating the entire technical request:
Which public channel should be used for a Benchmark allow-list request? The forum thread and dataset discussion are linked here.
Suggested default path
In practical terms, I think the least disruptive route is:
- Leave this forum request in place.
- Open a concise allow-list discussion in the CTGroundBench dataset’s Community tab and cross-link the two threads.
- State that
eval.yaml has passed validation and that inspect-ai is already registered.
- Ask whether the benchmark is already shortlisted and whether any schema or scorer changes are needed.
- Ask whether model-repository result PRs should be opened immediately.
- If requested—or as a small sanity check—open one or two result PRs for evaluated models with clear public Hugging Face repositories.
- Keep the API-only results and full prediction artifacts in CTGroundBench unless HF identifies a supported native route for them.
That route preserves the work already done, avoids assuming that this forum post was misplaced, and follows the most clearly documented successful public examples I could find.