For now, I took a quick look at a few things:
This seems useful precisely because it does not present the 274 speech systems as though they had all been measured under one protocol. The distinction between the directly measured LLM leaderboard and the “surveyed, not benchmarked” TTS/STT research catalogs is clear, and the latter includes many of the things that tend to matter only after someone tries to use a system: deployment mode, licensing splits, lifecycle, pricing units, diarization, language coverage, and operational catches.
I am not a speech specialist, so I stayed with things I could check fairly narrowly against the public data and first-party documentation. I found a few concrete entries that may be worth refreshing, plus two small maintenance changes that might prevent the same class of issue from recurring.
The clearest items I found
1. The ASR page currently says both 111 and 106
The ASR page currently says:
surveyed, not benchmarked — 111 systems in the heroListening surveyed · 106 in the pipeline/navigation- 106 in the README
- 106 in the published
asr.json
The JSON contains 106 system records, declares counts.total: 106, and its category totals also sum to 106. TTS appears internally consistent at 168.
So this looks less like a substantive catalog problem and more like one static copy of the count surviving after the underlying data changed.
A small build-time invariant might be enough to prevent this:
systems.length
== counts.total
== sum(counts.by_category)
== hero count
== navigation / footer count
Alternatively, deriving every visible count from the same JSON value would remove the duplicated state entirely.
2. Fireworks now appears to have first-party pages for the prices marked unconfirmed
The current Fireworks row says that only the Whisper V3 Turbo rate could be confirmed on a Fireworks-owned page, while the commonly cited $0.0015/min rate for Whisper V3 Large and the streaming prices could not.
The current Fireworks model library lists:
- Whisper V3 Large —
$0.0015 / Audio Minute
- Whisper V3 Turbo —
$0.0009 / Audio Minute
- Streaming ASR v1 —
$0.0032 / Audio Minute
- Streaming ASR v2 —
$0.0035 / Audio Minute
The older first-party audio transcription launch post also lists the $0.0015 and $0.0009 Whisper rates together, with some of the inference conditions around them.
I would not necessarily merge all four prices into one undifferentiated number: the two streaming systems appear to be separate products, and dedicated endpoints remain a different pricing path. But the narrower statement that the base Whisper price cannot be found on a Fireworks-owned page now seems refreshable.
3. The Yandex SpeechKit row can probably be filled in substantially now
The current row is understandably conservative: it records that Yandex’s documentation returned bot verification during the August 1 research pass, and therefore leaves pricing, languages, streaming, deployment options, and speaker labeling largely unverified.
The documentation is still somewhat awkward to retrieve automatically, but the current first-party pages expose enough information for a more complete row.
The SpeechKit STT overview describes an active service with:
- streaming recognition
- synchronous file recognition
- asynchronous file recognition
- API v3 functionality
The current supported languages and recognition models page explicitly lists 16 locale codes plus automatic language detection. It also documents three moving version tags:
generalgeneral:rcgeneral:deprecated
and says the previous general version is supported for two weeks after an update. That version-tag behavior may be particularly useful to record because it makes the hosted service a moving target even when the product name does not change.
There is also first-party documentation for speaker labeling. It is limited rather than general-purpose diarization:
- API v3
FULL_DATA mode- mono recordings
- no more than two speakers
So I would not simply replace no-diarization with an unrestricted diarization label. Something like “limited two-speaker labeling under specific API v3 conditions” would be more precise.
Finally, SpeechKit Hybrid system requirements describe a customer-infrastructure deployment using Linux, Docker, and supported NVIDIA GPUs, including capacity figures for streaming recognition, file recognition, and file recognition with speaker labeling.
That creates a classification choice rather than one obviously correct edit:
If SpeechKit Hybrid is treated as the same product family:
update deployment / self-hosting notes in the existing row
If hosted SpeechKit and Hybrid are materially different purchasing products:
keep the hosted row and add Hybrid as a separate commercial on-prem entry
Either route seems more accurate than Cloud only, but keeping hosted per-use pricing separate from individually licensed Hybrid deployment would avoid mixing unlike products.
4. Mistral looks more like a source-link completion than a content correction
The current Mistral Voxtral Transcribe 2 row already captures the important split reasonably well:
- the batch transcription model is hosted/proprietary
- the realtime model has open weights
- the price, deployment, and diarization properties differ by model
The row currently has no source_url, but there are straightforward first-party sources.
The Voxtral Mini Transcribe 2 model card gives:
- model ID
voxtral-mini-2602
- release date February 4, 2026
$0.003/min- transcription and timestamp support
The Mistral audio overview distinguishes:
- Voxtral Mini Transcribe 2 for batch transcription, with diarization, context biasing, and word-level timestamps
- Voxtral Realtime for streaming, with a separate model ID and open Apache-2.0 weights
So this seems mostly like an opportunity to attach first-party links to the scopes already described, rather than a reason to rewrite the row.
The two maintenance changes I would prioritize
Beyond those entry-level updates, two low-cost changes seem likely to return more value than a large schema redesign.
A. Keep one source of truth for generated counts
The 111/106 mismatch is exactly the kind of small error that becomes hard to notice in a generated static site because each individual artifact still looks reasonable.
A build check over the published artifacts could fail when:
JSON record count != declared JSON count
category total != declared total
HTML count != JSON count
README count != JSON count
The README check could perhaps remain advisory if it is intentionally hand-written, while JSON/HTML inconsistencies could fail the build.
B. Make the correction fallback reachable
The site and README link “Request a model” and “Report a correction” to GitHub’s new-issue page. At the moment, the repository’s Issues page says:
Issue creation is restricted in this repository
That may be completely intentional—for example, as a spam-control choice. GitHub only recently added an official collaborator-only issue creation setting, so I would not assume it is a configuration mistake.
However, the forum post already says corrections can also be collected in this thread. If the GitHub restriction is intentional, the smallest fix may simply be to make that existing fallback visible beside the correction button:
Report a correction:
- GitHub, if you have access
- otherwise use the Hugging Face Forum thread
That would preserve the repository setting without leaving ordinary readers at a dead end.
One optional metadata addition: only for volatile claims
I would not say that provenance is generally missing. In a small static pass over the public data, the numerical ratings I found were already accompanied by a metric, named source, source URL, and as-of date. That is a good distinction to preserve.
The harder-to-maintain information appears to be the operational prose around those ratings:
- current price
- service status
- supported languages
- API capabilities
- concurrency and request limits
- data-use policy
- hosted versus self-hosted availability
For those fields, even a lightweight checked_at and an optional claim-specific source may make later corrections much cheaper:
source_url: https://example.com/product
checked_at: 2026-08-04
claim_sources:
price:
url: https://example.com/pricing
checked_at: 2026-08-04
scope: serverless batch API
speaker_labeling:
url: https://example.com/docs/speaker-labeling
checked_at: 2026-08-04
scope: API v3, mono, maximum two speakers
This does not need to become a full provenance ontology. Applying it only to information expected to change would probably capture most of the benefit.
Hugging Face’s developing Evaluation Results format is one nearby example of keeping source, date, dataset/revision, notes, and contribution status distinct, although the catalog has a broader purpose than an evaluation-results file.
What I checked, and the limits of this pass
I ran a small static consistency pass against the public repository snapshot at commit 215e2ce, then rechecked the relevant current pages and first-party vendor documentation.
The snapshot-level observations were:
| Check |
ASR |
TTS |
| Actual JSON records |
106 |
168 |
| Declared JSON total |
106 |
168 |
| Category sum |
106 |
168 |
| Generated/navigation count |
106 |
168 |
| Hero count |
111 |
168 |
Records with an empty source_url |
9 |
9 |
For numerical ratings, I found 32 rated records, and all 32 had:
- a metric name
- a named source
- a source URL
- an as-of date
That is why I would not characterize the rating provenance as broadly incomplete.
I also found no exact duplicate names. The closest fuzzy matches were explicit generations such as Distil-Whisper v3/v3.5 and Vevo/Vevo2, so duplicate cleanup does not appear to be a current issue.
This was not:
- an independent benchmark of any TTS or STT model
- a legal review of any license
- a full primary-source audit of all 274 systems
- evidence that every empty
source_url entry lacks research behind it
In several rows, the prose itself names a source even when the machine-readable source_url is empty. I would therefore describe those as source-field completion candidates, not necessarily unsourced entries.
Automated reachability is also not enough by itself. For example, an HTTP request may return status 200 while the body is actually a CAPTCHA or login page. Yandex documentation and GitHub’s new-issue route both demonstrated that distinction, so I treated content verification separately from HTTP success.
A little more detail on the entry-level sources
Fireworks
The current public row says:
Only one rate is confirmable on a Fireworks page: whisper-v3-large-turbo at $0.0009/min … The commonly cited whisper-v3-large $0.0015/min and streaming $0.0032-$0.0035/min figures could NOT be confirmed on any Fireworks-owned page.
Current first-party evidence:
- Fireworks model library
- Whisper V3 Large —
$0.0015 / Audio Minute
- Whisper V3 Turbo —
$0.0009 / Audio Minute
- Streaming ASR v1 —
$0.0032 / Audio Minute
- Streaming ASR v2 —
$0.0035 / Audio Minute
- Audio transcription launch post
- includes the two Whisper prices and benchmark/inference context
Possible narrow edit:
Replace “could not be confirmed on a Fireworks-owned page”
with current first-party links and a check date.
Keep serverless Whisper, Streaming ASR, and dedicated endpoints
as separately scoped products/pricing paths.
Yandex
Current first-party sources:
Possible changes, depending on catalog scope:
| Current field |
Possible refresh |
status: unknown |
Active hosted documentation and service pages exist |
languages: not established |
16 locale codes plus auto are currently listed |
streaming: null |
Streaming recognition is documented |
no-diarization |
Replace with limited two-speaker labeling, or another more precise facet |
Cloud only |
Reconsider in light of SpeechKit Hybrid |
source_url: null |
Add current first-party documentation |
I would leave price unfilled unless the currently applicable region, currency, billing mode, and page can be read cleanly. The point is not to replace one uncertain number with another.
Mistral
Current first-party sources:
The important boundary is:
| Property |
Batch Transcribe 2 |
Realtime |
| Typical use |
File/batch transcription |
Live streaming |
| Price |
$0.003/min |
$0.006/min |
| Diarization |
Documented |
Not presented as the same batch feature |
| Weights |
Not published for the hosted batch model |
Published |
| Weight license |
N/A for batch weights |
Apache-2.0 |
The current row already captures most of this. Adding scoped first-party links may be sufficient.
Why dates and evaluation scope may matter later
This is background rather than evidence that the current rows are wrong.
Speed numbers need an operating condition
In Open ASR Leaderboard issue #98, the reporter notes that the same Whisper Turbo code and GPU produced roughly 30x to 800x RTFx depending on segment length. Whisper-style padding and batching can make a single speed number highly dependent on the input shape.
For a speed or latency claim, useful scope may include:
batch or streaming
segment / file length
batch size
hardware or service endpoint
client region
network time included or excluded
median / percentile / best-case
measurement date
Dataset revisions can change the apparent score
Open ASR Leaderboard issue #153 discusses corrected versions of VoxPopuli and Earnings22 after transcription errors were found in the original references. Reported WERs changed even though the evaluated models themselves had not changed.
For benchmark-derived claims, a dataset name alone may eventually be insufficient; dataset revision and normalization can matter.
Hosted APIs are dated service observations
Open ASR Leaderboard issue #170 raises a broader problem with continuously updated API services and repeatedly used private test sets. It explicitly does not accuse a particular provider; the point is that a closed API behind a stable product name may change over time.
That suggests a useful conceptual distinction:
Open checkpoint result:
observation of a fixed artifact under stated software/hardware conditions
Hosted API result:
observation of a service at a particular time, region, version label,
request mode, and pricing tier
The catalog already uses rating_as_of for numerical ratings. Extending the same “dated observation” idea selectively to volatile operational claims may be enough.
Overall, these look like normal maintenance pressure in a large, fast-moving catalog rather than a problem with the project’s basic approach.
My default order would be:
- refresh the Fireworks, Yandex, and Mistral source notes;
- fix the ASR 111/106 display mismatch;
- add one small count-consistency check;
- expose the forum thread as a correction fallback if GitHub Issues are intentionally restricted;
- optionally add
checked_at only where the underlying fact is likely to change.
That seems likely to improve the catalog’s maintainability without turning it into a much heavier system.