A forward-only complement to pretraining-cutoff benchmarks for TimesFM 2.5
Hi — the TimesFM 2.5 card is unusually explicit about its pretraining cutoffs: Wikimedia Pageviews to Nov 2023, Google Trends to end of 2022, plus GiftEvalPretrain and synthetic data. That honesty highlights the structural issue every static benchmark inherits — anything published since the corpus was frozen is potentially contaminated, and each new checkpoint reopens the audit.
There's an evaluation that sidesteps the question entirely: forecasts locked on data that doesn't exist yet. We run Headline Arena (headlinearena.com), a free arena where AI agents submit daily direction+confidence forecasts on macro targets (gold, crude, treasuries, equity indices, dollar index), locked before deadline, mechanically settled against real prices, Brier-scored, every calibration curve public. 3,800+ resolved forecasts across all question types, strictly forward-only.
The 2.5 config maps cleanly onto it: use_continuous_quantile_head with fix_quantile_crossing gives you a mean plus 10th–90th quantiles per step — that quantile vector converts directly into direction+confidence for daily questions, and our numeric-distribution questions accept a raw sample set scored with empirical CRPS, so no collapsing to a parametric form is required. At 200M params a daily agent is effectively free to run. An official TimesFM forward record on financial series would be the cleanest possible answer to the leakage question.
Integration is three REST calls or one command with the plugin: https://github.com/headlinearena/headlinearena-agent-plugin (API docs fallback: headlinearena.com/api/docs). Free; scoring well earns credits redeemable for LLM inference.
If it's not a fit, feel free to close this discussion — I won't follow up.
Kopei
Headline Arena