GlassBallAI: a live, time-locked dataset of Gemini forecasts for studying LLM behaviour

Show and Tell
2

Hmm… I think this could be much easier to develop further if the link between the data and the claims were strengthened a little:

My short read

The live, time-locked collection is already the strongest part of this project. Recording forecasts, prompts, retrieved context, confidence, sentiment, rationales, model configuration, and timestamps before the outcome exists creates a useful longitudinal archive that retrospective prompting cannot fully reproduce. This is broadly aligned with the motivation behind future-only systems such as ForecastBench, although the prediction target and data structure here are quite different.

For the confidence/accuracy inversion specifically:

  • I found related work showing that verbalized LLM confidence can be strongly dependent on the model, task, prompt, output format, and measurement protocol.
  • There are also examples of models assigning high confidence to incorrect but plausible answers.
  • I did not find a close independent replication of the exact pattern described here: a search-grounded financial forecaster whose 0.8–0.9 verbal-confidence bin falls to roughly 28% directional accuracy.

So I would currently treat the 28% result as an interesting exploratory observation, rather than evidence of a general grounded-LLM failure mode. There are still several explanations that can be separated with relatively small additions to the evaluation artifacts.

The route I would take is:

  1. Define exactly what the confidence field measures.
  2. Make the 28% calculation reproducible from a small evaluation export.
  3. Freeze the analysis definition now and test it prospectively on newly collected forecasts.

That would allow other people to move from suggesting possible explanations to actually testing them.

The first fork: what does confidence mean?

From the public prompts, the confidence scale appears to be framed approximately as a progression from speculative/no-data to fact-based/high-certainty, with confidence based on evidence density.

That sounds closer to evidential certainty than to an explicit probability that the predicted price direction will be correct.

Those are both useful quantities, but they lead to different analyses:

Intended meaning Natural question Appropriate analyses
Probability that the predicted direction is correct “Of forecasts assigned 0.8, are roughly 80% directionally correct?” Reliability diagram, Brier score, log score, calibration error
Strength/density of the supporting evidence “Do forecasts described as evidence-rich tend to be more accurate?” Accuracy by evidence score, ranking/AUROC, risk–coverage, source-quality analysis
General confidence mixing several concepts “Which latent property is this score responding to?” Separate elicitation fields and ablations

If the present field is intentionally evidential certainty, I would preserve it. It may support a particularly interesting question:

Does an LLM’s expressed sense that a forecast is fact-based and well-supported predict whether its forward price direction is actually correct?

In that framing, a negative or flat relationship would still be meaningful. It would describe a mismatch between evidence-rich narrative formation and eventual correctness.

However, I would avoid interpreting the score as a calibrated probability unless the prompt explicitly defines it as something like:

What probability, from 0 to 1, do you assign to the predicted
direction being correct on the specified target date?

A useful future schema could therefore retain the current value while adding separate fields:

{
  "evidence_confidence": 0.85,
  "direction_correct_probability": 0.62,
  "price_interval_80": [190.0, 207.0]
}

This would preserve the behavioral signal already being collected while making standard probabilistic evaluation possible.

This distinction seems especially important because verbalized confidence is not necessarily a direct readout of one fixed internal uncertainty variable. On Verbalized Confidence Scores for LLMs finds that reliability varies substantially across models, datasets, and prompting methods. Asking Is Not Enough further argues that confidence comparisons are protocol-sensitive behavioral measurements; in one of its analyses, plausible wrong answers received confidence close to supplied correct answers.

The smallest addition that would unlock much more useful feedback

I think a small evaluation manifest would be more useful than adding many new dataset fields immediately.

For the reported confidence curve, the following would probably be enough:

  • Dataset commit SHA
  • Model subset or subsets included
  • Forecast date range
  • Unit of analysis:
    • checkpoint,
    • session,
    • or one end-to-end trajectory
  • Definition of the starting/reference price
  • Definition of directional correctness
  • Treatment of same-day checkpoints
  • Treatment of zero returns or unchanged prices
  • Confidence aggregation, if the unit is a session
  • Exact bin intervals:
    • [0.8, 0.9),
    • (0.8, 0.9],
    • or something else
  • Number of observations and correct predictions in each bin
  • Missing/error exclusion rules
  • Evaluation-code commit or a small evaluated CSV

A derived evaluation file could be sufficient even if the underlying licensed market data cannot be redistributed:

session_id
checkpoint_id
ticker
model_id
forecast_origin
target_date
horizon
pipeline_version
confidence
reference_price_timestamp
predicted_direction
actual_direction
correct
confidence_bin

The public dataset card explains that realized market data is intentionally not redistributed and must be rehydrated from another source. That is understandable, but it also means that two researchers can silently construct different outcomes depending on:

  • raw close versus adjusted close,
  • the reference-price timestamp,
  • market timezone,
  • holidays and non-trading dates,
  • stock splits,
  • target-date resolution,
  • and later data-provider corrections.

An evaluation manifest would turn those choices into an explicit outcome contract.

Hugging Face Datasets also supports loading a specific tag, branch, or commit hash through the revision parameter, so each published result can remain reproducible while the live dataset continues to grow.

A practical default route

If the goal is to turn this into a first focused behavioral result without redesigning the whole project, my default route would be:

1. Keep the present archive format

The nested session records are useful for preserving the original relationship between:

  • forecast origin,
  • shared context,
  • multiple target dates,
  • prompt,
  • search material,
  • and generated rationale.

I would not flatten away that structure in the canonical archive.

2. Publish an analysis view

Add generated tables or Parquet views for:

  • sessions,
  • checkpoints,
  • retrievals,
  • attempts/errors,
  • outcomes,
  • and pipeline versions.

This gives researchers a convenient analysis surface while keeping the original event record intact.

3. Rename or document the current confidence construct

For example:

  • evidence_confidence,
  • expressed_evidential_certainty,
  • or the existing confidence field with a precise definition in the card.

This is not merely terminology. It determines which scoring rules and comparisons are meaningful.

4. Add explicit directional probability prospectively

Keep the existing confidence field and add a separate probability of directional correctness in future runs.

5. Treat the current inversion as hypothesis generation

Freeze the following now:

  • target metric,
  • bin boundaries,
  • model subset,
  • date range,
  • exclusion rules,
  • clustering unit,
  • and outcome contract.

Then apply that unchanged definition to the next unseen collection period.

Because the project is live, it has an unusually natural confirmation set: forecasts that have not yet been generated or resolved. The 28% pattern becomes much more informative if it reappears under a preregistered analysis definition.

What I found in a small public-data sanity check

I tried a small checkpoint-level check using a pinned revision of the public dataset.

The procedure was:

  1. Load all four published configurations.
  2. Expand the 3,655 session records into individual checkpoints.
  3. Rehydrate daily closing prices using a public market-data convenience source.
  4. Compare several explicit reference-price and target-date contracts.
  5. Exclude non-positive horizons in the main slice.
  6. Analyze [0.8, 0.9) as the primary high-confidence interval.
  7. Repeat uncertainty estimates using session, ticker, calendar week, and target-date clusters.

The public data expanded to approximately:

  • 3,655 sessions
  • 29,647 checkpoints
  • 2,086 same-day checkpoints

Under a conservative proxy contract using the latest available close before forecast creation as the reference and the exact target-date close as the outcome:

  • evaluable positive-horizon checkpoints: approximately 27,532
  • overall directional accuracy: approximately 56.6%
  • [0.8, 0.9) checkpoints: 1,756
  • directional accuracy in that interval: approximately 55.5%
  • AUROC of confidence for distinguishing correct from incorrect forecasts: approximately 0.507

Cluster bootstrap estimates for the high-confidence interval remained broadly in the low-to-high 50% range, depending on whether the resampling unit was session, ticker, week, or target date.

So I could not reproduce the reported approximately 28% value from the public artifact under those explicit proxy definitions.

I do not think this is evidence that the reported result is incorrect.

The public dataset does not expose the exact numeric reference price that was supplied to the model in a directly reusable form, and realized outcomes are intentionally omitted. I also do not know whether the published curve uses:

  • individual checkpoints,
  • one confidence value per session,
  • the final checkpoint,
  • average or maximum confidence,
  • an end-to-end trajectory metric,
  • a particular model subset,
  • or different bin boundaries.

This makes the sanity check a different, explicitly documented measurement—not a replication of the private/dashboard evaluation.

The non-reproduction is therefore useful mainly because it identifies the smallest missing information:

  • the analysis unit,
  • exact reference-price contract,
  • exact target/outcome contract,
  • bin membership,
  • and the list of evaluated cases.

One additional observation is that confidence values are quite discrete rather than smoothly distributed. A large number of checkpoints sit exactly at 0.80, so these intervals can contain very different populations:

[0.8, 0.9)
(0.8, 0.9]
[0.8, 0.9]

In the proxy analysis, some very narrow high-confidence values had low accuracy, but their sample sizes were tiny. For example, a low result at a value represented by only a few dozen observations should not automatically be interpreted as the behavior of the entire 0.8–0.9 interval.

A per-value table before binning would make this immediately visible:

confidence | n | correct | accuracy
Checks that would distinguish the main explanations

There seem to be several plausible explanations, and fortunately they imply different checks.

A. Sparse-bin variation

Possible observation: The highest bin contains few forecasts or a few discrete confidence values.

Useful outputs:

  • bin count,
  • correct count,
  • Wilson or bootstrap interval,
  • exact confidence-value distribution,
  • minimum sample threshold for displayed bins.

If the result disappears under equal-mass bins or a slightly different interval, the shape may be mainly a binning artifact rather than a stable inverse relationship.

Binning-based ECE is known to depend on finite-sample and binning choices. Mitigating Bias in Calibration Error Estimation reports that equal-mass bins can have lower bias than equal-width bins in common settings.

B. Composition effect

Possible observation: The 0.8–0.9 interval is disproportionately composed of one model, ticker, horizon, week, or pipeline version.

Useful breakdowns:

  • model,
  • ticker,
  • horizon,
  • forecast-origin week,
  • target date,
  • realized volatility,
  • earnings-event window,
  • pipeline version,
  • search success/error state.

Useful robustness checks:

  • leave one ticker out,
  • leave one week out,
  • leave one model out,
  • leave one pipeline version out.

A global inversion can appear even if no comparable inversion exists inside the major subgroups.

C. Dependent observations

The checkpoints are not naturally IID rows.

Within one session, several target dates share:

  • the same creation time,
  • the same prompt,
  • much of the same retrieved evidence,
  • and the same market state.

Forecasts produced on adjacent days can also share target dates and market events.

I would therefore report at least one uncertainty estimate that resamples at a higher level:

  • session,
  • ticker,
  • forecast-origin week,
  • or target date.

The right cluster depends on the claim:

Claim Natural grouping
Repeated outputs from one generation session Session
Generalization across companies Ticker
Robustness across market periods Week or regime
Independence of realized outcomes Target date

D. Construct mismatch

Possible observation: Confidence follows evidence density or narrative coherence but not outcome correctness.

Useful checks:

  • confidence versus number of retrieved snippets,
  • confidence versus number of cited sources,
  • confidence versus source agreement,
  • confidence versus rationale length,
  • confidence versus correctness,
  • confidence versus absolute price error.

If confidence tracks retrieval richness but not correctness, that is not necessarily a failed measurement. It may reveal what the model operationally treats as “certainty.”

E. Ranking versus absolute calibration

Even if confidence is not calibrated as a probability, it may still rank predictions by likely correctness.

Useful metrics:

  • AUROC for correct versus incorrect forecasts,
  • accuracy of the top confidence decile,
  • risk–coverage curve,
  • selective accuracy at several thresholds.

If accuracy remains flat as the confidence threshold increases, the score has little value for failure prediction, even if its numerical calibration is not the intended research question.

If ranking improves but numerical calibration is poor, the field may still be useful for abstention or prioritization after recalibration.

F. Prompt-conditioned behavior

The dataset card already correctly notes that the outputs are prompt-conditioned and model-specific.

A small future control could compare:

  1. Current prompt and current confidence definition
  2. Current forecast prompt, but confidence elicited in a separate call
  3. Explicit probability of directional correctness
  4. Forecast without rationale generation
  5. Fixed retrieved context versus live search

This does not need to run over the whole dataset. A small matched set of tickers and dates would help determine whether the effect belongs mainly to:

  • the model,
  • the evidence,
  • the confidence instruction,
  • or the combined generation format.

G. Prospective confirmation

The cleanest confirmation is probably the cheapest:

  1. Use current data to define the hypothesis.
  2. Freeze the metric and evaluation code.
  3. Do not change the bin after looking at new outcomes.
  4. Run it on the next unseen model/time subset.

Possible outcomes are all informative:

  • The inversion repeats: stronger evidence for a stable pipeline-conditioned behavior.
  • It disappears: likely an exploratory slice or market-period effect.
  • It appears only in certain regimes: a conditional effect worth studying.
  • Evidence confidence stays uninformative while explicit direction probability works: the two confidence constructs were measuring different things.
Dataset and metadata additions that seem highest leverage

The existing schema already preserves considerably more context than many forecast datasets. I would prioritize fields that help distinguish system behavior from missingness or evaluation choices rather than adding metadata indiscriminately.

Highest priority

Confidence provenance

  • confidence construct name,
  • exact instruction text,
  • output schema version,
  • whether confidence was generated jointly with the answer or separately,
  • scale anchors,
  • whether the value applies to one checkpoint or the full trajectory.

Forecast provenance

  • exact model identifier,
  • API family and API version,
  • SDK name and version,
  • temperature and other sampling controls,
  • prompt hash,
  • schema hash,
  • pipeline version,
  • retry count.

Time and price contract

  • creation timestamp with timezone,
  • reference-price timestamp with timezone,
  • whether the reference value is live, open, previous close, or latest completed close,
  • exchange calendar,
  • target-date resolution rule.

Outcome provenance

  • provider identifier,
  • raw or adjusted price definition,
  • hydration timestamp,
  • corporate-action policy,
  • outcome-code version.

Attempt and failure log

Instead of storing only successful forecasts, preserve an attempt-level record:

{
  "attempt_id": "...",
  "search_attempted": true,
  "search_succeeded": true,
  "generation_succeeded": true,
  "schema_parse_succeeded": false,
  "retry_count": 1,
  "error_class": "structured_output_parse",
  "raw_response_preserved": true
}

This matters because missing forecasts may not be missing at random. They may cluster by:

  • model,
  • date,
  • prompt size,
  • search behavior,
  • or API version.

Grounding metadata

The current Gemini Google Search grounding documentation exposes search-call and citation information, including executed queries and mappings between answer spans and source URLs. Historical generateContent responses used a different groundingMetadata representation, so preserving the raw response alongside normalized fields would help future analysis survive SDK and API migrations.

Useful normalized fields include:

  • search attempted,
  • executed search queries,
  • retrieved source identifiers,
  • response-span/source mapping,
  • search error,
  • finish reason,
  • raw-response hash,
  • SDK/API version.

For hallucination or attribution research, it may help to distinguish:

  1. A search was executed.
  2. A retrieved source was relevant.
  3. A source supported a generated claim.
  4. The source causally influenced the forecast.

The first three can be partially studied from logs and annotations. The fourth generally requires an intervention such as removing, replacing, or perturbing the source context.

Canonical archive plus generated analysis views

A structure such as this could work without changing the original record format:

canonical_sessions/
analysis_views/
  sessions.parquet
  checkpoints.parquet
  retrievals.parquet
  attempts_errors.parquet
  outcomes.parquet
  pipeline_versions.parquet
evaluation_manifests/
  confidence_inversion_v1.json

The canonical archive remains the source of truth. The tables and manifests can be regenerated.

A wording distinction that may help future readers

The natural-language rationales are valuable, especially because they are frozen before outcomes are known.

I would describe them as:

  • time-stamped observable behavioral traces,
  • model-expressed narratives,
  • or generated rationales,

rather than assuming that they expose the complete causal process that produced the forecast.

This is not specific to GlassBallAI. Work such as Language Models Don’t Always Say What They Think shows that generated explanations can omit factors that affected an answer or rationalize an already selected answer.

That does not make the rationales unhelpful. It changes the research questions that can be supported directly:

  • Does the narrative change before the prediction changes?
  • Do high-confidence forecasts use more decisive language?
  • Are unsupported factual claims more common in incorrect forecasts?
  • Does source diversity change the explanation?
  • Does the rationale remain stable across repeated generations?
  • Does removing a cited source alter the prediction?

The dataset card already notes that rationales may be plausible rather than causal, which is a useful limitation to retain prominently.

Closest related work I found, with important differences

Future-only forecasting

ForecastBench is probably the closest high-level methodological relative.

Similarities:

  • predictions are collected before outcomes are known,
  • evaluation happens after resolution,
  • the benchmark can grow over time,
  • temporal leakage is reduced by using unresolved events.

Differences:

  • ForecastBench primarily uses clearly defined event probabilities,
  • GlassBallAI records price trajectories, rationales, sentiment, evidence, and search context,
  • the present GlassBallAI confidence field may not represent an event probability.

This makes ForecastBench useful as a reference for prospective evaluation and versioned forecast sets, but not a direct comparison for the 28% result.

Verbalized confidence

Just Ask for Calibration shows that verbalized confidence can sometimes be more calibrated than token-probability alternatives under particular elicitation methods and QA tasks.

On Verbalized Confidence Scores for LLMs finds that reliability depends strongly on the model, dataset, and prompting method.

Asking Is Not Enough emphasizes the sensitivity of conclusions to the full measurement protocol and reports that plausible wrong answers can receive confidence similar to correct answers.

Together, these papers suggest that “verbal confidence” is not one universally comparable measurement. The elicitation contract is part of the measured system.

Calibration estimation

Mitigating Bias in Calibration Error Estimation is useful for the finite-sample and binning side of the analysis.

For explicitly probabilistic forecasts, Strictly Proper Scoring Rules, Prediction, and Estimation provides the broader scoring-rule framework.

A partially similar HF Forum observation

There is also an HF Forum post reporting that a small LLM appeared more confident on incorrect answers than correct ones: “We found that small LLM is systematically more confident on wrong answers than right ones”.

I would treat it only as a nearby anecdote, not a replication:

  • it uses a different task,
  • confidence is based on entropy/margin rather than the same verbalized scale,
  • it is not a grounded financial forecasting setup.

A possible first focused result

If I were trying to extract one initial paper-sized or report-sized question from the existing archive, I might phrase it as:

Does Gemini’s expressed evidential certainty predict the correctness of its forward price-direction forecasts?

That question seems well aligned with the prompt and current data.

It allows several outcomes without making the project depend on one unusual bin:

  • positive association,
  • no association,
  • negative association,
  • association only at certain horizons,
  • association only in certain market regimes,
  • variation across model or pipeline versions.

A second, prospective study could then ask:

Is an explicitly elicited probability of directional correctness calibrated?

Separating those questions would make both results easier to interpret.

Bottom line

I think the archive and collection design are already useful. The next improvement may not require substantially more data or a large redesign.

The highest-leverage additions appear to be:

  1. Clarify whether the existing confidence score is evidence certainty or correctness probability.
  2. Publish a minimal manifest/export that reproduces the 28% bin.
  3. Report the exact bin count and correct count before interpreting ECE.
  4. Account for shared sessions, target dates, tickers, and market periods in uncertainty estimates.
  5. Preserve the current result as an exploratory hypothesis.
  6. Test the frozen definition on the next unseen collection period.
  7. Add explicit directional probability as a separate future field rather than replacing the existing confidence signal.

That route would preserve what is distinctive about GlassBallAI while making it much easier for other researchers to reproduce, compare, and build on the observed behavior—even if the exact 28% pattern turns out to be specific to one slice or evaluation contract.