Cross-architectural runtime probability dynamics in transformer LLMs — two clusters not explained by parameter count

Research
1

I want to share a finding from a measurement framework I’ve been working
on, because the result is counterintuitive enough that I think it might
interest people thinking about architectural differences between
transformer LLMs.

The setup

I measured the runtime geometry of probability distributions across
eight open-source attention-based transformers ranging from 70M to 1.3B
parameters: Pythia-70M, DistilGPT-2, GPT-2, OPT-125M, Pythia-160M,
Qwen2.5-0.5B, TinyLlama-1.1B, and Phi-1.5.

For each (token, layer) point during inference, the framework computes
geometric properties of the probability distribution over the vocabulary:
entropy, concentration on the top candidates, competition between the
leading and runner-up tokens, dispersion above a 1% threshold. From these
metrics, a bicephalic operator separates two distinct geometric tensions
that probability distributions can carry, which I label G (concentration
pole) and D (competition pole). The ratio between mean G and mean D, what
I call the GD_ratio, becomes a per-model signature.

What I found

The eight models do not vary continuously on the GD_ratio. They partition
into two clusters with no overlap and roughly an order of magnitude of
gap between them:

GPT-2 GD_ratio 2.458

Phi-1.5 GD_ratio 1.764
DistilGPT-2 GD_ratio 1.577

Qwen-0.5B GD_ratio 0.079
OPT-125M GD_ratio 0.074
Pythia-70M GD_ratio 0.059
Pythia-160M GD_ratio 0.039
TinyLlama-1.1B GD_ratio 0.021

The cluster split appears on three independent components of the operator:
the GD_ratio itself, the mean G alone, and the mean D alone. The
separation is not an artifact of one metric.

The interesting part is what does not explain the clustering. Parameter
count does not. GPT-2 has 124M parameters and is in the upper cluster.
OPT-125M has 125M parameters and is in the lower cluster. Phi-1.5 has
1.3B parameters and sits with GPT-2. TinyLlama-1.1B has roughly the same
size as Phi-1.5 and sits with OPT.

What might explain it (hypothesis only)

The most parsimonious pattern I can see is that the upper cluster shares
characteristics of training corpus curation. Phi-1.5 was trained on
heavily curated synthetic data. GPT-2 and DistilGPT-2 share the original
GPT-2 WebText distribution and tokenizer, which had its own filtering
protocol. The lower cluster spans more heterogeneous training corpora,
including older (OPT, Pythia) and newer (Qwen, TinyLlama) architectures
trained on relatively unfiltered web text.

I want to be careful here: this is a hypothesis, not a finding. I do not
have an experimental setup that isolates training corpus from architecture
choices. The hypothesis is consistent with the data but cannot be
established by it.

Why this might matter

If the two-cluster structure generalizes, any tool or analysis that
implicitly assumes a single dynamic profile across transformer models
will produce inconsistent results depending on which cluster the target
model falls into. This includes calibration techniques, uncertainty
estimation methods, and probably some interpretability approaches that
were tuned on one architectural family and may not transfer cleanly to
the other.

Other observations in the same study

A few things worth noting briefly:

  • The framework also defines a five-state taxonomy of dynamic regimes
    (stable, hidden turbulence, surface branching, committed, full
    bifurcation). The full bifurcation state turns out to be consistently
    transient across architectures: on three primary models tested in depth,
    its self-transition probability is 0.023 (GPT-2) or exactly 0.000
    (OPT-125M, Qwen-0.5B). Models pass through this regime, they do not
    settle into it.

  • Three models tested under controlled hidden-state perturbation respond
    in qualitatively different ways. GPT-2 absorbs the perturbation with
    state percentages shifting by less than 1.5 points. OPT-125M converts
    the perturbation into surface dispersion (branching state rises +12.5
    points). Qwen-0.5B destabilizes its dominant state (stable state drops
    -18.8 points). Three architectural perturbation signatures, same input
    noise.

  • One model (Phi-1.5) produces an anomalous taxonomy distribution under
    the standard threshold rule. I report it openly as needing dedicated
    investigation rather than smoothing it over.

What I’m not claiming

The panel is eight models, all under 1.3B parameters. The two-cluster
structure could collapse, stretch, or restructure when extended to 7B+
models. I have not validated on non-transformer architectures within
this study. The work is single-author and has not been independently
replicated. The training-corpus hypothesis is offered, not established.

I included explicit “limited findings” and “rejected claims” sections in
the paper, listing five things in each category that initial intuitions
suggested but that the data either partially support or actively reject.
I treat this as central to the framework’s credibility.

Why I’m posting

I would be interested in hearing whether anyone working with larger or
more architecturally diverse models has observed similar partitioning
phenomena in their own measurements, whether on attention, hidden states,
gradients, or any other intermediate quantity. The two-cluster structure
felt unexpected enough that I want to understand whether it is a
transformer-wide phenomenon, an artifact of the parameter range I tested,
or something specific to the particular operator I defined.

I would also be interested in alternative interpretations of the cluster
split beyond the training-corpus hypothesis. Possible candidates I have
considered but cannot test from this panel alone: pre-norm vs post-norm
architecture, tokenizer differences, attention head configurations,
intermediate layer dimensionality, positional encoding choices.

Where the details are

Full methodology, all tables, the explicit limitations section, and the
list of rejected claims are in the preprint on Zenodo:

Happy to discuss the operator definition, the threshold methodology, the
cluster finding, or any concern about the panel size and statistical
robustness.

3

This is exceptional feedback. Thank you for taking the time to dissect the operator and the methodology with such rigor. You’ve pinpointed the exact confounds that keep me up at night, particularly regarding calibration and activation norms.

I want to address your points directly, as they align perfectly with my own internal audits (some of which are in the “Rejected Claims” section of the preprint, but deserve more prominence):

1. The Temperature/Calibration Confound (Priority #1)
You are absolutely right. GD_ratio could simply be a proxy for per-model “confidence temperature” driven by training objectives (e.g., label smoothing, final-LN gain).

  • Action: I will run the scalar temperature fitting test you suggested. If the clusters collapse after entropy-matching, then GD_ratio is indeed a calibration metric, not a dynamic one. If they hold, it proves there is a structural geometric difference beyond simple sharpness. This is the cheapest and most decisive test. I’ll update the repo with these results.

2. Vocabulary Invariance
Good catch on the vocab size disparity (32k vs 152k).

  • Current Mitigation: My operator uses relative thresholds (top-k competition, dispersion above 1% of the mass), which helps, but raw entropy is definitely biased by VV.

  • Action: I will recompute using entropy normalized by log⁡(V)log(V) and also try the “top-p mass geometry” approach you mentioned. This should isolate the shape of the distribution from its granularity.

3. Perturbation Magnitude & Nulls
This is a critical methodological flaw in many interpretability studies.

  • Action: I currently inject fixed Gaussian noise. I will switch to RMS-normalized noise (scaled to each layer’s activation RMS) to ensure equitable perturbation strength. I will also add the zero-perturbation null to baseline the measurement pipeline’s stability. This will strengthen the claim that the “three signatures” are architectural, not artifacts of scale.

4. Error Bars & Bootstrapping
Agreed. Visual separation on n=8n=8 is suggestive, not definitive.

  • Action: I will bootstrap the GD_ratio over prompts and seeds to generate Confidence Intervals (95% CI) for each model. This will quantify the “no overlap” claim statistically.

5. The Large Model Test (Gemma & Softcapping)
This is a brilliant concrete test. Gemma’s logit softcapping is a known architectural constraint that directly bounds concentration metrics.

  • Hypothesis: If Gemma clusters with the “Low GD” group solely due to softcapping, it confirms that architecture can override training corpus signals.

  • Action: I am currently limited by compute for full 12B runs, but I can run inference on smaller softcapped variants or use existing hidden-state logs if available. Alternatively, I can simulate softcapping on my current panel to see if it forces them into the lower cluster.

Next Steps:
I’m prioritizing the Temperature Normalization and RMS-scaled Perturbation audits this week. These are high-leverage controls that will either validate the core hypothesis or refine it into a more precise statement about calibration vs. dynamics.

I’ll post an update here once the bootstrapped CIs and temperature-calibrated ratios are ready. Thanks again for pushing the rigor bar higher this is exactly how robust science gets done.

Best,
Jean-Denis

4

Thanks for this — and you were right.

Ran the temperature normalization test you suggested as priority 1. The
result is a partial falsification of the strongest interpretation of the
finding.

Setup: fitted a per-model scalar temperature T to match a common target
entropy across the panel (target = 3.01), recomputed geometry on the
calibrated logits, recomputed clustering.

What happened: two models migrate between clusters after calibration.
GPT-2 and Phi-1.5 both move. The raw clustering structure is therefore
substantially driven by effective logit temperature, not purely by
runtime dynamics.

What this falsifies: “Raw GD_ratio directly measures model dynamics
independently of calibration.” This interpretation is rejected by the
test you proposed.

What may still hold: residual structure after calibration. Two clusters
still appear post-calibration, but their composition changes — so the
question becomes “is there a calibration-independent component to the
clustering, and if so what does it measure?” That requires the additional
controls you listed: vocab normalization (log V), top-p truncation,
perturbation magnitude-matched to per-layer activation RMS, bootstrap CIs
on the gap. None of those are done yet.

The published version of the V20 preprint (deposited yesterday) overstates
the dynamical interpretation of the raw GD_ratio. I am preparing a
revision that incorporates the temperature audit explicitly. The
falsification is going into the rejected-claims section. The remaining
structure question is moving into limited-findings with the controls you
specified as the protocol for elevation.

The attention-based findings from companion work (different observation
level than logits) are not affected by this confound and remain the
direction I will continue to develop.

Genuinely useful review. The “temperature first because it kills the
result or makes it much stronger” framing was the right priority.

7

Thanks for checking back and yes, I definitely kept going down the rabbit hole.

Your previous comments ended up changing the direction of the project in a meaningful way.

Following your suggestion, I treated logit calibration as a potential confound rather than assuming GD_ratio was measuring intrinsic dynamics. I then ran a series of follow-up audits:

  • Temperature normalization audit (V22.5): after matching entropy across models, a substantial part of the original structure changed. Temperature is indeed a major driver.

  • Vocabulary-invariant audit (V22.6): normalizing for vocabulary size and tokenizer granularity only partially altered the structure. The result was PARTIAL, suggesting vocabulary explains some, but not all, of the signal.

  • Bootstrap confidence audit (V22.7): only part of the pairwise differences remained statistically robust, so I downgraded the strength of the original claim rather than strengthening it.

  • Residual structure audit (V22.8b): with the pipeline corrected, the remaining signal is still strongly associated with effective temperature, so I’ve revised the interpretation again.

At this point I no longer describe GD_ratio as a direct measure of “internal dynamics.” A more accurate interpretation is that it captures aspects of output distribution geometry, with temperature acting as an important confounding factor.

The interesting part now is the residual after controlling for those obvious mechanisms. Rather than trying to defend the original interpretation, I’m using the negative results to refine the framework.

Your suggestion about per-token / entropy-bin calibration is still on my list. I think that’s probably the next meaningful control because it asks whether any residual survives after local calibration instead of only matching the global mean.

I’m also moving beyond logits toward hidden-state and attention analyses, but with the same philosophy you recommended: identify the boring explanations first (calibration, tokenizer effects, attention sinks, architectural constraints) before making stronger claims.

So yes the rabbit hole got deeper, but also narrower. The framework is becoming less ambitious in its claims and, I hope, more trustworthy because of that. Thanks again for pushing on the weak points rather than just the positive results.