Hmm… for now, I tried some experiments in Colab, and I couldn’t fully reproduce the issue using the Transformers path:
I don’t think the diffusion-decoding hypothesis was unreasonable. DiffusionGemma really does generate differently from a normal left-to-right autoregressive model: it iteratively denoises a block/canvas of tokens, can revise positions during denoising, and uses confidence/entropy to decide which positions to keep versus re-noise. That gives it some failure modes that an AR model does not have in exactly the same form. The Transformers documentation and the DiffusionGemma model card describe that generation structure.
However, in the checks I could do, the broad explanation “parallel diffusion sometimes merges adjacent words” did not reproduce the specific upcomingbunkering failure very well.
Those two checks split a surprisingly large part of the search space without changing the QG design.
Why I ended up prioritizing these branches
1. What the Transformers-side result does — and does not — tell us
I would describe my result as:
The Transformers-side checks were clean within the cases I tested.
I would not go as far as saying “Transformers is unaffected” or “this proves vLLM is the problem.”
I did not reproduce your complete 26B FP8 serving environment with the exact QG request end-to-end. What I could check was narrower:
- the specific word boundary in a direct canvas-style test;
- the core, weight-free sampler state transitions;
- entropy-bound selection;
- confidence/stability behavior around the relevant threshold.
Those did not expose an obvious Transformers-side mechanism that would specifically prefer upcomingbunkering.
There is also a useful caveat in the current Transformers DiffusionGemma documentation: DiffusionGemma exists in both Transformers and Diffusers, but Diffusers is now described as its primary home for new scheduling/sampling features, while the Transformers implementation mainly receives bug fixes.
So I am using Transformers here as a useful comparison path, not claiming it is an absolute oracle for every possible DiffusionGemma serving configuration.
2. Diffusion itself can create unusual local dynamics, but that is not yet the diagnosis
DiffusionGemma does not simply emit:
token 1 -> token 2 -> token 3 -> ...
once.
The decoder works on a canvas/block, repeatedly predicts positions, accepts sufficiently low-entropy positions, and re-noises others for another refinement pass. The current Transformers implementation makes that explicit in its DiffusionGemma generation code.
That means adjacent positions can influence later refinement in ways that are different from ordinary AR decoding. So something like a word-boundary instability was worth investigating.
But there is an important distinction:
mechanism plausibility is not target reproduction.
I could make generic surrogate systems exhibit fusion-like behavior under some conditions. But when I moved to the actual upcoming bunkering boundary, the direct test preferred the properly spaced form in every run I tried.
So I would currently rank the explanations roughly like this:
- QG/RAG pipeline, prompt, sampling, or backend difference
- model/runtime interaction specific to the actual serving path
- intrinsic DiffusionGemma word-boundary behavior
rather than starting at #3.
3. One vLLM top-k/top-p path looks especially interesting
This is the most concrete implementation-level thing I found, although I want to stress that it is still a source-level + synthetic candidate, not a diagnosis of your request.
vLLM added per-request top_k / top_p support for DiffusionGemma in PR #45429, merged on July 26, 2026.
The PR deliberately filters excluded logits to -inf before the DiffusionGemma denoising step.
The current vLLM DiffusionGemma sampler then computes entropy approximately like this:
log_probs = scaled.log_softmax(dim=-1)
probs = log_probs.exp()
token_entropy = -(probs * log_probs).sum(dim=-1)
You can see the current path in vllm/model_executor/models/diffusion_gemma.py.
That arithmetic has a potentially awkward interaction with filtered logits:
filtered logit = -inf
log probability = -inf
probability = 0
0 * -inf -> NaN
I reproduced that behavior with synthetic logits following the current vLLM computation.
It was not limited to an extreme top_p. With a large DiffusionGemma-like vocabulary, even a top_p extremely close to 1 can still remove a small tail, which is enough to introduce -inf entries.
The important part is that the entropy is not merely diagnostic output in the current DiffusionGemma sampler. It feeds directly into:
- mean-entropy confidence;
- entropy-bound token acceptance;
- which positions are retained;
- which positions are re-noised for the next denoising step.
In my synthetic state-transition checks, that produced a very large behavioral difference.
For example, using high-confidence synthetic rows and repeating across 100 random seeds:
current entropy path:
average top-token positions preserved: about 0.5 / 32
NaN-safe entropy path:
top-token positions preserved: 32 / 32
And in a small multi-step simulation, the NaN path tended to keep re-noising instead of satisfying the normal early-convergence condition, while the safe path converged quickly.
I also tried the relevant operation under compiled PyTorch paths rather than only eager execution; the NaN did not disappear there.
The Transformers implementation provides an interesting control: its entropy-bound sampler uses:
dist = torch.distributions.Categorical(logits=logits)
token_entropy = dist.entropy()
rather than the manual probs * log_probs expression. See the Transformers generation source.
I tested a NaN-safe equivalent against that path with randomized filtered synthetic states:
2,000 synthetic requests
32,000 token rows
acceptance masks matched in all tested requests
I also ran 1,000 filter-off controls and did not see an observable state/output difference between the existing expression and the safe version when no filtered -inf values were present.
So, if your QG path is activating top_p or top_k, this looks like a useful branch because it could be a local sampler-integration issue, rather than something requiring a redesign of the QG application.
But the condition matters a lot.
The current RedHatAI FP8 model’s generation_config.json contains the DiffusionGemma settings such as:
{
"confidence_threshold": 0.005,
"max_denoising_steps": 48,
"sampler_config": {
"_cls_name": "EntropyBoundSamplerConfig",
"entropy_bound": 0.1
},
"stability_threshold": 1,
"t_max": 0.8,
"t_min": 0.4
}
but it does not currently specify top_k or top_p.
So I would not assume this branch is active merely because you are using that checkpoint.
The useful split is:
QG effective top_p < 1 or top_k > 0,
while RAG does not
-> this branch becomes much more interesting
both effectively use top_p = 1 and top_k = 0
-> this branch drops sharply in priority
If the first case applies, a very cheap diagnostic A/B would be:
same known-bad QG request
same prompt
same model
same everything else
only:
top_p = 1
top_k = 0
If the symptom changes, that gives a strong lead.
If nothing changes, this particular branch can be deprioritized quickly.
One subtle point: PR #45429 reports a no-NaN test, and it also changed truncated-canvas padding to avoid an -inf * 0 NaN there. The entropy calculation above appears to be a different possible 0 * -inf location, so I would treat it as something worth reproducing on the exact vLLM build rather than assuming the existing test already covers it.
4. “The JSON request” and “the effective sampling configuration” may not be identical
This is why I would compare more than the application-side payload if possible.
vLLM can combine several layers of configuration:
model generation_config
+
server launch flags / HF overrides
+
OpenAI-compatible request
+
extra_body / non-standard sampling options
+
version-specific normalization/validation
=
effective generation parameters
The vLLM OpenAI-compatible server documentation is useful here because parameters outside the normal OpenAI schema can be supplied through vLLM-specific options such as extra_body.
The target RedHatAI model card also demonstrates that server-side DiffusionGemma behavior can be configured at launch time, for example through --hf-overrides, and its deployment example explicitly sets a default enable_thinking value.
So ideally I would compare:
QG client payload
RAG client payload
plus:
vLLM launch command
exact vLLM build
model revision
If server-resolved SamplingParams are easy to log, that is even better.
5. I would compare the rendered prompts, not only the human-readable prompts
This became more interesting than I initially expected.
The target checkpoint’s chat template has actually changed. For example, this RedHatAI model revision removed an empty thought-channel fragment from the non-thinking generation template.
That does not mean the old template causes merged words.
It does mean that:
"same model"
+
"similar visible prompt"
is not quite enough to prove that RAG and QG are presenting the same token sequence to the model.
For the comparison I would include:
exact model revision
chat-template revision
enable_thinking
final rendered prompt
If RAG and QG differ there, normalize that before changing model architecture or denoising parameters.
This is also low-cost: it does not require another model run if the rendered prompt can simply be logged.
6. The exact vLLM version matters unusually much here
DiffusionGemma support has been changing fairly quickly.
For example, PR #45965 changed vLLM’s stability/convergence behavior to better match the Hugging Face stability_threshold semantics.
And, separately, PR #45429 later added the top_k / top_p path discussed above.
So an exact version is useful for more than ordinary reproducibility. Different builds can literally have different diffusion-generation semantics.
For a packaged release:
vllm.__version__
is a good start.
For a nightly/custom container, I would prefer the commit hash or image digest as well.
That lets the investigation branch cleanly:
build before a relevant implementation change
-> that branch cannot explain the observation
build after the change
-> compare the request/configuration that activates it
7. Structured output should be separated into two completely different cases
I would distinguish:
A. Prompt-only JSON
For example:
Return only valid JSON matching this structure...
This is still ordinary model generation. The model is being asked to produce JSON, but decoding is not being constrained by a grammar.
B. Actual constrained/structured decoding
For example something conceptually like:
{
"response_format": {
"type": "json_schema",
"json_schema": {
...
}
}
}
or vLLM’s structured_outputs path.
That is a different mechanism.
This distinction matters for DiffusionGemma because vLLM has already had an integration problem at exactly this boundary.
The original structured-output failure is discussed in the chain leading to PR #45468. The current behavior intentionally rejects structured outputs for diffusion decoders with a clear request-time error rather than allowing the ordinary autoregressive grammar FSM to fail deeper in generation.
There is also an open feature request, #45572: Canvas-aware structured outputs / guided decoding for diffusion language models, which explains the architectural issue quite clearly: the existing grammar path assumes left-to-right token commitment, while diffusion decoding commits/refines canvas positions differently.
The practical workaround described there is essentially:
prompt-only JSON
+
external validation
+
retry/fallback when invalid
So if your “structured output” is only an instruction in the prompt, I would keep it in the ordinary generation branch.
If you are actually sending response_format / structured_outputs, I would separate that immediately, because it has its own backend compatibility story.
8. A raw non-streaming response is a particularly cheap discriminator
This may be the single easiest test in the whole tree.
Take one request that is known to produce the bad output and run the same thing without streaming, then preserve the raw server response before any application processing.
If the server itself returns:
... upcomingbunkering ...
then client display, SSE assembly, whitespace normalization, etc. become much less interesting.
If the server returns:
... upcoming bunkering ...
but the final QG consumer shows:
... upcomingbunkering ...
then the model/sampler investigation can mostly stop for that example, and I would inspect:
stream assembly
JSON extraction
string cleanup
whitespace normalization
formatters
question splitting/joining
instead.
I would only put streaming high on the list if RAG and QG actually use different transport paths.
DiffusionGemma produces blocks/multiple tokens differently from a normal AR stream, and vLLM’s early DiffusionGemma integration did need special handling around multi-token deltas/parsers. That makes streaming worth excluding cheaply, but I do not have evidence that streaming itself is the cause of your missing space.
9. I would leave FP8 fairly low in the first-pass ranking
The checkpoint you are using is an FP8 quantized version, so quantization is certainly part of the environment.
But I would not start there.
The RedHatAI model card reports evaluation of the quantized model against the original model, and more importantly for this particular debugging problem, your working RAG path and problematic QG path appear to be using the same checkpoint.
So there is already a more informative comparison available:
same weights
working path vs failing path
That makes request/rendering/runtime differences more attractive first.
If everything else is eventually aligned and the raw server output still reproduces the anomaly, then:
same request
same backend settings
FP8 checkpoint vs original/BF16 checkpoint
would become a useful later A/B.
I just would not pay that cost before the cheaper comparisons above.
10. I would avoid parameter fishing until the two paths are aligned
The current target generation_config.json is already quite specific about DiffusionGemma’s normal convergence behavior:
max_denoising_steps = 48
entropy_bound = 0.1
confidence_threshold = 0.005
stability_threshold = 1
t_max = 0.8
t_min = 0.4
I would therefore resist the temptation to immediately try:
more denoising steps
different entropy_bound
different confidence_threshold
different temperature schedule
all at once.
Those experiments can make output look better or worse without telling you why QG and RAG differed.
A cleaner control is:
make RAG and QG use the same effective generation settings first
and only tune diffusion-specific parameters after that if the failure remains.
11. If occasional surface corruption remains, you can protect the QG design rather than replacing it
Even if this eventually turns out to be a rare DiffusionGemma/runtime surface failure, I do not think that automatically implies replacing the fast QG path with the RAG path or abandoning diffusion generation.
A useful component boundary would be:
generation
|
v
preserve raw output
|
v
cheap output validator
|
+-- valid ------> use result
|
+-- invalid ----> retry / repair / fallback
For question generation, the validator can remain cheap.
Depending on your output contract, examples might include:
JSON parses
expected question count exists
required fields exist
question lengths are plausible
questions have expected terminal punctuation
no obvious malformed concatenation pattern
The important diagnostic detail is:
preserve the raw generation before repairing it.
I would avoid immediately adding a silent special-case such as:
text = text.replace("upcomingbunkering", "upcoming bunkering")
because it makes the application look fixed while removing the best evidence for determining where the corruption originated.
A validator/retry boundary lets you solve the operational problem and the diagnostic problem independently:
service reliability
!=
root-cause investigation
That seems particularly attractive here because DiffusionGemma’s speed is presumably part of why QG is interesting in the first place.
12. What would change my current ranking
This is roughly how I would interpret new observations:
| Observation |
What it would do to my ranking |
QG alone has effective top_p < 1 or top_k > 0 |
vLLM filter/entropy branch rises a lot |
Both paths effectively use top_p = 1, top_k = 0 |
that NaN branch falls a lot |
| QG/RAG rendered prompts differ |
chat-template/thinking branch rises |
| Same prompt + same effective params |
backend/model-specific branch rises |
| Raw non-streaming server response is already merged |
generation/runtime/model side rises |
| Raw response is correct but application output is merged |
client/parser/post-processing becomes the main branch |
Real response_format / structured_outputs is sent |
constrained-decoding compatibility branch |
| JSON is only requested in the prompt |
ordinary generation branch |
| vLLM build predates a candidate implementation change |
that candidate can be discarded |
| Same exact request reproduces through Transformers |
model/model-family hypothesis rises again |
| Same exact request stays clean through Transformers |
backend-specific integration becomes more likely |
| Everything above matches and FP8 is the remaining variable |
FP8 vs original becomes a useful later A/B |
But if both RAG and QG are effectively running unfiltered sampling, I would drop that hypothesis rather than trying to force it to fit.
Either way, I don’t think the evidence currently says “QG is the wrong use case for DiffusionGemma.” It looks more useful to first isolate which generation contract the two paths are actually exercising, and then keep any workaround at that component boundary.