arxiv:2609.04098

Why Gated DeltaNet Survives 4-Bit Quantization: NVFP4 W4A4 for the Recurrent Half of a Hybrid 27B LLM

Published on Sep 3

· Submitted by

David Mayboroda on Sep 4

Minima AI

Upvote

Authors:

Davyd Maiboroda

Abstract

Fully quantizing hybrid LLMs—including recurrent Gated DeltaNet layers—to 4-bit NVFP4 preserves accuracy across long-context and reasoning benchmarks by localizing outliers and exploiting robust delta-rule dynamics.

Generated by thinkingmachines/Inkling-Small

Hybrid LLMs pair softmax attention with linear-attention layers such as Gated DeltaNet (GDN), whose recurrent state summarizes the context in fixed size. Early community 4-bit quantizations of Qwen3.8-27B (48 GDN layers, 16 attention layers) left the GDN block in 8- or 16-bit precision -- especially its decay and write-strength gates -- on the intuition that errors in a recurrence accumulate over long contexts. We test that intuition by building Minima: NVFP4 W4A4 on all 496 linear layers, GDN included. Across perplexity at 4K/32K, MMLU-Pro, GSM8K, AIME'25, GPQA-Diamond, LiveCodeBench, and RULER retrieval to 64K, Minima matches BF16 within seed noise (5-task average -0.52) while being the smallest (17.5 GiB) and fastest-prefill (+14-19%) recipe we compare, and its 32K perplexity gap shrinks with position. A four-part mechanism study explains why: (i) NVFP4's 16-element block scaling localizes the residual stream's extreme outliers, equalizing activation error across layer roles; (ii) the supposedly fragile gate projections are the least sensitive -- softplus/exponential and sigmoid parameterizations compress ~11% GEMM error to ~2% output error; (iii) the delta-rule recurrence holds injected noise at a flat plateau over 32K tokens and forgets a state impulse within hundreds of steps, because each write overwrites the state along the current key direction; (iv) the per-token quantization cost washes out with context instead of compounding. We also repair a global-scale mismatch that arises when per-module-calibrated NVFP4 checkpoints are served by kernels that fuse those modules into one GEMM, and show calibrated FP8 KV-cache scales are performance-free. The result: a practical recipe -- quantize everything, ship KV scales -- and a mechanistic account of why the recurrent half of a hybrid LLM is the easy half to quantize. Checkpoint: https://huggingface.co/minima-ai/mnma_qwen3.8_27b_nvfp4

View arXiv page View PDF Add to collection

Community

dmayboroda

Paper author Paper submitter about 14 hours ago

Hi everyone,

TL;DR: the community assumption for hybrid LLMs has been that the recurrent half (Gated DeltaNet) is too fragile for 4-bit — early quants of Qwen3.8-27B all kept it at 8/16-bit. We quantized all 496 linear layers to NVFP4 W4A4 — GDN and its gate projections included — with calibration-only PTQ (no QAT, no distillation), and it matches BF16 within seed noise on MMLU-Pro, GSM8K, AIME'25, GPQA-Diamond, LiveCodeBench, and RULER retrieval to 64K, while being ~2.9× smaller than BF16 and 14–19% faster at prefill than the partial quants.

All four models were benchmarked head-to-head under one identical serving regime (vLLM, TP=1, single 96 GB Blackwell GPU):

Qwen/Qwen3.8-27B — the BF16 baseline
minima-ai/mnma_qwen3.8_27b_nvfp4 — ours, all 496 layers NVFP4 W4A4
unsloth/Qwen3.8-27B-NVFP4 — keeps GDN + attention at FP8 W8A8
RadixArk/Qwen3.8-27B-NVFP4 — likewise protects GDN/attention

Three findings that surprised us:

The "fragile" decay/write gates are actually the least sensitive layers in the model — their softplus/sigmoid parameterizations squash an ~11% GEMM error down to ~2% at the output.
Quantization noise does not accumulate in the recurrent state. The delta rule overwrites the state along each new key direction, so an injected error is forgotten within hundreds of tokens — the 32K perplexity gap actually shrinks with position in the context.
A serving trap: kernels that fuse GDN projections into one GEMM silently mis-scale per-module-calibrated NVFP4 checkpoints. We document the fix; our released checkpoint ships pre-harmonized and serves correctly out of the box.

Weights (17.5 GiB, single Blackwell GPU): https://huggingface.co/minima-ai/mnma_qwen3.8_27b_nvfp4