Configuration Parsing Warning:In adapter_config.json: "peft.task_type" must be a string

LFM3-1.7B-Base-KV800

Code: https://github.com/riceharvest/lfm3-kv800

A 15x smaller KV cache for LiquidAI/LFM3-1.7B-Base, with no training required: 398 concurrent 32K-token sequences on one 32 GB Intel Arc Pro B70, where stock vLLM fits 27.

0.1.1: bug fix, please update

0.1.0 had a bug in the vLLM integration: past token 384, layer 10 quantized keys and values from the wrong cache pages, because a per-step flush plan was shared across KV groups (one-line fix in lfm3_kv800/vllm_attn.py). Every multi-token generation past that point was affected, in all decode backends. The training hook did not have the bug, so trainer and engine disagreed: on long rollouts from my own finetune, 13 % of tokens fell outside the [0.5, 5] importance ratio band, now 0.5 %.

Measured again after the fix: needles (chunk 2048) 88/96 to 91/96, needles (chunk 8192) 79/96 to 88/96, rollouts (chunk 2048) 28/32 to 30/32, rollouts (chunk 8192) 28/32 to 31/32. Not measured again: capacity, throughput, and the cross-entropy and recovery tables, which did not go through this code path.

LFM3-1.7B-Base is a base model. If you customize it with RL, most of the cost is generating rollouts, and rollouts are limited by KV-cache memory. This repo ships 800h, a KV-cache format for LFM3 that stores about 800 bytes per token instead of 12,288, plus everything needed to use it:

what where status
decode kernels: SYCL (Intel XMX), Triton, pure torch GitHub: riceharvest/lfm3-kv800 (lfm3_kv800) tested on B70; Triton/torch parity on CPU and NVIDIA T4
vLLM plugin: vllm serve / LLM() with the 800h cache GitHub (lfm3_kv800.plugin) tested: multiprocess vllm serve, vLLM 0.32
training hook: the same cache inside your SFT/RL forward pass GitHub (lfm3_kv800.train) tested: 10/10 tests on B70
recovery LoRA trained with the cache in the loop this repo (adapter_model.safetensors) 3.5-minute probe, closes 40 % of the quality gap

The base weights are not modified or included. The adapter is optional.


Why this is the base to customize at home

To make a small model good at your task, the usual route is SFT on your examples, then RL against a checker: unit tests, a grader, a tool's exit code. On a home GPU the RL part is the expensive one.

  • RL is mostly generation. GRPO-style training samples a group of answers (often 8-16) for every prompt, scores them and updates. The update step is cheap for a 1.7B model. Most of the wall-clock goes to producing those rollouts.
  • Rollouts are limited by KV memory, not compute. Each sequence being generated holds its KV cache for the whole rollout. With long prompts or long reasoning traces, the cache fills the card long before the GPU's compute is busy. On a 32 GB card, stock LFM3 fits 27 sequences at 32K context: one or two GRPO groups at a time.
  • 800h moves that wall about 15x. The same card holds 398 sequences at 32K, or about 25 GRPO groups of 16, in a single batch. Bigger batches use the GPU better:
    • measured at 32K: stock tops out at 431 tok/s with its maximum of 27 sequences;
    • 800h reaches 5,124 tok/s with 398 sequences, 11.9x the total, on the same card.
  • What that buys you (derived from the measured numbers above, not a measured RL run):
    • a GRPO step with 64 prompts × 16 samples of 1K new tokens each is about 1M generated tokens;
    • at stock's 431 tok/s that is about 39 minutes of rollouts per step on this card;
    • at 800h's 5,124 tok/s it is about 3.3 minutes.
    • The bigger gain is in what fits at all: long-context tasks (32K repos, documents, multi-turn agents) that stock can only sample one group at a time.
  • The trainer and the sampler see the same model. RL quietly breaks when the sampler and the trainer compute different probabilities for the same tokens: the update is then off-policy, which makes RL noisy or unstable. The training hook runs the same 800h cache inside your training forward pass, so the log-probs match what vLLM sampled. The same hook makes your SFT teach the model to work well with the compressed cache; the recovery adapter shows that effect (40 % of the gap closed in 3.5 minutes).
  • One card does all of it.
    • A 1.7B model with LoRA trains comfortably next to the sampler on a single 24-32 GB GPU.
    • No multi-GPU setup and no cloud are needed: everything here, including the recovery run, ran on one Intel Arc Pro B70.
    • The kernels have Triton and pure-torch fallbacks for other GPUs. They run correctly on NVIDIA (checked on a T4) but are not yet tuned there.

Why LFM3-1.7B-Base

  • It's a base model. No chat or safety tuning to fight. You decide the format, tone and skills in your own SFT, and RL starts from a clean policy.
  • Its architecture is already cheap.
    • 13 of its 24 layers are short convolutions and 5 are 256-token sliding windows, so they hold a fixed, tiny state no matter how long the context gets.
    • Only 6 layers have a growing cache, and 800h shrinks those 15x.
    • The whole model's per-sequence state at 32K is about 50 MB with 800h, against about 680 MB stock.
  • It's small enough to iterate. Short training runs mean you can try a reward function, see what the model learns, and change it the same evening.

Customize it at home: the loop

  1. SFT your examples with the hook on (patch_for_training(model, mode="800h"), see examples/sft_kv800.py). Start from the recovery adapter if you like. The model learns your task and adapts to the cheap cache at the same time.
  2. Serve the SFT'd model with LFM3_KV800=1 vllm serve ... and sample large rollout batches against your checker.
  3. RL (GRPO or similar) with the same hook in the trainer, so it scores exactly what vLLM generated. examples/grpo_trl_kv800.py sketches this with TRL's colocate mode (untested).
  4. Deploy with the same 800h cache to keep the memory savings at inference, or merge and serve with a normal cache if you prefer.

Results at a glance (Intel Arc Pro B70 32 GB, vLLM 0.32, bf16 weights)

Memory and concurrency at 32K context

stock vLLM (bf16 KV) 800h
KV that grows with length 12,288 B/token 800 B/token (819.2 allocated)
KV per 32,832-token request (as vLLM allocates it) 678 MB 50 MB
concurrent 32K requests on one B70 (KV pool 21.6 GiB) 27 (vLLM's stated max; 33 at steady state) 398 measured all resident, 0 preemptions (steady-state capacity 449)

At the full 256K context (math, not measured)

LFM3 supports up to 262,144 tokens. Same B70 KV pool (21.58 GiB):

per sequence at 256K stock 800h
KV content (growing cache + windows + conv state) 3.00 GiB 0.22 GB
as vLLM allocates it (scaled from the measured 32K block counts) 5.04 GiB 0.24 GB
sequences that fit at 256K (content / as allocated) 7 / 4 105 / 97

So one card goes from 4 full-length 256K sequences to about 97, roughly 15x on raw KV and about 24x as vLLM actually allocates.

This is capacity arithmetic from the measured per-token sizes. Throughput and quality were measured only up to 32K, and recall beyond 32K is untested.

Throughput at 32K context

All c requests are decoding together for 136-183 steps; no preemptions at any point.

concurrency aggregate decode time per step prefill KV used vs stock
stock vLLM (Triton attention) 27 (its max) 431 tok/s 62.6 ms 11.3K tok/s 79.5 % 1.0x
800h 27 861 tok/s 31.4 ms 10.7K tok/s 7.3 % 2.0x
800h 64 1,719 tok/s 37.2 ms 12.3K tok/s 16.4 % 4.0x
800h 128 3,336 tok/s 38.4 ms 12.5K tok/s 29.3 % 7.7x
800h 398 5,124 tok/s 77.7 ms 12.4K tok/s 90.7 % 11.9x
  • Same batch size: at stock's own limit of 27 sequences, 800h decodes 2x faster, because each step reads 15x less KV.
  • Concurrency stock can't reach: 800h keeps scaling to 398 sequences, 11.9x stock's best total throughput on the same card.
  • Why stock uses Triton attention: this vLLM image has no SYCL attention kernels for LFM3's head size, so stock runs vLLM's built-in TRITON_ATTN backend.

Quality

test stock 800h, no training
cross-entropy, last 2K tokens of 8K passages (nats/token) 1.225 1.488 (+0.26)
cross-entropy, last 2K tokens of 32K passages 1.177 1.529 (+0.35)
needle recall, 96 items, 1K - 32K (vLLM, prefill chunk 2048) 96/96 91/96 (94.8 %, 95 % CI 88.4 - 97.8)
needle recall by length 1K / 4K / 16K / 32K (of 24) 24 / 24 / 24 / 24 23 / 23 / 23 / 22
rollout recall: needle in 8K/16K, 1,024 generated tokens, then the question (32 items) 32/32 30/32 (93.8 %)
greedy match vs HF reference, short prompts (13 and 16 tokens, 64 generated) 64/64 64/64

The cost is about +0.3 nats/token, roughly what a 2-3 bit weight quant costs a small model. Long-range recall mostly survives. The misses are truncated codes (for example " 739." for 7397), not garbage. Greedy text drifts from stock sooner on long prompts.

Recovery with the cache in the loop (this repo's adapter)

stock cache 800h cache gap vs original stock
original model 1.582 1.735 +0.153
+ recovery LoRA (1M tokens, 3.5 min) 1.616 1.674 +0.091 (40 % closed)

The eval here is CE on the last 2,048 tokens of four 4K held-out passages. Same-weights gap (cache on vs off) goes from +0.153 to +0.057. Stock-cache CE rises by 0.034: the adapter specializes to the 800h path, so use it only with the 800h cache. Most of the gain lands in the first 0.5M tokens. This is a starting point for your own fine-tune, not a finished model.


What 800h is

LFM3-1.7B has 24 layers: 13 short-conv layers (fixed state), 5 sliding-window layers (window 256, fixed state) and 6 full-attention layers (2, 6, 10, 14, 18, 22, with 4 KV heads of dim 128). Only the 6 full-attention layers have a cache that grows with context. 800h changes only those.

layers what is stored bytes per token
6, 10, 14, 22 K at 2 bits, asymmetric per channel over 256-token blocks, before RoPE (RoPE is applied after dequantization, in the kernel); V at 1 bit per token after a per-head 128-point Hadamard rotation, groups of 256 dims; fp8 (e4m3) scale + zero for both 200 each
2, 18 4 sink tokens + the last 256 tokens, exact bf16 fixed per request
all 6 the newest 128 - 383 tokens stay exact bf16 until their 256-token block is complete and 128 tokens old fixed per request

That is 4 x 200 = 800 B/token of growing cache, against 6 x 2,048 = 12,288 for stock bf16.

Per request, beyond the growing part, 800h keeps a fixed 10.8 MB: exact windows of 640 tokens on the 6 full layers, the stock 256-token windows on the 5 sliding layers, and conv state.

Ingredients come from published work: KIVI (per-channel K, per-token V, an exact residual window), QuaRot-style Hadamard rotation (it makes 1-bit V usable), and StreamingLLM attention sinks. The LFM3-specific part is which layers tolerate what, found with weight-only surgery tests and no calibration data:

  • layers 2 and 18 can be windowed almost for free; layers 6, 14 and 22 carry long-range recall;
  • merging KV heads works to 8K and collapses at 16-32K;
  • sharing KV across layers produces garbage;
  • 1-bit V needs the rotation;
  • a QJL 1-bit K sketch loses to 2-bit per-channel K at equal bytes.

Zero-training quality frontier (same eval, 8K CE, decode-faithful):

B/token config compression CE 8K ΔCE vs stock B/token saved per +0.01 nats (vs stock) marginal: B saved per +0.01 nats vs the row above
12,288 stock bf16 1.0x 1.225 - - -
6,336 int8 per token-head (what vLLM ships) 1.9x 1.225 +0.000 free free
1,920 2-bit K / 2-bit V 6.4x 1.274 +0.049 2,116 901
1,344 2-bit K / 1-bit V (Hadamard) 9.1x 1.455 +0.230 476 32
800 800h 15.4x 1.494 +0.269 427 139

Reading it:

  • int8 is free but stops at 1.9x.
  • 2/2-bit is the best value per unit of loss: 2,116 B/token saved for each 0.01 nats.
  • The step from 2/2 to 2/1-bit is the expensive one: it saves 576 B/token for +0.18 nats, only 32 B per 0.01.
  • 800h's last step is efficient again: windowing layers 2 and 18 instead of quantizing them saves 544 B/token for +0.04 nats, 139 B per 0.01. That makes 800h the cheapest point at about 427 B saved per 0.01 nats overall.
  • Choosing a point: if quality matters more than memory, 2/2-bit at 1,920 B/token (+0.05 nats) is the conservative option; the same kernels and layout ideas apply.

Cost split at 8K: windows on layers 2 and 18 add +0.07, 2-bit K about +0.05, and 1-bit V the rest.


Use it

Install

The B70 path: vLLM 0.32 image, with the SYCL op prebuilt for torch 2.14+xpu (bmg-g31), so no compiler is needed:

docker run -it --device /dev/dri -v /dev/dri:/dev/dri:ro --shm-size 8g \
  -v ~/.cache/huggingface:/root/.cache/huggingface --entrypoint bash ghcr.io/riceharvest/b70-vllm:v0.32.0-b70.7
git clone https://github.com/riceharvest/lfm3-kv800 && pip install --no-deps --no-build-isolation ./lfm3-kv800

Elsewhere (CUDA, CPU, another torch): pip install ./lfm3-kv800 --no-deps. Decode then uses Triton on NVIDIA and pure torch anywhere else.

Serve

LFM3_KV800=1 vllm serve LiquidAI/LFM3-1.7B-Base --trust-remote-code --dtype bfloat16 \
  --max-model-len 34816 --max-num-seqs 256 --max-num-batched-tokens 2048 --gpu-memory-utilization 0.85
  • With LFM3_KV800=1, the plugin turns on eager mode and turns off prefix caching and torch.compile; you don't need flags for those.
  • Python LLM(...) works the same way, since installing the package registers the plugin.
  • A smaller --max-num-batched-tokens (prefill chunk) gives better recall: chunk 2048 got 91/96 needles, chunk 8192 got 88/96. Later prefill chunks read earlier chunks back from the quantized cache.
env var effect
LFM3_KV800=1 use the 800h cache (the plugin's vendored LFM3 model)
LFM3KV_BACKEND=sycl|triton|torch force a decode backend (default order: sycl → triton → torch on Intel, triton → torch on CUDA)
LFM3KV_XMX=0 Intel: SIMT SYCL kernel instead of the XMX one

Train with the same cache (SFT / RL)

from lfm3_kv800.train import patch_for_training
h = patch_for_training(model, mode="800h")    # HF LFM3 model, plain or PEFT-wrapped
loss = model(input_ids=x, labels=x).loss       # forward sees exactly what the vLLM cache would hold
h.set_mode("stock"); h.remove()                 # compare / undo
  • What it does: a decode-faithful attention on the 6 full-attention layers. Each token sees exact K/V for the window the cache keeps exact, and quantized K/V beyond it, with straight-through gradients. The quantizer is bit-identical to the inference cache.
  • Inputs: padding, packed sequences and any length work.
  • RL: h.prefill(boundary) marks the prompt part, so rollouts scored by the trainer match what vLLM sampled.
  • Examples: examples/sft_kv800.py and examples/grpo_trl_kv800.py. The TRL GRPO example is untested.

Load the recovery adapter

from peft import PeftModel
from transformers import AutoModelForCausalLM
base = AutoModelForCausalLM.from_pretrained("LiquidAI/LFM3-1.7B-Base", trust_remote_code=True, dtype="bfloat16")
model = PeftModel.from_pretrained(base, "darioooooo0o/LFM3-1.7B-Base-KV800")
  • For vLLM serving: merge it (model.merge_and_unload(), then save and point vllm serve at it).
  • Adapter details:
    • LoRA rank 32, alpha 32, on q/k/v/o of all 11 attention layers (0.27 % of parameters).
    • Trained for 1M tokens: 488 sequences of 2,048 tokens, batch 1, lr 5e-5 AdamW (0.9, 0.95), 20 warmup steps, clip 1.0, bf16.
    • The 800h cache was in the loop, with the newest 128 tokens exact.
    • Data: 50/50 code and reasoning text, from shard 0 of codeparrot/github-code-clean and of nvidia/OpenCodeReasoning.

How it runs

  • Cache in vLLM. The 800h cache uses vLLM's normal paged allocator.
    • The 4 quantized layers use 1,280-token pages: five 256-token sub-blocks of 51,200 B, padded to 262,144 B. The 6 KB padding holds the sinks of layers 2 and 18.
    • The exact windows use vLLM sliding-window pages of the same 262,144 B.
    • Every KV group shares one page size, so nothing is padded.
  • Decode kernels.
    • On Intel, the quantized layer runs as one SYCL sub-group per (sequence, KV head, split), with QK and PV on XMX matrix units (DPAS, 4 query rows).
    • K is dequantized and roped in registers; 1-bit V becomes bf16 {0, 2} in two integer ops.
    • The exact window and sinks use a second kernel. A combine kernel merges the split softmaxes and undoes the Hadamard rotation with an in-register FWHT.
  • Prefill. Prefill uses torch SDPA over the dequantized history, the exact window and the chunk.

Decode kernel time per layer call at 32K context (B70):

c=1 c=16 c=64
800h quantized layer, SYCL XMX 0.20 ms 0.52 ms 1.26 ms (356 GB/s of cache)
800h quantized layer, Triton 0.37 ms 1.18 ms 3.7 ms
window + sinks layer, SYCL 0.11 ms 0.12 ms 0.17 ms
stock bf16 SDPA, one layer (for scale) 0.49 ms 1.8 ms not measured

Evaluation details

  • Needles: "# The secret number of the <key> is NNNN." is inserted at 25 % or 75 % depth into 1K/4K/16K/32K tokens of CPython stdlib source (argparse, difflib, inspect, typing). The prompt then asks for the number; decoding is greedy with 8 new tokens.
    • 96 items = seeds 3-14 × 4 lengths × 2 depths.
    • Seeds 3-5 were used during development; seeds 6-14 are held out (800h: 66/72 held out).
    • Stock vs 800h, McNemar exact test: p = 0.008.
  • Rollouts: the needle context, then 1,024 greedy generated tokens, then the question.
    • 32 items = seeds 0-7 × 8K/16K × 2 depths.
    • McNemar exact test: p = 0.125.
  • Cross-entropy: four local passages (CPython inspect.py, typing.py, the LibreOffice license, the man-db manual).
    • 8K and 32K versions; the loss is measured on the last 2,048 tokens.
    • Decode-faithful: every token sees quantized K/V beyond its exact window, the same as at decode.
  • Hardware: Intel Arc Pro B70 32 GB, image ghcr.io/riceharvest/b70-vllm:v0.32.0-b70.7 (vLLM 0.32.0, torch 2.14+xpu).
  • Raw per-item JSON and scripts: package/.../eval/.

Limitations

  • NVIDIA: correct, not yet fast. Checked on a free Kaggle Tesla T4 (sm_75) with nvidia/check_nvidia.ipynb.
    • Kernel parity passes for the Triton and torch backends, and the backend chain auto-selects Triton.
    • End-to-end needles at 1K/4K/16K: 800h 21/24, stock 24/24, the same pattern as on the B70.
    • The Triton kernels are untuned for NVIDIA: 25.6 ms per quantized-layer call at 64 × 32K on the T4, against 1.26 ms for the B70 SYCL kernel. They were written for Intel's 16-lane sub-groups.
    • Porting or tuning them for your GPU (num_warps, tile sizes, tensor-core dots) is a good job for a coding agent, and the 12-case parity test tells it when it's still correct.
  • Lossy. About +0.3 nats without training, and 8/96 needles missed at chunk 2048.
  • vLLM limits. The integration needs eager mode, prefix caching off and a single GPU. There is no XPU/CUDA graph capture yet, and no speculative decoding.
  • Tied to LFM3-1.7B-Base's architecture. The layer choices were found for this model.
  • Untested pieces. The TRL GRPO example and the SFT example script haven't been run end to end.
  • Upstream vLLM. vLLM has an open PR adding LFM3 (#64959). With LFM3_KV800=1 the plugin always uses its own vendored LFM3 model, because upstream's would ignore the 800h cache.

Files

file what
README.md this card
adapter_config.json, adapter_model.safetensors recovery LoRA (PEFT, 18 MB)
package/lfm3-kv800-src.tar.gz snapshot of the code; the live code is at https://github.com/riceharvest/lfm3-kv800
LICENSE LFM Open License v1.0

License

The adapter is a derivative of LFM3-1.7B-Base and is released under the LFM Open License v1.0, including that license's commercial-use threshold. The base weights are not included. The package's vLLM model file is a port of LFM3's architecture for vLLM. The kernels, cache format, training hook and evals are the author's own work.

References

  • KIVI: Liu et al., 2024, KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache (arXiv:2402.02750)
  • QuaRot: Ashkboos et al., 2024, QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs (arXiv:2404.00456)
  • StreamingLLM: Xiao et al., 2023, Efficient Streaming Language Models with Attention Sinks (arXiv:2309.17453)
  • QJL: Zandieh et al., 2024 (arXiv:2406.03482), tested here and not used

Credits

Made by @imdariotoo. I post my local-model releases on X, so follow there for the next ones.

Downloads last month
50
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for darioooooo0o/LFM3-1.7B-Base-KV800

Adapter
(8)
this model

Papers for darioooooo0o/LFM3-1.7B-Base-KV800