Instructions to use darioooooo0o/LFM3-1.7B-Base-KV800 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use darioooooo0o/LFM3-1.7B-Base-KV800 with PEFT:
Task type is invalid.
- Notebooks
- Google Colab
- Kaggle
Configuration Parsing Warning:In adapter_config.json: "peft.task_type" must be a string
LFM3-1.7B-Base-KV800
Code: https://github.com/riceharvest/lfm3-kv800
A 15x smaller KV cache for LiquidAI/LFM3-1.7B-Base, with no training required: 398 concurrent 32K-token sequences on one 32 GB Intel Arc Pro B70, where stock vLLM fits 27.
0.1.1: bug fix, please update
0.1.0 had a bug in the vLLM integration: past token 384, layer 10 quantized keys and values from the wrong cache pages, because a per-step flush plan was shared across KV groups (one-line fix in lfm3_kv800/vllm_attn.py). Every multi-token generation past that point was affected, in all decode backends. The training hook did not have the bug, so trainer and engine disagreed: on long rollouts from my own finetune, 13 % of tokens fell outside the [0.5, 5] importance ratio band, now 0.5 %.
Measured again after the fix: needles (chunk 2048) 88/96 to 91/96, needles (chunk 8192) 79/96 to 88/96, rollouts (chunk 2048) 28/32 to 30/32, rollouts (chunk 8192) 28/32 to 31/32. Not measured again: capacity, throughput, and the cross-entropy and recovery tables, which did not go through this code path.
LFM3-1.7B-Base is a base model. If you customize it with RL, most of the cost is generating rollouts, and rollouts are limited by KV-cache memory. This repo ships 800h, a KV-cache format for LFM3 that stores about 800 bytes per token instead of 12,288, plus everything needed to use it:
| what | where | status |
|---|---|---|
| decode kernels: SYCL (Intel XMX), Triton, pure torch | GitHub: riceharvest/lfm3-kv800 (lfm3_kv800) |
tested on B70; Triton/torch parity on CPU and NVIDIA T4 |
vLLM plugin: vllm serve / LLM() with the 800h cache |
GitHub (lfm3_kv800.plugin) |
tested: multiprocess vllm serve, vLLM 0.32 |
| training hook: the same cache inside your SFT/RL forward pass | GitHub (lfm3_kv800.train) |
tested: 10/10 tests on B70 |
| recovery LoRA trained with the cache in the loop | this repo (adapter_model.safetensors) |
3.5-minute probe, closes 40 % of the quality gap |
The base weights are not modified or included. The adapter is optional.
Why this is the base to customize at home
To make a small model good at your task, the usual route is SFT on your examples, then RL against a checker: unit tests, a grader, a tool's exit code. On a home GPU the RL part is the expensive one.
- RL is mostly generation. GRPO-style training samples a group of answers (often 8-16) for every prompt, scores them and updates. The update step is cheap for a 1.7B model. Most of the wall-clock goes to producing those rollouts.
- Rollouts are limited by KV memory, not compute. Each sequence being generated holds its KV cache for the whole rollout. With long prompts or long reasoning traces, the cache fills the card long before the GPU's compute is busy. On a 32 GB card, stock LFM3 fits 27 sequences at 32K context: one or two GRPO groups at a time.
- 800h moves that wall about 15x. The same card holds 398 sequences at 32K, or about 25 GRPO groups of 16, in a single batch. Bigger batches use the GPU better:
- measured at 32K: stock tops out at 431 tok/s with its maximum of 27 sequences;
- 800h reaches 5,124 tok/s with 398 sequences, 11.9x the total, on the same card.
- What that buys you (derived from the measured numbers above, not a measured RL run):
- a GRPO step with 64 prompts × 16 samples of 1K new tokens each is about 1M generated tokens;
- at stock's 431 tok/s that is about 39 minutes of rollouts per step on this card;
- at 800h's 5,124 tok/s it is about 3.3 minutes.
- The bigger gain is in what fits at all: long-context tasks (32K repos, documents, multi-turn agents) that stock can only sample one group at a time.
- The trainer and the sampler see the same model. RL quietly breaks when the sampler and the trainer compute different probabilities for the same tokens: the update is then off-policy, which makes RL noisy or unstable. The training hook runs the same 800h cache inside your training forward pass, so the log-probs match what vLLM sampled. The same hook makes your SFT teach the model to work well with the compressed cache; the recovery adapter shows that effect (40 % of the gap closed in 3.5 minutes).
- One card does all of it.
- A 1.7B model with LoRA trains comfortably next to the sampler on a single 24-32 GB GPU.
- No multi-GPU setup and no cloud are needed: everything here, including the recovery run, ran on one Intel Arc Pro B70.
- The kernels have Triton and pure-torch fallbacks for other GPUs. They run correctly on NVIDIA (checked on a T4) but are not yet tuned there.
Why LFM3-1.7B-Base
- It's a base model. No chat or safety tuning to fight. You decide the format, tone and skills in your own SFT, and RL starts from a clean policy.
- Its architecture is already cheap.
- 13 of its 24 layers are short convolutions and 5 are 256-token sliding windows, so they hold a fixed, tiny state no matter how long the context gets.
- Only 6 layers have a growing cache, and 800h shrinks those 15x.
- The whole model's per-sequence state at 32K is about 50 MB with 800h, against about 680 MB stock.
- It's small enough to iterate. Short training runs mean you can try a reward function, see what the model learns, and change it the same evening.
Customize it at home: the loop
- SFT your examples with the hook on (
patch_for_training(model, mode="800h"), seeexamples/sft_kv800.py). Start from the recovery adapter if you like. The model learns your task and adapts to the cheap cache at the same time. - Serve the SFT'd model with
LFM3_KV800=1 vllm serve ...and sample large rollout batches against your checker. - RL (GRPO or similar) with the same hook in the trainer, so it scores exactly what vLLM generated.
examples/grpo_trl_kv800.pysketches this with TRL's colocate mode (untested). - Deploy with the same 800h cache to keep the memory savings at inference, or merge and serve with a normal cache if you prefer.
Results at a glance (Intel Arc Pro B70 32 GB, vLLM 0.32, bf16 weights)
Memory and concurrency at 32K context
| stock vLLM (bf16 KV) | 800h | |
|---|---|---|
| KV that grows with length | 12,288 B/token | 800 B/token (819.2 allocated) |
| KV per 32,832-token request (as vLLM allocates it) | 678 MB | 50 MB |
| concurrent 32K requests on one B70 (KV pool 21.6 GiB) | 27 (vLLM's stated max; 33 at steady state) | 398 measured all resident, 0 preemptions (steady-state capacity 449) |
At the full 256K context (math, not measured)
LFM3 supports up to 262,144 tokens. Same B70 KV pool (21.58 GiB):
| per sequence at 256K | stock | 800h |
|---|---|---|
| KV content (growing cache + windows + conv state) | 3.00 GiB | 0.22 GB |
| as vLLM allocates it (scaled from the measured 32K block counts) | 5.04 GiB | 0.24 GB |
| sequences that fit at 256K (content / as allocated) | 7 / 4 | 105 / 97 |
So one card goes from 4 full-length 256K sequences to about 97, roughly 15x on raw KV and about 24x as vLLM actually allocates.
This is capacity arithmetic from the measured per-token sizes. Throughput and quality were measured only up to 32K, and recall beyond 32K is untested.
Throughput at 32K context
All c requests are decoding together for 136-183 steps; no preemptions at any point.
| concurrency | aggregate decode | time per step | prefill | KV used | vs stock | |
|---|---|---|---|---|---|---|
| stock vLLM (Triton attention) | 27 (its max) | 431 tok/s | 62.6 ms | 11.3K tok/s | 79.5 % | 1.0x |
| 800h | 27 | 861 tok/s | 31.4 ms | 10.7K tok/s | 7.3 % | 2.0x |
| 800h | 64 | 1,719 tok/s | 37.2 ms | 12.3K tok/s | 16.4 % | 4.0x |
| 800h | 128 | 3,336 tok/s | 38.4 ms | 12.5K tok/s | 29.3 % | 7.7x |
| 800h | 398 | 5,124 tok/s | 77.7 ms | 12.4K tok/s | 90.7 % | 11.9x |
- Same batch size: at stock's own limit of 27 sequences, 800h decodes 2x faster, because each step reads 15x less KV.
- Concurrency stock can't reach: 800h keeps scaling to 398 sequences, 11.9x stock's best total throughput on the same card.
- Why stock uses Triton attention: this vLLM image has no SYCL attention kernels for LFM3's head size, so stock runs vLLM's built-in
TRITON_ATTNbackend.
Quality
| test | stock | 800h, no training |
|---|---|---|
| cross-entropy, last 2K tokens of 8K passages (nats/token) | 1.225 | 1.488 (+0.26) |
| cross-entropy, last 2K tokens of 32K passages | 1.177 | 1.529 (+0.35) |
| needle recall, 96 items, 1K - 32K (vLLM, prefill chunk 2048) | 96/96 | 91/96 (94.8 %, 95 % CI 88.4 - 97.8) |
| needle recall by length 1K / 4K / 16K / 32K (of 24) | 24 / 24 / 24 / 24 | 23 / 23 / 23 / 22 |
| rollout recall: needle in 8K/16K, 1,024 generated tokens, then the question (32 items) | 32/32 | 30/32 (93.8 %) |
| greedy match vs HF reference, short prompts (13 and 16 tokens, 64 generated) | 64/64 | 64/64 |
The cost is about +0.3 nats/token, roughly what a 2-3 bit weight quant costs a small model. Long-range recall mostly survives. The misses are truncated codes (for example " 739." for 7397), not garbage. Greedy text drifts from stock sooner on long prompts.
Recovery with the cache in the loop (this repo's adapter)
| stock cache | 800h cache | gap vs original stock | |
|---|---|---|---|
| original model | 1.582 | 1.735 | +0.153 |
| + recovery LoRA (1M tokens, 3.5 min) | 1.616 | 1.674 | +0.091 (40 % closed) |
The eval here is CE on the last 2,048 tokens of four 4K held-out passages. Same-weights gap (cache on vs off) goes from +0.153 to +0.057. Stock-cache CE rises by 0.034: the adapter specializes to the 800h path, so use it only with the 800h cache. Most of the gain lands in the first 0.5M tokens. This is a starting point for your own fine-tune, not a finished model.
What 800h is
LFM3-1.7B has 24 layers: 13 short-conv layers (fixed state), 5 sliding-window layers (window 256, fixed state) and 6 full-attention layers (2, 6, 10, 14, 18, 22, with 4 KV heads of dim 128). Only the 6 full-attention layers have a cache that grows with context. 800h changes only those.
| layers | what is stored | bytes per token |
|---|---|---|
| 6, 10, 14, 22 | K at 2 bits, asymmetric per channel over 256-token blocks, before RoPE (RoPE is applied after dequantization, in the kernel); V at 1 bit per token after a per-head 128-point Hadamard rotation, groups of 256 dims; fp8 (e4m3) scale + zero for both | 200 each |
| 2, 18 | 4 sink tokens + the last 256 tokens, exact bf16 | fixed per request |
| all 6 | the newest 128 - 383 tokens stay exact bf16 until their 256-token block is complete and 128 tokens old | fixed per request |
That is 4 x 200 = 800 B/token of growing cache, against 6 x 2,048 = 12,288 for stock bf16.
Per request, beyond the growing part, 800h keeps a fixed 10.8 MB: exact windows of 640 tokens on the 6 full layers, the stock 256-token windows on the 5 sliding layers, and conv state.
Ingredients come from published work: KIVI (per-channel K, per-token V, an exact residual window), QuaRot-style Hadamard rotation (it makes 1-bit V usable), and StreamingLLM attention sinks. The LFM3-specific part is which layers tolerate what, found with weight-only surgery tests and no calibration data:
- layers 2 and 18 can be windowed almost for free; layers 6, 14 and 22 carry long-range recall;
- merging KV heads works to 8K and collapses at 16-32K;
- sharing KV across layers produces garbage;
- 1-bit V needs the rotation;
- a QJL 1-bit K sketch loses to 2-bit per-channel K at equal bytes.
Zero-training quality frontier (same eval, 8K CE, decode-faithful):
| B/token | config | compression | CE 8K | ΔCE vs stock | B/token saved per +0.01 nats (vs stock) | marginal: B saved per +0.01 nats vs the row above |
|---|---|---|---|---|---|---|
| 12,288 | stock bf16 | 1.0x | 1.225 | - | - | - |
| 6,336 | int8 per token-head (what vLLM ships) | 1.9x | 1.225 | +0.000 | free | free |
| 1,920 | 2-bit K / 2-bit V | 6.4x | 1.274 | +0.049 | 2,116 | 901 |
| 1,344 | 2-bit K / 1-bit V (Hadamard) | 9.1x | 1.455 | +0.230 | 476 | 32 |
| 800 | 800h | 15.4x | 1.494 | +0.269 | 427 | 139 |
Reading it:
- int8 is free but stops at 1.9x.
- 2/2-bit is the best value per unit of loss: 2,116 B/token saved for each 0.01 nats.
- The step from 2/2 to 2/1-bit is the expensive one: it saves 576 B/token for +0.18 nats, only 32 B per 0.01.
- 800h's last step is efficient again: windowing layers 2 and 18 instead of quantizing them saves 544 B/token for +0.04 nats, 139 B per 0.01. That makes 800h the cheapest point at about 427 B saved per 0.01 nats overall.
- Choosing a point: if quality matters more than memory, 2/2-bit at 1,920 B/token (+0.05 nats) is the conservative option; the same kernels and layout ideas apply.
Cost split at 8K: windows on layers 2 and 18 add +0.07, 2-bit K about +0.05, and 1-bit V the rest.
Use it
Install
The B70 path: vLLM 0.32 image, with the SYCL op prebuilt for torch 2.14+xpu (bmg-g31), so no compiler is needed:
docker run -it --device /dev/dri -v /dev/dri:/dev/dri:ro --shm-size 8g \
-v ~/.cache/huggingface:/root/.cache/huggingface --entrypoint bash ghcr.io/riceharvest/b70-vllm:v0.32.0-b70.7
git clone https://github.com/riceharvest/lfm3-kv800 && pip install --no-deps --no-build-isolation ./lfm3-kv800
Elsewhere (CUDA, CPU, another torch): pip install ./lfm3-kv800 --no-deps. Decode then uses Triton on NVIDIA and pure torch anywhere else.
Serve
LFM3_KV800=1 vllm serve LiquidAI/LFM3-1.7B-Base --trust-remote-code --dtype bfloat16 \
--max-model-len 34816 --max-num-seqs 256 --max-num-batched-tokens 2048 --gpu-memory-utilization 0.85
- With
LFM3_KV800=1, the plugin turns on eager mode and turns off prefix caching and torch.compile; you don't need flags for those. - Python
LLM(...)works the same way, since installing the package registers the plugin. - A smaller
--max-num-batched-tokens(prefill chunk) gives better recall: chunk 2048 got 91/96 needles, chunk 8192 got 88/96. Later prefill chunks read earlier chunks back from the quantized cache.
| env var | effect |
|---|---|
LFM3_KV800=1 |
use the 800h cache (the plugin's vendored LFM3 model) |
LFM3KV_BACKEND=sycl|triton|torch |
force a decode backend (default order: sycl → triton → torch on Intel, triton → torch on CUDA) |
LFM3KV_XMX=0 |
Intel: SIMT SYCL kernel instead of the XMX one |
Train with the same cache (SFT / RL)
from lfm3_kv800.train import patch_for_training
h = patch_for_training(model, mode="800h") # HF LFM3 model, plain or PEFT-wrapped
loss = model(input_ids=x, labels=x).loss # forward sees exactly what the vLLM cache would hold
h.set_mode("stock"); h.remove() # compare / undo
- What it does: a decode-faithful attention on the 6 full-attention layers. Each token sees exact K/V for the window the cache keeps exact, and quantized K/V beyond it, with straight-through gradients. The quantizer is bit-identical to the inference cache.
- Inputs: padding, packed sequences and any length work.
- RL:
h.prefill(boundary)marks the prompt part, so rollouts scored by the trainer match what vLLM sampled. - Examples:
examples/sft_kv800.pyandexamples/grpo_trl_kv800.py. The TRL GRPO example is untested.
Load the recovery adapter
from peft import PeftModel
from transformers import AutoModelForCausalLM
base = AutoModelForCausalLM.from_pretrained("LiquidAI/LFM3-1.7B-Base", trust_remote_code=True, dtype="bfloat16")
model = PeftModel.from_pretrained(base, "darioooooo0o/LFM3-1.7B-Base-KV800")
- For vLLM serving: merge it (
model.merge_and_unload(), then save and pointvllm serveat it). - Adapter details:
- LoRA rank 32, alpha 32, on q/k/v/o of all 11 attention layers (0.27 % of parameters).
- Trained for 1M tokens: 488 sequences of 2,048 tokens, batch 1, lr 5e-5 AdamW (0.9, 0.95), 20 warmup steps, clip 1.0, bf16.
- The 800h cache was in the loop, with the newest 128 tokens exact.
- Data: 50/50 code and reasoning text, from shard 0 of
codeparrot/github-code-cleanand ofnvidia/OpenCodeReasoning.
How it runs
- Cache in vLLM. The 800h cache uses vLLM's normal paged allocator.
- The 4 quantized layers use 1,280-token pages: five 256-token sub-blocks of 51,200 B, padded to 262,144 B. The 6 KB padding holds the sinks of layers 2 and 18.
- The exact windows use vLLM sliding-window pages of the same 262,144 B.
- Every KV group shares one page size, so nothing is padded.
- Decode kernels.
- On Intel, the quantized layer runs as one SYCL sub-group per (sequence, KV head, split), with QK and PV on XMX matrix units (DPAS, 4 query rows).
- K is dequantized and roped in registers; 1-bit V becomes bf16 {0, 2} in two integer ops.
- The exact window and sinks use a second kernel. A combine kernel merges the split softmaxes and undoes the Hadamard rotation with an in-register FWHT.
- Prefill. Prefill uses torch SDPA over the dequantized history, the exact window and the chunk.
Decode kernel time per layer call at 32K context (B70):
| c=1 | c=16 | c=64 | |
|---|---|---|---|
| 800h quantized layer, SYCL XMX | 0.20 ms | 0.52 ms | 1.26 ms (356 GB/s of cache) |
| 800h quantized layer, Triton | 0.37 ms | 1.18 ms | 3.7 ms |
| window + sinks layer, SYCL | 0.11 ms | 0.12 ms | 0.17 ms |
| stock bf16 SDPA, one layer (for scale) | 0.49 ms | 1.8 ms | not measured |
Evaluation details
- Needles: "# The secret number of the
<key>is NNNN." is inserted at 25 % or 75 % depth into 1K/4K/16K/32K tokens of CPython stdlib source (argparse, difflib, inspect, typing). The prompt then asks for the number; decoding is greedy with 8 new tokens.- 96 items = seeds 3-14 × 4 lengths × 2 depths.
- Seeds 3-5 were used during development; seeds 6-14 are held out (800h: 66/72 held out).
- Stock vs 800h, McNemar exact test: p = 0.008.
- Rollouts: the needle context, then 1,024 greedy generated tokens, then the question.
- 32 items = seeds 0-7 × 8K/16K × 2 depths.
- McNemar exact test: p = 0.125.
- Cross-entropy: four local passages (CPython
inspect.py,typing.py, the LibreOffice license, the man-db manual).- 8K and 32K versions; the loss is measured on the last 2,048 tokens.
- Decode-faithful: every token sees quantized K/V beyond its exact window, the same as at decode.
- Hardware: Intel Arc Pro B70 32 GB, image
ghcr.io/riceharvest/b70-vllm:v0.32.0-b70.7(vLLM 0.32.0, torch 2.14+xpu). - Raw per-item JSON and scripts:
package/.../eval/.
Limitations
- NVIDIA: correct, not yet fast. Checked on a free Kaggle Tesla T4 (sm_75) with
nvidia/check_nvidia.ipynb.- Kernel parity passes for the Triton and torch backends, and the backend chain auto-selects Triton.
- End-to-end needles at 1K/4K/16K: 800h 21/24, stock 24/24, the same pattern as on the B70.
- The Triton kernels are untuned for NVIDIA: 25.6 ms per quantized-layer call at 64 × 32K on the T4, against 1.26 ms for the B70 SYCL kernel. They were written for Intel's 16-lane sub-groups.
- Porting or tuning them for your GPU (num_warps, tile sizes, tensor-core dots) is a good job for a coding agent, and the 12-case parity test tells it when it's still correct.
- Lossy. About +0.3 nats without training, and 8/96 needles missed at chunk 2048.
- vLLM limits. The integration needs eager mode, prefix caching off and a single GPU. There is no XPU/CUDA graph capture yet, and no speculative decoding.
- Tied to LFM3-1.7B-Base's architecture. The layer choices were found for this model.
- Untested pieces. The TRL GRPO example and the SFT example script haven't been run end to end.
- Upstream vLLM. vLLM has an open PR adding LFM3 (#64959). With
LFM3_KV800=1the plugin always uses its own vendored LFM3 model, because upstream's would ignore the 800h cache.
Files
| file | what |
|---|---|
README.md |
this card |
adapter_config.json, adapter_model.safetensors |
recovery LoRA (PEFT, 18 MB) |
package/lfm3-kv800-src.tar.gz |
snapshot of the code; the live code is at https://github.com/riceharvest/lfm3-kv800 |
LICENSE |
LFM Open License v1.0 |
License
The adapter is a derivative of LFM3-1.7B-Base and is released under the LFM Open License v1.0, including that license's commercial-use threshold. The base weights are not included. The package's vLLM model file is a port of LFM3's architecture for vLLM. The kernels, cache format, training hook and evals are the author's own work.
References
- KIVI: Liu et al., 2024, KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache (arXiv:2402.02750)
- QuaRot: Ashkboos et al., 2024, QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs (arXiv:2404.00456)
- StreamingLLM: Xiao et al., 2023, Efficient Streaming Language Models with Attention Sinks (arXiv:2309.17453)
- QJL: Zandieh et al., 2024 (arXiv:2406.03482), tested here and not used
Credits
Made by @imdariotoo. I post my local-model releases on X, so follow there for the next ones.
- Downloads last month
- 50
Model tree for darioooooo0o/LFM3-1.7B-Base-KV800
Base model
LiquidAI/LFM3-1.7B-Base