groxaxo/LFM3-Audio-2B-GPTQ-Pro-INT4

LFM3-Audio-2B — GPTQ-Pro INT4

Native INT4. Real CUDA inference. Smaller VRAM. Actual audio to listen to.

Project overview

LFM3-Audio-2B GPTQ-Pro INT4 is my selective 4-bit quantization of Liquid AI's LFM3-Audio-2B for local speech recognition, speech generation and voice-agent experiments. It uses GPTQ-Pro to compress the language backbone while preserving the audio components in BF16.

The release includes the packed INT4 overlay, a custom CUDA inference loader, the quantization scripts, reproducible RTX 3090 benchmarks, and English/Spanish audio comparisons. The aim is lower GPU memory use with inspectable quality and speed measurements.

INT4 here means symmetric 4-bit projection weights with a group size of 64 across 142 projections in 24 decoder layers. It is a mixed-precision release: audio-sensitive components remain BF16, and inference requires the original base checkpoint plus the supplied overlay loader.

Why I made this

I've been experimenting with local voice AI, trying to get useful models running efficiently on hardware I already own. Liquid AI's LFM3-Audio-2B looked like a great candidate for an AI receptionist: speech in, text out, speech back.

So I ran its language backbone through my GPTQ-Pro quantization framework, kept the audio-sensitive pieces in BF16, and compared it against the untouched model on the same RTX 3090.

The result: 53.87% less allocated GPU model memory. What I didn't get was a free speed boost—INT4 ASR was actually a little slower, and Spanish accuracy took a small hit in these tests. I'm publishing both sides of the story, along with the code, metrics and original audio.

Important: This is a selective native INT4 overlay, not a standalone Transformers/GGUF/vLLM checkpoint. You need the original BF16 LFM3-Audio-2B weights plus the 755 MiB GPTQ-Pro language-backbone overlay and the included custom loader.

Hear the difference

I generated five English and five Spanish phrases in both BF16 and INT4. Click an A/B comparison to hear the original BF16 first, a 1.2-second pause, and then GPTQ-Pro INT4. The two versions use the same prompt and seed, and the combined comparison is volume-matched. Individual stems keep their generated gain (with lossy Opus encoding).

English — five call scenarios

Scenario	Compare A → B	BF16 (A)	INT4 (B)
01. Answering the phone	▶ Listen	A	B
02. Appointment availability	▶ Listen	A	B
03. Spelling a surname	▶ Listen	A	B
04. Callback request	▶ Listen	A	B
05. Confirming an appointment request	▶ Listen	A	B

Español — cinco escenarios de atención telefónica

Situación	Comparar A → B	BF16 (A)	INT4 (B)
01. Atender la llamada	▶ Escuchar	A	B
02. Consultar turnos	▶ Escuchar	A	B
03. Deletrear el apellido	▶ Escuchar	A	B
04. Devolver la llamada	▶ Escuchar	A	B
05. Confirmar el pedido	▶ Escuchar	A	B

All 40 audio files and checksums · Every prompt, seed and roundtrip transcript

My RTX 3090 results: BF16 vs GPTQ-Pro INT4

Both versions ran on GPU 0 of the same RTX 3090. For ASR I used 50 English and 50 Latin American Spanish human-recorded FLEURS test utterances, artificially encoded to 8-kHz telephone-style bandwidth before resampling to 16 kHz. The FLEURS clips were not used for quantization calibration.

Measured metric	BF16 original	GPTQ-Pro INT4
Allocated GPU model tensors	3.683 GiB	1.699 GiB
English ASR WER (50 clips / 1,076 words)	4.554%	4.926%
Spanish ASR WER (50 clips / 1,259 words)	2.542%	3.098%
English ASR generation RTF	0.0348	0.0387
Spanish ASR generation RTF	0.0361	0.0408
English TTS generation RTF (five synthetic phrases)	0.2721	0.2701
Spanish TTS generation RTF (five synthetic phrases)	0.2535	0.2660

RTF = model-generation seconds / audio duration (lower is faster). It excludes audio decoding, preprocessing, VAD, networking and telephony. A low generation-only RTF is not a verified time-to-first-speech or end-to-end call latency figure.

My take: the memory saving is real; the latency advantage is not. For a single live voice agent on a 24 GiB RTX 3090, I'd currently choose BF16. INT4 becomes interesting when I want to fit more model instances in VRAM, but I haven't established the concurrency gains yet.

And how does the actual voice hold up?

I ran automatic word-recovery checks on the generated WAV files and verified there was no digital clipping in the ten selected pairs.

Five TTS phrases per language	BF16 word recovery	INT4 word recovery
English	96.7%	96.7%
Spanish	98.6%	94.3%

The Spanish ES_01 recording is worth hearing: BF16 recovered 92.9% of the intended words while INT4 recovered 71.4%, and INT4 spoke for 7.68 s versus 5.60 s. I kept that example in the gallery rather than hiding it.

These are self-ASR intelligibility proxies, because I used BF16 LFM3 itself to transcribe the generated speech. This is not independent human listening, MOS/CMOS, a speaker similarity evaluation, or proof of production call quality. See the detailed RTF and audio-quality analysis and per-sample signal measurements.

How I quantized it

Component	Format
142 language projections across 24 decoder layers	GPTQ-Pro symmetric INT4, group size 64
Attention and feed-forward projection weights	Native packed INT4 CUDA kernels
Audio encoder, projector, depth decoder and LiquidCodec	Kept BF16
Embeddings, output head, normalization, gates and convolutions	Kept at original precision
Calibration	16 separate synthetic English/Spanish call-style samples
Published quant data	24 safe-tensor layer shards, ~755 MiB

This is a post-training quantization, not a new fine-tune. The original Liquid AI checkpoint is pinned to revision 9d49701c372d6011d900f6abb40c6b296fb706dd. The exact quantization runtime source used for this release is bundled here, commit c49c4b7bb142371f9a2c2dec81558ec7edc89d65.

Run it yourself

I pushed the actual code to GitHub, not just a model card.

• GPTQ-Pro — native CUDA quantization and kernels
• LFM3 inference code — ASR, TTS, conversation and A/B
• Portable single-model script · BF16/INT4 A/B script · Quantization pipeline

Tested with Ubuntu, Python 3.12, CUDA 12.8, PyTorch 2.10.0, Transformers 5.21.0, NVIDIA RTX 3090. You'll need Linux, CUDA capability 8.0 or higher and a compatible compiler/toolkit for the native extension.

# Set up an environment and download the GPTQ-Pro overlay + scripts:
mkdir lfm3-int4 && cd lfm3-int4
uv venv .venv --python 3.12
uv pip install --python .venv/bin/python huggingface_hub
.venv/bin/hf download groxaxo/LFM3-Audio-2B-GPTQ-Pro-INT4 --local-dir .
uv pip install --python .venv/bin/python torch==2.10.0 \
  --index-url https://download.pytorch.org/whl/cu128
uv pip install --python .venv/bin/python -r vendor/GPTQ-Pro/requirements.txt
uv pip install --python .venv/bin/python \
  transformers==5.21.0 soundfile==0.14.0 librosa==1.0.0 \
  huggingface_hub sentencepiece
uv pip install --python .venv/bin/python -e vendor/GPTQ-Pro --no-deps

# Original BF16 model is downloaded automatically at its pinned revision.
CUDA_VISIBLE_DEVICES=0 .venv/bin/python hf_infer.py \
  --mode asr --language en --audio /path/to/16khz_mono_call.wav

# Generate Spanish audio:
CUDA_VISIBLE_DEVICES=0 .venv/bin/python hf_infer.py \
  --mode tts --language es \
  --text "Hola, ¿en qué puedo ayudarte?" --out spanish_reply.wav

# Make your own BF16 vs INT4 listening comparison:
CUDA_VISIBLE_DEVICES=0 .venv/bin/python infer_ab.py \
  --language en --seed 2026102902 \
  --text "Kia ora, thanks for calling!" --outdir my_comparison

The first run also needs to download the original approximately 3.9 GB BF16 checkpoint, because this repository ships only a compressed language-backbone overlay. Don't load this directly with AutoModel.from_pretrained(this_repo) or an ordinary GGUF/vLLM GPTQ loader; it requires the custom overlay_runtime.py. To verify the download, see the manifest and SHA-256 checksums.

Want to reproduce the numbers?

Evidence	Link
Human ASR dataset, WER, real-time factors and paired uncertainty	100-clip comparison
Generated TTS recordings, seeds and word-recovery checks	Ten-pair report
Signal-level quality checks, raw RTF calculations	RTF and quality report
Actual audio, separately or side-by-side	Listening gallery
Quantization recipe and Python inference	GitHub repository
Additional historical project record	experimentos (private)

What I'd test next

I want to measure genuine simultaneous callers, time to first speech, English/Spanish accent variation and round-trip phone noise—not just fast batch inference. The original BF16 model itself sometimes code-switches to English or gets weekdays wrong during simulated booking conversations, so I'd never let either version confirm an appointment without an authoritative backend.

Credits and licensing

Big thanks to Liquid AI for the original audio model and to the GPTQModel contributors behind the native quantization ecosystem. The quantization, local tests and release were done independently; this is not an official Liquid AI checkpoint.

The base model and redistributed derivative use the LFM Open License v1.0, including its commercial-use restrictions. My vendored GPTQ-Pro source carries its separate Apache-2.0 license. The FLEURS human-speech dataset is CC BY 4.0 and the recordings are not redistributed in this repo. See NOTICE.

---

Built, tested and shared by @groxaxo.
If you get this working on another GPU, come share your numbers in the GitHub issues. I'd love to compare results.