halonc/Qwen3.8-27B-W3Rot-INT3-Paiton-RDNA4-abliterated

Qwen3.8-27B abliterated — Paiton W3Rot INT3 (W3A4) overlay

A 3-bit overlay for the Paiton paiton-w3rot-v1 format, built from the huihui-ai abliterated checkpoint and calibrated with GPTQ (block-diagonal Hessian, 65,536 tokens).

It is an overlay, not a full model: only the attention / GDN / MLP projections are here (64 shards, 1,296 tensors). embed_tokens, lm_head, norms, A_log, conv1d and dt_bias still come from the NVFP4 base (unsloth/Qwen3.8-27B-NVFP4, revision f0b7c9e).

Usage

Mount this directory at /models/w3rot-abl-gptq, mount the NVFP4 base at /models/target, and set PAITON_W3_A4_DIR=/models/w3rot-abl-gptq in the Paiton container. The runtime merges the overlay over the base; no other change is needed.

Results

Same harness, one Radeon AI PRO R9700 (32 GB), 262K context profile, thinking off:

Benchmark	This overlay	Official W3A4
MMLU-Pro, 14×200 random, 0-shot	57.86%	57.82%
GSM8K 5-shot (1,319)	93.10%	95.30%
HumanEval pass@1 (164)	90.24%	93.90%
Needle @ 61,440 tokens (160)	160/160	100

Single-stream decode (BetterBench 0.6.1, 20 passes, weighted): 155.0 tok/s vs 174.3 for the official 262K mode on the same card; 8-way concurrency 436.8 vs 458.9. The gap is host-side (virtualized guest on an older Xeon), not the weights: the official weights measured in the same container run at the same speed.

Build

Scripts and notes: https://github.com/halonc/paiton-vllm-plugin-27b-w3a4-abliterated

Licence and caveats

Apache-2.0. Derived from Apache-2.0 checkpoints; the upstream Paiton plugin is Apache-2.0. Abliteration removes a refusal direction, not policy; outputs are the user's responsibility. Research and personal use only.