eexian/gemma4-31b-w4-weighted-reasoning-lora

Gemma 4 31B W4 — weighted reasoning LoRA, checkpoint 246

Final adapter from the completed three-GPU training run on 2026-10-21. This repository contains the PEFT adapter, not a merged base model.

Base and adapter

• Base: google/gemma-4-31B-it-qat-w4a16-ct
• Pinned base revision: 52f3f65bc7a02d555763bc923bd1d9094898219d
• Base weight SHA256: 1b9b1d622a93f02c0d33f98e502f233b5d707443af6ddc464ed0bf5498506c20
• LoRA rank/alpha: 16/16; dropout: 0.05
• Targets: language-model attention projections (q_proj, k_proj, v_proj, o_proj) and MLP projections (gate_proj, up_proj, down_proj).
• Export: 820 FP32 tensors, 489,840,816 bytes.
• Adapter SHA256: 6a9414b2c6bd0b13f524ed00a6992c57efe8dc6c423d7c4b87a4ee083661c3f7

Training decoded the pinned W4 base into frozen BF16 weights and optimized only the LoRA parameters. Deployment uses the original packed W4 base plus the adapter through native LoRA loading. It does not merge or requantize the base weights.

Training

• Code: TheWayLost/gemmacodeagent.
• Dataset: Lanzo-T-H/dpsk_untitledagent_data_lanzo, revision d912c71174a4522199f07470dfb1b82eb17050b3.
• Training split: 180 tasks and 2,951 decision rows; the separate 20-task validation split was held out.
• One epoch, 246 optimizer updates on three PRO 6000 GPUs. Microbatch 1/GPU, accumulation 4, effective global batch 12. Distributed sampling pads one example.
• Learning rate: 2e-5; weight decay: 0.01; gradient clipping: 1; seed: 42.
• Final recorded training loss: 0.3582274. This is a training metric, not a task solve rate.
• Cross-entropy chunk checkpointing reduced activation memory without changing tokens, sample weights, or the objective; numerical checks and a three-rank stress test passed.

Loading

Use an inference runtime that supports both this Gemma 4 compressed-tensors checkpoint and Gemma 4 LoRA. The tested runtime is the competition-patched vLLM 0.19.1 with Transformers 5.13.1 and compressed-tensors 0.15.0.1. Compatibility with an arbitrary upstream runtime is not asserted.

Native vLLM options include:

--enable-lora --max-lora-rank 16 --lora-dtype auto
--lora-modules gemma4_w4_lora246=/absolute/path/to/this/adapter

auto loads the FP32 export in the base model's inference dtype. Keep the original base tokenizer and quantization metadata; do not apply Qwen-specific adapter-key remapping.

For the Kaggle Gemma 4 Developer Agent competition, place both adapter files in adapters/gemma4_w4_lora246/ and use:

model: gemma-4-31b-it-qat-w4a16-ct
adapter: gemma4_w4_lora246

Validation status

The final checkpoint passed the W4 loading/control test: zero-B LoRA versus the base had maximum raw-logit delta 0.0; trained versus zero-B LoRA had maximum raw-logit delta 2.5 on two fixed prompts. Native served-route effect, tool calls, thinking-budget enforcement, and four-request concurrency were also checked. These checks establish loading and adapter activation, not repair accuracy.

Repair evaluation on the held-out 20 tasks and the separate 129 Kaggle tasks is in progress. No final solve-rate claim is made in this release.

Attribution and terms

Consult the base model's terms and the training dataset's LICENSE.md and sources.json. The dataset combines generated trajectories and material from SWE-Gym, R2E-Gym, and SWE-rebench sources with mixed upstream terms. This card does not relicense third-party data or repository code.