ModelCloud/Ripple-65B-Preview-GPTQ-W4A16-v2

Ripple-65B-Preview — GPTQ W4A16

4-bit GPTQ (W4A16) quantization of ricursiveAI/Ripple-65B-Preview, produced with GPTQModel.

Ripple is a thin-and-deep hybrid-attention model. Beyond a Llama-style block it uses PeriNorm (four RMSNorms per layer), gated attention (g_proj), per-head Q/K RMSNorm, and a value-residual blend on v_proj. Eight projections are quantized per layer; all norms and the value-residual scalar stay in their original precision.

Benchmark results

Full test splits, greedy decoding, no answer-format hints. Both runs use ModelCloud/Evalution 0.0.17.

Benchmark	Samples	Metric	Score
MMLU-Pro	12,032 (full test split)	exact match	70.20% (8,446 / 12,032)
GSM8K Platinum	1,209 (full test split)	accuracy	95.29% (1,152 / 1,209)

MMLU-Pro is 5-shot with max_new_tokens=1024; GSM8K Platinum is 8-shot CoT with max_new_tokens=256. Each run was split into four strided shards and merged with deduplication, so every row of the test split is scored exactly once. Per-shard breakdowns and the exact scoring rules are in Evaluation.

Quantization

Setting	Value
Method	GPTQ
Bits	4
Group size	128
desc_act	false
Symmetric	yes
Pack dtype	int32
lm_head	not quantized (16-bit)
Calibration	256 samples (90,384 tokens)

The checkpoint is stored in the packed GPTQ layout (qweight, scales, qzeros, g_idx). There are no dequantized/expanded weight caches.

Usage

Ripple ships custom modeling code, so trust_remote_code=True is required.

from gptqmodel import GPTQModel

model = GPTQModel.load(
    "ModelCloud/Ripple-65B-Preview-GPTQ-W4A16-v2",
    trust_remote_code=True,
    device="cuda:0",
)

ids = model.tokenizer("The capital of France is", return_tensors="pt").input_ids
print(model.tokenizer.decode(model.generate(ids, max_new_tokens=32, do_sample=False)[0]))

Note on generate() The original upstream modeling_ricursive.py slices input_ids[:, cache_position] even when cache_position is None, which adds a spurious axis and crashes the first prefill under Transformers 5.x. The copy shipped in this repository already carries the Transformers 5.x fixes, so generate() and KV-cached decoding work out of the box. GPTQModel still installs its compatibility patch at load time for checkpoints that carry the original file; the patched file declares _ricursive_tf5_native_fix so the two do not stack.

Transformers 5.x compatibility

modeling_ricursive.py in this repository is not the original upstream file. It contains the following fixes:

• prepare_inputs_for_generation no longer slices input_ids with a None cache_position, rebuilds per-row position_ids from the padding mask, and moves inputs onto the model device.
• _update_causal_mask sizes the key axis from cache_position instead of the removed get_max_cache_shape(), and no longer masks an entire single-token decode step.
• _apply_sliding_window_mask anchors key positions at the first key the cache still holds after eviction, and is skipped when the window cannot bind.
• RicursiveForCausalLM inherits GenerationMixin, which Transformers 5 no longer injects into remote-code classes.

These changes are proposed upstream in ricursiveAI/Ripple-65B-Preview PR #1. The modeling_ricursive.py shipped here is byte-identical to that PR's file, and a drop-in copy plus a unified diff are mirrored at ModelCloud/Ripple-65B-Preview-tf5-patch.

The same fix is available in GPTQModel PR #3535, now merged into main (merge commit 6d483dc4), as a load-time runtime patch for checkpoints that still ship the original upstream file. Both implementations were compared on the same checkpoint and produce bit-identical logits and tokens across prefill, cached decode, uncached decode and padded batches.

Evaluation

Benchmarks were run with ModelCloud/Evalution.

Component	Version
GPTQModel (quantization)	7.9.0 + 8cf756c6f
GPTQModel (evaluation)	7.9.0 + 38b7bf8a9
GPTQModel PR #3535	merged into main at 6d483dc4
Evalution	0.0.17
Transformers	5.22.1
Torch	2.15.0+cu132

Results

Benchmark	Setting	Samples	Metric	Score
MMLU-Pro	mmlu_pro, 5-shot, greedy	12,032 (full test split)	exact match	70.20% (8446/12032)
GSM8K Platinum	gsm8k_platinum_cot, 8-shot CoT, greedy	1,209 (full test split)	accuracy	95.29% (1152/1209)

MMLU-Pro per-shard breakdown:

Shard	Samples	Correct	Accuracy
0/4	3,008	2,147	71.38%
1/4	3,008	2,100	69.81%
2/4	3,008	2,092	69.55%
3/4	3,008	2,107	70.05%
Total	12,032	8,446	70.20%

GSM8K Platinum per-shard breakdown:

Shard	Samples	Correct	Accuracy
0/4	303	290	95.71%
1/4	302	288	95.36%
2/4	302	284	94.04%
3/4	302	290	96.03%
Total	1,209	1,152	95.29%

Both runs used bfloat16 and no chat template. GSM8K Platinum generated at batch size 8 with max_new_tokens=256 and scored with numeric-format-insensitive answer matching against madrylab/gsm8k-platinum (test). MMLU-Pro generated at batch size 12 with max_new_tokens=1024 and scored on the option letter.

MMLU-Pro scoring note The mmlu_pro implementation in Evalution 0.0.17 needs two corrections for this model, both applied here: row_indices sharding support (so the 12,032 rows can be split across workers) and an answer-boundary trim. Ripple keeps writing new few-shot examples after answering, each ending in the answer is (X), so scoring the last match in the whole response grades a hallucinated follow-up. The prompt section separator \n\n\nQuestion: is used as a stop string and the response is trimmed at it before the option letter is extracted. The reported 70.20% is with that fix.

Ripple's ricursive model definition and the generate() compatibility patch were merged in GPTQModel PR #3535 on top of 7.9.0. Install GPTQModel from main, or pin the merge commit for reproducibility:

pip install "git+https://github.com/ModelCloud/GPTQModel.git@6d483dc483c87ed1cfce278caee14f43dc459007"

Install the evaluator:

pip install evalution==0.0.17

Run script

This is the sharded launcher used for the reported numbers, with the local checkout path generalized. It runs one strided shard of an Evalution task per worker, so four workers cover the 1,209-row test split without overlap. Run it from the root of a GPTQModel checkout, or point GPTQMODEL_ROOT at one.

"""Run one strided shard of an Evalution task on one GPU."""
import argparse, json, os, sys, time

sys.path.insert(0, os.environ.get("GPTQMODEL_ROOT", "."))
from tests.eval import evaluate, get_eval_task_results

p = argparse.ArgumentParser()
p.add_argument("--model", required=True)
p.add_argument("--task", default="gsm8k_platinum_cot")
p.add_argument("--shard", type=int, required=True)
p.add_argument("--num-shards", type=int, default=4)
p.add_argument("--total-rows", type=int, required=True)
p.add_argument("--max-new-tokens", type=int, default=256)
p.add_argument("--batch-size", type=int, default=8)
p.add_argument("--num-fewshot", type=int, default=None)
p.add_argument("--out", required=True)
a = p.parse_args()

rows = tuple(range(a.shard, a.total_rows, a.num_shards))
device = os.environ.get("EVAL_DEVICE", "cuda:0")
suite_kwargs = {"max_new_tokens": a.max_new_tokens, "stream": True, "row_indices": rows}
if a.num_fewshot is not None:
    suite_kwargs["num_fewshot"] = a.num_fewshot

t0 = time.time()
raw = evaluate(
    model_or_id_or_path=a.model,
    tasks=[a.task],
    backend="auto",
    batch_size=a.batch_size,
    trust_remote_code=True,
    apply_chat_template=False,
    model_args={"dtype": "bfloat16", "device": device, "trust_remote_code": True},
    suite_kwargs=suite_kwargs,
)
res = get_eval_task_results(raw)
print("RESULT_JSON", json.dumps(res), flush=True)
print(f"SHARD {a.shard} rows={len(rows)} elapsed={time.time() - t0:.0f}s", flush=True)
json.dump(raw, open(a.out, "w"), indent=2)

Launch four shards, one per GPU:

for s in 0 1 2 3; do
  EVAL_DEVICE=cuda:$s python shard_eval.py \
      --model ModelCloud/Ripple-65B-Preview-GPTQ-W4A16-v2 \
      --task gsm8k_platinum_cot --shard $s --num-shards 4 --total-rows 1209 \
      --batch-size 8 --max-new-tokens 256 --out shard_$s.json &
done
wait

Merge and score the shards, deduplicating by prompt/target and summing acc,num:

import hashlib, json, sys

expected = 1209
correct, seen = 0.0, {}
for f in sys.argv[1:]:
    for t in json.load(open(f))["tests"]:
        for s in t["samples"]:
            key = hashlib.sha1((s["prompt"] + "\x00" + str(s["target"])).encode()).hexdigest()
            if key in seen:
                raise SystemExit(f"duplicate sample across shards: {f} vs {seen[key]}")
            seen[key] = f
            correct += float(s["scores"].get("acc,num", 0.0))
n = len(seen)
print(f"rows={n} correct={correct:.0f} acc={correct/n:.4f}")
if n != expected:
    print(f"WARNING: expected {expected} rows, got {n}")

tests.eval resolves the benchmark data locally when it is available and otherwise falls back to the public dataset, so the script works on a clean machine.

Files

config.json                     model config (model_type: ricursive)
quantize_config.json            GPTQ settings
model-0000N-of-00009.safetensors  9 packed shards
model.safetensors.index.json
tokenizer.json / tokenizer_config.json / special_tokens_map.json / chat_template.jinja
configuration_ricursive.py      upstream custom config code
modeling_ricursive.py           custom modeling code with the Transformers 5.x fixes

License

Apache-2.0, inherited from the base model.