code-and-canvas/Spark-X2.6-4B-MLX-oQ6e-fp16

Spark-X2.6-4B oQ6e (MLX, fp16 KV)

This is a mixed-precision oQ quantisation of XHToken/Spark-X2.6-4B, converted for local inference on Apple Silicon with oMLX. It is the speed-oriented build: a 6-bit base with 32 selected tensors kept at 8 bits. It is the drafting model used by the LinkedIn agent on a 16 GB M2 Air.

The fp16 suffix refers to the KV cache dtype. The weights are affine quantised.

Quantisation

Property	Value
Weight format	6-bit affine base, group size 64
Higher-precision tensors	32 tensors at 8-bit (first layer plus selected attention and MLP projections)
KV cache	fp16
Model size on disk	3.2 GiB
Context window	65,536 tokens configured; prompts up to ~39.5k benchmarked
Architecture	Spark2_5ForCausalLM, 36 layers, hidden size 2,560
Source checkpoint	XHToken/Spark-X2.6-4B at 66e3ae7bee54e0a523d692cb8ef841f35409807b
Base model	XHToken/Spark-X2.6-4B-Base
License	Apache-2.0

Quantisation used the oMLX oQ pipeline with the same importance-matrix calibration data as the 8-bit build (oqe_code_multilingual, 1,024 samples, 512 tokens each). The 8-bit exceptions protect the layers most sensitive to quantisation, including the first layer and the g_proj and MLP projections in selected layers.

What is in this repository

• model.safetensors — the quantised weights
• config.json — Spark 2.5 model configuration, including the per-tensor quantisation map
• tokenizer files and chat_template.jinja
• configuration_spark.py and modeling_spark.py — the reference Spark 2.5 implementation
• oq_imatrix_report.json — quantisation calibration metadata
• LICENSE
• model.safetensors SHA-256: 0f162094003ade5f2a1851f38be62f8cf12f72486bfd5be9fa6bb9ce90c1b060

auto_map is intentionally omitted from config.json. Transformers will not load these weights through remote code. Use an oMLX build with native Spark 2.5 support, or an MLX runtime that implements spark2_5.

Running with oMLX

# Download this repository somewhere oMLX can see it.
hf download code-and-canvas/Spark-X2.6-4B-oQ6e-fp16 --local-dir ~/.omlx/models/Spark-X2.6-4B-oQ6e-fp16

# Start the OpenAI-compatible server.
omlx serve --model-dir ~/.omlx/models --port 8081

Then call the model as Spark-X2.6-4B-oQ6e-fp16:

curl -s http://127.0.0.1:8081/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Spark-X2.6-4B-oQ6e-fp16",
    "messages": [
      {"role": "user", "content": "Explain why a prefix cache helps long prompts."}
    ],
    "max_tokens": 256,
    "temperature": 0.7,
    "chat_template_kwargs": {"enable_thinking": false}
  }'

For this build, keep thinking off for structured output and tool calls. High thinking can hit the output budget on coding tasks; with thinking off the quicksort test passes 8/8.

Evaluation

Measured on a 16 GB M2 Air with oMLX 0.7.2, fp16 KV, prefix cache enabled, and 64 output tokens per generation. These are local results, not a claim about other hardware.

Suite	Result
Coding: strict duration parser	12 / 21
Coding: merge intervals	8 / 8
Reasoning + coding: quicksort	0 / 8 with high thinking (budget loop); 8 / 8 with thinking off in 13.2 s
Research synthesis	18 / 20
Prose launch post	17 / 20
Tool use: on-call flow	5 / 5

Generation throughput:

Prompt size	Throughput
Short (68 tokens)	22.02 tok/s
~1.3k tokens	21.11 tok/s
~19.8k tokens	14.72 tok/s
~39.5k tokens	14.53 tok/s

The full 65,536-token context was not benchmarked. Long prompts are limited by prefill time on the 16 GB machine.

Intended use

• Local text generation, coding help, summarisation, and agentic workflows on Apple Silicon.
• Latency-sensitive drafting where the 3.2 GiB memory footprint matters.
• Applications where data must stay on the machine.
• Drafting and rewriting tasks where a human reviews the output.

Limitations

• This is a 4B-parameter model. It is not a substitute for a larger model on hard reasoning, long-horizon planning, or high-stakes factual work.
• Mixed 6/8-bit quantisation can change behaviour more than the pure 8-bit build. Validate it on your own task before relying on it.
• High thinking is unreliable for structured coding tasks on this build. Use thinking off unless you have tested the specific task.
• The source model was trained mainly on English and Chinese. Other languages were not evaluated.
• The model can hallucinate facts, numbers, citations, and tool arguments. Grounding and review are the application's responsibility.
• The full 65,536-token window was not benchmarked; the longest measured prompt was about 39.5k tokens. The source configuration advertises a much larger maximum, which was not tested here.

Attribution

The base model is XHToken/Spark-X2.6-4B, released under Apache-2.0. This repository is a quantised derivative produced with oMLX. Please retain the original license and attribution when redistributing.

Model card and quantisation: Code & Canvas.