Sieve-2B-Plus

Sieve-2B-Plus is a decision model. It takes a state, a typed question and a closed set of options, and returns a calibrated probability for every option. It generates no text. It is the successor of Sieve-2B: the same architecture, trained on a much broader mix of decision tasks, with a larger adapter that also covers the Gated DeltaNet layers.

	parameters
run at inference (embeddings + blocks 1-16 of the base, adapter, head)	1.47 B
trained (LoRA 44,851,200 + scorer head 1,053,697)	45.9 M
full base model (Qwen/Qwen3.5-2B-Base text model), for reference	1.88 B

Not affiliated with TypeSafe AI.

How it works

The state and the question are encoded once. Every option is listed in the prompt, in an order fixed by a hash of each option's text. The encoded state is then copied once per option, and each option is scored as its own continuation of that copy. No option sees another, so reordering the options cannot change the answer: that holds by construction, not by training. A scalar head reads the last token of each option; a softmax over the options, divided by one built-in temperature, gives the probabilities.

from modeling_sieve import Sieve        # needs: transformers==5.17, torch, peft, safetensors, huggingface_hub

s = Sieve.from_pretrained("sthanika-ai/Sieve-2B-Plus")   # downloads Qwen3.5-2B-Base once, applies this adapter + head
s.choice(
    state="Customer: my invoice was charged twice and nobody answers the phone!",
    question="Which team should handle this?",
    options=["Billing", "Technical support", "Sales"],
)

Yes/no questions are answered as a 2-option choice. Inputs up to 32,768 tokens are scored; training used inputs up to 8,192.

Training

• Base: Qwen/Qwen3.5-2B-Base at revision b1485b2f, a hybrid of Gated DeltaNet and full-attention blocks; only blocks 1-16 of 24 run.
• Adapter: LoRA r=64, α=128 on the attention, MLP and Gated DeltaNet projections of blocks 1-16, plus a scalar head (LayerNorm → 512 → 1).
• Recipe: 1 epoch, AdamW, lr 5e-5 with 3% warmup and cosine decay to 10%, batch 8 questions, gradient-norm clip 1.0, trained through the same option fork that serves the model.
• Data: 163,425 questions from 154 decision tracks, sampled at 40% per track from a public + synthetic mix and ordered so every part of the epoch sees every track. It covers intent and topic classification, NLI and fact-checking, commonsense and knowledge questions, safety and moderation, preference judging, tool selection and routing, retrieval and long documents, and rule-based synthetic reasoning (dates, numbers, policies, lookups).
  • 36,984 questions (22.6%) come from the train or dev splits of datasets whose test splits appear in the Decision Index: HellaSwag 8,246, WinoGrande 8,246, iSarcasmEval 5,318, CLINC150 3,027, BANKING77 2,457, ESCI 2,374, GSM8K 2,024, SGD 1,408, RAGTruth 1,098, HoVer 968, ARC 906, When2Call 470, ContractNLI 442. No test split was used.
  • Each public dataset keeps its own licence.

Calibration

One temperature, T = 1.4785, fitted by NLL on a 9,976-question calibration split that covers every track and was never trained on. It never changes which option wins. On a separate 2,560-question validation split:

	NLL	ECE	confidence − accuracy	accuracy
raw (T = 1)	0.393	0.041	+0.039	85.08%
built in (T = 1.48)	0.371	0.012	−0.003	85.08%

Separate temperatures for yes/no questions or by option count did not do better, so the model ships one.

Decision Index

Self-run with the official kit, edition 0.3 public suite (37 benchmarks, 140,178 requests): public index 37.26, complete, 2 requests refused (options with identical text). The 0.3 board's main ranking also includes private tests that only the maintainers run.

area	0.3 public	0.2.1
Knowledge & Reasoning	16.6	21.0
Language Understanding	35.9	37.2
Retrieval & Classification	57.4	57.4
Tools & Automation	56.6	56.6
Arts & Human Taste	18.5	17.8
Index	37.26	38.65

Results: sthanika-ai/Sieve-2B-Plus-decision-index-results.

GSM8K. Under 0.2.1 this model scores 88 on GSM8K; under 0.3, where the wrong options are answers to other GSM8K problems, it scores 4 (accuracy at chance). Its GSM8K training rows used the 0.2.1 style of wrong options (fixed offsets of the right answer), and the model learned to recognise that pattern rather than to solve the problems. Treat it as having no GSM8K-level arithmetic.

Latency (single process, one request at a time, bf16, A100 80GB, 400 uniformly sampled suite requests): median 97 ms, mean 124 ms, 80th percentile 110 ms.

Limitations

• No arithmetic beyond what is in the options (see GSM8K above). Knowledge-heavy benchmarks (MMLU-Pro, GPQA, HLE) remain weak.
• Single training run, one seed.
• The 0.2.1 score of 38.65 is inflated by GSM8K; the 0.3 public index is the better number.
• Requires transformers==5.17 (qwen3_5 architecture) and peft>=0.21.

License

Apache-2.0 for the adapter, head and code in this repository. The Qwen3.5 base model is Apache-2.0. The training datasets keep their own licences.