Spaces:

---

AxiomicLabs

/

Open_SLM_Leaderboard

Like 156

Running

Adds TinyBrainBot-3M-Loop-v2 (tinybrainbot org): 2,964,481 parameters, trained from scratch, open weights, custom code (trust_remote_code=True).

This follows #260. Two things changed since then.

1. A bug in my model code, which I think explains the 27% on ArithMark-2. The old modeling file ignored attention_mask, so left-padded batches broke it. On the same 600 AM2 items it scored 56.3% right-padded and 35.5% left-padded. It now masks padding and counts positions from each row's first real token, so left and right padding give identical scores (83.2% / 83.2% on those items). The fix is in both repos.

2. The arithmetic now generalizes beyond ArithMark-3's format. I continued the #260 model from its pre-decay checkpoint for 1.5B more tokens, mostly math. The new data comes from my own generators, not from any benchmark's problems or code: bare expressions with order of operations, parentheses, negatives and 2-5 operands; number words; and the existing story problems with a quarter of their numbers in words.

Intelligence Index: 20.15

Math beyond ArithMark-3, all zero-shot and scored the same way for every model (my runs):

On ArithMark-1: its answer key mostly evaluates expressions left to right (14 * -17 + -1 * -18 = 4302), while AM2 uses operator precedence (25 + 10 * 5 = 75). This model follows precedence, so it is weak on AM1's longer expressions. I didn't train toward that.

This round aims at general arithmetic across the public benchmarks, not at the private set. If it still drops on HD1, that's useful to know, and I'll move on to general models.

All scores are zero-shot on the full sets: lm-eval 0.4.13 in float32 plus the official ArithMark scripts, re-run on the published Hub copy.

To reproduce:

lm_eval --model hf --model_args pretrained=nkthebass/tinybrainbot-3m-loop-v2,dtype=float32,trust_remote_code=True --tasks arc_easy,arc_challenge,hellaswag,piqa --num_fewshot 0
python bencharithmark-3.py --model nkthebass/tinybrainbot-3m-loop-v2 --dtype float32
python benchmark_arithmark-2.0.py --model nkthebass/tinybrainbot-3m-loop-v2