vistalab-system-one-e4b

A Turkish model for System One requests, fine-tuned from Gemma 4 E4B by VistaLab. Given a text and a question with named answer options, it is used by reading the probabilities it gives the options' letters in one forward pass. It is not meant for chat or free-form generation.

On system-one-bench-tr it scores 75.4% accuracy on the 25 main tasks, 6.5 points above its base model and 2.2 below Jev (TypeSafe's System One model), with a lower raw log-loss than Jev (0.62 against 0.81). It is weaker than Jev on knowledge and politics. A larger version is vistalab-system-one-12b.

Request format

A request is a text (the state: a message, a document, or an object of named fields) and one or more questions. Each question is one of three kinds:

• choice: one of several named options, each with a criterion that says when it applies;
• noul: yes or no, with a criterion for each (true and false);
• score: a level on an ordered scale, each level described.

The options are shown as lettered lines (A, B, C …), and the probability of an answer is the probability the model gives its letter as the next token. Choice and noul questions are asked a second time with the options reversed, and the two answers are averaged.

Using it with s1bench

The benchmark's package runs the model exactly as it was evaluated. The model is public, so no token is needed to download it. The outputs shown below come from this model, rounded to two decimals. The weights take 16 GB in bfloat16; use a CUDA GPU with at least 24 GB.

pip install git+https://github.com/UlkuTuncerKucuktas/system-one-bench-tr

from s1bench.llm import LLM

model = LLM("UlkuTuncerKucuktas/vistalab-system-one-e4b")
request = {
    "id": "1",
    "state": "Siparişim 10 gündür kargoda görünüyor. İptal edip paramı geri almak istiyorum.",
    "questions": {
        "talep": {
            "type": "choice",
            "instructions": "Müşteri bu mesajla ne istiyor?",
            "criteria": {
                "iade": "İade: Müşteri siparişi iptal edip parasını geri almak istiyor.",
                "kargo": "Kargo: Müşteri siparişinin nerede olduğunu öğrenmek istiyor.",
                "değişim": "Değişim: Müşteri ürünü başka bir ürünle değiştirmek istiyor.",
                "diğer": "Diğer: Mesaj bu seçeneklerin hiçbirine uymuyor.",
            },
        },
        "acil": {
            "type": "noul",
            "instructions": "Mesaj hemen ilgilenilmesi gereken bir durum mu bildiriyor?",
            "criteria": {
                "true": "Evet: Para kaybı, güvenlik ya da yasal bir risk var.",
                "false": "Hayır: Beklemesi zarar vermeyecek sıradan bir talep.",
            },
        },
    },
}
print(model.predict([request])[0])
# {'talep': {'iade': 0.96, 'kargo': 0.01, 'değişim': 0.0, 'diğer': 0.03}, 'acil': {'true': 0.39, 'false': 0.61}}

predict takes a list of requests and batches them. A score question lists its levels in order, and the answer keys are the level numbers, counted from 0:

review = {
    "id": "2",
    "state": "Ürün zamanında geldi ve fiyatına göre kaliteli, ama kutusu ezilmişti.",
    "questions": {
        "yıldız": {
            "type": "score",
            "instructions": "Yorumcu ürüne kaç yıldız verir?",
            "criteria": ["1 yıldız: Çok kötü.", "2 yıldız: Kötü.", "3 yıldız: Ne iyi ne kötü.", "4 yıldız: İyi.", "5 yıldız: Çok iyi."],
        },
    },
}
print(model.predict([review])[0])
# {'yıldız': {'0': 0.01, '1': 0.04, '2': 0.28, '3': 0.59, '4': 0.08}}

Using it with transformers only

The same readout in plain transformers:

import json

import torch
from transformers import AutoModelForImageTextToText, AutoTokenizer

REPO = "UlkuTuncerKucuktas/vistalab-system-one-e4b"
LETTERS = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
tokenizer = AutoTokenizer.from_pretrained(REPO)
model = AutoModelForImageTextToText.from_pretrained(REPO, dtype=torch.bfloat16, device_map="auto")


def letter_ids(letter):
    ids = {tokenizer.convert_tokens_to_ids(token) for token in [letter, "▁" + letter, "Ġ" + letter]}
    return [i for i in ids if i is not None and i != tokenizer.unk_token_id]


@torch.no_grad()
def ask(state, instructions, options):
    # options: {answer: criterion}; returns {answer: probability}
    text = state if isinstance(state, str) else json.dumps(state, ensure_ascii=False, indent=2)
    lines = [f"{letter}) {criterion}" for letter, criterion in zip(LETTERS, options.values())]
    prompt = f"{text}\n\n{instructions}\n\n" + "\n".join(lines) + "\n\nYalnızca doğru seçeneğin harfini yaz."
    chat = tokenizer.apply_chat_template([{"role": "user", "content": prompt}], tokenize=False, add_generation_prompt=True, enable_thinking=False)
    inputs = tokenizer(chat, return_tensors="pt", add_special_tokens=False).to(model.device)
    logits = model(**inputs).logits[0, -1].float()
    scores = torch.stack([logits[letter_ids(letter)].logsumexp(0) for letter in LETTERS[: len(options)]])
    return dict(zip(options, scores.softmax(0).tolist()))


def decide(state, instructions, options):
    # choice and yes-or-no questions are asked in both option orders and averaged, as in the benchmark
    forward = ask(state, instructions, options)
    backward = ask(state, instructions, dict(reversed(options.items())))
    return {answer: (forward[answer] + backward[answer]) / 2 for answer in options}


print(decide(
    "Tebrikler! 5.000 TL kazandınız. Ödülünüzü almak için hemen bu bağlantıya tıklayın.",
    "Bu mesaj spam mı?",
    {"true": "Evet: İstenmeyen reklam, dolandırıcılık ya da toplu gönderilmiş bir mesaj.", "false": "Hayır: Kişisel ya da beklenen bir mesaj."},
))
# {'true': 0.96, 'false': 0.04}

For a yes-or-no question put the yes criterion first, under true. For a score question call ask once, with the levels in ascending order.

Notes on use

• Give every answer a criterion; the model compares the text with them. Add "diğer" (other) or "belirtilmemiş" (not stated) when the right answer may be missing.
• The probabilities were fairly well calibrated on the benchmark. Check them on your own data before acting on a threshold.
• Several questions about the same text can share one request.
• The model was trained on the format above (lettered options, one criterion per line, the Turkish closing line); other formats were not tested.

Training

• Base: google/gemma-4-E4B-it, with a rank-32 LoRA on the attention and MLP layers of the language model, merged into the weights.
• Data: 50,000 real questions from about 40 public Turkish datasets and 163,680 synthetic questions written and labelled by open-weight models (vistalab-system-one-synthetic). Training texts that copy a benchmark text were removed.
• One pass over 213,337 questions in batches of 32, learning rate 5e-5, with the loss on the answer-letter probabilities against the (soft) labels; about 3 hours on one H200.
• Code: vistalab-system-one, python -m vistalab.train e4b.

Evaluation

system-one-bench-tr, test split. Accuracy (%) on the 25 main tasks and by area, raw log-loss on the main tasks (lower is better).

Model	Accuracy	Raw log-loss	Routing	Opinion	Safety	Fact-checking	Judging	Workflow	Politics	Language	Knowledge
vistalab-system-one-e4b	75.4	0.62	87.8	63.3	92.1	57.5	79.5	89.7	76.6	76.3	66.4
Jev	77.6	0.81	87.4	62.4	90.6	51.2	81.7	87.3	86.8	78.9	86.9
Gemma 4 E4B (base)	68.9	1.61	83.2	59.1	81.7	44.4	72.0	86.1	80.2	69.1	63.0

Two training runs of this recipe for the 12B scored 77.6 and 78.1, so differences of under a point are within run-to-run variation.

Limitations

• Evaluated only on system-one-bench-tr. The model never saw a benchmark question, but for a few tasks it was trained on other splits of the same datasets (the training splits of TrGLUE and Lavoir) or on other texts from the same sources (other WebFAQ sites, fact-check reports about other claims).
• Weaker than its base model on politics, and well behind Jev on exam-style knowledge (66.4% against 86.9%).
• Trained for Turkish; other languages were not tested.

License

CC BY-NC 4.0, because several training sources are non-commercial. The base model, Gemma 4, is released by Google under Apache 2.0.