Instructions to use webmp3/Sakura-MicroQuality-Kolibri-1-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use webmp3/Sakura-MicroQuality-Kolibri-1-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf webmp3/Sakura-MicroQuality-Kolibri-1-GGUF:IQ2_XS
# Run inference directly in the terminal:
llama cli -hf webmp3/Sakura-MicroQuality-Kolibri-1-GGUF:IQ2_XS

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf webmp3/Sakura-MicroQuality-Kolibri-1-GGUF:IQ2_XS
# Run inference directly in the terminal:
llama cli -hf webmp3/Sakura-MicroQuality-Kolibri-1-GGUF:IQ2_XS

Build from source code

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf webmp3/Sakura-MicroQuality-Kolibri-1-GGUF:IQ2_XS
# Run inference directly in the terminal:
./build/bin/llama cli -hf webmp3/Sakura-MicroQuality-Kolibri-1-GGUF:IQ2_XS

Use Docker

docker model run hf.co/webmp3/Sakura-MicroQuality-Kolibri-1-GGUF:IQ2_XS

Llama new
LM Studio
Jan

vLLM

How to use webmp3/Sakura-MicroQuality-Kolibri-1-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "webmp3/Sakura-MicroQuality-Kolibri-1-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "webmp3/Sakura-MicroQuality-Kolibri-1-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/webmp3/Sakura-MicroQuality-Kolibri-1-GGUF:IQ2_XS

Ollama
How to use webmp3/Sakura-MicroQuality-Kolibri-1-GGUF with Ollama:
```
ollama run hf.co/webmp3/Sakura-MicroQuality-Kolibri-1-GGUF:IQ2_XS
```
Unsloth Desktop

How to use webmp3/Sakura-MicroQuality-Kolibri-1-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf webmp3/Sakura-MicroQuality-Kolibri-1-GGUF:IQ2_XS

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "webmp3/Sakura-MicroQuality-Kolibri-1-GGUF:IQ2_XS"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use webmp3/Sakura-MicroQuality-Kolibri-1-GGUF with Docker Model Runner:
```
docker model run hf.co/webmp3/Sakura-MicroQuality-Kolibri-1-GGUF:IQ2_XS
```

Lemonade

How to use webmp3/Sakura-MicroQuality-Kolibri-1-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull webmp3/Sakura-MicroQuality-Kolibri-1-GGUF:IQ2_XS

Run and chat with the model

lemonade run user.Sakura-MicroQuality-Kolibri-1-GGUF-IQ2_XS

List all available models

lemonade list

Hermes Agent

How to use webmp3/Sakura-MicroQuality-Kolibri-1-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf webmp3/Sakura-MicroQuality-Kolibri-1-GGUF:IQ2_XS

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default webmp3/Sakura-MicroQuality-Kolibri-1-GGUF:IQ2_XS

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use webmp3/Sakura-MicroQuality-Kolibri-1-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf webmp3/Sakura-MicroQuality-Kolibri-1-GGUF:IQ2_XS

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "webmp3/Sakura-MicroQuality-Kolibri-1-GGUF:IQ2_XS" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Sakura MicroQuality — Kolibri-1 (GGUF, three sizes)

Made in Germany: Kolibri-1 is the open 78B model of the German company Aleph Alpha, a mixture-of-experts model (3.5B active parameters, 384 routed experts plus one shared expert per layer) for German and English, with a thinking mode and tool calling. This repository holds three GGUF files of different size made from the Q8_0 GGUF of the original model (near-lossless; all quality numbers use it as the reference) with our measured mixed-codec method. Pick one on the Files page; the table below says what each file is. Independent community quantization, not an official Aleph Alpha release. Part of the Sakura Micro line.

Sakura MicroQuality

Smaller than the standard quant of the same class, and much closer to the original. Every file here is measured against the standard llama.cpp quant of its size class (same importance matrix, same texts, same machine) and only files that beat it are published. Measured as KL divergence to the Q8_0 source on 4 held-out texts (en, dev, wiki, de); lower is better.

File	Size	Standard quant of its class	KL divergence of this file	Better on
`Sakura-Kolibri-1-IQ2_XS-20.94GiB.gguf`	20.94 GiB	IQ2_XS (21.41 GiB)	59 % lower	4 of 4 texts
`Sakura-Kolibri-1-IQ3_XXS-28.75GiB.gguf`	28.75 GiB	IQ3_XXS (28.12 GiB)	53 % lower	4 of 4 texts
`Sakura-Kolibri-1-IQ4_XS-38.07GiB.gguf`	38.07 GiB	IQ4_XS (38.91 GiB)	40 % lower	4 of 4 texts

Needs a patched llama.cpp for now. The kolibri1 architecture is not in mainline llama.cpp yet (feature request #29922). Apply the patch of Hob-forge/Kolibri-1-GGUF to llama.cpp commit 836d571, the same as for the other community GGUF files of this model. This card will be updated when upstream support lands.

Quality at a glance

Same method, same texts, same machine for every file; lower KL divergence means closer to the near-lossless Q8_0 reference:

Sakura-Kolibri-1-IQ2_XS-20.94GiB.gguf vs llama.cpp default IQ2_XS (21.41 GiB): KL divergence to the near-lossless Q8_0 reference 59 % lower on average over all 4 texts (0.46 GiB smaller).
Sakura-Kolibri-1-IQ3_XXS-28.75GiB.gguf vs llama.cpp default IQ3_XXS (28.12 GiB): KL divergence to the near-lossless Q8_0 reference 53 % lower on average over all 4 texts (0.63 GiB larger).
Sakura-Kolibri-1-IQ4_XS-38.07GiB.gguf vs llama.cpp default IQ4_XS (38.91 GiB): KL divergence to the near-lossless Q8_0 reference 40 % lower on average over all 4 texts (0.84 GiB smaller).

Why: the bit budget per tensor is allocated from measured sensitivity (see How it was made), not from fixed rules. Full table below.

Which file should I take?

File	For
`Sakura-Kolibri-1-IQ2_XS-20.94GiB.gguf` (20.94 GiB, 2.30 bpw)	smallest: for tight memory budgets; expect visible quality loss
`Sakura-Kolibri-1-IQ3_XXS-28.75GiB.gguf` (28.75 GiB, 3.16 bpw)	middle: best balance of size and quality for most machines
`Sakura-Kolibri-1-IQ4_XS-38.07GiB.gguf` (38.06 GiB, 4.19 bpw)	largest: closest to the Q8_0 reference of the three

The files

File	Size	bits/weight	KLD en	KLD dev	KLD wiki	KLD de	same top token (avg)	Tensor types by size
`Sakura-Kolibri-1-IQ2_XS-20.94GiB.gguf`	20.94 GiB	2.30	0.0758	0.0381	0.2237	0.0987	90.5 %	IQ2_XXS 70%, IQ2_XS 11%, IQ3_XXS 8%, Q8_0 5%, IQ4_XS 4%, Q2_K 2%
`Sakura-Kolibri-1-IQ3_XXS-28.75GiB.gguf`	28.75 GiB	3.16	0.0452	0.0206	0.1283	0.0607	93.5 %	IQ3_XXS 69%, Q8_0 8%, IQ2_XS 5%, IQ2_S 5%, IQ4_XS 4%, IQ3_S 4%, Q2_K 2%, IQ2_XXS 2%
`Sakura-Kolibri-1-IQ4_XS-38.07GiB.gguf`	38.06 GiB	4.19	0.0291	0.0173	0.0886	0.0376	95.0 %	IQ4_XS 65%, IQ3_S 14%, Q4_K 8%, Q8_0 7%, IQ3_XXS 4%

"Tensor types by size" lists the share of the file's bytes per ggml type (all are standard llama.cpp types, so the files run in stock llama.cpp). Quality columns: mean KL divergence of the quantized model's next-token distribution against the Q8_0 source on held-out text (lower is better) and how often the most likely token is unchanged (higher is better).

Comparison at similar size

Same texts, same context, same chunks, measured by us with llama-perplexity (near-lossless Q8_0 source = reference; PPL of the Q8_0 reference: en: 2.460, dev: 1.928, wiki: 24.848, de: 2.908). The reference quants are: llama.cpp default recipe, same importance matrix (ours), measured the same way. Sorted by size.

Quant	Source	Size	KLD en	KLD dev	KLD wiki	KLD de	PPL en	same top token (avg)
`Sakura-Kolibri-1-IQ2_XS-20.94GiB.gguf`	this repository	20.94 GiB	0.0758	0.0381	0.2237	0.0987	2.462	90.5 %
IQ2_XS	llama.cpp default recipe, same importance matrix (ours)	21.41 GiB	0.1859	0.1113	0.4540	0.2548	2.653	84.6 %
IQ3_XXS	llama.cpp default recipe, same importance matrix (ours)	28.12 GiB	0.0917	0.0519	0.2667	0.1187	2.453	89.6 %
`Sakura-Kolibri-1-IQ3_XXS-28.75GiB.gguf`	this repository	28.75 GiB	0.0452	0.0206	0.1283	0.0607	2.457	93.5 %
`Sakura-Kolibri-1-IQ4_XS-38.07GiB.gguf`	this repository	38.07 GiB	0.0291	0.0173	0.0886	0.0376	2.456	95.0 %
IQ4_XS	llama.cpp default recipe, same importance matrix (ours)	38.91 GiB	0.0566	0.0253	0.1550	0.0616	2.514	93.2 %

Method: 12 chunks of 512 tokens per text; texts: English held-out text, developer-text held-out set, English encyclopedia held-out text (not used in any calibration), German held-out text (not used in any calibration). Single measurement on one machine; small KLD differences at similar size are not a quality ranking.

New comparison point

Hob-forge now lists a conventional Q2_K file for Kolibri-1. The published file sizes show Sakura IQ2_XS is substantially smaller:

Model	Quant	Published file size
Sakura MicroQuality	IQ2_XS	20.94 GiB (about 22.5 GB)
Hob-forge	Q2_K	28.6 GB (about 26.6 GiB)

That makes Sakura IQ2_XS about 5.7 GiB smaller by the published sizes.

The Sakura KL and top-token measurements elsewhere on this card remain as published: they use the Q8_0 source as reference and four held-out texts. Hob-forge reports its Q2_K KL against an independent float64 PyTorch reference on a 67-token prompt. Since the reference and evaluation protocol differ, Hob-forge's published KL figures are not directly comparable with the Sakura KL figures. A same-reference benchmark has not yet been run, so this size comparison makes no relative quality claim.

How it was made

The source is the Q8_0 GGUF from Hob-forge/Kolibri-1-GGUF (converted from Aleph Alpha's FP8 checkpoint; near-lossless). We did not convert from the 156 GB BF16 weights, so the reference for every number on this card is that Q8_0. The importance matrix was computed by us (llama-imatrix) on German and English text, code and the model's own chat answers with thinking.
Every large weight matrix was quantized once per candidate type (Q2_K, IQ2_XXS, IQ2_XS, IQ2_S, IQ3_XXS, IQ3_S, Q3_K, IQ4_XS, Q4_K, Q8_0) with llama-quantize and the importance matrix; the error of each choice was estimated per matrix (importance-weighted) and scaled by a sensitivity factor measured with real KL divergence: every group of tensors (for example the FFN down projections, the attention value projections, the first and last layers) was moved alone to a lower-bit type, and the KL divergence it caused was compared with its quantization error.
An exact budget allocation (multiple-choice knapsack over bytes) picked one type per matrix for each target size; the final model was assembled from the stored tensors without re-quantizing. We do not publish per-tensor choices. Norms and small tensors stay at high precision.

Limits

Measured only with KL divergence and perplexity on short held-out texts and not with downstream benchmarks; do not read it as a task-quality claim.
Quantization always costs some quality; the smallest file costs the most.
Requires a patched llama.cpp until upstream support lands (see the banner at the top). All numbers on this card were measured with that build (Vulkan backend).
The experts hold almost the whole file. About 5 % of the experts (mostly in the first three layers) were never selected by our calibration texts; they were given the average importance of the others, so every file is complete. Requantizing from a Q8_0 source adds a very small extra error compared with converting from BF16; the quality columns compare every file against that same Q8_0 source.
Recommended settings from the model card: temperature 1.0, top-p 0.97, top-k 128; reasoning effort is set per request (none, low, medium, high).

Credits

Original model: Kolibri-1 (Aleph Alpha, Apache-2.0), Apache-2.0 license (included as LICENSE).
Q8_0 source and the llama.cpp patch: Hob-forge/Kolibri-1-GGUF (Hob-forge / Seraphiel102). Reference quants: llama-quantize default recipes with our importance matrix.
GGUF format and tools: ggml-org/llama.cpp (MIT).

中文说明 · 樱花 (Simplified Chinese)

English above. 本节为上文的中文翻译(Sakura = 樱花 yīnghuā);完整的独立中文版见 README_zh.md。

Sakura MicroQuality — Kolibri-1(GGUF,三种大小)

德国制造: Kolibri-1 是德国公司 Aleph Alpha 的开源 78B 模型,为混合专家(MoE)模型(3.5B 激活参数,每层 384 个路由专家加 1 个共享专家),面向德语和英语,带有思考模式和工具调用。本仓库包含 三个不同大小的 GGUF 文件,它们由原模型的 Q8_0 GGUF(近乎无损;所有质量数据均以它为参照)通过我们实测得出的混合编码方法制作。请在 Files 页面选择其中一个;下表说明了每个文件的用途。这是独立的社区量化版本,不是 Aleph Alpha 的官方发布。属于 Sakura Micro 系列。

Sakura MicroQuality

比同级别的标准量化更小,并且更接近原始模型。 这里的每个文件都与其大小级别的标准 llama.cpp 量化(相同的重要性矩阵、相同的文本、同一台机器)进行了对比测量,只有优于它的文件才会发布。测量方式为在 4 个留出文本(en、dev、wiki、de)上相对于 Q8_0 源文件的 KL 散度;越低越好。

文件	大小	同级别的标准量化	此文件的 KL 散度	更优的文本数
`Sakura-Kolibri-1-IQ2_XS-20.94GiB.gguf`	20.94 GiB	IQ2_XS (21.41 GiB)	低 59 %	4 个中的 4 个
`Sakura-Kolibri-1-IQ3_XXS-28.75GiB.gguf`	28.75 GiB	IQ3_XXS (28.12 GiB)	低 53 %	4 个中的 4 个
`Sakura-Kolibri-1-IQ4_XS-38.07GiB.gguf`	38.07 GiB	IQ4_XS (38.91 GiB)	低 40 %	4 个中的 4 个

目前需要打过补丁的 llama.cpp。 kolibri1 架构尚未进入 llama.cpp 主线(功能请求 #29922)。请将 Hob-forge/Kolibri-1-GGUF 的补丁应用到 llama.cpp 提交 836d571 上,与该模型的其他社区 GGUF 文件相同。上游支持落地后,本卡片会更新。

质量概览

每个文件使用相同的方法、相同的文本、同一台机器;KL 散度越低,越接近近乎无损的 Q8_0 参照:

Sakura-Kolibri-1-IQ2_XS-20.94GiB.gguf 对比 llama.cpp 默认的 IQ2_XS (21.41 GiB):相对于近乎无损的 Q8_0 参照,在全部 4 个文本上平均 KL 散度 低 59 %(小 0.46 GiB)。
Sakura-Kolibri-1-IQ3_XXS-28.75GiB.gguf 对比 llama.cpp 默认的 IQ3_XXS (28.12 GiB):相对于近乎无损的 Q8_0 参照,在全部 4 个文本上平均 KL 散度 低 53 %(大 0.63 GiB)。
Sakura-Kolibri-1-IQ4_XS-38.07GiB.gguf 对比 llama.cpp 默认的 IQ4_XS (38.91 GiB):相对于近乎无损的 Q8_0 参照,在全部 4 个文本上平均 KL 散度 低 40 %(小 0.84 GiB)。

原因:每个张量的比特预算是根据实测的敏感度分配的(见 制作方式),而不是按固定规则。完整表格见下文。

我该选哪个文件?

文件	适用场景
`Sakura-Kolibri-1-IQ2_XS-20.94GiB.gguf`(20.94 GiB,2.30 bpw)	最小:适合内存预算紧张的情况;预计质量会有明显损失
`Sakura-Kolibri-1-IQ3_XXS-28.75GiB.gguf`(28.75 GiB,3.16 bpw)	居中:对大多数机器而言,大小与质量的最佳平衡
`Sakura-Kolibri-1-IQ4_XS-38.07GiB.gguf`(38.06 GiB,4.19 bpw)	最大:三者中最接近 Q8_0 参照

文件

文件	大小	bits/weight	KLD en	KLD dev	KLD wiki	KLD de	same top token (avg)	各张量类型(按大小)
`Sakura-Kolibri-1-IQ2_XS-20.94GiB.gguf`	20.94 GiB	2.30	0.0758	0.0381	0.2237	0.0987	90.5 %	IQ2_XXS 70%, IQ2_XS 11%, IQ3_XXS 8%, Q8_0 5%, IQ4_XS 4%, Q2_K 2%
`Sakura-Kolibri-1-IQ3_XXS-28.75GiB.gguf`	28.75 GiB	3.16	0.0452	0.0206	0.1283	0.0607	93.5 %	IQ3_XXS 69%, Q8_0 8%, IQ2_XS 5%, IQ2_S 5%, IQ4_XS 4%, IQ3_S 4%, Q2_K 2%, IQ2_XXS 2%
`Sakura-Kolibri-1-IQ4_XS-38.07GiB.gguf`	38.06 GiB	4.19	0.0291	0.0173	0.0886	0.0376	95.0 %	IQ4_XS 65%, IQ3_S 14%, Q4_K 8%, Q8_0 7%, IQ3_XXS 4%

“各张量类型(按大小)”列出每个 ggml 类型占文件字节数的比例(全部是标准的 llama.cpp 类型,因此这些文件可在原版 llama.cpp 中运行)。质量列:量化模型的下一个 token 分布相对于 Q8_0 源文件在留出文本上的平均 KL 散度(越低越好),以及最可能的 token 保持不变的频率(越高越好)。

相近大小的比较

相同的文本、相同的上下文、相同的分块,由我们使用 llama-perplexity 测量(近乎无损的 Q8_0 源文件 = 参照;Q8_0 参照的 PPL:en: 2.460,dev: 1.928,wiki: 24.848,de: 2.908)。参照量化为:llama.cpp 默认配方、相同的重要性矩阵(我们的)、以相同方式测量。按大小排序。

量化	来源	大小	KLD en	KLD dev	KLD wiki	KLD de	PPL en	same top token (avg)
`Sakura-Kolibri-1-IQ2_XS-20.94GiB.gguf`	本仓库	20.94 GiB	0.0758	0.0381	0.2237	0.0987	2.462	90.5 %
IQ2_XS	llama.cpp 默认配方,相同的重要性矩阵(我们的)	21.41 GiB	0.1859	0.1113	0.4540	0.2548	2.653	84.6 %
IQ3_XXS	llama.cpp 默认配方,相同的重要性矩阵(我们的)	28.12 GiB	0.0917	0.0519	0.2667	0.1187	2.453	89.6 %
`Sakura-Kolibri-1-IQ3_XXS-28.75GiB.gguf`	本仓库	28.75 GiB	0.0452	0.0206	0.1283	0.0607	2.457	93.5 %
`Sakura-Kolibri-1-IQ4_XS-38.07GiB.gguf`	本仓库	38.07 GiB	0.0291	0.0173	0.0886	0.0376	2.456	95.0 %
IQ4_XS	llama.cpp 默认配方,相同的重要性矩阵(我们的)	38.91 GiB	0.0566	0.0253	0.1550	0.0616	2.514	93.2 %

方法:每个文本取 12 个块,每块 512 个 token;文本为:英文留出文本、开发者文本留出集、英文百科留出文本(未用于任何校准)、德文留出文本(未用于任何校准)。这是在一台机器上的单次测量;相近大小下 KLD 的细小差异并不构成质量排名。

新的比较点

Hob-forge 现在列出了 Kolibri-1 的一个常规 Q2_K 文件。从已发布的文件大小看,Sakura IQ2_XS 要小得多:

模型	量化	已发布的文件大小
Sakura MicroQuality	IQ2_XS	20.94 GiB (about 22.5 GB)
Hob-forge	Q2_K	28.6 GB (about 26.6 GiB)

按已发布的大小,Sakura IQ2_XS 因此约小 5.7 GiB。

本卡片其他位置的 Sakura KL 和 top-token 测量保持原样:它们以 Q8_0 源文件为参照,使用四个留出文本。Hob-forge 报告的 Q2_K KL 是相对于独立的 float64 PyTorch 参照、在一个 67 个 token 的提示上测得的。由于参照和评测协议不同,Hob-forge 已发布的 KL 数值与 Sakura 的 KL 数值不能直接比较。尚未运行使用相同参照的基准测试,因此这一大小比较不作任何相对质量的声明。

制作方式

来源是 Hob-forge/Kolibri-1-GGUF 的 Q8_0 GGUF(由 Aleph Alpha 的 FP8 检查点转换而来;近乎无损)。我们没有从 156 GB 的 BF16 权重转换,因此本卡片上每个数据的参照都是该 Q8_0。重要性矩阵由我们自己(llama-imatrix)在德语和英语文本、代码以及模型自己带思考的聊天回答上计算。
每个大型权重矩阵针对每种候选类型(Q2_K、IQ2_XXS、IQ2_XS、IQ2_S、IQ3_XXS、IQ3_S、Q3_K、IQ4_XS、Q4_K、Q8_0)使用 llama-quantize 和重要性矩阵各量化一次;每种选择的误差按矩阵估计(按重要性加权),并乘以一个 用真实 KL 散度测得的敏感度系数:每组张量(例如 FFN down 投影、注意力 value 投影、首尾几层)被单独移到较低比特的类型,并将它造成的 KL 散度与其量化误差进行比较。
精确的预算分配(以字节为单位的多选背包问题)为每个目标大小为每个矩阵挑选一种类型;最终模型由已存储的张量组装而成,无需重新量化。我们不公布逐张量的选择。归一化层和小张量保持高精度。

局限

仅用短的留出文本上的 KL 散度和困惑度测量,没有使用下游基准测试;请勿将其理解为任务质量的声明。
量化总会损失一些质量;最小的文件损失最大。
在上游支持落地之前,需要打过补丁的 llama.cpp(见顶部横幅)。本卡片上的所有数据都是用该构建(Vulkan 后端)测得的。
专家几乎占了整个文件。约 5 % 的专家(主要在前三层)从未被我们的校准文本选中;它们被赋予其他专家的平均重要性,因此每个文件都是完整的。与从 BF16 转换相比,从 Q8_0 源文件重新量化会增加极小的额外误差;质量列将每个文件与同一个 Q8_0 源文件进行比较。
模型卡片推荐的设置:temperature 1.0、top-p 0.97、top-k 128;推理强度按请求设置(none、low、medium、high)。

致谢

原始模型:Kolibri-1 (Aleph Alpha, Apache-2.0),Apache-2.0 许可证(以 LICENSE 随附)。
Q8_0 源文件和 llama.cpp 补丁:Hob-forge/Kolibri-1-GGUF(Hob-forge / Seraphiel102)。参照量化:llama-quantize 默认配方,配以我们的重要性矩阵。
GGUF 格式和工具:ggml-org/llama.cpp (MIT)。