Instructions to use leeaandrob/ternary-bonsai-27b-gguf with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use leeaandrob/ternary-bonsai-27b-gguf with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf leeaandrob/ternary-bonsai-27b-gguf:Q2_0
# Run inference directly in the terminal:
llama cli -hf leeaandrob/ternary-bonsai-27b-gguf:Q2_0

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf leeaandrob/ternary-bonsai-27b-gguf:Q2_0
# Run inference directly in the terminal:
llama cli -hf leeaandrob/ternary-bonsai-27b-gguf:Q2_0

Build from source code

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf leeaandrob/ternary-bonsai-27b-gguf:Q2_0
# Run inference directly in the terminal:
./build/bin/llama cli -hf leeaandrob/ternary-bonsai-27b-gguf:Q2_0

Use Docker

docker model run hf.co/leeaandrob/ternary-bonsai-27b-gguf:Q2_0

Llama new
LM Studio
Jan
Ollama
How to use leeaandrob/ternary-bonsai-27b-gguf with Ollama:
```
ollama run hf.co/leeaandrob/ternary-bonsai-27b-gguf:Q2_0
```
Unsloth Desktop

How to use leeaandrob/ternary-bonsai-27b-gguf with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf leeaandrob/ternary-bonsai-27b-gguf:Q2_0

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "leeaandrob/ternary-bonsai-27b-gguf:Q2_0"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use leeaandrob/ternary-bonsai-27b-gguf with Docker Model Runner:
```
docker model run hf.co/leeaandrob/ternary-bonsai-27b-gguf:Q2_0
```

Lemonade

How to use leeaandrob/ternary-bonsai-27b-gguf with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull leeaandrob/ternary-bonsai-27b-gguf:Q2_0

Run and chat with the model

lemonade run user.ternary-bonsai-27b-gguf-Q2_0

List all available models

lemonade list

Hermes Agent

How to use leeaandrob/ternary-bonsai-27b-gguf with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf leeaandrob/ternary-bonsai-27b-gguf:Q2_0

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default leeaandrob/ternary-bonsai-27b-gguf:Q2_0

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use leeaandrob/ternary-bonsai-27b-gguf with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf leeaandrob/ternary-bonsai-27b-gguf:Q2_0

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "leeaandrob/ternary-bonsai-27b-gguf:Q2_0" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Ternary-Bonsai-27B GGUF (NeuroGrid mirror)

A mirror of the prism-ml/Ternary-Bonsai-27B-gguf weights (Apache-2.0, base Qwen/Qwen3.6-27B), kept here because these are the exact files the NeuroGrid engine serves in production and pinning them makes a deployment reproducible. The weights are not ours and are not modified. Credit and licence belong upstream.

Which file to download

Three quantized files with two different block geometries share the same 2-bit weights. Picking the wrong one gets you a load error, not bad output.

File	Block	bpw	Loads on
`Ternary-Bonsai-27B-Q2_g64.gguf` (7.59 GB)	64	2.25	upstream `llama.cpp` (`GGML_TYPE_Q2_0 = 42`) and NeuroGrid
`Ternary-Bonsai-27B-Q2_0.gguf` (7.17 GB)	128	2.125	the PrismML fork and NeuroGrid
`Ternary-Bonsai-27B-PQ2_0.gguf` (7.17 GB)	128	2.125	the same group-128 layout under its own name
`Ternary-Bonsai-27B-mmproj-Q8_0.gguf` (0.63 GB)			the CLIP vision tower, pair it with any of the above

The Q2_0 name means two different byte layouts depending on who wrote the file: upstream settled on group 64, the PrismML fork uses group 128. Both are here so you do not have to guess. NeuroGrid reads both and refuses a file its device kernel cannot run at load time rather than serving it wrong.

Serving on NeuroGrid: take Q2_0 (group 128), which is what the device GEMV kernel wants.
Comparing against llama.cpp: take Q2_g64.

Running it on NeuroGrid

One process serves chat, vision and embeddings:

neurogrid -solo \
  -bonsai-gguf   Ternary-Bonsai-27B-Q2_0.gguf \
  -bonsai-mmproj Ternary-Bonsai-27B-mmproj-Q8_0.gguf \
  -bonsai-max-seq 16384 -max-seq-len 16384 \
  -http-port 8080 -host 127.0.0.1

Vision is enabled by the tower being loaded, not by the model name you advertise, so -bonsai-mmproj is the whole switch. Images arrive as OpenAI-style image_url content parts on /v1/chat/completions.

On CUDA the useful environment is NG_QWEN35_GPU=1, NG_QWEN35_GPU_LMHEAD=1, NG_QWEN35_GRAPH=1 and NG_CLIP_GPU=1; add NG_QWEN35_BATCH=8 with NG_QWEN35_BATCH_GRAPH=1 to run up to 8 concurrent sequences through one decode step.

Running it on llama.cpp

llama-server -m Ternary-Bonsai-27B-Q2_g64.gguf \
  --mmproj Ternary-Bonsai-27B-mmproj-Q8_0.gguf -c 16384

Group-64 only. The 7.17 GB files will not load on ggml-org/llama.cpp master.

Downloads last month: 71

GGUF

Model size

27B params

Architecture

qwen35

Hardware compatibility

2-bit

View +1 variant

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for leeaandrob/ternary-bonsai-27b-gguf

Base model

Qwen/Qwen3.6-27B

Quantized

prism-ml/Ternary-Bonsai-27B-gguf

Quantized

(5)

this model