WaveCut/FLUX-3-Image-Klein-OrbitQuant-W4A8

FLUX.3 Image Klein OrbitQuant W4A8

OrbitQuant 4-bit weights for FLUX 3 Image Klein, the distilled 5.6B model for text-to-image generation and image editing with up to 10 reference images. The transformer runs 8-bit activations over 4-bit weights, rounded with GPTQ. The Qwen3-VL text encoder uses 6-bit weights and activations. The runtime uses the native flux3 pipeline of Black Forest Labs (16 steps, no guidance) with their prompt upsampler, and the ComfyUI-OrbitQuant nodes. The bundle is 7.9 GB. The original is 20.3 GB.

On one RTX 5090 a 1024×1024 image takes 2.11 s (BF16: 4.30 s) with 13.0 GiB of device memory at the peak (BF16: 29.0 GiB). With --offload the peak is 6.0 GiB and the image takes 5.25 s. These times exclude the prompt upsampler (Usage). The Performance section has the details.

Usage

Use Linux, Python 3.12 and an NVIDIA GPU with compute capability 8.0 or newer. The recipe is the installation of Black Forest Labs with orbitquant added. It was tested with Torch 2.11.0 (CUDA 13.0) and vLLM 0.21.0 on an RTX 5090 (driver 595.91.07).

git clone https://github.com/black-forest-labs/flux-3-image-klein.git && cd flux-3-image-klein
git checkout bfc8889a4497f846416f253a3972a1941ac698b9
python3.12 -m venv .venv && . .venv/bin/activate
pip install -e . --extra-index-url https://download.pytorch.org/whl/cu130
hf download WaveCut/FLUX-3-Image-Klein-OrbitQuant-W4A8 --local-dir weights/orbitquant
hf download black-forest-labs/flux-3-image-klein-pup --local-dir weights/pup
pip install -r weights/orbitquant/requirements.txt
python -m orbitquant.cli.main kernels-install

The commands below send the prompt through the prompt upsampler of Black Forest Labs before the model, as their CLI does. The model is trained on the captions of the upsampler. Pass --no-pup only when the prompt already is such a caption. The upsampler is an 18 GB model on vLLM. On a 32 GB GPU it does not fit next to the model without offload, so run_flux3.py runs it in its own process (python -m flux3.pup), which exits before the model loads. On the RTX 5090 one call takes 98 to 117 s, most of it the start of vLLM (the first call takes 210 s because vLLM compiles). The device total peaks at 26.2 GiB while the upsampler runs.

Text to image:

python weights/orbitquant/run_flux3.py --prompt "A man walking with his dog, afternoon sun" --seed 7 --output out.png

Editing. The first reference sets the output size unless you set --width and --height. Up to 10 references:

python weights/orbitquant/run_flux3.py --prompt "put a red bandana around the dog's neck" --image examples/inputs/3.png --seed 22 --output out.png
python weights/orbitquant/run_flux3.py --prompt "print the dog from the second image onto the mug" \
  --image examples/inputs/2.png --image examples/inputs/3.png --seed 23 --output out.png

Layout with bounding boxes (the format is in the README of Black Forest Labs):

python weights/orbitquant/run_flux3.py --prompt "hot-air balloons over a green valley at sunrise" \
  --boxes examples/inputs/balloons_boxes.json --seed 1 --output out.png

Each image comes with a .json record: the caption that the model received, the size, the steps and the seed. The upsampler can word its caption differently from run to run, so the same request and seed can give another image. The caption from the record with --no-pup and the same seed repeats the image.

To caption on another machine, or to reuse a caption, run the two steps by hand. The caption can also come from another vision-language model with the system prompts of Black Forest Labs.

python -m flux3.pup --prompt "A man walking with his dog, afternoon sun" --out prompt.txt
python weights/orbitquant/run_flux3.py --prompt "$(cat prompt.txt)" --no-pup --seed 7 --output out.png

Python, inside flux-3-image-klein:

import sys

sys.path.insert(0, "weights/orbitquant")
from flux3_orbitquant import Klein, upsample

caption = upsample("A man walking with his dog, afternoon sun", pup="weights/pup")
model = Klein("weights/orbitquant")
image = model.generate(caption, seed=7)
image.save("out.png")

upsample also takes reference image paths and a boxes file. generate also takes images, boxes, width, height and steps.

For many images, keep the upsampler loaded in the same process, as Models of Black Forest Labs does. On the RTX 5090 it fits next to the offloaded model with pup_memory=0.66. Loading takes 75 s. Then text to image takes 9.3 s and an edit with one reference 9.8 s, upsampling included, and the device total peaks at 25.8 GiB. vLLM starts its engine in a new process that imports the script again, so the code needs the main guard:

import sys

sys.path.insert(0, "weights/orbitquant")
from flux3_orbitquant import Klein

if __name__ == "__main__":
    model = Klein("weights/orbitquant", pup="weights/pup", pup_memory=0.66, offload=True)
    for seed in (7, 8):
        model.generate("A man walking with his dog, afternoon sun", seed=seed).save(f"out_{seed}.png")

Options of run_flux3.py (Klein takes offload and attention as arguments):

• --no-pup: send the prompt as typed. Use it only with an upsampler caption.
• --pup-memory: the share of the GPU that vLLM reserves while the upsampler runs (default 0.85, as in python -m flux3.pup).
• --offload: keep the transformer, the text encoder and the autoencoder in host memory. Each moves to the GPU for its own stage.
• --attention int8: INT8 Q·K attention. Text to image takes 1.93 s instead of 2.11 s, and the image changes slightly.

ComfyUI

ComfyUI-OrbitQuant lists this release as WaveCut/FLUX-3-Image-Klein-OrbitQuant-W4A8. Connect OrbitQuant Model Loader to OrbitQuant Generate Image and save the result with a standard save node. Connect up to 10 images to reference_images for editing. The generator runs in the Python environment that python_executable selects. Install the packages of the Usage section there. The node sends the prompt as typed: write an upsampler caption, or make one with python -m flux3.pup.

• profile: default, model-offload or int8-attention.
• keep_loaded (on by default): the worker keeps the model for the next request. The first text-to-image request takes 26.3 s because it loads the model. A later one takes 3.0 s. A request with another model or profile replaces the kept model. A video or audio run, a run with keep_loaded off, or an interrupt stops the worker and frees its GPU memory.

On the RTX 5090 the device total peaks at 12.6 GiB with the default profile. With model-offload it peaks at 6.2 GiB, and a request with keep_loaded off takes 27.2 s. These values include the 1.0 GiB that the idle ComfyUI server holds.

Quality

Mean LPIPS to BF16, same caption and seed (lower is closer): the 11 examples of the original repository with their published captions, and the 22 tasks of the OpenRouter benchmark below:

Configuration	11 BFL examples	22 OpenRouter tasks
FP8 (variant of Black Forest Labs)	0.037	0.048
OrbitQuant W4A8	0.107	0.074
OrbitQuant W4A8, INT8 attention	0.104	0.072
OrbitQuant W4A8, nearest rounding (previous transformer)	0.103	0.123

GPTQ brings 29 of the 33 images closer to BF16 than nearest rounding. On BFL's gig poster it sets the title on one line instead of two; this one image raises the mean over the 11 examples.

The images differ from BF16 in details and sometimes in layout. Small lettering in BFL's typography examples, same seed:

Prompts of the OpenRouter Image Benchmarks, turned into captions by BFL's upsampler. Both columns use the same caption and seed. Edits show their reference images at the left.

Performance

One RTX 5090 (575 W, driver 595.91.07), Torch 2.12.1 with CUDA 13.0. One process per configuration in one session, batch size 1, 16 steps. Time is the warm median of 9 runs after 2 warm-up runs (5 runs with offload). It covers prompt encoding, sampling and VAE decoding, with CUDA synchronized at both ends. Device peak is the total from NVML at 0.1 s intervals, including the CUDA context. Tensor peak is the PyTorch allocator peak. Host RAM is the peak resident set of the process, including loading.

Configuration	Text to image	Edit, 1 reference	Device peak	Tensor peak	Host RAM peak	Checkpoint
BF16 (original)	4.30 s	9.33 s	29.0 GiB	23.0 GiB	11.2 GiB	20.3 GB
FP8 (variant of Black Forest Labs)	4.09 s	9.54 s	25.3 GiB	18.6 GiB	6.7 GiB	15.5 GB
OrbitQuant W4A8	2.11 s	4.73 s	13.0 GiB	10.4 GiB	5.6 GiB	7.9 GB
OrbitQuant W4A8, INT8 attention	1.93 s	3.95 s	12.4 GiB	10.4 GiB	5.6 GiB	7.9 GB
BF16 (original), offload	15.07 s	21.90 s	13.9 GiB	11.9 GiB	25.0 GiB	20.3 GB
OrbitQuant W4A8, offload	5.25 s	8.71 s	6.0 GiB	4.4 GiB	13.9 GiB	7.9 GB

The edit uses one 1024×1024 reference, which doubles the token count. Memory values are the larger of the two workloads. A shorter run in the environment of the Usage section (Torch 2.11.0) gave the same times for OrbitQuant W4A8: 2.11 s for text to image and 4.73 s for the edit. The loader drops the vision tower and the decoder layers after 32 from the text encoder. The pipeline encodes text only and reads hidden states 4 to 32. The loader also enables expandable allocator segments when PYTORCH_CUDA_ALLOC_CONF is empty. Other GPUs are untested.

Components

Component	Precision	Size
Transformer, 5.5B parameters	OrbitQuant 4-bit weights (GPTQ rounding) and 8-bit activations for the 258 block projections. INT4 for the 6 modulation projections. BF16 input, time and output layers.	2.94 GB
Text encoder, Qwen3-VL-4B-Instruct	OrbitQuant W6A6 for the language model. BF16 for layers 6 and 16 and for the vision tower, which the runtime does not load.	4.59 GB
Autoencoder	FP32, unchanged	0.34 GB

GPTQ keeps the codebook, rotations, bit width and packed layout of OrbitQuant and chooses each 4-bit code against the inputs that the projection sees in BF16 sampling. The calibration ran BFL's pipeline at revision bfc8889 (16 steps) on 25 requests: 8 text-to-image examples of BFL that are not in the evaluation, 12 text-to-image requests of our own (7 with lettering) and 5 edits of crops from BFL's sample sheet, all captioned by BFL's upsampler. The Hessian-weighted output error of the projections falls from 0.0046 with nearest rounding to 0.0006 on average. dit/gptq-info.json lists the calibration requests and the error of every projection group.

The text encoder keeps 6 bits because its residual stream carries values near 11,000 in the first token.

Files

• dit/: the packed transformer (model.safetensors), its codebooks and rotations, the OrbitQuant configuration and manifest, and the GPTQ report (gptq-info.json).
• text_encoder/: the quantized Qwen3-VL-4B-Instruct with its tokenizer and processor files.
• ae.safetensors: the autoencoder of the original model.
• flux3_orbitquant.py: the loader and upsample. run_flux3.py: the command-line entry point. The fused transformer blocks come from orbitquant 0.14.
• manifest.json: the components and the runtime pins. SHA256SUMS: the checksums of all files.

Limits and license

• The fused blocks run one sample at a time. The base model with guidance, which packs two samples, is not supported.
• Small text, counts and fine spatial relations can fail, as in the original model.

This release is a modified version of FLUX 3 Image Klein under the FLUX Kommunity License v.1.0. The license permits non-commercial use, and commercial output generation by Qualifying Users with less than US$5M annual revenue, under its conditions. Read the license before any commercial use, including its content-filter and disclosure requirements. Keep the attribution in NOTICE.md. Black Forest Labs did not create, endorse or validate this release. The Qwen3-VL-4B-Instruct text encoder is under the Apache License 2.0.