Strike1999/BeyondFreq-L2I-COCO512

BeyondFreq-L2I — COCO512

Visual blueprint generator associated with Unbiased Object Detection Beyond Frequency with Visually Prompted Image Synthesis, ICLR 2026.

• Paper
• Code and installation

The checkpoint generates 512×512 images from an RGB class-color blueprint and an object-count text prompt. It contains the Stable Diffusion 1.5 U-Net, the learned blueprint ControlNet, the VAE and the CLIP text weights, trained for 100,000 updates on COCO.

This release covers the generator component. It does not bundle the representation-score planner or detector training from the full paper. This is the original 100k checkpoint, before the separate G→D cycle fine-tuning experiments.

Files and loading

Weights are serialized into three safetensors shards. Tensor names, values and dtypes match the original checkpoint exactly; no quantization or precision conversion was applied. weights_manifest.json records the source digest and each shard's size and SHA256 hash. The index is beyondfreq_l2i_coco512_100k.safetensors.index.json.

Use the BeyondFreq-L2I generator release of the linked code repository, or extract the accompanying beyondfreq_l2i_source.zip. The native LDM architecture and bundled tokenizer assets are required; these shards are not a Diffusers pipeline checkpoint. The source archive makes this model release runnable independently of the GitHub code update.

from beyond_freq.generator import BeyondFreqL2I

# After downloading this model repository:
generator = BeyondFreqL2I("path/to/downloaded/model", device="cuda")
# generator.sample(rgb_blueprints_BCHW, prompts, image_ids, seed=451)

Default inference uses 50 DDIM steps, guidance 4.5, blueprint strength 1.5, and BF16 denoising on CUDA. Weights retain their original precision. The package also supports strict loading of the original Lightning checkpoint.

Conditioning conventions

COCO category order defines the HSV class palette. Repeated instances use a brightness decrement of 0.02. The existing checkpoint was trained with opaque rectangle fills, annotation-area painting order and object-count prompts. Preserve these conventions when using the pretrained weights. The code guide explains their relationship to the paper's visual blueprint description.

Checkpoint verification benchmark

Measured on 5000 COCO val2017 layouts, seed 451, 512×512, DDIM 50, guidance 4.5 and blueprint strength 1.5. Layouts are shared non-crowd annotations capped at 30 objects, using frozen DEIM and independent YOLOv8x.

Metric	Value
Frozen DEIM layout AP	52.50
Independent YOLOv8x layout AP	48.46
Clean-FID 5K	25.16
KID	0.005285
CLIPScore	27.56

This benchmark uses a different detector and evaluation protocol from the paper's tables. The values verify the released checkpoint and are not a claim to reproduce those tables.

License

The model is derived from Stable Diffusion 1.5. Its OpenRAIL-M terms and restrictions are retained in LICENSE. Third-party code notices are provided in the code repository.

Citation

@inproceedings{cai2026unbiased,
  title={Unbiased Object Detection Beyond Frequency with Visually Prompted Image Synthesis},
  author={Cai, Xinhao and Li, Liulei and Pei, Gensheng and Chen, Tao and Pan, Jinshan and Yao, Yazhou and Wang, Wenguan},
  booktitle={ICLR},
  year={2026}
}