Text Generation
Transformers
Safetensors
English
Chinese
Russian
yue2
music-generation
orbitquant
quantization
4-bit precision
custom-code
8-bit precision
Instructions to use WaveCut/YuE2-3B-OrbitQuant-W4A4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use WaveCut/YuE2-3B-OrbitQuant-W4A4 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="WaveCut/YuE2-3B-OrbitQuant-W4A4")# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("WaveCut/YuE2-3B-OrbitQuant-W4A4", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use WaveCut/YuE2-3B-OrbitQuant-W4A4 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "WaveCut/YuE2-3B-OrbitQuant-W4A4" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "WaveCut/YuE2-3B-OrbitQuant-W4A4", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/WaveCut/YuE2-3B-OrbitQuant-W4A4
- SGLang
How to use WaveCut/YuE2-3B-OrbitQuant-W4A4 with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "WaveCut/YuE2-3B-OrbitQuant-W4A4" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "WaveCut/YuE2-3B-OrbitQuant-W4A4", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "WaveCut/YuE2-3B-OrbitQuant-W4A4" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "WaveCut/YuE2-3B-OrbitQuant-W4A4", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use WaveCut/YuE2-3B-OrbitQuant-W4A4 with Docker Model Runner:
docker model run hf.co/WaveCut/YuE2-3B-OrbitQuant-W4A4
Publish OrbitQuant 0.9.5 byte-pair decode and optional RMS activation fusion
Browse filesThis view is limited to 50 files because it contains too many changes. See raw diff
- .gitattributes +3 -0
- README.md +24 -4
- SHA256SUMS +60 -14
- evaluation/runtime-v3/contract-checks.json +1 -0
- evaluation/runtime-v3/fast-rms0-s8.json +304 -0
- evaluation/runtime-v3/fast-rms1-s8.json +304 -0
- evaluation/runtime-v3/memory-rms1-s8.json +304 -0
- examples/runtime-v3/fast-rms.wav +3 -0
- src/OPTIMIZATION_IDEAS.md +11 -0
- src/kernel-source/orbitquant-packed-matmul/CARD.md +10 -0
- src/kernel-source/orbitquant-packed-matmul/orbitquant_packed_matmul_cuda/packed_matmul.cu +58 -4
- src/kernel-source/orbitquant-packed-matmul/tests/test_decode_gemv.py +15 -2
- src/kernel-source/rms-quant/.gitattributes +3 -0
- src/kernel-source/rms-quant/.gitignore +1 -0
- src/kernel-source/rms-quant/CARD.md +15 -0
- src/kernel-source/rms-quant/build.toml +22 -0
- src/kernel-source/rms-quant/cmake/_ops.py.in +9 -0
- src/kernel-source/rms-quant/cmake/add_build_metadata.py +56 -0
- src/kernel-source/rms-quant/cmake/build-variants.cmake +328 -0
- src/kernel-source/rms-quant/cmake/compile-metal.cmake +104 -0
- src/kernel-source/rms-quant/cmake/get_gpu_lang.cmake +9 -0
- src/kernel-source/rms-quant/cmake/get_gpu_lang.py +20 -0
- src/kernel-source/rms-quant/cmake/hipify.py +76 -0
- src/kernel-source/rms-quant/cmake/kernel.cmake +317 -0
- src/kernel-source/rms-quant/cmake/metallib_to_header.py +73 -0
- src/kernel-source/rms-quant/cmake/utils.cmake +611 -0
- src/kernel-source/rms-quant/compat.py +26 -0
- src/kernel-source/rms-quant/flake.nix +11 -0
- src/kernel-source/rms-quant/metadata-cann.json +21 -0
- src/kernel-source/rms-quant/metadata-cpu.json +21 -0
- src/kernel-source/rms-quant/metadata-cuda.json +21 -0
- src/kernel-source/rms-quant/metadata-metal.json +21 -0
- src/kernel-source/rms-quant/metadata-neuron.json +21 -0
- src/kernel-source/rms-quant/metadata-rocm.json +21 -0
- src/kernel-source/rms-quant/metadata-xpu.json +21 -0
- src/kernel-source/rms-quant/orbitquant_gemv_cuda/fused_ops.cu +67 -0
- src/kernel-source/rms-quant/orbitquant_gemv_cuda/orbitquant_gemv.cu +74 -0
- src/kernel-source/rms-quant/orbitquant_gemv_cuda/qkv_decode.cu +31 -0
- src/kernel-source/rms-quant/orbitquant_gemv_cuda/rmsquant.cu +126 -0
- src/kernel-source/rms-quant/tests/__init__.py +0 -0
- src/kernel-source/rms-quant/tests/test_fused_ops.py +23 -0
- src/kernel-source/rms-quant/tests/test_orbitquant_gemv.py +54 -0
- src/kernel-source/rms-quant/tests/test_qkv_decode.py +20 -0
- src/kernel-source/rms-quant/tests/test_rmsquant.py +56 -0
- src/kernel-source/rms-quant/torch-ext/registration.h +39 -0
- src/kernel-source/rms-quant/torch-ext/torch_binding.cpp +20 -0
- src/kernel-source/rms-quant/torch-ext/torch_binding.h +13 -0
- src/kernel-source/rms-quant/torch-ext/yue2_qkv_fused/__init__.py +34 -0
- src/requirements.txt +1 -1
- src/run.py +7 -2
.gitattributes
CHANGED
|
@@ -44,3 +44,6 @@ src/wheels/orbitquant-0.9.2-py3-none-any.whl filter=lfs diff=lfs merge=lfs -text
|
|
| 44 |
src/runtime/kernels/fast/torch210-cxx11-cu128-x86_64-linux/_yue2_qkv_fused_cuda_46ceaaf_dirty.abi3.so filter=lfs diff=lfs merge=lfs -text
|
| 45 |
src/wheels/orbitquant-0.9.3-py3-none-any.whl filter=lfs diff=lfs merge=lfs -text
|
| 46 |
src/wheels/orbitquant-0.9.4-py3-none-any.whl filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
|
|
|
|
|
|
| 44 |
src/runtime/kernels/fast/torch210-cxx11-cu128-x86_64-linux/_yue2_qkv_fused_cuda_46ceaaf_dirty.abi3.so filter=lfs diff=lfs merge=lfs -text
|
| 45 |
src/wheels/orbitquant-0.9.3-py3-none-any.whl filter=lfs diff=lfs merge=lfs -text
|
| 46 |
src/wheels/orbitquant-0.9.4-py3-none-any.whl filter=lfs diff=lfs merge=lfs -text
|
| 47 |
+
examples/runtime-v3/fast-rms.wav filter=lfs diff=lfs merge=lfs -text
|
| 48 |
+
src/runtime/kernels/rms/torch210-cxx11-cu128-x86_64-linux/_yue2_qkv_fused_cuda_cf0db58.abi3.so filter=lfs diff=lfs merge=lfs -text
|
| 49 |
+
src/wheels/orbitquant-0.9.5-py3-none-any.whl filter=lfs diff=lfs merge=lfs -text
|
README.md
CHANGED
|
@@ -29,7 +29,7 @@ The autoregressive branch plans an ABC score and generates semantic audio tokens
|
|
| 29 |
|
| 30 |
2,818,572,288 transformer weights occupy 1,409,286,144 packed bytes. The complete model checkpoint is **3,035,926,840 bytes**, versus **7,261,441,640 bytes** upstream: 2.39× smaller (58.2% reduction). The separate 530,512,720-byte VAE checkpoint is unchanged and downloaded at a pinned revision.
|
| 31 |
|
| 32 |
-
Compatible Q/K/V and gate/up projections are grouped: 224 physical packed modules represent 392 logical projections without duplicated packed storage. OrbitQuant 0.9.
|
| 33 |
|
| 34 |
The default VAE core is 512 frames with the original 16-frame halo. On the recorded Russian song this was waveform-identical to 1024 frames, while lowering the decoder's isolated memory peak. This is a tested example, not a universal bitwise-equivalence guarantee.
|
| 35 |
|
|
@@ -40,11 +40,11 @@ The default VAE core is 512 frames with the original 16-frame halo. On the recor
|
|
| 40 |
- `examples/`: Russian WAV comparisons and the default prompt in `examples/prompts/`.
|
| 41 |
- `evaluation/`: recorded metrics and validation evidence.
|
| 42 |
|
| 43 |
-
Use `src/run.py` for packed loading. The inherited `auto_map` for the original dense model has been removed: it cannot load these packed weights. Earlier clean-load reports retain their original pinned revision and paths; they describe the earlier release. Runtime v2
|
| 44 |
|
| 45 |
## Supported runtime
|
| 46 |
|
| 47 |
-
**Runtime
|
| 48 |
|
| 49 |
```bash
|
| 50 |
hf download WaveCut/YuE2-3B-OrbitQuant-W4A4 --local-dir YuE2-W4A4
|
|
@@ -72,12 +72,32 @@ The output includes `audio.wav` and generation artifacts. `--prompt` accepts JSO
|
|
| 72 |
```bash
|
| 73 |
python src/run.py --profile fused --output song-fused
|
| 74 |
python src/run.py --profile memory --output song-memory
|
|
|
|
|
|
|
| 75 |
python src/run.py --profile fast --allocator expandable --output song-fast-expandable
|
| 76 |
python src/run.py --profile reference --output song-reference
|
| 77 |
```
|
| 78 |
|
|
|
|
|
|
|
| 79 |
`--ode-steps` and `--vae-precision fp32|fp16|bf16` override the profile. VAE checkpoint weights remain FP32 even with FP16 compute. `--low-memory` retains the older unfused reference alternative; it cannot be combined with fast/fused/memory. `--offload-ar` is also available independently. In Python, set `PYTORCH_ALLOC_CONF=expandable_segments:True` **before importing torch** to reproduce the memory CLI allocator. Offloading is sequential; one pipeline instance is not a concurrent request server.
|
| 80 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 81 |
## Runtime v2.1: model-lifetime fix
|
| 82 |
|
| 83 |
OrbitQuant 0.9.4 removes strong references that retained discarded quantized layers. In the YuE2 load/fuse/close probe, retained layers fell from 504 to zero and allocated tensor memory after close fell from 2.36 GB to zero. Arithmetic and weights are unchanged. CPU regression and fullgraph checks passed (574 tests / 85 skips in the full local suite); the PyPI release workflow passed.
|
|
@@ -146,6 +166,6 @@ Earlier unoptimized packed execution was substantially slower than BF16, illustr
|
|
| 146 |
|
| 147 |
## Provenance and license
|
| 148 |
|
| 149 |
-
Source pins are in `source-lock.json`; runtime changes are in `src/runtime/OPTIMIZATIONS.md`; standalone kernel sources and tests are in `src/kernel-source/`. The runtime includes the verified OrbitQuant 0.9.
|
| 150 |
|
| 151 |
Model weights are **CC BY-NC 4.0**, inherited from YuE2. This release does not grant commercial rights. Runtime and kernel components retain their respective code licenses. Credit M-A-P / YuE2 and OrbitQuant when using this derivative.
|
|
|
|
| 29 |
|
| 30 |
2,818,572,288 transformer weights occupy 1,409,286,144 packed bytes. The complete model checkpoint is **3,035,926,840 bytes**, versus **7,261,441,640 bytes** upstream: 2.39× smaller (58.2% reduction). The separate 530,512,720-byte VAE checkpoint is unchanged and downloaded at a pinned revision.
|
| 31 |
|
| 32 |
+
Compatible Q/K/V and gate/up projections are grouped: 224 physical packed modules represent 392 logical projections without duplicated packed storage. OrbitQuant 0.9.5 with native kernels 1.0.2 provides a byte-pair lookup DP4A W4A4 GEMV for 1–8 rows at input widths 1024–16384 (smaller inputs retain the previous GEMV); larger inputs retain the packed Tensor Core path. No persistent decoded BF16 or INT8 weight caches are used. Original AR CUDA graphs and prefix caching were already present upstream and are not credited as new optimizations.
|
| 33 |
|
| 34 |
The default VAE core is 512 frames with the original 16-frame halo. On the recorded Russian song this was waveform-identical to 1024 frames, while lowering the decoder's isolated memory peak. This is a tested example, not a universal bitwise-equivalence guarantee.
|
| 35 |
|
|
|
|
| 40 |
- `examples/`: Russian WAV comparisons and the default prompt in `examples/prompts/`.
|
| 41 |
- `evaluation/`: recorded metrics and validation evidence.
|
| 42 |
|
| 43 |
+
Use `src/run.py` for packed loading. The inherited `auto_map` for the original dense model has been removed: it cannot load these packed weights. Earlier clean-load reports retain their original pinned revision and paths; they describe the earlier release. Runtime v2 changed numerical execution and sampling defaults; v3 adds the byte-pair kernel and optional RMS/activation fusion. Weights are unchanged.
|
| 44 |
|
| 45 |
## Supported runtime
|
| 46 |
|
| 47 |
+
**Runtime v3 validated on NVIDIA RTX PRO 4500 Blackwell (SM120); earlier runtimes were validated on RTX PRO 6000 Blackwell Server Edition and RTX 5090. Linux x86_64, glibc ≥ 2.34, Python 3.12, PyTorch 2.10.0 / CUDA 12.8.** Packaged kernels use Python ABI3. Other GPUs and Torch/CUDA combinations require rebuilding and validation; the runner rejects unsupported GPU capability. This is not a generic `AutoModel.from_pretrained` integration.
|
| 48 |
|
| 49 |
```bash
|
| 50 |
hf download WaveCut/YuE2-3B-OrbitQuant-W4A4 --local-dir YuE2-W4A4
|
|
|
|
| 72 |
```bash
|
| 73 |
python src/run.py --profile fused --output song-fused
|
| 74 |
python src/run.py --profile memory --output song-memory
|
| 75 |
+
python src/run.py --profile fast --fuse-rms-quant --output song-rms
|
| 76 |
+
python src/run.py --profile memory --fuse-rms-quant --output song-rms-memory
|
| 77 |
python src/run.py --profile fast --allocator expandable --output song-fast-expandable
|
| 78 |
python src/run.py --profile reference --output song-reference
|
| 79 |
```
|
| 80 |
|
| 81 |
+
`--fuse-rms-quant` is an experimental fast/memory option: it merges RMSNorm into activation quantization, keeps intermediate products in FP32 and can change the sampled arrangement. The flag is recorded in saved configuration and runtime reports. In Python pass `fuse_rms_quant=True` to `load_pipeline`.
|
| 82 |
+
|
| 83 |
`--ode-steps` and `--vae-precision fp32|fp16|bf16` override the profile. VAE checkpoint weights remain FP32 even with FP16 compute. `--low-memory` retains the older unfused reference alternative; it cannot be combined with fast/fused/memory. `--offload-ar` is also available independently. In Python, set `PYTORCH_ALLOC_CONF=expandable_segments:True` **before importing torch** to reproduce the memory CLI allocator. Offloading is sequential; one pipeline instance is not a concurrent request server.
|
| 84 |
|
| 85 |
+
## Runtime v3: byte-pair decode and optional RMS fusion
|
| 86 |
+
|
| 87 |
+
Native 1.0.2 replaces two nibble lookups with one byte-pair lookup in eligible decode projections. `--fuse-rms-quant` additionally folds RMSNorm into activation quantization, including its epsilon handling. The option is implemented directly in the runtime and supports fast/memory profiles, two-branch CFG, cancellation, and expert staging. It is disabled by default because the changed intermediate rounding can produce a different arrangement.
|
| 88 |
+
|
| 89 |
+
RTX PRO 4500 Blackwell, OrbitQuant 0.9.5, 8 midpoint steps, FP16 VAE, expandable allocator. Two requests per fresh process, warm request shown; loading excluded. Fast with and without RMS fusion generate different arrangements. The two RMS-fused rows have identical semantic tokens, latents, and decoded audio.
|
| 90 |
+
|
| 91 |
+
| Profile | Audio seconds | Warm generation seconds | NVML peak GiB |
|
| 92 |
+
|---|---:|---:|---:|
|
| 93 |
+
| fast | 187.64 | 30.49 | 5.55 |
|
| 94 |
+
| fast + RMS/activation fusion | 183.48 | 28.50 | 5.41 |
|
| 95 |
+
| memory + RMS/activation fusion | 183.48 | 29.82 | 5.33 |
|
| 96 |
+
|
| 97 |
+
The independent interleaved kernel comparison reduced identical 187.64 s audio from 31.82 to 30.34 s. Adding the byte-pair kernel to the same RMS-fused 183.48 s song reduced 29.81 to 28.28 s. Those are separate timing experiments; their pooled-process memory is not compared with this table. The saved 128-position native decode comparison was exact and improved 3.148 to 2.943 ms per token. RMS fusion itself changes rounding: its earlier 128-position probe versus unfused fast had mean KL 0.0266 nats and top-1 agreement 84.38%; whole-song normalized log-band energy differed by 2.76 dB. These are diagnostics, not perceptual scores.
|
| 98 |
+
|
| 99 |
+
Measurements and positive/negative runtime contract checks are in `evaluation/runtime-v3/`. The Russian RMS-fused example is [available here](examples/runtime-v3/fast-rms.wav). Earlier listening milestones and numerical policies remain available below.
|
| 100 |
+
|
| 101 |
## Runtime v2.1: model-lifetime fix
|
| 102 |
|
| 103 |
OrbitQuant 0.9.4 removes strong references that retained discarded quantized layers. In the YuE2 load/fuse/close probe, retained layers fell from 504 to zero and allocated tensor memory after close fell from 2.36 GB to zero. Arithmetic and weights are unchanged. CPU regression and fullgraph checks passed (574 tests / 85 skips in the full local suite); the PyPI release workflow passed.
|
|
|
|
| 166 |
|
| 167 |
## Provenance and license
|
| 168 |
|
| 169 |
+
Source pins are in `source-lock.json`; runtime changes are in `src/runtime/OPTIMIZATIONS.md`; standalone kernel sources and tests are in `src/kernel-source/`. The runtime includes the verified OrbitQuant 0.9.5 wheel and native kernels 1.0.2; `src/runtime-lock.json` records their provenance. Weights were generated with OrbitQuant 0.9.2, so the original weight source lock and quantization manifest intentionally retain that version. Original licenses and third-party notices are retained.
|
| 170 |
|
| 171 |
Model weights are **CC BY-NC 4.0**, inherited from YuE2. This release does not grant commercial rights. Runtime and kernel components retain their respective code licenses. Credit M-A-P / YuE2 and OrbitQuant when using this derivative.
|
SHA256SUMS
CHANGED
|
@@ -1,5 +1,5 @@
|
|
| 1 |
060985741d20e70613b4c189c7de106cabd3fb2109fbbfb6d705c9d619417dd0 LICENSE
|
| 2 |
-
|
| 3 |
14d3fd9f6fee86b4260b69b0979735b99ffec8c6b4db254567047a4215cd9af3 THIRD_PARTY_NOTICES.md
|
| 4 |
e479ab4f807b93d4c945ae9a14c5269dafb34751ee059d29f9883a5e812f2e56 config.json
|
| 5 |
5e375ad03c84d8118f3b37e56a3d01cf137b5aa1526b5725d91b17857839616d evaluation/ar-fused-fixed.json
|
|
@@ -37,10 +37,15 @@ baa19f27eca20bc767b67057910e10d78f066e9f6e5ed45df639f0ec9678552d evaluation/run
|
|
| 37 |
142bb2808102bbd36a5266d7d28b2173f3522a514081957600be06d842339530 evaluation/runtime-v2/pro4500-reference.json
|
| 38 |
96a39130f0b58355a872edb608713bc57f65e7b0ef921081b022f14b27bc4df3 evaluation/runtime-v2/reference.json
|
| 39 |
a1badc0e42720b32b8259a55cbfee8a9e2d9537296ee517ed6c575b432291d29 evaluation/runtime-v2/spectra.json
|
|
|
|
|
|
|
|
|
|
|
|
|
| 40 |
00da89c0d8a7242efb498fc30f221fab25b0becc127a61079a528d333be229df evaluation/vae-chunk-benchmark.json
|
| 41 |
6daa417ffd2972c8a7f681083ca59f131a03b9f5caf62871a9f37ec30905179a examples/orbitquant-ru.wav
|
| 42 |
4879336415f868fe92e0d0d21733dea194c09f8deb10484c27add434950b4bb6 examples/original-ru.wav
|
| 43 |
a67f3becd1edcc3c42c2602735b47d455bf76cf1258862a3fe5978cf87fb16b0 examples/prompts/default-ru.json
|
|
|
|
| 44 |
ccc842772f029049fadf9b288393bcdd9f79271b361c0f1875ec1218e4ba5170 generation_config.json
|
| 45 |
da9858d516047d82096d01c112a61bd67f26d289039464d668a1d45f91738ecc licenses/SnakeBeta-NVIDIA-MIT.txt
|
| 46 |
a1fac33b7bcd791b74fb33aeb439f825e7277e239fc119fb7d2ab6f084a0c101 licenses/stable-audio-tools-MIT.txt
|
|
@@ -48,7 +53,7 @@ e169774b8e79f423e6e8586221558acd28a35bacc67aae912d9a8a905231134a model.safetens
|
|
| 48 |
c9fe5177b33b8727cdf97d946ba173645a490e8f30181c55a3f172b9ed852979 orbitquant_manifest.json
|
| 49 |
b2b1b8dfb5cc5f024bafc373121c6aba3f66f9a5a0269e243470a1de16a33186 qwen.tiktoken
|
| 50 |
bd90ffcd3768e450363a5b583cb6153966417f818b08427d0e8f180b1d33944a source-lock.json
|
| 51 |
-
|
| 52 |
49a3cfbb640f0c2dc098630907db27f5e60b7fb810ba3d1d31dcff98ac5579c8 src/evaluation_harness/ar_bench.py
|
| 53 |
7faa379989e0a77c28be169a1ea91a0f4b960c7f1d949479a8b18511083179c4 src/evaluation_harness/ar_quality.py
|
| 54 |
863e3137fb6a429c7439208484aa79dcfc1fc81fa3286f4dfe39301bcd165fd0 src/evaluation_harness/asr_evaluate.py
|
|
@@ -94,7 +99,7 @@ d0193d3790e87ca7e4bd7b3afa2eb399a6507150273dbce54e548829520d811c src/kernel-sou
|
|
| 94 |
4de6806c3d18b3b6421ed8a4b3e50e1308e9963721a4b93ea30a8419a84c0e79 src/kernel-source/orbitquant-gemv/torch-ext/registration.h
|
| 95 |
4228ded5775db7e8c4071a412e8b87575bd02c5bc2197009fcb08734cbffd134 src/kernel-source/orbitquant-gemv/torch-ext/torch_binding.cpp
|
| 96 |
ad6d0f6601ee0e2e126164fad6e4ac572fc5820544ce71f79e2ba152bb08e7a1 src/kernel-source/orbitquant-gemv/torch-ext/torch_binding.h
|
| 97 |
-
|
| 98 |
7a9ffddcd684ec325c9ef87b3ebdb2fdbbb0c2747dfc6e1d63144a920252b565 src/kernel-source/orbitquant-packed-matmul/benchmarks/benchmark.py
|
| 99 |
1e51ee864385d2116ca5329c8b31091c8bdb81f9446094e9a92121e0b2390842 src/kernel-source/orbitquant-packed-matmul/benchmarks/benchmark_decode.py
|
| 100 |
59f8eafa81f046711b70b2340b48c6b9be1775353feae95e4310f62331dee2be src/kernel-source/orbitquant-packed-matmul/build.toml
|
|
@@ -116,16 +121,52 @@ dcf9baa1b69fbf4bf3621ee4037bab9d8002fdabf3802f8a22c3636dcb9d4c3d src/kernel-sou
|
|
| 116 |
bf91916d187fc1c6f387029a354d57379297384b8e414deab074858a284b274d src/kernel-source/orbitquant-packed-matmul/orbitquant_packed_matmul_cpu/packed_matmul_x86.cpp
|
| 117 |
a90525ad3c2c30fe12cd4266d50c3ac391b2861ea37e885be0da3cb599df7eb0 src/kernel-source/orbitquant-packed-matmul/orbitquant_packed_matmul_cpu/packed_matmul_x86_avx512.cpp
|
| 118 |
0781a49f03f263d384fc1e66a3eb5ed196c25373d2811b7880b4496662ea048c src/kernel-source/orbitquant-packed-matmul/orbitquant_packed_matmul_cpu/quantize_activations_cpu.cpp
|
| 119 |
-
|
| 120 |
02d02f3b8c271507fd63de99609b155e201a089f476a3e4a4338bc568cebdae8 src/kernel-source/orbitquant-packed-matmul/orbitquant_packed_matmul_metal/packed_matmul.metal
|
| 121 |
4e600bb69569f831f0cab0e4298ec82a19e9a4406dc3d77bb1b6d11413cdb5dc src/kernel-source/orbitquant-packed-matmul/orbitquant_packed_matmul_metal/packed_matmul.mm
|
| 122 |
33b4a2c75b1d34f0810274230455e137ff920e0f90a95009a3d0ce5bd681126d src/kernel-source/orbitquant-packed-matmul/scripts/prepare_wheel_project.py
|
| 123 |
32aa8cb8f2c2cdd29c68e70413227d96d15df6688c3e8bf8028c60c99e20d33b src/kernel-source/orbitquant-packed-matmul/tests/__init__.py
|
| 124 |
-
|
| 125 |
7e192d41a6a93f1b63f395c21383c7722a0c69d90c696629a57a2c5aeaf6794f src/kernel-source/orbitquant-packed-matmul/tests/test_packed_matmul.py
|
| 126 |
414ecd0cbfc1a6bc66e0a6e2aeaa1bb3ac433d0172099af7071806bb41234950 src/kernel-source/orbitquant-packed-matmul/torch-ext/orbitquant_packed_matmul/__init__.py
|
| 127 |
8782bd8bef9d52708606053d1e85bf44c032bec14cf22beeed420e26dcd110bf src/kernel-source/orbitquant-packed-matmul/torch-ext/torch_binding.cpp
|
| 128 |
8af7352374d8694a4870c89d6998ae8df4fe34ccb2a08b261fe4e24913da251f src/kernel-source/orbitquant-packed-matmul/torch-ext/torch_binding.h
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 129 |
9dad1cdeac2b3861bf7cce889ec8ae50133999eb76a93a7bfd8528ca7a5364c9 src/kernel-source/yue2-fast/.gitattributes
|
| 130 |
44575cf5b28512d75644bf54a517dcef304ff809fd511747621b4d64f19aac66 src/kernel-source/yue2-fast/.gitignore
|
| 131 |
89de5c024ac81e58a2f9e2a2484f424d6823fee075c4403e29e279e04e07c88c src/kernel-source/yue2-fast/CARD.md
|
|
@@ -164,20 +205,25 @@ da90c2dc55ecca5ffd2f7b15d1968180d5744a5b31ade45a7d221041096bdebe src/kernel-sou
|
|
| 164 |
ac54a84ce2bb7a0742a1cd46df398aeb23d83ebde9ed8c7cc0f7d914b59837dd src/kernel-source/yue2-fast/torch-ext/torch_binding.h
|
| 165 |
a368cb32d65829479d15c19451fe3199cf7a2d6510891a9596aa08fc4ed746dd src/kernel-source/yue2-fast/torch-ext/yue2_qkv_fused/__init__.py
|
| 166 |
62d2bd758c8fff1a0d64584a699343076fca9a7880db5d8e19263bc30ae79d7c src/modeling_yue2.py
|
| 167 |
-
|
| 168 |
-
|
| 169 |
-
|
| 170 |
-
|
| 171 |
-
|
| 172 |
026e36da4a42cf2095b75ebacb250cf8e20f5011763b2f0e44247afabdb9d112 src/runtime/fuse_packed.py
|
| 173 |
414ecd0cbfc1a6bc66e0a6e2aeaa1bb3ac433d0172099af7071806bb41234950 src/runtime/kernels/base/orbitquant_packed_matmul/__init__.py
|
| 174 |
c1df0cc119cdfe0471754469458fa83fc718c6f42feca0b79a0d07cb2c068ce5 src/runtime/kernels/base/orbitquant_packed_matmul/_ops.py
|
| 175 |
-
|
| 176 |
a368cb32d65829479d15c19451fe3199cf7a2d6510891a9596aa08fc4ed746dd src/runtime/kernels/fast/torch210-cxx11-cu128-x86_64-linux/__init__.py
|
| 177 |
44aec945e18569118863f1eda191d273eabea41e5b6589771e5104eab3936d5c src/runtime/kernels/fast/torch210-cxx11-cu128-x86_64-linux/_ops.py
|
| 178 |
b0a4d667dc1e0b1701f0bf9371254c293e30e974892e5681341980c9fe4d00ef src/runtime/kernels/fast/torch210-cxx11-cu128-x86_64-linux/_yue2_qkv_fused_cuda_46ceaaf_dirty.abi3.so
|
| 179 |
0a68364ee298078b690fc2360e714d05da28fb932bf3ad6eb0c9c5b3e1bb7b24 src/runtime/kernels/fast/torch210-cxx11-cu128-x86_64-linux/metadata.json
|
| 180 |
0c560f96b857c188c4a8297ff27d1299619957c3459a5e433e132329fbfdf066 src/runtime/kernels/fast/torch210-cxx11-cu128-x86_64-linux/yue2_qkv_fused/__init__.py
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 181 |
060985741d20e70613b4c189c7de106cabd3fb2109fbbfb6d705c9d619417dd0 src/runtime/licenses/LICENSE
|
| 182 |
14d3fd9f6fee86b4260b69b0979735b99ffec8c6b4db254567047a4215cd9af3 src/runtime/licenses/THIRD_PARTY_NOTICES.md
|
| 183 |
da9858d516047d82096d01c112a61bd67f26d289039464d668a1d45f91738ecc src/runtime/licenses/licenses/SnakeBeta-NVIDIA-MIT.txt
|
|
@@ -185,7 +231,7 @@ a1fac33b7bcd791b74fb33aeb439f825e7277e239fc119fb7d2ab6f084a0c101 src/runtime/li
|
|
| 185 |
e6a6b3c798f697ee3011e6b817d0620ca79824c823e56c6c0748889c0a5b4407 src/runtime/yue2/__init__.py
|
| 186 |
3a25793081ed10325225e4e7bd9303243998abecd3a1a5b7a15139d664b70604 src/runtime/yue2/cli.py
|
| 187 |
617ee6914d534a234f1ff5d5f4685391eac5dc9f6be667429949eabf1e541517 src/runtime/yue2/compact_sampling.py
|
| 188 |
-
|
| 189 |
e1722959ad0b0bf27eeaaf019cb9f109212a2450f03dd145866b1a0239151f38 src/runtime/yue2/fast.py
|
| 190 |
e6523f113132df6c04878103bca1a8b3de519fefcbc0117e97455f0dd49d1fba src/runtime/yue2/modeling_vae.py
|
| 191 |
62d2bd758c8fff1a0d64584a699343076fca9a7880db5d8e19263bc30ae79d7c src/runtime/yue2/modeling_yue2.py
|
|
@@ -197,7 +243,7 @@ e00d32bf22baac46c7d4dafaeecd4f3d5a4753d651b3dc8f4164b9f3774980ad src/runtime/yu
|
|
| 197 |
63731de35665ac426130c781f3ba7ff1159c58d34a9b3e6ee2e78ff0c5881ff9 src/runtime/yue2/sampling.py
|
| 198 |
a5bcc181b16412b45bc560e5264ba0b303667b5fd81957991ac0266c87fe2a5d src/runtime/yue2/storage.py
|
| 199 |
fc88387f96e5db2020a0c3f5374b82726206026d83159b68b83e4136bb741acc src/runtime/yue2/tokenization_yue2.py
|
| 200 |
-
|
| 201 |
-
|
| 202 |
46ce45128d77c97565aac511de8f000c098cc978b2694304c8475cea8b01a1d1 weights_manifest.json
|
| 203 |
203830cebde6e3644eb291925362990d198e66c3b6006cd11f1aaa21904bcc61 yue2_generation_config.json
|
|
|
|
| 1 |
060985741d20e70613b4c189c7de106cabd3fb2109fbbfb6d705c9d619417dd0 LICENSE
|
| 2 |
+
09ebd5b472d0ee3b2e4ceda1bbb8169dee42196ea31a84a39dc4a0c6b5e54105 README.md
|
| 3 |
14d3fd9f6fee86b4260b69b0979735b99ffec8c6b4db254567047a4215cd9af3 THIRD_PARTY_NOTICES.md
|
| 4 |
e479ab4f807b93d4c945ae9a14c5269dafb34751ee059d29f9883a5e812f2e56 config.json
|
| 5 |
5e375ad03c84d8118f3b37e56a3d01cf137b5aa1526b5725d91b17857839616d evaluation/ar-fused-fixed.json
|
|
|
|
| 37 |
142bb2808102bbd36a5266d7d28b2173f3522a514081957600be06d842339530 evaluation/runtime-v2/pro4500-reference.json
|
| 38 |
96a39130f0b58355a872edb608713bc57f65e7b0ef921081b022f14b27bc4df3 evaluation/runtime-v2/reference.json
|
| 39 |
a1badc0e42720b32b8259a55cbfee8a9e2d9537296ee517ed6c575b432291d29 evaluation/runtime-v2/spectra.json
|
| 40 |
+
d27cd1f8f8c5b8dc667e5bedb7c5a4ac1c72aae8cac83c25b794caba544b58c9 evaluation/runtime-v3/contract-checks.json
|
| 41 |
+
04f7ce93db3d265c80fe5302f8cd7bf8151deb02e51ff9c7096cbeb7273ab64f evaluation/runtime-v3/fast-rms0-s8.json
|
| 42 |
+
a9a2dc4587cf5c842b4eb48f1ef1abe655109426c0b9ae042e30871a0d49048f evaluation/runtime-v3/fast-rms1-s8.json
|
| 43 |
+
370cc04e8caf6fb345197e9e76161452abe62d077a1388dde6385472b5935276 evaluation/runtime-v3/memory-rms1-s8.json
|
| 44 |
00da89c0d8a7242efb498fc30f221fab25b0becc127a61079a528d333be229df evaluation/vae-chunk-benchmark.json
|
| 45 |
6daa417ffd2972c8a7f681083ca59f131a03b9f5caf62871a9f37ec30905179a examples/orbitquant-ru.wav
|
| 46 |
4879336415f868fe92e0d0d21733dea194c09f8deb10484c27add434950b4bb6 examples/original-ru.wav
|
| 47 |
a67f3becd1edcc3c42c2602735b47d455bf76cf1258862a3fe5978cf87fb16b0 examples/prompts/default-ru.json
|
| 48 |
+
30a70e2224b008e08f10ba0e0f9b8243b06b32a4fcbd7381e980d68cd5a7974e examples/runtime-v3/fast-rms.wav
|
| 49 |
ccc842772f029049fadf9b288393bcdd9f79271b361c0f1875ec1218e4ba5170 generation_config.json
|
| 50 |
da9858d516047d82096d01c112a61bd67f26d289039464d668a1d45f91738ecc licenses/SnakeBeta-NVIDIA-MIT.txt
|
| 51 |
a1fac33b7bcd791b74fb33aeb439f825e7277e239fc119fb7d2ab6f084a0c101 licenses/stable-audio-tools-MIT.txt
|
|
|
|
| 53 |
c9fe5177b33b8727cdf97d946ba173645a490e8f30181c55a3f172b9ed852979 orbitquant_manifest.json
|
| 54 |
b2b1b8dfb5cc5f024bafc373121c6aba3f66f9a5a0269e243470a1de16a33186 qwen.tiktoken
|
| 55 |
bd90ffcd3768e450363a5b583cb6153966417f818b08427d0e8f180b1d33944a source-lock.json
|
| 56 |
+
331c06a248811f778f9f5ed280101e86bc0dd580e80db85619893148ff885720 src/OPTIMIZATION_IDEAS.md
|
| 57 |
49a3cfbb640f0c2dc098630907db27f5e60b7fb810ba3d1d31dcff98ac5579c8 src/evaluation_harness/ar_bench.py
|
| 58 |
7faa379989e0a77c28be169a1ea91a0f4b960c7f1d949479a8b18511083179c4 src/evaluation_harness/ar_quality.py
|
| 59 |
863e3137fb6a429c7439208484aa79dcfc1fc81fa3286f4dfe39301bcd165fd0 src/evaluation_harness/asr_evaluate.py
|
|
|
|
| 99 |
4de6806c3d18b3b6421ed8a4b3e50e1308e9963721a4b93ea30a8419a84c0e79 src/kernel-source/orbitquant-gemv/torch-ext/registration.h
|
| 100 |
4228ded5775db7e8c4071a412e8b87575bd02c5bc2197009fcb08734cbffd134 src/kernel-source/orbitquant-gemv/torch-ext/torch_binding.cpp
|
| 101 |
ad6d0f6601ee0e2e126164fad6e4ac572fc5820544ce71f79e2ba152bb08e7a1 src/kernel-source/orbitquant-gemv/torch-ext/torch_binding.h
|
| 102 |
+
ed46fd7deb251b44cda8b238e0b4dc44b9b0f84c692043e82cd5e4b58cd5a6ca src/kernel-source/orbitquant-packed-matmul/CARD.md
|
| 103 |
7a9ffddcd684ec325c9ef87b3ebdb2fdbbb0c2747dfc6e1d63144a920252b565 src/kernel-source/orbitquant-packed-matmul/benchmarks/benchmark.py
|
| 104 |
1e51ee864385d2116ca5329c8b31091c8bdb81f9446094e9a92121e0b2390842 src/kernel-source/orbitquant-packed-matmul/benchmarks/benchmark_decode.py
|
| 105 |
59f8eafa81f046711b70b2340b48c6b9be1775353feae95e4310f62331dee2be src/kernel-source/orbitquant-packed-matmul/build.toml
|
|
|
|
| 121 |
bf91916d187fc1c6f387029a354d57379297384b8e414deab074858a284b274d src/kernel-source/orbitquant-packed-matmul/orbitquant_packed_matmul_cpu/packed_matmul_x86.cpp
|
| 122 |
a90525ad3c2c30fe12cd4266d50c3ac391b2861ea37e885be0da3cb599df7eb0 src/kernel-source/orbitquant-packed-matmul/orbitquant_packed_matmul_cpu/packed_matmul_x86_avx512.cpp
|
| 123 |
0781a49f03f263d384fc1e66a3eb5ed196c25373d2811b7880b4496662ea048c src/kernel-source/orbitquant-packed-matmul/orbitquant_packed_matmul_cpu/quantize_activations_cpu.cpp
|
| 124 |
+
4036bdf8916a85aa623e90522cb54833c32a0ee25312fd023679d9802beb77cc src/kernel-source/orbitquant-packed-matmul/orbitquant_packed_matmul_cuda/packed_matmul.cu
|
| 125 |
02d02f3b8c271507fd63de99609b155e201a089f476a3e4a4338bc568cebdae8 src/kernel-source/orbitquant-packed-matmul/orbitquant_packed_matmul_metal/packed_matmul.metal
|
| 126 |
4e600bb69569f831f0cab0e4298ec82a19e9a4406dc3d77bb1b6d11413cdb5dc src/kernel-source/orbitquant-packed-matmul/orbitquant_packed_matmul_metal/packed_matmul.mm
|
| 127 |
33b4a2c75b1d34f0810274230455e137ff920e0f90a95009a3d0ce5bd681126d src/kernel-source/orbitquant-packed-matmul/scripts/prepare_wheel_project.py
|
| 128 |
32aa8cb8f2c2cdd29c68e70413227d96d15df6688c3e8bf8028c60c99e20d33b src/kernel-source/orbitquant-packed-matmul/tests/__init__.py
|
| 129 |
+
2ec1604258719eb1388496249b1c7cb59f6a53e8501189f10d4374c3e9700043 src/kernel-source/orbitquant-packed-matmul/tests/test_decode_gemv.py
|
| 130 |
7e192d41a6a93f1b63f395c21383c7722a0c69d90c696629a57a2c5aeaf6794f src/kernel-source/orbitquant-packed-matmul/tests/test_packed_matmul.py
|
| 131 |
414ecd0cbfc1a6bc66e0a6e2aeaa1bb3ac433d0172099af7071806bb41234950 src/kernel-source/orbitquant-packed-matmul/torch-ext/orbitquant_packed_matmul/__init__.py
|
| 132 |
8782bd8bef9d52708606053d1e85bf44c032bec14cf22beeed420e26dcd110bf src/kernel-source/orbitquant-packed-matmul/torch-ext/torch_binding.cpp
|
| 133 |
8af7352374d8694a4870c89d6998ae8df4fe34ccb2a08b261fe4e24913da251f src/kernel-source/orbitquant-packed-matmul/torch-ext/torch_binding.h
|
| 134 |
+
9dad1cdeac2b3861bf7cce889ec8ae50133999eb76a93a7bfd8528ca7a5364c9 src/kernel-source/rms-quant/.gitattributes
|
| 135 |
+
44575cf5b28512d75644bf54a517dcef304ff809fd511747621b4d64f19aac66 src/kernel-source/rms-quant/.gitignore
|
| 136 |
+
163420273229ab3481f39d2f35984e0e93ca88848bb09595ab46404e4f8edfcb src/kernel-source/rms-quant/CARD.md
|
| 137 |
+
85ed8c1d9e3e9e5b0b61e638d18898fb71a5624d34be61b72f49a64020f6e959 src/kernel-source/rms-quant/build.toml
|
| 138 |
+
933e544bcbad82151547591748831765cc21b6d55564055989859ccbe905a0d7 src/kernel-source/rms-quant/cmake/_ops.py.in
|
| 139 |
+
f2ac9825685c35b0c96cd47f9b9bc7790754f16e3236ad558f97d0d1f9ff74de src/kernel-source/rms-quant/cmake/add_build_metadata.py
|
| 140 |
+
c6a0ea1116893886ab52b4559821d4e42c8830ed644980e58f22ac58b3cce80a src/kernel-source/rms-quant/cmake/build-variants.cmake
|
| 141 |
+
3c4a3ab628be03ff2be0771b43281e4f8e0c01bfd1530c7ca18967cbd22ac285 src/kernel-source/rms-quant/cmake/compile-metal.cmake
|
| 142 |
+
28f11b0b855fe9ea47b2a5b46bf50556388380b91396e1007a2302c4908b01be src/kernel-source/rms-quant/cmake/get_gpu_lang.cmake
|
| 143 |
+
2f8fb8e6d51d8265b34ce54a630bd9154839ccc3ba331e57634b996589bf4739 src/kernel-source/rms-quant/cmake/get_gpu_lang.py
|
| 144 |
+
5cfc2220d20f0587327f37ca27e0d74cdbdea2540ce6ae2032569fb995b79ac8 src/kernel-source/rms-quant/cmake/hipify.py
|
| 145 |
+
20d526820f9e4867bae370c55280bc8e853908658fe6caff29dab7101e94f46c src/kernel-source/rms-quant/cmake/kernel.cmake
|
| 146 |
+
0defa64521b9093e4509eadd3059e609cc04dfab4b085426d7e7f1770971f0eb src/kernel-source/rms-quant/cmake/metallib_to_header.py
|
| 147 |
+
757019d6d92eb0c09c26d43b9a83b1c8838ce5392b78f47388cfd81c235fca08 src/kernel-source/rms-quant/cmake/utils.cmake
|
| 148 |
+
0c560f96b857c188c4a8297ff27d1299619957c3459a5e433e132329fbfdf066 src/kernel-source/rms-quant/compat.py
|
| 149 |
+
bc6a259827061020c2f5ae9a5dc1e947bb0c81abe81dec1abf507b15270dfa16 src/kernel-source/rms-quant/flake.nix
|
| 150 |
+
1d1d6efe831924e14fd729ec59035c9bcc9b65ce9497f2c99219ab9bea71ad3d src/kernel-source/rms-quant/metadata-cann.json
|
| 151 |
+
03e00877b6f8b923c2162dbd564089f694047d9ebdb5dadea81bd8f5e0140617 src/kernel-source/rms-quant/metadata-cpu.json
|
| 152 |
+
13e39897751174039c2f62a6013f9ddd605596e659f29d2547388bb79fdfb34b src/kernel-source/rms-quant/metadata-cuda.json
|
| 153 |
+
7932dba1c62fbd904ad0dd2bd2ead57a03f050eaca5c0b5cbc9e1d76f6c9ae0d src/kernel-source/rms-quant/metadata-metal.json
|
| 154 |
+
0acdd9dfef658f58dd087fbe95a0886ae4e8495ba0ba666eb5365e36518ad591 src/kernel-source/rms-quant/metadata-neuron.json
|
| 155 |
+
99234c78d40650bfcc9c8e430b3857cc769f06473dee850a943baaa182dc5080 src/kernel-source/rms-quant/metadata-rocm.json
|
| 156 |
+
314e0552e15b459f9ca3e31029ab73ccfa6efa4188cc0c35e8ec5aa8af7a2a4e src/kernel-source/rms-quant/metadata-xpu.json
|
| 157 |
+
6128f79d7ec43624eff68f40f13037f4c261ae748ddb6882abf7a5b7841cd82f src/kernel-source/rms-quant/orbitquant_gemv_cuda/fused_ops.cu
|
| 158 |
+
b930e00662b3e0f2ad381f8769f86c750761c3ce476eb7ae42671c50eb43f473 src/kernel-source/rms-quant/orbitquant_gemv_cuda/orbitquant_gemv.cu
|
| 159 |
+
062ff955c58c2efa111c611cbd733afd259e138f862a56255e5d5f8b45f90de5 src/kernel-source/rms-quant/orbitquant_gemv_cuda/qkv_decode.cu
|
| 160 |
+
92e9a8e1c80976451255939b190cf8032a12e9ce36c2c2d224ddbda7edb2cd8d src/kernel-source/rms-quant/orbitquant_gemv_cuda/rmsquant.cu
|
| 161 |
+
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 src/kernel-source/rms-quant/tests/__init__.py
|
| 162 |
+
6760124522ecb74b0e7ae5a63d0d5094ff4c98f1de48fc69dfa6548f22f2eff1 src/kernel-source/rms-quant/tests/test_fused_ops.py
|
| 163 |
+
b2275e6926b515bebf8ae4de0adeaf10a84c40e1441b10430541a68323c51a9e src/kernel-source/rms-quant/tests/test_orbitquant_gemv.py
|
| 164 |
+
aac2795a2000382e177737078a5698abb9bafd83fbae2a56c858d6b2ca5d3f95 src/kernel-source/rms-quant/tests/test_qkv_decode.py
|
| 165 |
+
608939f6f4cb359018d2ea49a1312c5f9dd979b8b5c348a341550347b0c756c8 src/kernel-source/rms-quant/tests/test_rmsquant.py
|
| 166 |
+
4de6806c3d18b3b6421ed8a4b3e50e1308e9963721a4b93ea30a8419a84c0e79 src/kernel-source/rms-quant/torch-ext/registration.h
|
| 167 |
+
5b24f09a78fa634db9963ba96178fb6f2b99085cfff1f6c35b5a6209e1a720d9 src/kernel-source/rms-quant/torch-ext/torch_binding.cpp
|
| 168 |
+
0b95faa0b766fba75859b66c4121638a372df28807207b318cb4a5833506d5e4 src/kernel-source/rms-quant/torch-ext/torch_binding.h
|
| 169 |
+
02e9eb8b6490f31c840fbaa70371189882303141f52d2e47043499dcf4f5be38 src/kernel-source/rms-quant/torch-ext/yue2_qkv_fused/__init__.py
|
| 170 |
9dad1cdeac2b3861bf7cce889ec8ae50133999eb76a93a7bfd8528ca7a5364c9 src/kernel-source/yue2-fast/.gitattributes
|
| 171 |
44575cf5b28512d75644bf54a517dcef304ff809fd511747621b4d64f19aac66 src/kernel-source/yue2-fast/.gitignore
|
| 172 |
89de5c024ac81e58a2f9e2a2484f424d6823fee075c4403e29e279e04e07c88c src/kernel-source/yue2-fast/CARD.md
|
|
|
|
| 205 |
ac54a84ce2bb7a0742a1cd46df398aeb23d83ebde9ed8c7cc0f7d914b59837dd src/kernel-source/yue2-fast/torch-ext/torch_binding.h
|
| 206 |
a368cb32d65829479d15c19451fe3199cf7a2d6510891a9596aa08fc4ed746dd src/kernel-source/yue2-fast/torch-ext/yue2_qkv_fused/__init__.py
|
| 207 |
62d2bd758c8fff1a0d64584a699343076fca9a7880db5d8e19263bc30ae79d7c src/modeling_yue2.py
|
| 208 |
+
0c0e34419a8d8c3f95cfb3113f2453616d8a906c67ee919cad4f47be127ee939 src/requirements.txt
|
| 209 |
+
8f9c3e42202c02e33dcaa25ef4090ef5d8f8e53f4747f3757104091960893c21 src/run.py
|
| 210 |
+
919ee3f67cf15f6bc2cdb9b7e8eb2a9387fe1401dd452758b56d980fa5516c80 src/runtime-lock.json
|
| 211 |
+
fd1b581582dfe3e2503e7f7625d807129c90135692bddeb411bda559a05becee src/runtime/OPTIMIZATIONS.md
|
| 212 |
+
e70020c928aee0ba295d3aa71abb9d1331c9cc28898c88f66f1dbe7ffe41cb6b src/runtime/fast_ops.py
|
| 213 |
026e36da4a42cf2095b75ebacb250cf8e20f5011763b2f0e44247afabdb9d112 src/runtime/fuse_packed.py
|
| 214 |
414ecd0cbfc1a6bc66e0a6e2aeaa1bb3ac433d0172099af7071806bb41234950 src/runtime/kernels/base/orbitquant_packed_matmul/__init__.py
|
| 215 |
c1df0cc119cdfe0471754469458fa83fc718c6f42feca0b79a0d07cb2c068ce5 src/runtime/kernels/base/orbitquant_packed_matmul/_ops.py
|
| 216 |
+
96839947a05b59c11a781346184fc344ed6af96404081ba1e8b9ad5fa29d828a src/runtime/kernels/base/orbitquant_packed_matmul/_orbitquant_packed_matmul_cuda_rel_cu128_t210.abi3.so
|
| 217 |
a368cb32d65829479d15c19451fe3199cf7a2d6510891a9596aa08fc4ed746dd src/runtime/kernels/fast/torch210-cxx11-cu128-x86_64-linux/__init__.py
|
| 218 |
44aec945e18569118863f1eda191d273eabea41e5b6589771e5104eab3936d5c src/runtime/kernels/fast/torch210-cxx11-cu128-x86_64-linux/_ops.py
|
| 219 |
b0a4d667dc1e0b1701f0bf9371254c293e30e974892e5681341980c9fe4d00ef src/runtime/kernels/fast/torch210-cxx11-cu128-x86_64-linux/_yue2_qkv_fused_cuda_46ceaaf_dirty.abi3.so
|
| 220 |
0a68364ee298078b690fc2360e714d05da28fb932bf3ad6eb0c9c5b3e1bb7b24 src/runtime/kernels/fast/torch210-cxx11-cu128-x86_64-linux/metadata.json
|
| 221 |
0c560f96b857c188c4a8297ff27d1299619957c3459a5e433e132329fbfdf066 src/runtime/kernels/fast/torch210-cxx11-cu128-x86_64-linux/yue2_qkv_fused/__init__.py
|
| 222 |
+
02e9eb8b6490f31c840fbaa70371189882303141f52d2e47043499dcf4f5be38 src/runtime/kernels/rms/torch210-cxx11-cu128-x86_64-linux/__init__.py
|
| 223 |
+
1c4350939f2e52581663e8694cb869926bf2dcc04beb80b9c09233571dd1695f src/runtime/kernels/rms/torch210-cxx11-cu128-x86_64-linux/_ops.py
|
| 224 |
+
1ca79342c379a231d86fa5c8e94e45134c3624d7b85ca5e55997004ec69ff542 src/runtime/kernels/rms/torch210-cxx11-cu128-x86_64-linux/_yue2_qkv_fused_cuda_cf0db58.abi3.so
|
| 225 |
+
40e4a06eb58f41ef286f95a434bfb06d10742387e9cb47219d2d6b088a59bf3d src/runtime/kernels/rms/torch210-cxx11-cu128-x86_64-linux/metadata.json
|
| 226 |
+
0c560f96b857c188c4a8297ff27d1299619957c3459a5e433e132329fbfdf066 src/runtime/kernels/rms/torch210-cxx11-cu128-x86_64-linux/yue2_qkv_fused/__init__.py
|
| 227 |
060985741d20e70613b4c189c7de106cabd3fb2109fbbfb6d705c9d619417dd0 src/runtime/licenses/LICENSE
|
| 228 |
14d3fd9f6fee86b4260b69b0979735b99ffec8c6b4db254567047a4215cd9af3 src/runtime/licenses/THIRD_PARTY_NOTICES.md
|
| 229 |
da9858d516047d82096d01c112a61bd67f26d289039464d668a1d45f91738ecc src/runtime/licenses/licenses/SnakeBeta-NVIDIA-MIT.txt
|
|
|
|
| 231 |
e6a6b3c798f697ee3011e6b817d0620ca79824c823e56c6c0748889c0a5b4407 src/runtime/yue2/__init__.py
|
| 232 |
3a25793081ed10325225e4e7bd9303243998abecd3a1a5b7a15139d664b70604 src/runtime/yue2/cli.py
|
| 233 |
617ee6914d534a234f1ff5d5f4685391eac5dc9f6be667429949eabf1e541517 src/runtime/yue2/compact_sampling.py
|
| 234 |
+
20f1764f1c9298fcff0411e755b99fc9a040d1819079461e527cfe1b20f03017 src/runtime/yue2/cuda_graph.py
|
| 235 |
e1722959ad0b0bf27eeaaf019cb9f109212a2450f03dd145866b1a0239151f38 src/runtime/yue2/fast.py
|
| 236 |
e6523f113132df6c04878103bca1a8b3de519fefcbc0117e97455f0dd49d1fba src/runtime/yue2/modeling_vae.py
|
| 237 |
62d2bd758c8fff1a0d64584a699343076fca9a7880db5d8e19263bc30ae79d7c src/runtime/yue2/modeling_yue2.py
|
|
|
|
| 243 |
63731de35665ac426130c781f3ba7ff1159c58d34a9b3e6ee2e78ff0c5881ff9 src/runtime/yue2/sampling.py
|
| 244 |
a5bcc181b16412b45bc560e5264ba0b303667b5fd81957991ac0266c87fe2a5d src/runtime/yue2/storage.py
|
| 245 |
fc88387f96e5db2020a0c3f5374b82726206026d83159b68b83e4136bb741acc src/runtime/yue2/tokenization_yue2.py
|
| 246 |
+
5f314395707eff25911314e7d1dabac7deeec96097dbece8efd929043820836b src/runtime/yue2_orbit.py
|
| 247 |
+
fe3f929910881e2fe0d9cfad5329e2c904441aa947bfa8d4494df7cbc97ac187 src/wheels/orbitquant-0.9.5-py3-none-any.whl
|
| 248 |
46ce45128d77c97565aac511de8f000c098cc978b2694304c8475cea8b01a1d1 weights_manifest.json
|
| 249 |
203830cebde6e3644eb291925362990d198e66c3b6006cd11f1aaa21904bcc61 yue2_generation_config.json
|
evaluation/runtime-v3/contract-checks.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"status": "passed", "checks": ["invalid profiles and overrides", "CFG two branches", "greedy min/max budget", "compact stochastic legal vocabulary", "cancellation cleanup and next request", "expert staging exception restoration"]}
|
evaluation/runtime-v3/fast-rms0-s8.json
ADDED
|
@@ -0,0 +1,304 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
[
|
| 2 |
+
{
|
| 3 |
+
"profile": "fast",
|
| 4 |
+
"fuse_rms_quant": false,
|
| 5 |
+
"repeat": 0,
|
| 6 |
+
"wall_seconds": 32.138701032847166,
|
| 7 |
+
"audio_seconds": 187.63866666666667,
|
| 8 |
+
"nvml_peak": 5757927424,
|
| 9 |
+
"torch_allocated_peak": 4513181184,
|
| 10 |
+
"timing": {
|
| 11 |
+
"abc": {
|
| 12 |
+
"seconds": 7.440518390387297,
|
| 13 |
+
"prefill_seconds": 0.8290309868752956,
|
| 14 |
+
"ttft_seconds": 0.9952553361654282,
|
| 15 |
+
"output_tokens": 1629,
|
| 16 |
+
"content_tokens": 1628,
|
| 17 |
+
"output_tps": 218.93635826565125,
|
| 18 |
+
"prefix_tokens": 423,
|
| 19 |
+
"cfg_branches": 1,
|
| 20 |
+
"execution": "cuda_graph",
|
| 21 |
+
"attention": "flash"
|
| 22 |
+
},
|
| 23 |
+
"semantic": {
|
| 24 |
+
"seconds": 17.255344545468688,
|
| 25 |
+
"prefill_seconds": 0.17433544248342514,
|
| 26 |
+
"ttft_seconds": 0.17487437278032303,
|
| 27 |
+
"output_tokens": 4692,
|
| 28 |
+
"content_tokens": 4691,
|
| 29 |
+
"output_tps": 271.91575268962885,
|
| 30 |
+
"prefix_tokens": 2053,
|
| 31 |
+
"cfg_branches": 1,
|
| 32 |
+
"execution": "cuda_graph",
|
| 33 |
+
"attention": "flash"
|
| 34 |
+
},
|
| 35 |
+
"nar_seconds": 3.986683363094926,
|
| 36 |
+
"vae_seconds": 3.4521421883255243,
|
| 37 |
+
"load": {
|
| 38 |
+
"resolve_and_integrity_seconds": 2.3701207600533962,
|
| 39 |
+
"mot_load_seconds": 0.8900818899273872
|
| 40 |
+
},
|
| 41 |
+
"e2e_seconds": 32.137739622965455
|
| 42 |
+
},
|
| 43 |
+
"semantic_sha256": "2cadb2f7623adcfe95baf6a8febb0532bae42e8fe1fe716d885c4a36790bd4f6",
|
| 44 |
+
"waveform_sha256": "102fb2bd604da4cfa44320973ce7e02afd21782d87a32a5dcab2129e79c416dc",
|
| 45 |
+
"latent_sha256": "56b429ba02eda8d86408a06131950753b56eb056966e43b3fb305d2edb0645b5",
|
| 46 |
+
"config": {
|
| 47 |
+
"generation": {
|
| 48 |
+
"abc": {
|
| 49 |
+
"temperature": 0.7,
|
| 50 |
+
"top_p": 0.9,
|
| 51 |
+
"top_k": 30,
|
| 52 |
+
"repetition_penalty": 1.005,
|
| 53 |
+
"penalty_window": 100,
|
| 54 |
+
"min_tokens": 32,
|
| 55 |
+
"max_tokens": 4096
|
| 56 |
+
},
|
| 57 |
+
"semantic": {
|
| 58 |
+
"temperature": 1.0,
|
| 59 |
+
"top_p": 0.95,
|
| 60 |
+
"top_k": 100,
|
| 61 |
+
"repetition_penalty": 1.2,
|
| 62 |
+
"penalty_window": 50,
|
| 63 |
+
"min_tokens": 200,
|
| 64 |
+
"max_tokens": 9000
|
| 65 |
+
},
|
| 66 |
+
"ode_steps": 8,
|
| 67 |
+
"ode_method": "midpoint",
|
| 68 |
+
"context": 24576,
|
| 69 |
+
"version": "yue2-native-v1"
|
| 70 |
+
},
|
| 71 |
+
"overrides": {
|
| 72 |
+
"ode_steps": 8
|
| 73 |
+
},
|
| 74 |
+
"cot": "full",
|
| 75 |
+
"cfg_scale": 1.0,
|
| 76 |
+
"cfg_negative": "same_instruction_and_exact_abc",
|
| 77 |
+
"backend": "torch",
|
| 78 |
+
"quantization": "orbitquant",
|
| 79 |
+
"model_dtype": "bfloat16",
|
| 80 |
+
"vae_dtype": "float32",
|
| 81 |
+
"vae_decode": "halo_crop",
|
| 82 |
+
"vae_core_frames": 512,
|
| 83 |
+
"vae_halo_frames": 16,
|
| 84 |
+
"device": "cuda:0",
|
| 85 |
+
"memory_budget_gib": 30.0,
|
| 86 |
+
"offload_ar": false,
|
| 87 |
+
"runtime_sha256": "e48937c70a5ca16669e6e722b392614379112aeb8f3a6a617120b4cdf1cd5c6a",
|
| 88 |
+
"decoder_release": "standard",
|
| 89 |
+
"validation_status": "unvalidated",
|
| 90 |
+
"runtime_profile": "fast",
|
| 91 |
+
"fuse_rms_quant": false,
|
| 92 |
+
"runtime_bundle_sha256": "fa427b99412fb298a91661cad5254f0b9a2c649d569a495de04b65708cac9904",
|
| 93 |
+
"vae_compute_dtype": "torch.float16",
|
| 94 |
+
"expert_staging": false,
|
| 95 |
+
"orbitquant": {
|
| 96 |
+
"quant_method": "orbitquant",
|
| 97 |
+
"weight_bits": 4,
|
| 98 |
+
"activation_bits": 4,
|
| 99 |
+
"rotation": "rpbh",
|
| 100 |
+
"rotation_seed": 0,
|
| 101 |
+
"block_size": "paper",
|
| 102 |
+
"codebook": "lloyd_max",
|
| 103 |
+
"codebook_version": 2,
|
| 104 |
+
"codebook_dtype": "float32",
|
| 105 |
+
"row_norm_dtype": "bfloat16",
|
| 106 |
+
"activation_norm_dtype": "float32",
|
| 107 |
+
"activation_eps": 1e-10,
|
| 108 |
+
"weight_pack_dtype": "uint8",
|
| 109 |
+
"target_policy": "universal",
|
| 110 |
+
"adaln_policy": "int4_rtn",
|
| 111 |
+
"adaln_group_size": 64,
|
| 112 |
+
"modules_to_convert": [],
|
| 113 |
+
"modules_to_use_adaln": [],
|
| 114 |
+
"modules_to_not_convert": [],
|
| 115 |
+
"modules_dtype_dict": {},
|
| 116 |
+
"artifact_format_version": 1,
|
| 117 |
+
"runtime_mode": "auto_fused",
|
| 118 |
+
"activation_kernel_backend": "auto",
|
| 119 |
+
"packed_matmul_block_m": 64,
|
| 120 |
+
"packed_matmul_block_n": 64,
|
| 121 |
+
"packed_matmul_block_k": 128,
|
| 122 |
+
"packed_matmul_num_warps": 4,
|
| 123 |
+
"weight_row_tile_size": 256,
|
| 124 |
+
"w4a4_int8_weight_cache": false,
|
| 125 |
+
"lowbit_boundary_protection": "auto",
|
| 126 |
+
"lowbit_protected_blocks": 4,
|
| 127 |
+
"lowbit_protected_bits": 4,
|
| 128 |
+
"lowbit_interior_protection": "auto"
|
| 129 |
+
}
|
| 130 |
+
},
|
| 131 |
+
"runtime": {
|
| 132 |
+
"fuse_rms_quant": false,
|
| 133 |
+
"orbitquant_version": "0.9.5",
|
| 134 |
+
"profile": "fast",
|
| 135 |
+
"allocator_backend": "native",
|
| 136 |
+
"modules": 224,
|
| 137 |
+
"logical_projections": 392,
|
| 138 |
+
"fused": true,
|
| 139 |
+
"effective_modes": {
|
| 140 |
+
"native_packed_matmul": 224
|
| 141 |
+
},
|
| 142 |
+
"activation_backends": {
|
| 143 |
+
"native_cuda_int8_surrogate": 168,
|
| 144 |
+
"triton_cuda_packed_w4": 56
|
| 145 |
+
},
|
| 146 |
+
"decoded_weight_caches": 0,
|
| 147 |
+
"int8_weight_caches": 0,
|
| 148 |
+
"packed_weight_bytes": 1409286144
|
| 149 |
+
},
|
| 150 |
+
"allocator_config": "expandable_segments:True",
|
| 151 |
+
"expandable_segments": 8
|
| 152 |
+
},
|
| 153 |
+
{
|
| 154 |
+
"profile": "fast",
|
| 155 |
+
"fuse_rms_quant": false,
|
| 156 |
+
"repeat": 1,
|
| 157 |
+
"wall_seconds": 30.492892453446984,
|
| 158 |
+
"audio_seconds": 187.63866666666667,
|
| 159 |
+
"nvml_peak": 5956501504,
|
| 160 |
+
"torch_allocated_peak": 4796496896,
|
| 161 |
+
"timing": {
|
| 162 |
+
"abc": {
|
| 163 |
+
"seconds": 6.603233300149441,
|
| 164 |
+
"prefill_seconds": 0.15628607757389545,
|
| 165 |
+
"ttft_seconds": 0.15696118772029877,
|
| 166 |
+
"output_tokens": 1629,
|
| 167 |
+
"content_tokens": 1628,
|
| 168 |
+
"output_tps": 246.697326287583,
|
| 169 |
+
"prefix_tokens": 423,
|
| 170 |
+
"cfg_branches": 1,
|
| 171 |
+
"execution": "cuda_graph",
|
| 172 |
+
"attention": "flash"
|
| 173 |
+
},
|
| 174 |
+
"semantic": {
|
| 175 |
+
"seconds": 17.143573760986328,
|
| 176 |
+
"prefill_seconds": 0.14921116270124912,
|
| 177 |
+
"ttft_seconds": 0.14976098202168941,
|
| 178 |
+
"output_tokens": 4692,
|
| 179 |
+
"content_tokens": 4691,
|
| 180 |
+
"output_tps": 273.6885590726477,
|
| 181 |
+
"prefix_tokens": 2053,
|
| 182 |
+
"cfg_branches": 1,
|
| 183 |
+
"execution": "cuda_graph",
|
| 184 |
+
"attention": "flash"
|
| 185 |
+
},
|
| 186 |
+
"nar_seconds": 3.957493392750621,
|
| 187 |
+
"vae_seconds": 2.4542733766138554,
|
| 188 |
+
"load": {
|
| 189 |
+
"resolve_and_integrity_seconds": 2.3701207600533962,
|
| 190 |
+
"mot_load_seconds": 0.8900818899273872
|
| 191 |
+
},
|
| 192 |
+
"e2e_seconds": 30.487808683887124
|
| 193 |
+
},
|
| 194 |
+
"semantic_sha256": "2cadb2f7623adcfe95baf6a8febb0532bae42e8fe1fe716d885c4a36790bd4f6",
|
| 195 |
+
"waveform_sha256": "102fb2bd604da4cfa44320973ce7e02afd21782d87a32a5dcab2129e79c416dc",
|
| 196 |
+
"latent_sha256": "56b429ba02eda8d86408a06131950753b56eb056966e43b3fb305d2edb0645b5",
|
| 197 |
+
"config": {
|
| 198 |
+
"generation": {
|
| 199 |
+
"abc": {
|
| 200 |
+
"temperature": 0.7,
|
| 201 |
+
"top_p": 0.9,
|
| 202 |
+
"top_k": 30,
|
| 203 |
+
"repetition_penalty": 1.005,
|
| 204 |
+
"penalty_window": 100,
|
| 205 |
+
"min_tokens": 32,
|
| 206 |
+
"max_tokens": 4096
|
| 207 |
+
},
|
| 208 |
+
"semantic": {
|
| 209 |
+
"temperature": 1.0,
|
| 210 |
+
"top_p": 0.95,
|
| 211 |
+
"top_k": 100,
|
| 212 |
+
"repetition_penalty": 1.2,
|
| 213 |
+
"penalty_window": 50,
|
| 214 |
+
"min_tokens": 200,
|
| 215 |
+
"max_tokens": 9000
|
| 216 |
+
},
|
| 217 |
+
"ode_steps": 8,
|
| 218 |
+
"ode_method": "midpoint",
|
| 219 |
+
"context": 24576,
|
| 220 |
+
"version": "yue2-native-v1"
|
| 221 |
+
},
|
| 222 |
+
"overrides": {
|
| 223 |
+
"ode_steps": 8
|
| 224 |
+
},
|
| 225 |
+
"cot": "full",
|
| 226 |
+
"cfg_scale": 1.0,
|
| 227 |
+
"cfg_negative": "same_instruction_and_exact_abc",
|
| 228 |
+
"backend": "torch",
|
| 229 |
+
"quantization": "orbitquant",
|
| 230 |
+
"model_dtype": "bfloat16",
|
| 231 |
+
"vae_dtype": "float32",
|
| 232 |
+
"vae_decode": "halo_crop",
|
| 233 |
+
"vae_core_frames": 512,
|
| 234 |
+
"vae_halo_frames": 16,
|
| 235 |
+
"device": "cuda:0",
|
| 236 |
+
"memory_budget_gib": 30.0,
|
| 237 |
+
"offload_ar": false,
|
| 238 |
+
"runtime_sha256": "e48937c70a5ca16669e6e722b392614379112aeb8f3a6a617120b4cdf1cd5c6a",
|
| 239 |
+
"decoder_release": "standard",
|
| 240 |
+
"validation_status": "unvalidated",
|
| 241 |
+
"runtime_profile": "fast",
|
| 242 |
+
"fuse_rms_quant": false,
|
| 243 |
+
"runtime_bundle_sha256": "fa427b99412fb298a91661cad5254f0b9a2c649d569a495de04b65708cac9904",
|
| 244 |
+
"vae_compute_dtype": "torch.float16",
|
| 245 |
+
"expert_staging": false,
|
| 246 |
+
"orbitquant": {
|
| 247 |
+
"quant_method": "orbitquant",
|
| 248 |
+
"weight_bits": 4,
|
| 249 |
+
"activation_bits": 4,
|
| 250 |
+
"rotation": "rpbh",
|
| 251 |
+
"rotation_seed": 0,
|
| 252 |
+
"block_size": "paper",
|
| 253 |
+
"codebook": "lloyd_max",
|
| 254 |
+
"codebook_version": 2,
|
| 255 |
+
"codebook_dtype": "float32",
|
| 256 |
+
"row_norm_dtype": "bfloat16",
|
| 257 |
+
"activation_norm_dtype": "float32",
|
| 258 |
+
"activation_eps": 1e-10,
|
| 259 |
+
"weight_pack_dtype": "uint8",
|
| 260 |
+
"target_policy": "universal",
|
| 261 |
+
"adaln_policy": "int4_rtn",
|
| 262 |
+
"adaln_group_size": 64,
|
| 263 |
+
"modules_to_convert": [],
|
| 264 |
+
"modules_to_use_adaln": [],
|
| 265 |
+
"modules_to_not_convert": [],
|
| 266 |
+
"modules_dtype_dict": {},
|
| 267 |
+
"artifact_format_version": 1,
|
| 268 |
+
"runtime_mode": "auto_fused",
|
| 269 |
+
"activation_kernel_backend": "auto",
|
| 270 |
+
"packed_matmul_block_m": 64,
|
| 271 |
+
"packed_matmul_block_n": 64,
|
| 272 |
+
"packed_matmul_block_k": 128,
|
| 273 |
+
"packed_matmul_num_warps": 4,
|
| 274 |
+
"weight_row_tile_size": 256,
|
| 275 |
+
"w4a4_int8_weight_cache": false,
|
| 276 |
+
"lowbit_boundary_protection": "auto",
|
| 277 |
+
"lowbit_protected_blocks": 4,
|
| 278 |
+
"lowbit_protected_bits": 4,
|
| 279 |
+
"lowbit_interior_protection": "auto"
|
| 280 |
+
}
|
| 281 |
+
},
|
| 282 |
+
"runtime": {
|
| 283 |
+
"fuse_rms_quant": false,
|
| 284 |
+
"orbitquant_version": "0.9.5",
|
| 285 |
+
"profile": "fast",
|
| 286 |
+
"allocator_backend": "native",
|
| 287 |
+
"modules": 224,
|
| 288 |
+
"logical_projections": 392,
|
| 289 |
+
"fused": true,
|
| 290 |
+
"effective_modes": {
|
| 291 |
+
"native_packed_matmul": 224
|
| 292 |
+
},
|
| 293 |
+
"activation_backends": {
|
| 294 |
+
"native_cuda_int8_surrogate": 168,
|
| 295 |
+
"triton_cuda_packed_w4": 56
|
| 296 |
+
},
|
| 297 |
+
"decoded_weight_caches": 0,
|
| 298 |
+
"int8_weight_caches": 0,
|
| 299 |
+
"packed_weight_bytes": 1409286144
|
| 300 |
+
},
|
| 301 |
+
"allocator_config": "expandable_segments:True",
|
| 302 |
+
"expandable_segments": 11
|
| 303 |
+
}
|
| 304 |
+
]
|
evaluation/runtime-v3/fast-rms1-s8.json
ADDED
|
@@ -0,0 +1,304 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
[
|
| 2 |
+
{
|
| 3 |
+
"profile": "fast",
|
| 4 |
+
"fuse_rms_quant": true,
|
| 5 |
+
"repeat": 0,
|
| 6 |
+
"wall_seconds": 29.99516544304788,
|
| 7 |
+
"audio_seconds": 183.47866666666667,
|
| 8 |
+
"nvml_peak": 5709692928,
|
| 9 |
+
"torch_allocated_peak": 4458537472,
|
| 10 |
+
"timing": {
|
| 11 |
+
"abc": {
|
| 12 |
+
"seconds": 6.863483566790819,
|
| 13 |
+
"prefill_seconds": 0.8106300234794617,
|
| 14 |
+
"ttft_seconds": 0.9729824792593718,
|
| 15 |
+
"output_tokens": 1573,
|
| 16 |
+
"content_tokens": 1572,
|
| 17 |
+
"output_tps": 229.18390999156898,
|
| 18 |
+
"prefix_tokens": 423,
|
| 19 |
+
"cfg_branches": 1,
|
| 20 |
+
"execution": "cuda_graph",
|
| 21 |
+
"attention": "flash"
|
| 22 |
+
},
|
| 23 |
+
"semantic": {
|
| 24 |
+
"seconds": 15.786963351070881,
|
| 25 |
+
"prefill_seconds": 0.12841831147670746,
|
| 26 |
+
"ttft_seconds": 0.1289493814110756,
|
| 27 |
+
"output_tokens": 4588,
|
| 28 |
+
"content_tokens": 4587,
|
| 29 |
+
"output_tps": 290.61953828433894,
|
| 30 |
+
"prefix_tokens": 1997,
|
| 31 |
+
"cfg_branches": 1,
|
| 32 |
+
"execution": "cuda_graph",
|
| 33 |
+
"attention": "flash"
|
| 34 |
+
},
|
| 35 |
+
"nar_seconds": 3.9035949781537056,
|
| 36 |
+
"vae_seconds": 3.437227487564087,
|
| 37 |
+
"load": {
|
| 38 |
+
"resolve_and_integrity_seconds": 2.3494524490088224,
|
| 39 |
+
"mot_load_seconds": 0.8739112019538879
|
| 40 |
+
},
|
| 41 |
+
"e2e_seconds": 29.99417370557785
|
| 42 |
+
},
|
| 43 |
+
"semantic_sha256": "b3a0fe62887f254806d06db9720f4ed69c5c1ce0c0431cedcc55b040b2325f34",
|
| 44 |
+
"waveform_sha256": "73844f9a945ace67ed8f58a53b80d4e39a1db9b8bff75f76de30efbfe8d72d15",
|
| 45 |
+
"latent_sha256": "90cb11118fa43a7f2b6ce99a5d4915dfc6b2dc7958fc18616fdc02ee19252aab",
|
| 46 |
+
"config": {
|
| 47 |
+
"generation": {
|
| 48 |
+
"abc": {
|
| 49 |
+
"temperature": 0.7,
|
| 50 |
+
"top_p": 0.9,
|
| 51 |
+
"top_k": 30,
|
| 52 |
+
"repetition_penalty": 1.005,
|
| 53 |
+
"penalty_window": 100,
|
| 54 |
+
"min_tokens": 32,
|
| 55 |
+
"max_tokens": 4096
|
| 56 |
+
},
|
| 57 |
+
"semantic": {
|
| 58 |
+
"temperature": 1.0,
|
| 59 |
+
"top_p": 0.95,
|
| 60 |
+
"top_k": 100,
|
| 61 |
+
"repetition_penalty": 1.2,
|
| 62 |
+
"penalty_window": 50,
|
| 63 |
+
"min_tokens": 200,
|
| 64 |
+
"max_tokens": 9000
|
| 65 |
+
},
|
| 66 |
+
"ode_steps": 8,
|
| 67 |
+
"ode_method": "midpoint",
|
| 68 |
+
"context": 24576,
|
| 69 |
+
"version": "yue2-native-v1"
|
| 70 |
+
},
|
| 71 |
+
"overrides": {
|
| 72 |
+
"ode_steps": 8
|
| 73 |
+
},
|
| 74 |
+
"cot": "full",
|
| 75 |
+
"cfg_scale": 1.0,
|
| 76 |
+
"cfg_negative": "same_instruction_and_exact_abc",
|
| 77 |
+
"backend": "torch",
|
| 78 |
+
"quantization": "orbitquant",
|
| 79 |
+
"model_dtype": "bfloat16",
|
| 80 |
+
"vae_dtype": "float32",
|
| 81 |
+
"vae_decode": "halo_crop",
|
| 82 |
+
"vae_core_frames": 512,
|
| 83 |
+
"vae_halo_frames": 16,
|
| 84 |
+
"device": "cuda:0",
|
| 85 |
+
"memory_budget_gib": 30.0,
|
| 86 |
+
"offload_ar": false,
|
| 87 |
+
"runtime_sha256": "e48937c70a5ca16669e6e722b392614379112aeb8f3a6a617120b4cdf1cd5c6a",
|
| 88 |
+
"decoder_release": "standard",
|
| 89 |
+
"validation_status": "unvalidated",
|
| 90 |
+
"runtime_profile": "fast",
|
| 91 |
+
"fuse_rms_quant": true,
|
| 92 |
+
"runtime_bundle_sha256": "fa427b99412fb298a91661cad5254f0b9a2c649d569a495de04b65708cac9904",
|
| 93 |
+
"vae_compute_dtype": "torch.float16",
|
| 94 |
+
"expert_staging": false,
|
| 95 |
+
"orbitquant": {
|
| 96 |
+
"quant_method": "orbitquant",
|
| 97 |
+
"weight_bits": 4,
|
| 98 |
+
"activation_bits": 4,
|
| 99 |
+
"rotation": "rpbh",
|
| 100 |
+
"rotation_seed": 0,
|
| 101 |
+
"block_size": "paper",
|
| 102 |
+
"codebook": "lloyd_max",
|
| 103 |
+
"codebook_version": 2,
|
| 104 |
+
"codebook_dtype": "float32",
|
| 105 |
+
"row_norm_dtype": "bfloat16",
|
| 106 |
+
"activation_norm_dtype": "float32",
|
| 107 |
+
"activation_eps": 1e-10,
|
| 108 |
+
"weight_pack_dtype": "uint8",
|
| 109 |
+
"target_policy": "universal",
|
| 110 |
+
"adaln_policy": "int4_rtn",
|
| 111 |
+
"adaln_group_size": 64,
|
| 112 |
+
"modules_to_convert": [],
|
| 113 |
+
"modules_to_use_adaln": [],
|
| 114 |
+
"modules_to_not_convert": [],
|
| 115 |
+
"modules_dtype_dict": {},
|
| 116 |
+
"artifact_format_version": 1,
|
| 117 |
+
"runtime_mode": "auto_fused",
|
| 118 |
+
"activation_kernel_backend": "auto",
|
| 119 |
+
"packed_matmul_block_m": 64,
|
| 120 |
+
"packed_matmul_block_n": 64,
|
| 121 |
+
"packed_matmul_block_k": 128,
|
| 122 |
+
"packed_matmul_num_warps": 4,
|
| 123 |
+
"weight_row_tile_size": 256,
|
| 124 |
+
"w4a4_int8_weight_cache": false,
|
| 125 |
+
"lowbit_boundary_protection": "auto",
|
| 126 |
+
"lowbit_protected_blocks": 4,
|
| 127 |
+
"lowbit_protected_bits": 4,
|
| 128 |
+
"lowbit_interior_protection": "auto"
|
| 129 |
+
}
|
| 130 |
+
},
|
| 131 |
+
"runtime": {
|
| 132 |
+
"fuse_rms_quant": true,
|
| 133 |
+
"orbitquant_version": "0.9.5",
|
| 134 |
+
"profile": "fast",
|
| 135 |
+
"allocator_backend": "native",
|
| 136 |
+
"modules": 224,
|
| 137 |
+
"logical_projections": 392,
|
| 138 |
+
"fused": true,
|
| 139 |
+
"effective_modes": {
|
| 140 |
+
"native_packed_matmul": 224
|
| 141 |
+
},
|
| 142 |
+
"activation_backends": {
|
| 143 |
+
"native_cuda_int8_surrogate": 168,
|
| 144 |
+
"triton_cuda_packed_w4": 56
|
| 145 |
+
},
|
| 146 |
+
"decoded_weight_caches": 0,
|
| 147 |
+
"int8_weight_caches": 0,
|
| 148 |
+
"packed_weight_bytes": 1409286144
|
| 149 |
+
},
|
| 150 |
+
"allocator_config": "expandable_segments:True",
|
| 151 |
+
"expandable_segments": 10
|
| 152 |
+
},
|
| 153 |
+
{
|
| 154 |
+
"profile": "fast",
|
| 155 |
+
"fuse_rms_quant": true,
|
| 156 |
+
"repeat": 1,
|
| 157 |
+
"wall_seconds": 28.4998283572495,
|
| 158 |
+
"audio_seconds": 183.47866666666667,
|
| 159 |
+
"nvml_peak": 5804064768,
|
| 160 |
+
"torch_allocated_peak": 4741853184,
|
| 161 |
+
"timing": {
|
| 162 |
+
"abc": {
|
| 163 |
+
"seconds": 6.031709000468254,
|
| 164 |
+
"prefill_seconds": 0.13977668620646,
|
| 165 |
+
"ttft_seconds": 0.14045371487736702,
|
| 166 |
+
"output_tokens": 1573,
|
| 167 |
+
"content_tokens": 1572,
|
| 168 |
+
"output_tps": 260.78844318880186,
|
| 169 |
+
"prefix_tokens": 423,
|
| 170 |
+
"cfg_branches": 1,
|
| 171 |
+
"execution": "cuda_graph",
|
| 172 |
+
"attention": "flash"
|
| 173 |
+
},
|
| 174 |
+
"semantic": {
|
| 175 |
+
"seconds": 15.868235632777214,
|
| 176 |
+
"prefill_seconds": 0.11507388576865196,
|
| 177 |
+
"ttft_seconds": 0.11560502462089062,
|
| 178 |
+
"output_tokens": 4588,
|
| 179 |
+
"content_tokens": 4587,
|
| 180 |
+
"output_tps": 289.13107330742486,
|
| 181 |
+
"prefix_tokens": 1997,
|
| 182 |
+
"cfg_branches": 1,
|
| 183 |
+
"execution": "cuda_graph",
|
| 184 |
+
"attention": "flash"
|
| 185 |
+
},
|
| 186 |
+
"nar_seconds": 3.8611518684774637,
|
| 187 |
+
"vae_seconds": 2.4071050081402063,
|
| 188 |
+
"load": {
|
| 189 |
+
"resolve_and_integrity_seconds": 2.3494524490088224,
|
| 190 |
+
"mot_load_seconds": 0.8739112019538879
|
| 191 |
+
},
|
| 192 |
+
"e2e_seconds": 28.494855623692274
|
| 193 |
+
},
|
| 194 |
+
"semantic_sha256": "b3a0fe62887f254806d06db9720f4ed69c5c1ce0c0431cedcc55b040b2325f34",
|
| 195 |
+
"waveform_sha256": "73844f9a945ace67ed8f58a53b80d4e39a1db9b8bff75f76de30efbfe8d72d15",
|
| 196 |
+
"latent_sha256": "90cb11118fa43a7f2b6ce99a5d4915dfc6b2dc7958fc18616fdc02ee19252aab",
|
| 197 |
+
"config": {
|
| 198 |
+
"generation": {
|
| 199 |
+
"abc": {
|
| 200 |
+
"temperature": 0.7,
|
| 201 |
+
"top_p": 0.9,
|
| 202 |
+
"top_k": 30,
|
| 203 |
+
"repetition_penalty": 1.005,
|
| 204 |
+
"penalty_window": 100,
|
| 205 |
+
"min_tokens": 32,
|
| 206 |
+
"max_tokens": 4096
|
| 207 |
+
},
|
| 208 |
+
"semantic": {
|
| 209 |
+
"temperature": 1.0,
|
| 210 |
+
"top_p": 0.95,
|
| 211 |
+
"top_k": 100,
|
| 212 |
+
"repetition_penalty": 1.2,
|
| 213 |
+
"penalty_window": 50,
|
| 214 |
+
"min_tokens": 200,
|
| 215 |
+
"max_tokens": 9000
|
| 216 |
+
},
|
| 217 |
+
"ode_steps": 8,
|
| 218 |
+
"ode_method": "midpoint",
|
| 219 |
+
"context": 24576,
|
| 220 |
+
"version": "yue2-native-v1"
|
| 221 |
+
},
|
| 222 |
+
"overrides": {
|
| 223 |
+
"ode_steps": 8
|
| 224 |
+
},
|
| 225 |
+
"cot": "full",
|
| 226 |
+
"cfg_scale": 1.0,
|
| 227 |
+
"cfg_negative": "same_instruction_and_exact_abc",
|
| 228 |
+
"backend": "torch",
|
| 229 |
+
"quantization": "orbitquant",
|
| 230 |
+
"model_dtype": "bfloat16",
|
| 231 |
+
"vae_dtype": "float32",
|
| 232 |
+
"vae_decode": "halo_crop",
|
| 233 |
+
"vae_core_frames": 512,
|
| 234 |
+
"vae_halo_frames": 16,
|
| 235 |
+
"device": "cuda:0",
|
| 236 |
+
"memory_budget_gib": 30.0,
|
| 237 |
+
"offload_ar": false,
|
| 238 |
+
"runtime_sha256": "e48937c70a5ca16669e6e722b392614379112aeb8f3a6a617120b4cdf1cd5c6a",
|
| 239 |
+
"decoder_release": "standard",
|
| 240 |
+
"validation_status": "unvalidated",
|
| 241 |
+
"runtime_profile": "fast",
|
| 242 |
+
"fuse_rms_quant": true,
|
| 243 |
+
"runtime_bundle_sha256": "fa427b99412fb298a91661cad5254f0b9a2c649d569a495de04b65708cac9904",
|
| 244 |
+
"vae_compute_dtype": "torch.float16",
|
| 245 |
+
"expert_staging": false,
|
| 246 |
+
"orbitquant": {
|
| 247 |
+
"quant_method": "orbitquant",
|
| 248 |
+
"weight_bits": 4,
|
| 249 |
+
"activation_bits": 4,
|
| 250 |
+
"rotation": "rpbh",
|
| 251 |
+
"rotation_seed": 0,
|
| 252 |
+
"block_size": "paper",
|
| 253 |
+
"codebook": "lloyd_max",
|
| 254 |
+
"codebook_version": 2,
|
| 255 |
+
"codebook_dtype": "float32",
|
| 256 |
+
"row_norm_dtype": "bfloat16",
|
| 257 |
+
"activation_norm_dtype": "float32",
|
| 258 |
+
"activation_eps": 1e-10,
|
| 259 |
+
"weight_pack_dtype": "uint8",
|
| 260 |
+
"target_policy": "universal",
|
| 261 |
+
"adaln_policy": "int4_rtn",
|
| 262 |
+
"adaln_group_size": 64,
|
| 263 |
+
"modules_to_convert": [],
|
| 264 |
+
"modules_to_use_adaln": [],
|
| 265 |
+
"modules_to_not_convert": [],
|
| 266 |
+
"modules_dtype_dict": {},
|
| 267 |
+
"artifact_format_version": 1,
|
| 268 |
+
"runtime_mode": "auto_fused",
|
| 269 |
+
"activation_kernel_backend": "auto",
|
| 270 |
+
"packed_matmul_block_m": 64,
|
| 271 |
+
"packed_matmul_block_n": 64,
|
| 272 |
+
"packed_matmul_block_k": 128,
|
| 273 |
+
"packed_matmul_num_warps": 4,
|
| 274 |
+
"weight_row_tile_size": 256,
|
| 275 |
+
"w4a4_int8_weight_cache": false,
|
| 276 |
+
"lowbit_boundary_protection": "auto",
|
| 277 |
+
"lowbit_protected_blocks": 4,
|
| 278 |
+
"lowbit_protected_bits": 4,
|
| 279 |
+
"lowbit_interior_protection": "auto"
|
| 280 |
+
}
|
| 281 |
+
},
|
| 282 |
+
"runtime": {
|
| 283 |
+
"fuse_rms_quant": true,
|
| 284 |
+
"orbitquant_version": "0.9.5",
|
| 285 |
+
"profile": "fast",
|
| 286 |
+
"allocator_backend": "native",
|
| 287 |
+
"modules": 224,
|
| 288 |
+
"logical_projections": 392,
|
| 289 |
+
"fused": true,
|
| 290 |
+
"effective_modes": {
|
| 291 |
+
"native_packed_matmul": 224
|
| 292 |
+
},
|
| 293 |
+
"activation_backends": {
|
| 294 |
+
"native_cuda_int8_surrogate": 168,
|
| 295 |
+
"triton_cuda_packed_w4": 56
|
| 296 |
+
},
|
| 297 |
+
"decoded_weight_caches": 0,
|
| 298 |
+
"int8_weight_caches": 0,
|
| 299 |
+
"packed_weight_bytes": 1409286144
|
| 300 |
+
},
|
| 301 |
+
"allocator_config": "expandable_segments:True",
|
| 302 |
+
"expandable_segments": 12
|
| 303 |
+
}
|
| 304 |
+
]
|
evaluation/runtime-v3/memory-rms1-s8.json
ADDED
|
@@ -0,0 +1,304 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
[
|
| 2 |
+
{
|
| 3 |
+
"profile": "memory",
|
| 4 |
+
"fuse_rms_quant": true,
|
| 5 |
+
"repeat": 0,
|
| 6 |
+
"wall_seconds": 31.11274246312678,
|
| 7 |
+
"audio_seconds": 183.47866666666667,
|
| 8 |
+
"nvml_peak": 5692915712,
|
| 9 |
+
"torch_allocated_peak": 3750597120,
|
| 10 |
+
"timing": {
|
| 11 |
+
"abc": {
|
| 12 |
+
"seconds": 6.880559112876654,
|
| 13 |
+
"prefill_seconds": 0.8072306606918573,
|
| 14 |
+
"ttft_seconds": 0.9844317771494389,
|
| 15 |
+
"output_tokens": 1573,
|
| 16 |
+
"content_tokens": 1572,
|
| 17 |
+
"output_tps": 228.61514219915674,
|
| 18 |
+
"prefix_tokens": 423,
|
| 19 |
+
"cfg_branches": 1,
|
| 20 |
+
"execution": "cuda_graph",
|
| 21 |
+
"attention": "flash"
|
| 22 |
+
},
|
| 23 |
+
"semantic": {
|
| 24 |
+
"seconds": 15.86077463068068,
|
| 25 |
+
"prefill_seconds": 0.1298980787396431,
|
| 26 |
+
"ttft_seconds": 0.13040952757000923,
|
| 27 |
+
"output_tokens": 4588,
|
| 28 |
+
"content_tokens": 4587,
|
| 29 |
+
"output_tps": 289.2670822725826,
|
| 30 |
+
"prefix_tokens": 1997,
|
| 31 |
+
"cfg_branches": 1,
|
| 32 |
+
"execution": "cuda_graph",
|
| 33 |
+
"attention": "flash"
|
| 34 |
+
},
|
| 35 |
+
"nar_seconds": 5.559019308537245,
|
| 36 |
+
"vae_seconds": 2.8005002085119486,
|
| 37 |
+
"load": {
|
| 38 |
+
"resolve_and_integrity_seconds": 2.3766595162451267,
|
| 39 |
+
"mot_load_seconds": 0.9255613535642624
|
| 40 |
+
},
|
| 41 |
+
"e2e_seconds": 31.11171407625079
|
| 42 |
+
},
|
| 43 |
+
"semantic_sha256": "b3a0fe62887f254806d06db9720f4ed69c5c1ce0c0431cedcc55b040b2325f34",
|
| 44 |
+
"waveform_sha256": "73844f9a945ace67ed8f58a53b80d4e39a1db9b8bff75f76de30efbfe8d72d15",
|
| 45 |
+
"latent_sha256": "90cb11118fa43a7f2b6ce99a5d4915dfc6b2dc7958fc18616fdc02ee19252aab",
|
| 46 |
+
"config": {
|
| 47 |
+
"generation": {
|
| 48 |
+
"abc": {
|
| 49 |
+
"temperature": 0.7,
|
| 50 |
+
"top_p": 0.9,
|
| 51 |
+
"top_k": 30,
|
| 52 |
+
"repetition_penalty": 1.005,
|
| 53 |
+
"penalty_window": 100,
|
| 54 |
+
"min_tokens": 32,
|
| 55 |
+
"max_tokens": 4096
|
| 56 |
+
},
|
| 57 |
+
"semantic": {
|
| 58 |
+
"temperature": 1.0,
|
| 59 |
+
"top_p": 0.95,
|
| 60 |
+
"top_k": 100,
|
| 61 |
+
"repetition_penalty": 1.2,
|
| 62 |
+
"penalty_window": 50,
|
| 63 |
+
"min_tokens": 200,
|
| 64 |
+
"max_tokens": 9000
|
| 65 |
+
},
|
| 66 |
+
"ode_steps": 8,
|
| 67 |
+
"ode_method": "midpoint",
|
| 68 |
+
"context": 24576,
|
| 69 |
+
"version": "yue2-native-v1"
|
| 70 |
+
},
|
| 71 |
+
"overrides": {
|
| 72 |
+
"ode_steps": 8
|
| 73 |
+
},
|
| 74 |
+
"cot": "full",
|
| 75 |
+
"cfg_scale": 1.0,
|
| 76 |
+
"cfg_negative": "same_instruction_and_exact_abc",
|
| 77 |
+
"backend": "torch",
|
| 78 |
+
"quantization": "orbitquant",
|
| 79 |
+
"model_dtype": "bfloat16",
|
| 80 |
+
"vae_dtype": "float32",
|
| 81 |
+
"vae_decode": "halo_crop",
|
| 82 |
+
"vae_core_frames": 512,
|
| 83 |
+
"vae_halo_frames": 16,
|
| 84 |
+
"device": "cuda:0",
|
| 85 |
+
"memory_budget_gib": 30.0,
|
| 86 |
+
"offload_ar": true,
|
| 87 |
+
"runtime_sha256": "e48937c70a5ca16669e6e722b392614379112aeb8f3a6a617120b4cdf1cd5c6a",
|
| 88 |
+
"decoder_release": "standard",
|
| 89 |
+
"validation_status": "unvalidated",
|
| 90 |
+
"runtime_profile": "memory",
|
| 91 |
+
"fuse_rms_quant": true,
|
| 92 |
+
"runtime_bundle_sha256": "fa427b99412fb298a91661cad5254f0b9a2c649d569a495de04b65708cac9904",
|
| 93 |
+
"vae_compute_dtype": "torch.float16",
|
| 94 |
+
"expert_staging": true,
|
| 95 |
+
"orbitquant": {
|
| 96 |
+
"quant_method": "orbitquant",
|
| 97 |
+
"weight_bits": 4,
|
| 98 |
+
"activation_bits": 4,
|
| 99 |
+
"rotation": "rpbh",
|
| 100 |
+
"rotation_seed": 0,
|
| 101 |
+
"block_size": "paper",
|
| 102 |
+
"codebook": "lloyd_max",
|
| 103 |
+
"codebook_version": 2,
|
| 104 |
+
"codebook_dtype": "float32",
|
| 105 |
+
"row_norm_dtype": "bfloat16",
|
| 106 |
+
"activation_norm_dtype": "float32",
|
| 107 |
+
"activation_eps": 1e-10,
|
| 108 |
+
"weight_pack_dtype": "uint8",
|
| 109 |
+
"target_policy": "universal",
|
| 110 |
+
"adaln_policy": "int4_rtn",
|
| 111 |
+
"adaln_group_size": 64,
|
| 112 |
+
"modules_to_convert": [],
|
| 113 |
+
"modules_to_use_adaln": [],
|
| 114 |
+
"modules_to_not_convert": [],
|
| 115 |
+
"modules_dtype_dict": {},
|
| 116 |
+
"artifact_format_version": 1,
|
| 117 |
+
"runtime_mode": "auto_fused",
|
| 118 |
+
"activation_kernel_backend": "auto",
|
| 119 |
+
"packed_matmul_block_m": 64,
|
| 120 |
+
"packed_matmul_block_n": 64,
|
| 121 |
+
"packed_matmul_block_k": 128,
|
| 122 |
+
"packed_matmul_num_warps": 4,
|
| 123 |
+
"weight_row_tile_size": 256,
|
| 124 |
+
"w4a4_int8_weight_cache": false,
|
| 125 |
+
"lowbit_boundary_protection": "auto",
|
| 126 |
+
"lowbit_protected_blocks": 4,
|
| 127 |
+
"lowbit_protected_bits": 4,
|
| 128 |
+
"lowbit_interior_protection": "auto"
|
| 129 |
+
}
|
| 130 |
+
},
|
| 131 |
+
"runtime": {
|
| 132 |
+
"fuse_rms_quant": true,
|
| 133 |
+
"orbitquant_version": "0.9.5",
|
| 134 |
+
"profile": "memory",
|
| 135 |
+
"allocator_backend": "native",
|
| 136 |
+
"modules": 224,
|
| 137 |
+
"logical_projections": 392,
|
| 138 |
+
"fused": true,
|
| 139 |
+
"effective_modes": {
|
| 140 |
+
"native_packed_matmul": 224
|
| 141 |
+
},
|
| 142 |
+
"activation_backends": {
|
| 143 |
+
"native_cuda_int8_surrogate": 168,
|
| 144 |
+
"triton_cuda_packed_w4": 56
|
| 145 |
+
},
|
| 146 |
+
"decoded_weight_caches": 0,
|
| 147 |
+
"int8_weight_caches": 0,
|
| 148 |
+
"packed_weight_bytes": 1409286144
|
| 149 |
+
},
|
| 150 |
+
"allocator_config": "expandable_segments:True",
|
| 151 |
+
"expandable_segments": 9
|
| 152 |
+
},
|
| 153 |
+
{
|
| 154 |
+
"profile": "memory",
|
| 155 |
+
"fuse_rms_quant": true,
|
| 156 |
+
"repeat": 1,
|
| 157 |
+
"wall_seconds": 29.819518825039268,
|
| 158 |
+
"audio_seconds": 183.47866666666667,
|
| 159 |
+
"nvml_peak": 5726470144,
|
| 160 |
+
"torch_allocated_peak": 4033912832,
|
| 161 |
+
"timing": {
|
| 162 |
+
"abc": {
|
| 163 |
+
"seconds": 6.047573072835803,
|
| 164 |
+
"prefill_seconds": 0.13665939681231976,
|
| 165 |
+
"ttft_seconds": 0.13733276538550854,
|
| 166 |
+
"output_tokens": 1573,
|
| 167 |
+
"content_tokens": 1572,
|
| 168 |
+
"output_tps": 260.10433955160056,
|
| 169 |
+
"prefix_tokens": 423,
|
| 170 |
+
"cfg_branches": 1,
|
| 171 |
+
"execution": "cuda_graph",
|
| 172 |
+
"attention": "flash"
|
| 173 |
+
},
|
| 174 |
+
"semantic": {
|
| 175 |
+
"seconds": 15.967649232596159,
|
| 176 |
+
"prefill_seconds": 0.11747091263532639,
|
| 177 |
+
"ttft_seconds": 0.11801005341112614,
|
| 178 |
+
"output_tokens": 4588,
|
| 179 |
+
"content_tokens": 4587,
|
| 180 |
+
"output_tps": 287.3309610680897,
|
| 181 |
+
"prefix_tokens": 1997,
|
| 182 |
+
"cfg_branches": 1,
|
| 183 |
+
"execution": "cuda_graph",
|
| 184 |
+
"attention": "flash"
|
| 185 |
+
},
|
| 186 |
+
"nar_seconds": 5.237515918910503,
|
| 187 |
+
"vae_seconds": 2.0264065731316805,
|
| 188 |
+
"load": {
|
| 189 |
+
"resolve_and_integrity_seconds": 2.3766595162451267,
|
| 190 |
+
"mot_load_seconds": 0.9255613535642624
|
| 191 |
+
},
|
| 192 |
+
"e2e_seconds": 29.814160283654928
|
| 193 |
+
},
|
| 194 |
+
"semantic_sha256": "b3a0fe62887f254806d06db9720f4ed69c5c1ce0c0431cedcc55b040b2325f34",
|
| 195 |
+
"waveform_sha256": "73844f9a945ace67ed8f58a53b80d4e39a1db9b8bff75f76de30efbfe8d72d15",
|
| 196 |
+
"latent_sha256": "90cb11118fa43a7f2b6ce99a5d4915dfc6b2dc7958fc18616fdc02ee19252aab",
|
| 197 |
+
"config": {
|
| 198 |
+
"generation": {
|
| 199 |
+
"abc": {
|
| 200 |
+
"temperature": 0.7,
|
| 201 |
+
"top_p": 0.9,
|
| 202 |
+
"top_k": 30,
|
| 203 |
+
"repetition_penalty": 1.005,
|
| 204 |
+
"penalty_window": 100,
|
| 205 |
+
"min_tokens": 32,
|
| 206 |
+
"max_tokens": 4096
|
| 207 |
+
},
|
| 208 |
+
"semantic": {
|
| 209 |
+
"temperature": 1.0,
|
| 210 |
+
"top_p": 0.95,
|
| 211 |
+
"top_k": 100,
|
| 212 |
+
"repetition_penalty": 1.2,
|
| 213 |
+
"penalty_window": 50,
|
| 214 |
+
"min_tokens": 200,
|
| 215 |
+
"max_tokens": 9000
|
| 216 |
+
},
|
| 217 |
+
"ode_steps": 8,
|
| 218 |
+
"ode_method": "midpoint",
|
| 219 |
+
"context": 24576,
|
| 220 |
+
"version": "yue2-native-v1"
|
| 221 |
+
},
|
| 222 |
+
"overrides": {
|
| 223 |
+
"ode_steps": 8
|
| 224 |
+
},
|
| 225 |
+
"cot": "full",
|
| 226 |
+
"cfg_scale": 1.0,
|
| 227 |
+
"cfg_negative": "same_instruction_and_exact_abc",
|
| 228 |
+
"backend": "torch",
|
| 229 |
+
"quantization": "orbitquant",
|
| 230 |
+
"model_dtype": "bfloat16",
|
| 231 |
+
"vae_dtype": "float32",
|
| 232 |
+
"vae_decode": "halo_crop",
|
| 233 |
+
"vae_core_frames": 512,
|
| 234 |
+
"vae_halo_frames": 16,
|
| 235 |
+
"device": "cuda:0",
|
| 236 |
+
"memory_budget_gib": 30.0,
|
| 237 |
+
"offload_ar": true,
|
| 238 |
+
"runtime_sha256": "e48937c70a5ca16669e6e722b392614379112aeb8f3a6a617120b4cdf1cd5c6a",
|
| 239 |
+
"decoder_release": "standard",
|
| 240 |
+
"validation_status": "unvalidated",
|
| 241 |
+
"runtime_profile": "memory",
|
| 242 |
+
"fuse_rms_quant": true,
|
| 243 |
+
"runtime_bundle_sha256": "fa427b99412fb298a91661cad5254f0b9a2c649d569a495de04b65708cac9904",
|
| 244 |
+
"vae_compute_dtype": "torch.float16",
|
| 245 |
+
"expert_staging": true,
|
| 246 |
+
"orbitquant": {
|
| 247 |
+
"quant_method": "orbitquant",
|
| 248 |
+
"weight_bits": 4,
|
| 249 |
+
"activation_bits": 4,
|
| 250 |
+
"rotation": "rpbh",
|
| 251 |
+
"rotation_seed": 0,
|
| 252 |
+
"block_size": "paper",
|
| 253 |
+
"codebook": "lloyd_max",
|
| 254 |
+
"codebook_version": 2,
|
| 255 |
+
"codebook_dtype": "float32",
|
| 256 |
+
"row_norm_dtype": "bfloat16",
|
| 257 |
+
"activation_norm_dtype": "float32",
|
| 258 |
+
"activation_eps": 1e-10,
|
| 259 |
+
"weight_pack_dtype": "uint8",
|
| 260 |
+
"target_policy": "universal",
|
| 261 |
+
"adaln_policy": "int4_rtn",
|
| 262 |
+
"adaln_group_size": 64,
|
| 263 |
+
"modules_to_convert": [],
|
| 264 |
+
"modules_to_use_adaln": [],
|
| 265 |
+
"modules_to_not_convert": [],
|
| 266 |
+
"modules_dtype_dict": {},
|
| 267 |
+
"artifact_format_version": 1,
|
| 268 |
+
"runtime_mode": "auto_fused",
|
| 269 |
+
"activation_kernel_backend": "auto",
|
| 270 |
+
"packed_matmul_block_m": 64,
|
| 271 |
+
"packed_matmul_block_n": 64,
|
| 272 |
+
"packed_matmul_block_k": 128,
|
| 273 |
+
"packed_matmul_num_warps": 4,
|
| 274 |
+
"weight_row_tile_size": 256,
|
| 275 |
+
"w4a4_int8_weight_cache": false,
|
| 276 |
+
"lowbit_boundary_protection": "auto",
|
| 277 |
+
"lowbit_protected_blocks": 4,
|
| 278 |
+
"lowbit_protected_bits": 4,
|
| 279 |
+
"lowbit_interior_protection": "auto"
|
| 280 |
+
}
|
| 281 |
+
},
|
| 282 |
+
"runtime": {
|
| 283 |
+
"fuse_rms_quant": true,
|
| 284 |
+
"orbitquant_version": "0.9.5",
|
| 285 |
+
"profile": "memory",
|
| 286 |
+
"allocator_backend": "native",
|
| 287 |
+
"modules": 224,
|
| 288 |
+
"logical_projections": 392,
|
| 289 |
+
"fused": true,
|
| 290 |
+
"effective_modes": {
|
| 291 |
+
"native_packed_matmul": 224
|
| 292 |
+
},
|
| 293 |
+
"activation_backends": {
|
| 294 |
+
"native_cuda_int8_surrogate": 168,
|
| 295 |
+
"triton_cuda_packed_w4": 56
|
| 296 |
+
},
|
| 297 |
+
"decoded_weight_caches": 0,
|
| 298 |
+
"int8_weight_caches": 0,
|
| 299 |
+
"packed_weight_bytes": 1409286144
|
| 300 |
+
},
|
| 301 |
+
"allocator_config": "expandable_segments:True",
|
| 302 |
+
"expandable_segments": 15
|
| 303 |
+
}
|
| 304 |
+
]
|
examples/runtime-v3/fast-rms.wav
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:30a70e2224b008e08f10ba0e0f9b8243b06b32a4fcbd7381e980d68cd5a7974e
|
| 3 |
+
size 70455896
|
src/OPTIMIZATION_IDEAS.md
CHANGED
|
@@ -98,3 +98,14 @@ Observed initial microbench (RTX PRO 6000 Blackwell, same 256 forced tokens): ba
|
|
| 98 |
- Autocast outside VAE alone was a no-op because decode disables it. Real inner autocast FP16 measured~.58s decoder vs~1.06sFP32; preserve FP32 weights and label compute precision.
|
| 99 |
- NAR V-view backing-storage hypothesis is ALREADY covered by current runtime's contiguous cache materialization. Guard rejected stale-source patch before inference; no new optimization credited.
|
| 100 |
- Bidirectional expert staging with deferred NAR restore after acoustic AR prefix improves some memory at transfer cost. Allocator experiments running to test fragmentation directly.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 98 |
- Autocast outside VAE alone was a no-op because decode disables it. Real inner autocast FP16 measured~.58s decoder vs~1.06sFP32; preserve FP32 weights and label compute precision.
|
| 99 |
- NAR V-view backing-storage hypothesis is ALREADY covered by current runtime's contiguous cache materialization. Guard rejected stale-source patch before inference; no new optimization credited.
|
| 100 |
- Bidirectional expert staging with deferred NAR restore after acoustic AR prefix improves some memory at transfer cost. Allocator experiments running to test fragmentation directly.
|
| 101 |
+
|
| 102 |
+
|
| 103 |
+
## Measured follow-up: byte-pair lookup and RMS fusion
|
| 104 |
+
|
| 105 |
+
- Byte-pair LUT: selected. Native1.0.2 halves codebook lookups for K1024..16384; shorter inputs use the original kernel. Actual native decoder3.148->2.943ms, saved logits exact. Full interleaved song31.82->30.34s, identical PCM.
|
| 106 |
+
- RMSNorm plus packed activation quantization: selected as an explicit fast/memory option. FP32 intermediate products can change musical arrangement. Early Hadamard stages use warp shuffles and the final reduction stages run in one warp. Combined with byte-pair GEMV, the same fused song29.81->28.28s in interleaved measurements.
|
| 107 |
+
- Warp-resident16-entry lookup: only1.8% model gain; short-K full-mask shuffle bug repaired. Slower than byte-pair lookup; retained as research evidence.
|
| 108 |
+
- 8/16warps perblock: roughly2.99/3.00ms, slower than4warps at2.94ms. Larger blocks amortize setup but reduce useful scheduling flexibility.
|
| 109 |
+
- UINT16 pair-table storage:2.921ms in one fixed-token probe. About0.7% below the4warpUINT32 result; requires interleaved confirmation before selection.
|
| 110 |
+
- Generic quantizer warp rewrite: dynamic branching blocked unrolling; static specialization removed most regression but gave negligible model improvement. Not selected.
|
| 111 |
+
- Future investigations: fuse device sampling/history maintenance if profiling justifies it; use per-shape table-layout selection; test attention-cache compression separately with spectral/listening milestones.
|
src/kernel-source/orbitquant-packed-matmul/CARD.md
CHANGED
|
@@ -45,6 +45,16 @@ The CUDA package also exports the operations used by OrbitQuant's W4A4 runtime:
|
|
| 45 |
- `matmul_packed_w4a4_int8`: direct packed A4/W4 CUDA MMA with fused token norm,
|
| 46 |
row norm, surrogate scales, and bias epilogue.
|
| 47 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 48 |
On CUDA compute capability 8.0 or newer, OrbitQuant normally combines
|
| 49 |
`quantize_activations_int8` with chunked packed-weight decode and Torch's
|
| 50 |
CUTLASS-backed INT8 matmul. The direct packed MMA operation remains available
|
|
|
|
| 45 |
- `matmul_packed_w4a4_int8`: direct packed A4/W4 CUDA MMA with fused token norm,
|
| 46 |
row norm, surrogate scales, and bias epilogue.
|
| 47 |
|
| 48 |
+
For one to eight rows, row-major weights, and input width at most 16384,
|
| 49 |
+
`matmul_packed_w4a4_int8` dispatches a DP4A GEMV. Native release 1.0.2 uses two
|
| 50 |
+
256-entry shared tables (2 KiB per block) for widths 1024–16384 to decode a packed byte into a pair
|
| 51 |
+
of signed INT8 values. Smaller widths retain the previous 16-entry tables.
|
| 52 |
+
Aligned inputs retain 32-bit vector loads; unaligned
|
| 53 |
+
contiguous views use the byte-load path. The packed format, output dtypes,
|
| 54 |
+
norms, and bias behavior are unchanged. Larger batches and K-major weights
|
| 55 |
+
continue to use the existing Tensor Core path. Set
|
| 56 |
+
`ORBITQUANT_W4A4_DISABLE_GEMV=1` before process startup to disable GEMV.
|
| 57 |
+
|
| 58 |
On CUDA compute capability 8.0 or newer, OrbitQuant normally combines
|
| 59 |
`quantize_activations_int8` with chunked packed-weight decode and Torch's
|
| 60 |
CUTLASS-backed INT8 matmul. The direct packed MMA operation remains available
|
src/kernel-source/orbitquant-packed-matmul/orbitquant_packed_matmul_cuda/packed_matmul.cu
CHANGED
|
@@ -65,6 +65,52 @@ __global__ void orbitquant_packed_w4a4_gemv(T* out, const uint8_t* x, const uint
|
|
| 65 |
}
|
| 66 |
}
|
| 67 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 68 |
inline bool orbitquant_w4a4_gemv_disabled() {
|
| 69 |
static const bool disabled = []() {
|
| 70 |
const char *value = std::getenv("ORBITQUANT_W4A4_DISABLE_GEMV");
|
|
@@ -1873,17 +1919,25 @@ void matmul_packed_w4a4_int8(
|
|
| 1873 |
!orbitquant_w4a4_gemv_disabled()) {
|
| 1874 |
const dim3 gemv_grid((out_features + 3) / 4, rows);
|
| 1875 |
const auto gemv_stream = at::cuda::getCurrentCUDAStream();
|
| 1876 |
-
#define ORBITQUANT_LAUNCH_DECODE_GEMV(T) \
|
| 1877 |
-
|
| 1878 |
out.data_ptr<T>(), packed_activations.data_ptr<uint8_t>(), \
|
| 1879 |
packed_weight_indices.data_ptr<uint8_t>(), token_norms.data_ptr<float>(), \
|
| 1880 |
row_norms.data_ptr<c10::BFloat16>(), activation_codes.data_ptr<int8_t>(), \
|
| 1881 |
weight_codes.data_ptr<int8_t>(), has_bias ? bias.data_ptr<T>() : nullptr, \
|
| 1882 |
out_features, in_features / 2, activation_scale, weight_scale)
|
| 1883 |
if (out.scalar_type() == torch::kBFloat16) {
|
| 1884 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1885 |
} else {
|
| 1886 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1887 |
}
|
| 1888 |
#undef ORBITQUANT_LAUNCH_DECODE_GEMV
|
| 1889 |
C10_CUDA_KERNEL_LAUNCH_CHECK();
|
|
|
|
| 65 |
}
|
| 66 |
}
|
| 67 |
|
| 68 |
+
// Pair tables amortize their initialization once K reaches 1024.
|
| 69 |
+
template<class T>
|
| 70 |
+
__global__ void orbitquant_packed_w4a4_gemv_byte_pairs(T* out, const uint8_t* x, const uint8_t* w,
|
| 71 |
+
const float* xn, const c10::BFloat16* wn, const int8_t* ac, const int8_t* wc,
|
| 72 |
+
const T* bias, int n, int packed_k, float as, float ws) {
|
| 73 |
+
const int lane = threadIdx.x & 31;
|
| 74 |
+
const int col = blockIdx.x * 4 + threadIdx.x / 32;
|
| 75 |
+
const int row = blockIdx.y;
|
| 76 |
+
__shared__ uint32_t a_pairs[256], w_pairs[256];
|
| 77 |
+
for (int byte = threadIdx.x; byte < 256; byte += blockDim.x) {
|
| 78 |
+
a_pairs[byte] = uint32_t(uint8_t(ac[byte & 15])) | (uint32_t(uint8_t(ac[byte >> 4])) << 8);
|
| 79 |
+
w_pairs[byte] = uint32_t(uint8_t(wc[byte & 15])) | (uint32_t(uint8_t(wc[byte >> 4])) << 8);
|
| 80 |
+
}
|
| 81 |
+
__syncthreads();
|
| 82 |
+
if (col >= n) return;
|
| 83 |
+
int sum = 0;
|
| 84 |
+
if (packed_k % 4 == 0 && (reinterpret_cast<uintptr_t>(x) & 3) == 0 && (reinterpret_cast<uintptr_t>(w) & 3) == 0) {
|
| 85 |
+
const uint32_t* xp = reinterpret_cast<const uint32_t*>(x + row * packed_k);
|
| 86 |
+
const uint32_t* wp = reinterpret_cast<const uint32_t*>(w + col * packed_k);
|
| 87 |
+
for (int k = lane; k < packed_k / 4; k += 32) {
|
| 88 |
+
const uint32_t av = xp[k], bv = wp[k];
|
| 89 |
+
// One table load decodes both nibbles of a byte to signed INT8 lanes.
|
| 90 |
+
const uint32_t al = a_pairs[av & 255] | (a_pairs[(av >> 8) & 255] << 16);
|
| 91 |
+
const uint32_t ah = a_pairs[(av >> 16) & 255] | (a_pairs[av >> 24] << 16);
|
| 92 |
+
const uint32_t bl = w_pairs[bv & 255] | (w_pairs[(bv >> 8) & 255] << 16);
|
| 93 |
+
const uint32_t bh = w_pairs[(bv >> 16) & 255] | (w_pairs[bv >> 24] << 16);
|
| 94 |
+
sum = __dp4a(static_cast<int>(al), static_cast<int>(bl), sum);
|
| 95 |
+
sum = __dp4a(static_cast<int>(ah), static_cast<int>(bh), sum);
|
| 96 |
+
}
|
| 97 |
+
} else {
|
| 98 |
+
for (int k = lane; k < packed_k; k += 32) {
|
| 99 |
+
const uint8_t a = x[row * packed_k + k];
|
| 100 |
+
const uint8_t b = w[col * packed_k + k];
|
| 101 |
+
sum = __dp4a(static_cast<int>(a_pairs[a]), static_cast<int>(w_pairs[b]), sum);
|
| 102 |
+
}
|
| 103 |
+
}
|
| 104 |
+
#pragma unroll
|
| 105 |
+
for (int offset = 16; offset; offset >>= 1) sum += __shfl_down_sync(0xffffffff, sum, offset);
|
| 106 |
+
if (lane == 0) {
|
| 107 |
+
float value = static_cast<float>(sum);
|
| 108 |
+
value *= xn[row] * static_cast<float>(wn[col]) * (as * ws);
|
| 109 |
+
if (bias) value += static_cast<float>(bias[col]);
|
| 110 |
+
out[row * n + col] = static_cast<T>(value);
|
| 111 |
+
}
|
| 112 |
+
}
|
| 113 |
+
|
| 114 |
inline bool orbitquant_w4a4_gemv_disabled() {
|
| 115 |
static const bool disabled = []() {
|
| 116 |
const char *value = std::getenv("ORBITQUANT_W4A4_DISABLE_GEMV");
|
|
|
|
| 1919 |
!orbitquant_w4a4_gemv_disabled()) {
|
| 1920 |
const dim3 gemv_grid((out_features + 3) / 4, rows);
|
| 1921 |
const auto gemv_stream = at::cuda::getCurrentCUDAStream();
|
| 1922 |
+
#define ORBITQUANT_LAUNCH_DECODE_GEMV(KERNEL, T) \
|
| 1923 |
+
KERNEL<T><<<gemv_grid, 128, 0, gemv_stream>>>( \
|
| 1924 |
out.data_ptr<T>(), packed_activations.data_ptr<uint8_t>(), \
|
| 1925 |
packed_weight_indices.data_ptr<uint8_t>(), token_norms.data_ptr<float>(), \
|
| 1926 |
row_norms.data_ptr<c10::BFloat16>(), activation_codes.data_ptr<int8_t>(), \
|
| 1927 |
weight_codes.data_ptr<int8_t>(), has_bias ? bias.data_ptr<T>() : nullptr, \
|
| 1928 |
out_features, in_features / 2, activation_scale, weight_scale)
|
| 1929 |
if (out.scalar_type() == torch::kBFloat16) {
|
| 1930 |
+
if (in_features >= 1024) {
|
| 1931 |
+
ORBITQUANT_LAUNCH_DECODE_GEMV(orbitquant_packed_w4a4_gemv_byte_pairs, c10::BFloat16);
|
| 1932 |
+
} else {
|
| 1933 |
+
ORBITQUANT_LAUNCH_DECODE_GEMV(orbitquant_packed_w4a4_gemv, c10::BFloat16);
|
| 1934 |
+
}
|
| 1935 |
} else {
|
| 1936 |
+
if (in_features >= 1024) {
|
| 1937 |
+
ORBITQUANT_LAUNCH_DECODE_GEMV(orbitquant_packed_w4a4_gemv_byte_pairs, c10::Half);
|
| 1938 |
+
} else {
|
| 1939 |
+
ORBITQUANT_LAUNCH_DECODE_GEMV(orbitquant_packed_w4a4_gemv, c10::Half);
|
| 1940 |
+
}
|
| 1941 |
}
|
| 1942 |
#undef ORBITQUANT_LAUNCH_DECODE_GEMV
|
| 1943 |
C10_CUDA_KERNEL_LAUNCH_CHECK();
|
src/kernel-source/orbitquant-packed-matmul/tests/test_decode_gemv.py
CHANGED
|
@@ -7,7 +7,17 @@ from orbitquant_packed_matmul import matmul_packed_w4a4_int8, supports_device
|
|
| 7 |
|
| 8 |
@pytest.mark.kernels_ci
|
| 9 |
@pytest.mark.parametrize(
|
| 10 |
-
"rows,n,k",
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 11 |
)
|
| 12 |
@pytest.mark.parametrize("dtype", [torch.bfloat16, torch.float16])
|
| 13 |
@pytest.mark.parametrize(
|
|
@@ -20,7 +30,10 @@ def test_decode_against_integer_oracle(rows, n, k, dtype, offset, bias_enabled,
|
|
| 20 |
torch.manual_seed(42)
|
| 21 |
x = torch.randint(0, 256, (rows * k // 2 + offset,), dtype=torch.uint8)[offset:]
|
| 22 |
w = torch.randint(0, 256, (n * k // 2 + offset,), dtype=torch.uint8)[offset:]
|
| 23 |
-
|
|
|
|
|
|
|
|
|
|
| 24 |
wc = ac.flip(0)
|
| 25 |
|
| 26 |
def unpack(values, count):
|
|
|
|
| 7 |
|
| 8 |
@pytest.mark.kernels_ci
|
| 9 |
@pytest.mark.parametrize(
|
| 10 |
+
"rows,n,k",
|
| 11 |
+
[
|
| 12 |
+
(1, 129, 64),
|
| 13 |
+
(8, 3, 64),
|
| 14 |
+
(1, 129, 512),
|
| 15 |
+
(8, 129, 1024),
|
| 16 |
+
(2, 1024, 2048),
|
| 17 |
+
(8, 2048, 6144),
|
| 18 |
+
(1, 257, 16384),
|
| 19 |
+
(9, 129, 128),
|
| 20 |
+
],
|
| 21 |
)
|
| 22 |
@pytest.mark.parametrize("dtype", [torch.bfloat16, torch.float16])
|
| 23 |
@pytest.mark.parametrize(
|
|
|
|
| 30 |
torch.manual_seed(42)
|
| 31 |
x = torch.randint(0, 256, (rows * k // 2 + offset,), dtype=torch.uint8)[offset:]
|
| 32 |
w = torch.randint(0, 256, (n * k // 2 + offset,), dtype=torch.uint8)[offset:]
|
| 33 |
+
# Exercise every code, including signed INT8 extremes and mixed nibble signs.
|
| 34 |
+
ac = torch.tensor(
|
| 35 |
+
[-128, 127, -1, 0, 1, -64, 64, -32, 32, -16, 16, -8, 8, -4, 4, 2], dtype=torch.int8
|
| 36 |
+
)
|
| 37 |
wc = ac.flip(0)
|
| 38 |
|
| 39 |
def unpack(values, count):
|
src/kernel-source/rms-quant/.gitattributes
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
*.so filter=lfs diff=lfs merge=lfs -text
|
| 2 |
+
*.pth filter=lfs diff=lfs merge=lfs -text
|
| 3 |
+
*.bin filter=lfs diff=lfs merge=lfs -text
|
src/kernel-source/rms-quant/.gitignore
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
build
|
src/kernel-source/rms-quant/CARD.md
ADDED
|
@@ -0,0 +1,15 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
library_name: kernels
|
| 4 |
+
---
|
| 5 |
+
# OrbitQuant packed GEMV
|
| 6 |
+
|
| 7 |
+
Bias-free W4A4 integer-surrogate matrix-vector evaluation for 1-8 activation rows. Reads packed nibbles directly, accumulates exact INT32 dot products, and applies the existing OrbitQuant token/row norms and surrogate scales. Output is BF16 or FP16.
|
| 8 |
+
|
| 9 |
+
The selected YuE2 dispatch uses it only for bias-free row-major packed weights. Larger batches and other layouts retain the original OrbitQuant implementation. The build targets SM120; other architectures require a rebuild and validation.
|
| 10 |
+
|
| 11 |
+
On RTX 5090, CUDA-graph layer timings improved from 0.1355 to 0.0102 ms (2048x2048) and from 0.4617 to 0.0144 ms (2048x6144), including activation preparation. BF16 reference timings were 0.0083 and 0.0105 ms: these isolated small matrices are still slightly slower than BF16. Whole-model grouped projections are measured separately.
|
| 12 |
+
|
| 13 |
+
Tests compare exact results without bias to the existing packed native kernel and exercise CUDA graph capture and invalid weight dimensions. Do not infer musical quality or whole-model speed from the microbenchmarks.
|
| 14 |
+
|
| 15 |
+
Build with kernel-builder-generated project files: `python setup.py build_kernel`. Python ABI3 and manylinux_2_34 checks pass for the supplied binary; manylinux_2_28 does not, due to a GLIBC_2.32 symbol from the Ubuntu build environment.
|
src/kernel-source/rms-quant/build.toml
ADDED
|
@@ -0,0 +1,22 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
[general]
|
| 2 |
+
name = "yue2-qkv-fused"
|
| 3 |
+
license = "Apache-2.0"
|
| 4 |
+
backends = [
|
| 5 |
+
"cuda",
|
| 6 |
+
]
|
| 7 |
+
version = 1
|
| 8 |
+
edition = 5
|
| 9 |
+
|
| 10 |
+
[general.hub]
|
| 11 |
+
repo-id = "WaveCut/orbitquant-gemv"
|
| 12 |
+
|
| 13 |
+
[torch]
|
| 14 |
+
src = [
|
| 15 |
+
"torch-ext/torch_binding.cpp",
|
| 16 |
+
"torch-ext/torch_binding.h",
|
| 17 |
+
]
|
| 18 |
+
|
| 19 |
+
[kernel.orbitquant_gemv]
|
| 20 |
+
backend = "cuda"
|
| 21 |
+
depends = ["torch"]
|
| 22 |
+
src = ["orbitquant_gemv_cuda/orbitquant_gemv.cu", "orbitquant_gemv_cuda/fused_ops.cu", "orbitquant_gemv_cuda/qkv_decode.cu", "orbitquant_gemv_cuda/rmsquant.cu"]
|
src/kernel-source/rms-quant/cmake/_ops.py.in
ADDED
|
@@ -0,0 +1,9 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import torch
|
| 2 |
+
from . import @OPS_NAME@
|
| 3 |
+
ops = torch.ops.@OPS_NAME@
|
| 4 |
+
|
| 5 |
+
def add_op_namespace_prefix(op_name: str):
|
| 6 |
+
"""
|
| 7 |
+
Prefix op by namespace.
|
| 8 |
+
"""
|
| 9 |
+
return f"@OPS_NAME@::{op_name}"
|
src/kernel-source/rms-quant/cmake/add_build_metadata.py
ADDED
|
@@ -0,0 +1,56 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import argparse
|
| 2 |
+
import json
|
| 3 |
+
import sys
|
| 4 |
+
|
| 5 |
+
|
| 6 |
+
def main():
|
| 7 |
+
parser = argparse.ArgumentParser(
|
| 8 |
+
description="Write a metadata JSON file with GPU architecture information, "
|
| 9 |
+
"reading from a source file and writing to a destination."
|
| 10 |
+
)
|
| 11 |
+
parser.add_argument(
|
| 12 |
+
"input",
|
| 13 |
+
help="Path to the source metadata JSON file to read from.",
|
| 14 |
+
)
|
| 15 |
+
parser.add_argument(
|
| 16 |
+
"destination",
|
| 17 |
+
help="Path to write the output metadata JSON file to.",
|
| 18 |
+
)
|
| 19 |
+
|
| 20 |
+
parser.add_argument(
|
| 21 |
+
"--archs",
|
| 22 |
+
help="Semicolon-separated list of GPU architectures/capabilities.",
|
| 23 |
+
)
|
| 24 |
+
|
| 25 |
+
args = parser.parse_args()
|
| 26 |
+
|
| 27 |
+
archs = (
|
| 28 |
+
sorted(set(a for a in args.archs.split(";") if a))
|
| 29 |
+
if args.archs is not None
|
| 30 |
+
else None
|
| 31 |
+
)
|
| 32 |
+
|
| 33 |
+
try:
|
| 34 |
+
with open(args.input) as f:
|
| 35 |
+
data = json.load(f)
|
| 36 |
+
except FileNotFoundError:
|
| 37 |
+
print(f"Error: input metadata file not found: {args.input}", file=sys.stderr)
|
| 38 |
+
sys.exit(1)
|
| 39 |
+
except json.JSONDecodeError as e:
|
| 40 |
+
print(f"Error: failed to parse input metadata JSON: {e}", file=sys.stderr)
|
| 41 |
+
sys.exit(1)
|
| 42 |
+
|
| 43 |
+
if archs is not None:
|
| 44 |
+
data["backend"]["archs"] = archs
|
| 45 |
+
|
| 46 |
+
try:
|
| 47 |
+
with open(args.destination, "w") as f:
|
| 48 |
+
json.dump(data, f, indent=2)
|
| 49 |
+
f.write("\n")
|
| 50 |
+
except OSError as e:
|
| 51 |
+
print(f"Error: failed to write output metadata JSON: {e}", file=sys.stderr)
|
| 52 |
+
sys.exit(1)
|
| 53 |
+
|
| 54 |
+
|
| 55 |
+
if __name__ == "__main__":
|
| 56 |
+
main()
|
src/kernel-source/rms-quant/cmake/build-variants.cmake
ADDED
|
@@ -0,0 +1,328 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Generate a standardized build variant name following the pattern:
|
| 2 |
+
# torch<VERSION>-[cxx11-]<COMPUTE>-<ARCH>-<OS>
|
| 3 |
+
# or, when compiled against the Torch stable ABI:
|
| 4 |
+
# torch-stable-abi<VERSION>-<COMPUTE>-<ARCH>-<OS>
|
| 5 |
+
#
|
| 6 |
+
# Arguments:
|
| 7 |
+
# OUT_BUILD_NAME - Output variable name
|
| 8 |
+
# TORCH_VERSION - PyTorch version (e.g., "2.7.1"); ignored when TORCH_STABLE_ABI is set
|
| 9 |
+
# COMPUTE_FRAMEWORK - One of: cuda, rocm, metal, xpu, cpu
|
| 10 |
+
# COMPUTE_VERSION - Version of compute framework (e.g., "12.4" for CUDA, "6.0" for ROCm)
|
| 11 |
+
# Optional for CPU-only builds (pass empty string or omit)
|
| 12 |
+
# Optional keyword arguments:
|
| 13 |
+
# TORCH_STABLE_ABI - Stable ABI version the extension was compiled against (e.g., "2.11");
|
| 14 |
+
# when set, TORCH_VERSION is ignored and the prefix becomes
|
| 15 |
+
# torch-stable-abi<VERSION> (e.g., "2.11" -> "torch-stable-abi211")
|
| 16 |
+
# Example output: torch27-cxx11-cu124-x86_64-linux (Linux)
|
| 17 |
+
# torch27-cu124-x86_64-windows (Windows)
|
| 18 |
+
# torch27-metal-aarch64-darwin (macOS)
|
| 19 |
+
# torch-stable-abi211-cu124-x86_64-linux (Linux, stable ABI)
|
| 20 |
+
#
|
| 21 |
+
function(generate_build_name OUT_BUILD_NAME TORCH_VERSION COMPUTE_FRAMEWORK COMPUTE_VERSION)
|
| 22 |
+
cmake_parse_arguments(ARG "" "TORCH_STABLE_ABI" "" ${ARGN})
|
| 23 |
+
|
| 24 |
+
if(ARG_TORCH_STABLE_ABI)
|
| 25 |
+
# Flatten the stable ABI version (e.g., "2.11" -> "211") and ignore TORCH_VERSION
|
| 26 |
+
string(REPLACE "." ";" VERSION_LIST "${ARG_TORCH_STABLE_ABI}")
|
| 27 |
+
list(LENGTH VERSION_LIST VERSION_COMPONENTS)
|
| 28 |
+
if(VERSION_COMPONENTS LESS 2)
|
| 29 |
+
list(APPEND VERSION_LIST "0")
|
| 30 |
+
endif()
|
| 31 |
+
list(GET VERSION_LIST 0 MAJOR)
|
| 32 |
+
list(GET VERSION_LIST 1 MINOR)
|
| 33 |
+
set(TORCH_PREFIX "torch-stable-abi${MAJOR}${MINOR}")
|
| 34 |
+
else()
|
| 35 |
+
# Flatten version by removing dots and padding to 2 components
|
| 36 |
+
string(REPLACE "." ";" VERSION_LIST "${TORCH_VERSION}")
|
| 37 |
+
list(LENGTH VERSION_LIST VERSION_COMPONENTS)
|
| 38 |
+
|
| 39 |
+
# Pad to at least 2 components
|
| 40 |
+
if(VERSION_COMPONENTS LESS 2)
|
| 41 |
+
list(APPEND VERSION_LIST "0")
|
| 42 |
+
endif()
|
| 43 |
+
|
| 44 |
+
# Take first 2 components and join without dots
|
| 45 |
+
list(GET VERSION_LIST 0 MAJOR)
|
| 46 |
+
list(GET VERSION_LIST 1 MINOR)
|
| 47 |
+
set(TORCH_PREFIX "torch${MAJOR}${MINOR}")
|
| 48 |
+
endif()
|
| 49 |
+
|
| 50 |
+
# Generate compute string
|
| 51 |
+
if(COMPUTE_FRAMEWORK STREQUAL "cuda")
|
| 52 |
+
# Flatten CUDA version (e.g., "12.4" -> "124")
|
| 53 |
+
string(REPLACE "." ";" COMPUTE_VERSION_LIST "${COMPUTE_VERSION}")
|
| 54 |
+
list(LENGTH COMPUTE_VERSION_LIST COMPUTE_COMPONENTS)
|
| 55 |
+
if(COMPUTE_COMPONENTS GREATER_EQUAL 2)
|
| 56 |
+
list(GET COMPUTE_VERSION_LIST 0 COMPUTE_MAJOR)
|
| 57 |
+
list(GET COMPUTE_VERSION_LIST 1 COMPUTE_MINOR)
|
| 58 |
+
set(COMPUTE_STRING "cu${COMPUTE_MAJOR}${COMPUTE_MINOR}")
|
| 59 |
+
else()
|
| 60 |
+
list(GET COMPUTE_VERSION_LIST 0 COMPUTE_MAJOR)
|
| 61 |
+
set(COMPUTE_STRING "cu${COMPUTE_MAJOR}0")
|
| 62 |
+
endif()
|
| 63 |
+
elseif(COMPUTE_FRAMEWORK STREQUAL "rocm")
|
| 64 |
+
# Flatten ROCm version (e.g., "6.0" -> "60")
|
| 65 |
+
string(REPLACE "." ";" COMPUTE_VERSION_LIST "${COMPUTE_VERSION}")
|
| 66 |
+
list(LENGTH COMPUTE_VERSION_LIST COMPUTE_COMPONENTS)
|
| 67 |
+
if(COMPUTE_COMPONENTS GREATER_EQUAL 2)
|
| 68 |
+
list(GET COMPUTE_VERSION_LIST 0 COMPUTE_MAJOR)
|
| 69 |
+
list(GET COMPUTE_VERSION_LIST 1 COMPUTE_MINOR)
|
| 70 |
+
set(COMPUTE_STRING "rocm${COMPUTE_MAJOR}${COMPUTE_MINOR}")
|
| 71 |
+
else()
|
| 72 |
+
list(GET COMPUTE_VERSION_LIST 0 COMPUTE_MAJOR)
|
| 73 |
+
set(COMPUTE_STRING "rocm${COMPUTE_MAJOR}0")
|
| 74 |
+
endif()
|
| 75 |
+
elseif(COMPUTE_FRAMEWORK STREQUAL "xpu")
|
| 76 |
+
# Flatten XPU version (e.g., "2025.2" -> "202552")
|
| 77 |
+
string(REPLACE "." ";" COMPUTE_VERSION_LIST "${COMPUTE_VERSION}")
|
| 78 |
+
list(LENGTH COMPUTE_VERSION_LIST COMPUTE_COMPONENTS)
|
| 79 |
+
if(COMPUTE_COMPONENTS GREATER_EQUAL 2)
|
| 80 |
+
list(GET COMPUTE_VERSION_LIST 0 COMPUTE_MAJOR)
|
| 81 |
+
list(GET COMPUTE_VERSION_LIST 1 COMPUTE_MINOR)
|
| 82 |
+
set(COMPUTE_STRING "xpu${COMPUTE_MAJOR}${COMPUTE_MINOR}")
|
| 83 |
+
else()
|
| 84 |
+
list(GET COMPUTE_VERSION_LIST 0 COMPUTE_MAJOR)
|
| 85 |
+
set(COMPUTE_STRING "xpu${COMPUTE_MAJOR}0")
|
| 86 |
+
endif()
|
| 87 |
+
elseif(COMPUTE_FRAMEWORK STREQUAL "metal")
|
| 88 |
+
set(COMPUTE_STRING "metal")
|
| 89 |
+
elseif(COMPUTE_FRAMEWORK STREQUAL "cpu")
|
| 90 |
+
set(COMPUTE_STRING "cpu")
|
| 91 |
+
else()
|
| 92 |
+
message(FATAL_ERROR "Unknown compute framework: ${COMPUTE_FRAMEWORK}")
|
| 93 |
+
endif()
|
| 94 |
+
|
| 95 |
+
# Detect from target system (CMAKE_SYSTEM_* variables refer to target, not host)
|
| 96 |
+
# Normalize architecture name
|
| 97 |
+
if(CMAKE_SYSTEM_PROCESSOR MATCHES "^(x86_64|amd64|AMD64)$")
|
| 98 |
+
set(CPU_ARCH "x86_64")
|
| 99 |
+
elseif(CMAKE_SYSTEM_PROCESSOR MATCHES "^(aarch64|arm64|ARM64)$")
|
| 100 |
+
set(CPU_ARCH "aarch64")
|
| 101 |
+
else()
|
| 102 |
+
message(FATAL_ERROR "Unsupported architecture: ${CMAKE_SYSTEM_PROCESSOR}")
|
| 103 |
+
endif()
|
| 104 |
+
|
| 105 |
+
# Normalize OS name
|
| 106 |
+
if(CMAKE_SYSTEM_NAME STREQUAL "Windows")
|
| 107 |
+
set(OS_NAME "windows")
|
| 108 |
+
elseif(CMAKE_SYSTEM_NAME STREQUAL "Linux")
|
| 109 |
+
set(OS_NAME "linux")
|
| 110 |
+
elseif(CMAKE_SYSTEM_NAME STREQUAL "Darwin")
|
| 111 |
+
set(OS_NAME "darwin")
|
| 112 |
+
else()
|
| 113 |
+
message(WARNING "Unknown OS ${CMAKE_SYSTEM_NAME}, using as-is")
|
| 114 |
+
string(TOLOWER "${CMAKE_SYSTEM_NAME}" OS_NAME)
|
| 115 |
+
endif()
|
| 116 |
+
|
| 117 |
+
set(ARCH_OS_STRING "${CPU_ARCH}-${OS_NAME}")
|
| 118 |
+
|
| 119 |
+
# Assemble the final build name
|
| 120 |
+
# For non-stable-ABI Linux builds, include cxx11 ABI indicator for compatibility
|
| 121 |
+
if(NOT ARG_TORCH_STABLE_ABI AND ARCH_OS_STRING MATCHES "-linux$")
|
| 122 |
+
set(BUILD_NAME "${TORCH_PREFIX}-cxx11-${COMPUTE_STRING}-${ARCH_OS_STRING}")
|
| 123 |
+
else()
|
| 124 |
+
set(BUILD_NAME "${TORCH_PREFIX}-${COMPUTE_STRING}-${ARCH_OS_STRING}")
|
| 125 |
+
endif()
|
| 126 |
+
|
| 127 |
+
set(${OUT_BUILD_NAME} "${BUILD_NAME}" PARENT_SCOPE)
|
| 128 |
+
message(STATUS "Generated build name: ${BUILD_NAME}")
|
| 129 |
+
endfunction()
|
| 130 |
+
|
| 131 |
+
#
|
| 132 |
+
# Create a custom install target for the huggingface/kernels library layout.
|
| 133 |
+
# This installs the extension into a directory structure suitable for kernel hub discovery:
|
| 134 |
+
# <PREFIX>/<BUILD_VARIANT_NAME>
|
| 135 |
+
#
|
| 136 |
+
# Arguments:
|
| 137 |
+
# TARGET_NAME - Name of the target to create the install rule for
|
| 138 |
+
# PACKAGE_NAME - Python package name (e.g., "activation")
|
| 139 |
+
# BUILD_VARIANT_NAME - Build variant name (e.g., "torch271-cxx11-cu124-x86_64-linux")
|
| 140 |
+
# INSTALL_PREFIX - Base installation directory (defaults to CMAKE_INSTALL_PREFIX)
|
| 141 |
+
# GPU_ARCHS - List of GPU architectures that were compiled
|
| 142 |
+
# (optional; when provided for CUDA/ROCm, metadata.json will include
|
| 143 |
+
# a "backend" key with the type and arch list)
|
| 144 |
+
#
|
| 145 |
+
function(add_kernels_install_target TARGET_NAME PACKAGE_NAME BUILD_VARIANT_NAME)
|
| 146 |
+
set(oneValueArgs INSTALL_PREFIX)
|
| 147 |
+
set(multiValueArgs DATA_EXTENSIONS GPU_ARCHS)
|
| 148 |
+
cmake_parse_arguments(ARG "" "${oneValueArgs}" "${multiValueArgs}" ${ARGN})
|
| 149 |
+
|
| 150 |
+
if(NOT ARG_INSTALL_PREFIX)
|
| 151 |
+
set(ARG_INSTALL_PREFIX "${CMAKE_INSTALL_PREFIX}")
|
| 152 |
+
endif()
|
| 153 |
+
|
| 154 |
+
gpu_lang_to_backend(_BACKEND ${GPU_LANG})
|
| 155 |
+
|
| 156 |
+
# Always include 'py' extension for Python files
|
| 157 |
+
set(ALL_EXTENSIONS ${ARG_DATA_EXTENSIONS})
|
| 158 |
+
list(APPEND ALL_EXTENSIONS "py")
|
| 159 |
+
|
| 160 |
+
# Set the installation directory
|
| 161 |
+
set(KERNEL_INSTALL_DIR "${ARG_INSTALL_PREFIX}/${BUILD_VARIANT_NAME}")
|
| 162 |
+
|
| 163 |
+
message(STATUS "Using PACKAGE_NAME: ${PACKAGE_NAME}")
|
| 164 |
+
|
| 165 |
+
# Install the compiled extension using CMake's install() command
|
| 166 |
+
# This will be triggered by the standard INSTALL target
|
| 167 |
+
install(TARGETS ${TARGET_NAME}
|
| 168 |
+
LIBRARY DESTINATION "${KERNEL_INSTALL_DIR}"
|
| 169 |
+
RUNTIME DESTINATION "${KERNEL_INSTALL_DIR}"
|
| 170 |
+
COMPONENT ${TARGET_NAME})
|
| 171 |
+
|
| 172 |
+
# Install data files with specified extensions
|
| 173 |
+
foreach(ext IN LISTS ALL_EXTENSIONS)
|
| 174 |
+
file(GLOB_RECURSE DATA_FILES RELATIVE "${CMAKE_SOURCE_DIR}/torch-ext/${PACKAGE_NAME}" "${CMAKE_SOURCE_DIR}/torch-ext/${PACKAGE_NAME}/*.${ext}")
|
| 175 |
+
foreach(data_file IN LISTS DATA_FILES)
|
| 176 |
+
get_filename_component(data_file_dir "${data_file}" DIRECTORY)
|
| 177 |
+
install(FILES "${CMAKE_SOURCE_DIR}/torch-ext/${PACKAGE_NAME}/${data_file}"
|
| 178 |
+
DESTINATION "${KERNEL_INSTALL_DIR}/${data_file_dir}"
|
| 179 |
+
COMPONENT ${TARGET_NAME})
|
| 180 |
+
endforeach()
|
| 181 |
+
endforeach()
|
| 182 |
+
|
| 183 |
+
message(STATUS "GPU archs: ${ARG_GPU_ARCHS}")
|
| 184 |
+
|
| 185 |
+
# Add the GPU archs to matadata.json when applicable.
|
| 186 |
+
if((GPU_LANG STREQUAL "CUDA" OR GPU_LANG STREQUAL "HIP") AND ARG_GPU_ARCHS)
|
| 187 |
+
list(JOIN ARG_GPU_ARCHS ";" _GPU_ARCHS_STR)
|
| 188 |
+
install(CODE "
|
| 189 |
+
file(MAKE_DIRECTORY \"${KERNEL_INSTALL_DIR}\")
|
| 190 |
+
execute_process(
|
| 191 |
+
COMMAND \"${Python3_EXECUTABLE}\"
|
| 192 |
+
\"${CMAKE_CURRENT_LIST_DIR}/cmake/add_build_metadata.py\"
|
| 193 |
+
\"${CMAKE_SOURCE_DIR}/metadata-${_BACKEND}.json\"
|
| 194 |
+
\"${KERNEL_INSTALL_DIR}/metadata.json\"
|
| 195 |
+
--archs \"${_GPU_ARCHS_STR}\"
|
| 196 |
+
RESULT_VARIABLE _METADATA_RESULT
|
| 197 |
+
ERROR_VARIABLE _METADATA_ERROR
|
| 198 |
+
)
|
| 199 |
+
if(NOT _METADATA_RESULT EQUAL 0)
|
| 200 |
+
message(WARNING \"Failed to add GPU arch metadata: \${_METADATA_ERROR}\")
|
| 201 |
+
endif()
|
| 202 |
+
" COMPONENT ${TARGET_NAME})
|
| 203 |
+
else()
|
| 204 |
+
install(CODE "
|
| 205 |
+
file(MAKE_DIRECTORY \"${KERNEL_INSTALL_DIR}\")
|
| 206 |
+
execute_process(
|
| 207 |
+
COMMAND \"${Python3_EXECUTABLE}\"
|
| 208 |
+
\"${CMAKE_CURRENT_LIST_DIR}/cmake/add_build_metadata.py\"
|
| 209 |
+
\"${CMAKE_SOURCE_DIR}/metadata-${_BACKEND}.json\"
|
| 210 |
+
\"${KERNEL_INSTALL_DIR}/metadata.json\"
|
| 211 |
+
RESULT_VARIABLE _METADATA_RESULT
|
| 212 |
+
ERROR_VARIABLE _METADATA_ERROR
|
| 213 |
+
)
|
| 214 |
+
if(NOT _METADATA_RESULT EQUAL 0)
|
| 215 |
+
message(WARNING \"Failed to write metadata: \${_METADATA_ERROR}\")
|
| 216 |
+
endif()
|
| 217 |
+
" COMPONENT ${TARGET_NAME})
|
| 218 |
+
endif()
|
| 219 |
+
|
| 220 |
+
# Compatibility with older kernels and direct Python imports.
|
| 221 |
+
install(FILES ${CMAKE_SOURCE_DIR}/compat.py
|
| 222 |
+
DESTINATION "${KERNEL_INSTALL_DIR}/${PACKAGE_NAME}"
|
| 223 |
+
RENAME "__init__.py"
|
| 224 |
+
COMPONENT ${TARGET_NAME})
|
| 225 |
+
|
| 226 |
+
message(STATUS "Added install rules for ${TARGET_NAME} -> ${BUILD_VARIANT_NAME}")
|
| 227 |
+
endfunction()
|
| 228 |
+
|
| 229 |
+
#
|
| 230 |
+
# Add install rules for local development with huggingface/kernels.
|
| 231 |
+
# This installs the extension into the layout expected by get_local_kernel():
|
| 232 |
+
# ${CMAKE_SOURCE_DIR}/build/<BUILD_VARIANT_NAME>/
|
| 233 |
+
#
|
| 234 |
+
# This allows developers to use get_local_kernel() from the kernels library to load
|
| 235 |
+
# locally built kernels without needing to publish to the hub.
|
| 236 |
+
#
|
| 237 |
+
# This uses the standard CMake install() command, so it works with the default
|
| 238 |
+
# "install" target that is always available.
|
| 239 |
+
#
|
| 240 |
+
# Arguments:
|
| 241 |
+
# TARGET_NAME - Name of the target to create the install rule for
|
| 242 |
+
# PACKAGE_NAME - Python package name (e.g., "activation")
|
| 243 |
+
# BUILD_VARIANT_NAME - Build variant name (e.g., "torch271-cxx11-cu124-x86_64-linux")
|
| 244 |
+
# GPU_ARCHS - List of GPU architectures that were compiled
|
| 245 |
+
# (optional; when provided for CUDA/ROCm, metadata.json will include
|
| 246 |
+
# a "backend" key with the type and arch list)
|
| 247 |
+
#
|
| 248 |
+
function(add_local_install_target TARGET_NAME PACKAGE_NAME BUILD_VARIANT_NAME)
|
| 249 |
+
set(oneValueArgs)
|
| 250 |
+
set(multiValueArgs DATA_EXTENSIONS GPU_ARCHS)
|
| 251 |
+
cmake_parse_arguments(ARG "" "${oneValueArgs}" "${multiValueArgs}" ${ARGN})
|
| 252 |
+
|
| 253 |
+
# Always include 'py' extension for Python files
|
| 254 |
+
set(ALL_EXTENSIONS ${ARG_DATA_EXTENSIONS})
|
| 255 |
+
list(APPEND ALL_EXTENSIONS "py")
|
| 256 |
+
|
| 257 |
+
# Define your local, folder based, installation directory
|
| 258 |
+
set(LOCAL_INSTALL_DIR "${CMAKE_SOURCE_DIR}/build/${BUILD_VARIANT_NAME}")
|
| 259 |
+
# Variant directory is where metadata.json should go (for kernels upload discovery)
|
| 260 |
+
set(VARIANT_DIR "${CMAKE_SOURCE_DIR}/build/${BUILD_VARIANT_NAME}")
|
| 261 |
+
|
| 262 |
+
# Create a custom target for local installation
|
| 263 |
+
add_custom_target(local_install
|
| 264 |
+
COMMENT "Installing files to local directory..."
|
| 265 |
+
)
|
| 266 |
+
|
| 267 |
+
gpu_lang_to_backend(_BACKEND ${GPU_LANG})
|
| 268 |
+
|
| 269 |
+
# Copy data files with specified extensions
|
| 270 |
+
foreach(ext IN LISTS ALL_EXTENSIONS)
|
| 271 |
+
file(GLOB_RECURSE DATA_FILES RELATIVE "${CMAKE_SOURCE_DIR}/torch-ext/${PACKAGE_NAME}" "${CMAKE_SOURCE_DIR}/torch-ext/${PACKAGE_NAME}/*.${ext}")
|
| 272 |
+
foreach(data_file IN LISTS DATA_FILES)
|
| 273 |
+
get_filename_component(data_file_dir "${data_file}" DIRECTORY)
|
| 274 |
+
add_custom_command(TARGET local_install POST_BUILD
|
| 275 |
+
COMMAND ${CMAKE_COMMAND} -E make_directory
|
| 276 |
+
${LOCAL_INSTALL_DIR}/${data_file_dir}
|
| 277 |
+
COMMAND ${CMAKE_COMMAND} -E copy_if_different
|
| 278 |
+
${CMAKE_SOURCE_DIR}/torch-ext/${PACKAGE_NAME}/${data_file}
|
| 279 |
+
${LOCAL_INSTALL_DIR}/${data_file_dir}/
|
| 280 |
+
COMMENT "Copying ${data_file} to ${LOCAL_INSTALL_DIR}/${data_file_dir}"
|
| 281 |
+
)
|
| 282 |
+
endforeach()
|
| 283 |
+
endforeach()
|
| 284 |
+
|
| 285 |
+
|
| 286 |
+
# Add the GPU archs to matadata.json when applicable.
|
| 287 |
+
if((GPU_LANG STREQUAL "CUDA" OR GPU_LANG STREQUAL "HIP") AND ARG_GPU_ARCHS)
|
| 288 |
+
list(JOIN ARG_GPU_ARCHS ";" _GPU_ARCHS_STR)
|
| 289 |
+
add_custom_command(TARGET local_install POST_BUILD
|
| 290 |
+
COMMAND ${CMAKE_COMMAND} -E make_directory ${VARIANT_DIR}
|
| 291 |
+
COMMAND ${Python3_EXECUTABLE}
|
| 292 |
+
${CMAKE_CURRENT_LIST_DIR}/cmake/add_build_metadata.py
|
| 293 |
+
${CMAKE_SOURCE_DIR}/metadata-${_BACKEND}.json
|
| 294 |
+
${VARIANT_DIR}/metadata.json
|
| 295 |
+
--archs "${_GPU_ARCHS_STR}"
|
| 296 |
+
COMMENT "Writing metadata.json with GPU arch info to ${VARIANT_DIR}"
|
| 297 |
+
)
|
| 298 |
+
else()
|
| 299 |
+
add_custom_command(TARGET local_install POST_BUILD
|
| 300 |
+
COMMAND ${CMAKE_COMMAND} -E make_directory ${VARIANT_DIR}
|
| 301 |
+
COMMAND ${Python3_EXECUTABLE}
|
| 302 |
+
${CMAKE_CURRENT_LIST_DIR}/cmake/add_build_metadata.py
|
| 303 |
+
${CMAKE_SOURCE_DIR}/metadata-${_BACKEND}.json
|
| 304 |
+
${VARIANT_DIR}/metadata.json
|
| 305 |
+
COMMENT "Writing metadata.json to ${VARIANT_DIR}"
|
| 306 |
+
)
|
| 307 |
+
endif()
|
| 308 |
+
|
| 309 |
+
add_custom_command(TARGET local_install POST_BUILD
|
| 310 |
+
# Copy the shared library
|
| 311 |
+
COMMAND ${CMAKE_COMMAND} -E copy_if_different
|
| 312 |
+
$<TARGET_FILE:${TARGET_NAME}>
|
| 313 |
+
${LOCAL_INSTALL_DIR}/
|
| 314 |
+
|
| 315 |
+
# Compatibility with older kernels and direct Python imports.
|
| 316 |
+
COMMAND ${CMAKE_COMMAND} -E copy_if_different
|
| 317 |
+
${CMAKE_SOURCE_DIR}/compat.py
|
| 318 |
+
${VARIANT_DIR}/${PACKAGE_NAME}/__init__.py
|
| 319 |
+
|
| 320 |
+
COMMENT "Copying shared library and Python files to ${LOCAL_INSTALL_DIR}"
|
| 321 |
+
COMMAND_EXPAND_LISTS
|
| 322 |
+
)
|
| 323 |
+
|
| 324 |
+
# Create both directories: variant dir for metadata.json, package dir for binaries
|
| 325 |
+
file(MAKE_DIRECTORY ${VARIANT_DIR})
|
| 326 |
+
file(MAKE_DIRECTORY ${LOCAL_INSTALL_DIR})
|
| 327 |
+
message(STATUS "Added install rules for ${TARGET_NAME} -> build/${BUILD_VARIANT_NAME}")
|
| 328 |
+
endfunction()
|
src/kernel-source/rms-quant/cmake/compile-metal.cmake
ADDED
|
@@ -0,0 +1,104 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Metal shader compilation function
|
| 2 |
+
function(compile_metal_shaders TARGET_NAME METAL_SOURCES EXTRA_INCLUDE_DIRS)
|
| 3 |
+
if(NOT DEFINED METAL_TOOLCHAIN)
|
| 4 |
+
execute_process(
|
| 5 |
+
COMMAND "xcodebuild" "-showComponent" "MetalToolchain"
|
| 6 |
+
OUTPUT_VARIABLE FIND_METAL_OUT
|
| 7 |
+
RESULT_VARIABLE FIND_METAL_ERROR_CODE
|
| 8 |
+
ERROR_VARIABLE FIND_METAL_STDERR
|
| 9 |
+
OUTPUT_STRIP_TRAILING_WHITESPACE)
|
| 10 |
+
|
| 11 |
+
if(NOT FIND_METAL_ERROR_CODE EQUAL 0)
|
| 12 |
+
message(FATAL_ERROR "${ERR_MSG}: ${FIND_METAL_STDERR}")
|
| 13 |
+
endif()
|
| 14 |
+
|
| 15 |
+
# Extract the Toolchain Search Path value and append Metal.xctoolchain
|
| 16 |
+
string(REGEX MATCH "Toolchain Search Path: ([^\n]+)" MATCH_RESULT "${FIND_METAL_OUT}")
|
| 17 |
+
set(METAL_TOOLCHAIN "${CMAKE_MATCH_1}/Metal.xctoolchain")
|
| 18 |
+
endif()
|
| 19 |
+
|
| 20 |
+
# Set Metal compiler flags
|
| 21 |
+
set(METAL_FLAGS "-std=metal4.0" "-O2")
|
| 22 |
+
|
| 23 |
+
# Output directory for compiled metallib
|
| 24 |
+
set(METALLIB_OUTPUT_DIR "${CMAKE_BINARY_DIR}/metallib")
|
| 25 |
+
file(MAKE_DIRECTORY ${METALLIB_OUTPUT_DIR})
|
| 26 |
+
|
| 27 |
+
foreach(INC ${EXTRA_INCLUDE_DIRS})
|
| 28 |
+
list(APPEND METAL_FLAGS "-I${INC}")
|
| 29 |
+
endforeach()
|
| 30 |
+
|
| 31 |
+
# Separate .metal files from .h files and compile .metal files to .air
|
| 32 |
+
set(AIR_FILES)
|
| 33 |
+
set(METAL_FILES)
|
| 34 |
+
set(HEADER_FILES)
|
| 35 |
+
|
| 36 |
+
foreach(SOURCE_FILE ${METAL_SOURCES})
|
| 37 |
+
if(SOURCE_FILE MATCHES "\\.metal$")
|
| 38 |
+
list(APPEND METAL_FILES ${SOURCE_FILE})
|
| 39 |
+
elseif(SOURCE_FILE MATCHES "\\.h$")
|
| 40 |
+
list(APPEND HEADER_FILES ${SOURCE_FILE})
|
| 41 |
+
endif()
|
| 42 |
+
endforeach()
|
| 43 |
+
|
| 44 |
+
foreach(METAL_FILE ${METAL_FILES})
|
| 45 |
+
get_filename_component(METAL_NAME ${METAL_FILE} NAME_WE)
|
| 46 |
+
set(AIR_FILE "${CMAKE_BINARY_DIR}/${METAL_NAME}.air")
|
| 47 |
+
|
| 48 |
+
# Include header files as dependencies
|
| 49 |
+
set(ALL_DEPENDENCIES ${CMAKE_CURRENT_SOURCE_DIR}/${METAL_FILE})
|
| 50 |
+
foreach(HEADER_FILE ${HEADER_FILES})
|
| 51 |
+
list(APPEND ALL_DEPENDENCIES ${CMAKE_CURRENT_SOURCE_DIR}/${HEADER_FILE})
|
| 52 |
+
endforeach()
|
| 53 |
+
|
| 54 |
+
add_custom_command(
|
| 55 |
+
OUTPUT ${AIR_FILE}
|
| 56 |
+
COMMAND "${METAL_TOOLCHAIN}/usr/bin/metal" ${METAL_FLAGS}
|
| 57 |
+
-c ${CMAKE_CURRENT_SOURCE_DIR}/${METAL_FILE}
|
| 58 |
+
-o ${AIR_FILE}
|
| 59 |
+
DEPENDS ${ALL_DEPENDENCIES}
|
| 60 |
+
COMMENT "Compiling Metal shader ${METAL_FILE} to ${AIR_FILE}"
|
| 61 |
+
VERBATIM
|
| 62 |
+
)
|
| 63 |
+
|
| 64 |
+
list(APPEND AIR_FILES ${AIR_FILE})
|
| 65 |
+
endforeach()
|
| 66 |
+
|
| 67 |
+
# Link all .air files into a single .metallib
|
| 68 |
+
set(METALLIB_FILE "${METALLIB_OUTPUT_DIR}/${TARGET_NAME}.metallib")
|
| 69 |
+
add_custom_command(
|
| 70 |
+
OUTPUT ${METALLIB_FILE}
|
| 71 |
+
COMMAND "${METAL_TOOLCHAIN}/usr/bin/metallib" ${AIR_FILES}
|
| 72 |
+
-o ${METALLIB_FILE}
|
| 73 |
+
DEPENDS ${AIR_FILES}
|
| 74 |
+
COMMENT "Linking Metal library ${METALLIB_FILE}"
|
| 75 |
+
VERBATIM
|
| 76 |
+
)
|
| 77 |
+
|
| 78 |
+
# Generate C++ header with embedded metallib data
|
| 79 |
+
set(METALLIB_HEADER "${CMAKE_BINARY_DIR}/${TARGET_NAME}_metallib.h")
|
| 80 |
+
set(METALLIB_TO_HEADER_SCRIPT "${CMAKE_CURRENT_SOURCE_DIR}/cmake/metallib_to_header.py")
|
| 81 |
+
|
| 82 |
+
add_custom_command(
|
| 83 |
+
OUTPUT ${METALLIB_HEADER}
|
| 84 |
+
COMMAND ${Python3_EXECUTABLE} ${METALLIB_TO_HEADER_SCRIPT} ${METALLIB_FILE} ${METALLIB_HEADER} ${TARGET_NAME}
|
| 85 |
+
DEPENDS ${METALLIB_FILE} ${METALLIB_TO_HEADER_SCRIPT}
|
| 86 |
+
COMMENT "Generating embedded Metal library header ${METALLIB_HEADER}"
|
| 87 |
+
VERBATIM
|
| 88 |
+
)
|
| 89 |
+
|
| 90 |
+
# Create a custom target for the metallib
|
| 91 |
+
add_custom_target(${TARGET_NAME}_metallib ALL DEPENDS ${METALLIB_FILE} ${METALLIB_HEADER})
|
| 92 |
+
|
| 93 |
+
# Add dependency to main target
|
| 94 |
+
add_dependencies(${TARGET_NAME} ${TARGET_NAME}_metallib)
|
| 95 |
+
|
| 96 |
+
# Add the generated header to include directories
|
| 97 |
+
target_include_directories(${TARGET_NAME} PRIVATE ${CMAKE_BINARY_DIR})
|
| 98 |
+
|
| 99 |
+
# Pass the metallib header and namespace as compile definitions
|
| 100 |
+
target_compile_definitions(${TARGET_NAME} PRIVATE
|
| 101 |
+
EMBEDDED_METALLIB_HEADER="${TARGET_NAME}_metallib.h"
|
| 102 |
+
EMBEDDED_METALLIB_NAMESPACE=${TARGET_NAME}_metal
|
| 103 |
+
)
|
| 104 |
+
endfunction()
|
src/kernel-source/rms-quant/cmake/get_gpu_lang.cmake
ADDED
|
@@ -0,0 +1,9 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#
|
| 2 |
+
# Get the GPU language from Torch.
|
| 3 |
+
#
|
| 4 |
+
function(get_gpu_lang OUT)
|
| 5 |
+
run_python_script(PYTHON_OUT
|
| 6 |
+
"${CMAKE_CURRENT_SOURCE_DIR}/cmake/get_gpu_lang.py"
|
| 7 |
+
"Cannot detect GPU language")
|
| 8 |
+
set(${OUT} ${PYTHON_OUT} PARENT_SCOPE)
|
| 9 |
+
endfunction()
|
src/kernel-source/rms-quant/cmake/get_gpu_lang.py
ADDED
|
@@ -0,0 +1,20 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
|
| 3 |
+
import sys
|
| 4 |
+
|
| 5 |
+
try:
|
| 6 |
+
import torch
|
| 7 |
+
except ImportError:
|
| 8 |
+
print("Torch is required for configuring a kernel build.", file=sys.stderr)
|
| 9 |
+
sys.exit(1)
|
| 10 |
+
|
| 11 |
+
if torch.version.cuda is not None:
|
| 12 |
+
print("CUDA")
|
| 13 |
+
elif torch.version.hip is not None:
|
| 14 |
+
print("HIP")
|
| 15 |
+
elif torch.backends.mps.is_available():
|
| 16 |
+
print("METAL")
|
| 17 |
+
elif hasattr(torch.version, "xpu") and torch.version.xpu is not None:
|
| 18 |
+
print("SYCL")
|
| 19 |
+
else:
|
| 20 |
+
print("CPU")
|
src/kernel-source/rms-quant/cmake/hipify.py
ADDED
|
@@ -0,0 +1,76 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
# SPDX-License-Identifier: Apache-2.0
|
| 3 |
+
|
| 4 |
+
# From vLLM: https://github.com/vllm-project/vllm/blob/main/cmake/hipify.py
|
| 5 |
+
|
| 6 |
+
#
|
| 7 |
+
# A command line tool for running pytorch's hipify preprocessor on CUDA
|
| 8 |
+
# source files.
|
| 9 |
+
#
|
| 10 |
+
# See https://github.com/ROCm/hipify_torch
|
| 11 |
+
# and <torch install dir>/utils/hipify/hipify_python.py
|
| 12 |
+
#
|
| 13 |
+
|
| 14 |
+
import argparse
|
| 15 |
+
import os
|
| 16 |
+
import shutil
|
| 17 |
+
|
| 18 |
+
from torch.utils.hipify.hipify_python import hipify
|
| 19 |
+
|
| 20 |
+
if __name__ == '__main__':
|
| 21 |
+
parser = argparse.ArgumentParser()
|
| 22 |
+
|
| 23 |
+
# Project directory where all the source + include files live.
|
| 24 |
+
parser.add_argument(
|
| 25 |
+
"-p",
|
| 26 |
+
"--project_dir",
|
| 27 |
+
help="The project directory.",
|
| 28 |
+
)
|
| 29 |
+
|
| 30 |
+
# Directory where hipified files are written.
|
| 31 |
+
parser.add_argument(
|
| 32 |
+
"-o",
|
| 33 |
+
"--output_dir",
|
| 34 |
+
help="The output directory.",
|
| 35 |
+
)
|
| 36 |
+
|
| 37 |
+
# Source files to convert.
|
| 38 |
+
parser.add_argument("sources",
|
| 39 |
+
help="Source files to hipify.",
|
| 40 |
+
nargs="*",
|
| 41 |
+
default=[])
|
| 42 |
+
|
| 43 |
+
args = parser.parse_args()
|
| 44 |
+
|
| 45 |
+
# Limit include scope to project_dir only
|
| 46 |
+
includes = [os.path.join(args.project_dir, '*')]
|
| 47 |
+
|
| 48 |
+
# Get absolute path for all source files.
|
| 49 |
+
extra_files = [os.path.abspath(s) for s in args.sources]
|
| 50 |
+
|
| 51 |
+
# Copy sources from project directory to output directory.
|
| 52 |
+
# The directory might already exist to hold object files so we ignore that.
|
| 53 |
+
shutil.copytree(args.project_dir, args.output_dir, dirs_exist_ok=True)
|
| 54 |
+
|
| 55 |
+
hipify_result = hipify(project_directory=args.project_dir,
|
| 56 |
+
output_directory=args.output_dir,
|
| 57 |
+
header_include_dirs=[],
|
| 58 |
+
includes=includes,
|
| 59 |
+
extra_files=extra_files,
|
| 60 |
+
show_detailed=True,
|
| 61 |
+
is_pytorch_extension=True,
|
| 62 |
+
hipify_extra_files_only=True)
|
| 63 |
+
|
| 64 |
+
hipified_sources = []
|
| 65 |
+
for source in args.sources:
|
| 66 |
+
s_abs = os.path.abspath(source)
|
| 67 |
+
hipified_s_abs = (hipify_result[s_abs].hipified_path if
|
| 68 |
+
(s_abs in hipify_result
|
| 69 |
+
and hipify_result[s_abs].hipified_path is not None)
|
| 70 |
+
else s_abs)
|
| 71 |
+
hipified_sources.append(hipified_s_abs)
|
| 72 |
+
|
| 73 |
+
assert (len(hipified_sources) == len(args.sources))
|
| 74 |
+
|
| 75 |
+
# Print hipified source files.
|
| 76 |
+
print("\n".join(hipified_sources))
|
src/kernel-source/rms-quant/cmake/kernel.cmake
ADDED
|
@@ -0,0 +1,317 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
function(accumulate_gpu_archs OUT_ACC ACC EXTRA_ARCHS)
|
| 2 |
+
list(APPEND ACC ${EXTRA_ARCHS})
|
| 3 |
+
list(REMOVE_DUPLICATES ACC)
|
| 4 |
+
list(SORT ACC)
|
| 5 |
+
set(${OUT_ACC} ${ACC} PARENT_SCOPE)
|
| 6 |
+
endfunction()
|
| 7 |
+
|
| 8 |
+
function(cuda_kernel_component SRC_VAR)
|
| 9 |
+
set(oneValueArgs CUDA_MINVER NAME)
|
| 10 |
+
set(multiValueArgs SOURCES INCLUDES CUDA_CAPABILITIES CUDA_FLAGS CXX_FLAGS)
|
| 11 |
+
cmake_parse_arguments(KERNEL "${options}" "${oneValueArgs}" "${multiValueArgs}" ${ARGN})
|
| 12 |
+
|
| 13 |
+
if(NOT KERNEL_SOURCES)
|
| 14 |
+
message(FATAL_ERROR "cuda_kernel_component: SOURCES argument is required")
|
| 15 |
+
endif()
|
| 16 |
+
|
| 17 |
+
# Bail out if this component is not supported by the CUDA version.
|
| 18 |
+
if(KERNEL_CUDA_MINVER)
|
| 19 |
+
if(CUDA_VERSION VERSION_LESS ${KERNEL_CUDA_MINVER})
|
| 20 |
+
return()
|
| 21 |
+
endif()
|
| 22 |
+
endif()
|
| 23 |
+
|
| 24 |
+
set(_KERNEL_SRC ${KERNEL_SOURCES})
|
| 25 |
+
|
| 26 |
+
if(KERNEL_INCLUDES)
|
| 27 |
+
# TODO: check if CLion support this:
|
| 28 |
+
# https://youtrack.jetbrains.com/issue/CPP-16510/CLion-does-not-handle-per-file-include-directories
|
| 29 |
+
set_source_files_properties(
|
| 30 |
+
${_KERNEL_SRC}
|
| 31 |
+
PROPERTIES INCLUDE_DIRECTORIES "${KERNEL_INCLUDES}")
|
| 32 |
+
endif()
|
| 33 |
+
|
| 34 |
+
# Determine CUDA architectures
|
| 35 |
+
if(KERNEL_CUDA_CAPABILITIES)
|
| 36 |
+
cuda_archs_loose_intersection(_KERNEL_ARCHS "${KERNEL_CUDA_CAPABILITIES}" "${CUDA_ARCHS}")
|
| 37 |
+
if(NOT _KERNEL_ARCHS)
|
| 38 |
+
message(FATAL_ERROR "CUDA kernel: ${KERNEL_NAME}, empty set of capabilities after intersection (kernel: ${KERNEL_CUDA_CAPABILITIES}, supported: ${CUDA_ARCHS})")
|
| 39 |
+
endif()
|
| 40 |
+
else()
|
| 41 |
+
set(_KERNEL_ARCHS "${CUDA_KERNEL_ARCHS}")
|
| 42 |
+
endif()
|
| 43 |
+
message(STATUS "CUDA kernel: ${KERNEL_NAME}, capabilities: ${_KERNEL_ARCHS}")
|
| 44 |
+
set_gencode_flags_for_srcs(SRCS "${_KERNEL_SRC}" CUDA_ARCHS "${_KERNEL_ARCHS}")
|
| 45 |
+
|
| 46 |
+
accumulate_gpu_archs(_ALL_GPU_ARCHS "${ALL_GPU_ARCHS}" "${_KERNEL_ARCHS}")
|
| 47 |
+
set(ALL_GPU_ARCHS ${_ALL_GPU_ARCHS} PARENT_SCOPE)
|
| 48 |
+
|
| 49 |
+
# Apply CUDA-specific compile flags
|
| 50 |
+
if(KERNEL_CUDA_FLAGS)
|
| 51 |
+
set(_CUDA_FLAGS "${KERNEL_CUDA_FLAGS}")
|
| 52 |
+
# -static-global-template-stub is not supported on CUDA < 12.8. Remove this
|
| 53 |
+
# once we don't support CUDA 12.6 anymore.
|
| 54 |
+
if(CUDA_VERSION VERSION_LESS 12.8)
|
| 55 |
+
string(REGEX REPLACE "-static-global-template-stub=(true|false)" "" _CUDA_FLAGS "${_CUDA_FLAGS}")
|
| 56 |
+
endif()
|
| 57 |
+
|
| 58 |
+
foreach(_SRC ${_KERNEL_SRC})
|
| 59 |
+
if(_SRC MATCHES ".*\\.cu$")
|
| 60 |
+
set_property(
|
| 61 |
+
SOURCE ${_SRC}
|
| 62 |
+
APPEND PROPERTY
|
| 63 |
+
COMPILE_OPTIONS "$<$<COMPILE_LANGUAGE:CUDA>:${_CUDA_FLAGS}>"
|
| 64 |
+
)
|
| 65 |
+
endif()
|
| 66 |
+
endforeach()
|
| 67 |
+
endif()
|
| 68 |
+
|
| 69 |
+
# Apply CXX-specific compile flags
|
| 70 |
+
if(KERNEL_CXX_FLAGS)
|
| 71 |
+
foreach(_SRC ${_KERNEL_SRC})
|
| 72 |
+
set_property(
|
| 73 |
+
SOURCE ${_SRC}
|
| 74 |
+
APPEND PROPERTY
|
| 75 |
+
COMPILE_OPTIONS "$<$<COMPILE_LANGUAGE:CXX>:${KERNEL_CXX_FLAGS}>"
|
| 76 |
+
)
|
| 77 |
+
endforeach()
|
| 78 |
+
endif()
|
| 79 |
+
|
| 80 |
+
set(_TMP_SRC ${${SRC_VAR}})
|
| 81 |
+
list(APPEND _TMP_SRC ${_KERNEL_SRC})
|
| 82 |
+
set(${SRC_VAR} ${_TMP_SRC} PARENT_SCOPE)
|
| 83 |
+
endfunction()
|
| 84 |
+
|
| 85 |
+
function(hip_kernel_component SRC_VAR)
|
| 86 |
+
set(options SUPPORTS_HIPIFY)
|
| 87 |
+
set(oneValueArgs CUDA_MINVER NAME)
|
| 88 |
+
set(multiValueArgs SOURCES INCLUDES CXX_FLAGS HIP_FLAGS ROCM_ARCHS)
|
| 89 |
+
cmake_parse_arguments(KERNEL "${options}" "${oneValueArgs}" "${multiValueArgs}" ${ARGN})
|
| 90 |
+
|
| 91 |
+
if(NOT KERNEL_SOURCES)
|
| 92 |
+
message(FATAL_ERROR "hip_kernel_component: SOURCES argument is required")
|
| 93 |
+
endif()
|
| 94 |
+
|
| 95 |
+
set(_KERNEL_SRC ${KERNEL_SOURCES})
|
| 96 |
+
|
| 97 |
+
if(KERNEL_INCLUDES)
|
| 98 |
+
# TODO: check if CLion support this:
|
| 99 |
+
# https://youtrack.jetbrains.com/issue/CPP-16510/CLion-does-not-handle-per-file-include-directories
|
| 100 |
+
set_source_files_properties(
|
| 101 |
+
${_KERNEL_SRC}
|
| 102 |
+
PROPERTIES INCLUDE_DIRECTORIES "${KERNEL_INCLUDES}")
|
| 103 |
+
endif()
|
| 104 |
+
|
| 105 |
+
# Apply HIP-specific compile flags
|
| 106 |
+
if(KERNEL_HIP_FLAGS)
|
| 107 |
+
foreach(_SRC ${_KERNEL_SRC})
|
| 108 |
+
if(_SRC MATCHES ".*\\.(cu|hip)$")
|
| 109 |
+
set_property(
|
| 110 |
+
SOURCE ${_SRC}
|
| 111 |
+
APPEND PROPERTY
|
| 112 |
+
COMPILE_OPTIONS "$<$<COMPILE_LANGUAGE:HIP>:${KERNEL_HIP_FLAGS}>"
|
| 113 |
+
)
|
| 114 |
+
endif()
|
| 115 |
+
endforeach()
|
| 116 |
+
endif()
|
| 117 |
+
|
| 118 |
+
# Determine ROCm architectures
|
| 119 |
+
if(KERNEL_ROCM_ARCHS)
|
| 120 |
+
hip_archs_loose_intersection(_KERNEL_ARCHS "${KERNEL_ROCM_ARCHS}" "${ROCM_ARCHS}")
|
| 121 |
+
if(NOT _KERNEL_ARCHS)
|
| 122 |
+
message(FATAL_ERROR "ROCm kernel: ${KERNEL_NAME}, empty set of architectures after intersection (kernel: ${KERNEL_ROCM_ARCHS}, supported: ${ROCM_ARCHS})")
|
| 123 |
+
endif()
|
| 124 |
+
else()
|
| 125 |
+
set(_KERNEL_ARCHS "${ROCM_ARCHS}")
|
| 126 |
+
endif()
|
| 127 |
+
message(STATUS "ROCm kernel: ${KERNEL_NAME}, archs: ${_KERNEL_ARCHS}")
|
| 128 |
+
|
| 129 |
+
accumulate_gpu_archs(_ALL_GPU_ARCHS "${ALL_GPU_ARCHS}" "${_KERNEL_ARCHS}")
|
| 130 |
+
set(ALL_GPU_ARCHS ${_ALL_GPU_ARCHS} PARENT_SCOPE)
|
| 131 |
+
|
| 132 |
+
foreach(_SRC ${_KERNEL_SRC})
|
| 133 |
+
if(_SRC MATCHES ".*\\.(cu|hip)$")
|
| 134 |
+
foreach(_ARCH ${_KERNEL_ARCHS})
|
| 135 |
+
set_property(
|
| 136 |
+
SOURCE ${_SRC}
|
| 137 |
+
APPEND PROPERTY
|
| 138 |
+
COMPILE_OPTIONS "$<$<COMPILE_LANGUAGE:HIP>:--offload-arch=${_ARCH}>"
|
| 139 |
+
)
|
| 140 |
+
endforeach()
|
| 141 |
+
endif()
|
| 142 |
+
endforeach()
|
| 143 |
+
|
| 144 |
+
set(_TMP_SRC ${${SRC_VAR}})
|
| 145 |
+
list(APPEND _TMP_SRC ${_KERNEL_SRC})
|
| 146 |
+
set(${SRC_VAR} ${_TMP_SRC} PARENT_SCOPE)
|
| 147 |
+
endfunction()
|
| 148 |
+
|
| 149 |
+
|
| 150 |
+
function(xpu_kernel_component SRC_VAR)
|
| 151 |
+
set(options)
|
| 152 |
+
set(oneValueArgs)
|
| 153 |
+
set(multiValueArgs SOURCES INCLUDES CXX_FLAGS SYCL_FLAGS)
|
| 154 |
+
cmake_parse_arguments(KERNEL "${options}" "${oneValueArgs}" "${multiValueArgs}" ${ARGN})
|
| 155 |
+
|
| 156 |
+
if(NOT KERNEL_SOURCES)
|
| 157 |
+
message(FATAL_ERROR "xpu_kernel_component: SOURCES argument is required")
|
| 158 |
+
endif()
|
| 159 |
+
|
| 160 |
+
set(_KERNEL_SRC ${KERNEL_SOURCES})
|
| 161 |
+
|
| 162 |
+
# Handle per-file include directories if specified
|
| 163 |
+
if(KERNEL_INCLUDES)
|
| 164 |
+
# TODO: check if CLion support this:
|
| 165 |
+
# https://youtrack.jetbrains.com/issue/CPP-16510/CLion-does-not-handle-per-file-include-directories
|
| 166 |
+
set_source_files_properties(
|
| 167 |
+
${_KERNEL_SRC}
|
| 168 |
+
PROPERTIES INCLUDE_DIRECTORIES "${KERNEL_INCLUDES}")
|
| 169 |
+
endif()
|
| 170 |
+
|
| 171 |
+
# Apply CXX-specific compile flags
|
| 172 |
+
if(KERNEL_CXX_FLAGS)
|
| 173 |
+
foreach(_SRC ${_KERNEL_SRC})
|
| 174 |
+
set_property(
|
| 175 |
+
SOURCE ${_SRC}
|
| 176 |
+
APPEND PROPERTY
|
| 177 |
+
COMPILE_OPTIONS "$<$<COMPILE_LANGUAGE:CXX>:${KERNEL_CXX_FLAGS}>"
|
| 178 |
+
)
|
| 179 |
+
endforeach()
|
| 180 |
+
endif()
|
| 181 |
+
|
| 182 |
+
# Add SYCL-specific compilation flags for XPU sources
|
| 183 |
+
if(KERNEL_SYCL_FLAGS)
|
| 184 |
+
# Use kernel-specific SYCL flags plus SYCL flags from the parent scope.
|
| 185 |
+
set(_SYCL_FLAGS ${sycl_flags})
|
| 186 |
+
list(APPEND _SYCL_FLAGS ${KERNEL_SYCL_FLAGS})
|
| 187 |
+
foreach(_SRC ${_KERNEL_SRC})
|
| 188 |
+
if(_SRC MATCHES ".*\\.(cpp|cxx|cc)$")
|
| 189 |
+
set_property(
|
| 190 |
+
SOURCE ${_SRC}
|
| 191 |
+
APPEND PROPERTY
|
| 192 |
+
COMPILE_OPTIONS "$<$<COMPILE_LANGUAGE:CXX>:${_SYCL_FLAGS}>"
|
| 193 |
+
)
|
| 194 |
+
endif()
|
| 195 |
+
endforeach()
|
| 196 |
+
else()
|
| 197 |
+
# Use default SYCL flags (from parent scope variable sycl_flags)
|
| 198 |
+
foreach(_SRC ${_KERNEL_SRC})
|
| 199 |
+
if(_SRC MATCHES ".*\\.(cpp|cxx|cc)$")
|
| 200 |
+
set_property(
|
| 201 |
+
SOURCE ${_SRC}
|
| 202 |
+
APPEND PROPERTY
|
| 203 |
+
COMPILE_OPTIONS "$<$<COMPILE_LANGUAGE:CXX>:${sycl_flags}>"
|
| 204 |
+
)
|
| 205 |
+
endif()
|
| 206 |
+
endforeach()
|
| 207 |
+
endif()
|
| 208 |
+
|
| 209 |
+
# Append to parent scope SRC variable
|
| 210 |
+
set(_TMP_SRC ${${SRC_VAR}})
|
| 211 |
+
list(APPEND _TMP_SRC ${_KERNEL_SRC})
|
| 212 |
+
set(${SRC_VAR} ${_TMP_SRC} PARENT_SCOPE)
|
| 213 |
+
endfunction()
|
| 214 |
+
|
| 215 |
+
function(cpu_kernel_component SRC_VAR)
|
| 216 |
+
set(options)
|
| 217 |
+
set(oneValueArgs)
|
| 218 |
+
set(multiValueArgs SOURCES INCLUDES CXX_FLAGS)
|
| 219 |
+
cmake_parse_arguments(KERNEL "${options}" "${oneValueArgs}" "${multiValueArgs}" ${ARGN})
|
| 220 |
+
|
| 221 |
+
if(NOT KERNEL_SOURCES)
|
| 222 |
+
message(FATAL_ERROR "cpu_kernel_component: SOURCES argument is required")
|
| 223 |
+
endif()
|
| 224 |
+
|
| 225 |
+
set(_KERNEL_SRC ${KERNEL_SOURCES})
|
| 226 |
+
|
| 227 |
+
# Handle per-file include directories if specified
|
| 228 |
+
if(KERNEL_INCLUDES)
|
| 229 |
+
# TODO: check if CLion support this:
|
| 230 |
+
# https://youtrack.jetbrains.com/issue/CPP-16510/CLion-does-not-handle-per-file-include-directories
|
| 231 |
+
set_source_files_properties(
|
| 232 |
+
${_KERNEL_SRC}
|
| 233 |
+
PROPERTIES INCLUDE_DIRECTORIES "${KERNEL_INCLUDES}")
|
| 234 |
+
endif()
|
| 235 |
+
|
| 236 |
+
# Apply CXX-specific compile flags
|
| 237 |
+
if(KERNEL_CXX_FLAGS)
|
| 238 |
+
foreach(_SRC ${_KERNEL_SRC})
|
| 239 |
+
set_property(
|
| 240 |
+
SOURCE ${_SRC}
|
| 241 |
+
APPEND PROPERTY
|
| 242 |
+
COMPILE_OPTIONS "$<$<COMPILE_LANGUAGE:CXX>:${KERNEL_CXX_FLAGS}>"
|
| 243 |
+
)
|
| 244 |
+
endforeach()
|
| 245 |
+
endif()
|
| 246 |
+
|
| 247 |
+
# Append to parent scope SRC variable
|
| 248 |
+
set(_TMP_SRC ${${SRC_VAR}})
|
| 249 |
+
list(APPEND _TMP_SRC ${_KERNEL_SRC})
|
| 250 |
+
set(${SRC_VAR} ${_TMP_SRC} PARENT_SCOPE)
|
| 251 |
+
endfunction()
|
| 252 |
+
|
| 253 |
+
function(metal_kernel_component SRC_VAR)
|
| 254 |
+
set(options)
|
| 255 |
+
set(oneValueArgs)
|
| 256 |
+
set(multiValueArgs SOURCES INCLUDES CXX_FLAGS)
|
| 257 |
+
cmake_parse_arguments(KERNEL "${options}" "${oneValueArgs}" "${multiValueArgs}" ${ARGN})
|
| 258 |
+
|
| 259 |
+
if(NOT KERNEL_SOURCES)
|
| 260 |
+
message(FATAL_ERROR "metal_kernel_component: SOURCES argument is required")
|
| 261 |
+
endif()
|
| 262 |
+
|
| 263 |
+
set(_KERNEL_SRC ${KERNEL_SOURCES})
|
| 264 |
+
|
| 265 |
+
# Separate Metal shader files from other sources
|
| 266 |
+
set(_METAL_SRC)
|
| 267 |
+
set(_CPP_SRC)
|
| 268 |
+
|
| 269 |
+
foreach(_SRC_FILE IN LISTS _KERNEL_SRC)
|
| 270 |
+
if(_SRC_FILE MATCHES "\\.(metal|h)$")
|
| 271 |
+
list(APPEND _METAL_SRC ${_SRC_FILE})
|
| 272 |
+
else()
|
| 273 |
+
list(APPEND _CPP_SRC ${_SRC_FILE})
|
| 274 |
+
endif()
|
| 275 |
+
endforeach()
|
| 276 |
+
|
| 277 |
+
# Handle per-file include directories if specified (for C++ sources only)
|
| 278 |
+
if(KERNEL_INCLUDES AND _CPP_SRC)
|
| 279 |
+
# TODO: check if CLion support this:
|
| 280 |
+
# https://youtrack.jetbrains.com/issue/CPP-16510/CLion-does-not-handle-per-file-include-directories
|
| 281 |
+
set_source_files_properties(
|
| 282 |
+
${_CPP_SRC}
|
| 283 |
+
PROPERTIES INCLUDE_DIRECTORIES "${KERNEL_INCLUDES}")
|
| 284 |
+
endif()
|
| 285 |
+
|
| 286 |
+
# Apply CXX-specific compile flags
|
| 287 |
+
if(KERNEL_CXX_FLAGS AND _CPP_SRC)
|
| 288 |
+
foreach(_SRC ${_CPP_SRC})
|
| 289 |
+
set_property(
|
| 290 |
+
SOURCE ${_SRC}
|
| 291 |
+
APPEND PROPERTY
|
| 292 |
+
COMPILE_OPTIONS "$<$<COMPILE_LANGUAGE:CXX>:${KERNEL_CXX_FLAGS}>"
|
| 293 |
+
)
|
| 294 |
+
endforeach()
|
| 295 |
+
endif()
|
| 296 |
+
|
| 297 |
+
# Add C++ sources to main source list
|
| 298 |
+
if(_CPP_SRC)
|
| 299 |
+
set(_TMP_SRC ${${SRC_VAR}})
|
| 300 |
+
list(APPEND _TMP_SRC ${_CPP_SRC})
|
| 301 |
+
set(${SRC_VAR} ${_TMP_SRC} PARENT_SCOPE)
|
| 302 |
+
endif()
|
| 303 |
+
|
| 304 |
+
# Keep track of Metal sources for later compilation
|
| 305 |
+
if(_METAL_SRC)
|
| 306 |
+
set(_TMP_METAL ${ALL_METAL_SOURCES})
|
| 307 |
+
list(APPEND _TMP_METAL ${_METAL_SRC})
|
| 308 |
+
set(ALL_METAL_SOURCES ${_TMP_METAL} PARENT_SCOPE)
|
| 309 |
+
endif()
|
| 310 |
+
|
| 311 |
+
# Keep the includes directory for the Metal sources
|
| 312 |
+
if(KERNEL_INCLUDES AND _METAL_SRC)
|
| 313 |
+
set(_TMP_METAL_INCLUDES ${METAL_INCLUDE_DIRS})
|
| 314 |
+
list(APPEND _TMP_METAL_INCLUDES ${KERNEL_INCLUDES})
|
| 315 |
+
set(METAL_INCLUDE_DIRS ${_TMP_METAL_INCLUDES} PARENT_SCOPE)
|
| 316 |
+
endif()
|
| 317 |
+
endfunction()
|
src/kernel-source/rms-quant/cmake/metallib_to_header.py
ADDED
|
@@ -0,0 +1,73 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
import sys
|
| 3 |
+
import os
|
| 4 |
+
|
| 5 |
+
def convert_metallib_to_header(metallib_path: str, header_path: str, target_name: str) -> None:
|
| 6 |
+
"""Convert a metallib binary file to a C++ header with embedded data."""
|
| 7 |
+
|
| 8 |
+
# Read the metallib binary data
|
| 9 |
+
with open(metallib_path, 'rb') as f:
|
| 10 |
+
data: bytes = f.read()
|
| 11 |
+
|
| 12 |
+
# Generate the header content
|
| 13 |
+
header_content: str = """// Auto-generated file containing embedded Metal library
|
| 14 |
+
#pragma once
|
| 15 |
+
#include <cstddef>
|
| 16 |
+
#include <Metal/Metal.h>
|
| 17 |
+
|
| 18 |
+
namespace """ + target_name + """_metal {
|
| 19 |
+
static const unsigned char metallib_data[] = {
|
| 20 |
+
"""
|
| 21 |
+
|
| 22 |
+
# Convert binary data to C array format
|
| 23 |
+
bytes_per_line: int = 16
|
| 24 |
+
for i in range(0, len(data), bytes_per_line):
|
| 25 |
+
chunk: bytes = data[i:i + bytes_per_line]
|
| 26 |
+
hex_values: str = ', '.join('0x{:02x}'.format(b) for b in chunk)
|
| 27 |
+
header_content += " " + hex_values + ","
|
| 28 |
+
if i + bytes_per_line < len(data):
|
| 29 |
+
header_content += "\n"
|
| 30 |
+
|
| 31 |
+
header_content += """
|
| 32 |
+
};
|
| 33 |
+
static const size_t metallib_data_len = """ + str(len(data)) + """;
|
| 34 |
+
|
| 35 |
+
// Convenience function to create Metal library from embedded data
|
| 36 |
+
inline id<MTLLibrary> createLibrary(id<MTLDevice> device, NSError** error = nullptr) {
|
| 37 |
+
dispatch_data_t libraryData = dispatch_data_create(
|
| 38 |
+
metallib_data,
|
| 39 |
+
metallib_data_len,
|
| 40 |
+
dispatch_get_main_queue(),
|
| 41 |
+
^{ /* No cleanup needed for static data */ });
|
| 42 |
+
|
| 43 |
+
NSError* localError = nil;
|
| 44 |
+
id<MTLLibrary> library = [device newLibraryWithData:libraryData error:&localError];
|
| 45 |
+
|
| 46 |
+
if (error) {
|
| 47 |
+
*error = localError;
|
| 48 |
+
}
|
| 49 |
+
|
| 50 |
+
return library;
|
| 51 |
+
}
|
| 52 |
+
} // namespace """ + target_name + """_metal
|
| 53 |
+
"""
|
| 54 |
+
|
| 55 |
+
# Write the header file
|
| 56 |
+
dir_path: str = os.path.dirname(header_path)
|
| 57 |
+
if dir_path:
|
| 58 |
+
os.makedirs(dir_path, exist_ok=True)
|
| 59 |
+
with open(header_path, 'w') as f:
|
| 60 |
+
f.write(header_content)
|
| 61 |
+
|
| 62 |
+
print("Generated {} ({} bytes)".format(header_path, len(data)))
|
| 63 |
+
|
| 64 |
+
if __name__ == "__main__":
|
| 65 |
+
if len(sys.argv) != 4:
|
| 66 |
+
print("Usage: metallib_to_header.py <metallib_path> <header_path> <target_name>")
|
| 67 |
+
sys.exit(1)
|
| 68 |
+
|
| 69 |
+
metallib_path: str = sys.argv[1]
|
| 70 |
+
header_path: str = sys.argv[2]
|
| 71 |
+
target_name: str = sys.argv[3]
|
| 72 |
+
|
| 73 |
+
convert_metallib_to_header(metallib_path, header_path, target_name)
|
src/kernel-source/rms-quant/cmake/utils.cmake
ADDED
|
@@ -0,0 +1,611 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Vendored from vLLM:
|
| 2 |
+
#
|
| 3 |
+
# https://github.com/vllm-project/vllm/blob/main/cmake/utils.cmake
|
| 4 |
+
#
|
| 5 |
+
# Attempt to find the python package that uses the same python executable as
|
| 6 |
+
# `EXECUTABLE` and is one of the `SUPPORTED_VERSIONS`.
|
| 7 |
+
#
|
| 8 |
+
macro (find_python_from_executable EXECUTABLE SUPPORTED_VERSIONS)
|
| 9 |
+
file(REAL_PATH ${EXECUTABLE} EXECUTABLE)
|
| 10 |
+
set(Python3_EXECUTABLE ${EXECUTABLE})
|
| 11 |
+
find_package(Python3 COMPONENTS Interpreter Development.Module Development.SABIModule)
|
| 12 |
+
if (NOT Python3_FOUND)
|
| 13 |
+
message(FATAL_ERROR "Unable to find python matching: ${EXECUTABLE}.")
|
| 14 |
+
endif()
|
| 15 |
+
set(_VER "${Python3_VERSION_MAJOR}.${Python3_VERSION_MINOR}")
|
| 16 |
+
set(_SUPPORTED_VERSIONS_LIST ${SUPPORTED_VERSIONS} ${ARGN})
|
| 17 |
+
if (NOT _VER IN_LIST _SUPPORTED_VERSIONS_LIST)
|
| 18 |
+
message(FATAL_ERROR
|
| 19 |
+
"Python version (${_VER}) is not one of the supported versions: "
|
| 20 |
+
"${_SUPPORTED_VERSIONS_LIST}.")
|
| 21 |
+
endif()
|
| 22 |
+
message(STATUS "Found python matching: ${EXECUTABLE}.")
|
| 23 |
+
endmacro()
|
| 24 |
+
|
| 25 |
+
#
|
| 26 |
+
# Run `EXPR` in python. The standard output of python is stored in `OUT` and
|
| 27 |
+
# has trailing whitespace stripped. If an error is encountered when running
|
| 28 |
+
# python, a fatal message `ERR_MSG` is issued.
|
| 29 |
+
#
|
| 30 |
+
function (run_python OUT EXPR ERR_MSG)
|
| 31 |
+
if(Python3_EXECUTABLE)
|
| 32 |
+
set(_PYTHON_EXECUTABLE "${Python3_EXECUTABLE}")
|
| 33 |
+
elseif(Python_EXECUTABLE)
|
| 34 |
+
set(_PYTHON_EXECUTABLE "${Python_EXECUTABLE}")
|
| 35 |
+
else()
|
| 36 |
+
message(FATAL_ERROR "No Python executable found. Set Python3_EXECUTABLE or Python_EXECUTABLE.")
|
| 37 |
+
endif()
|
| 38 |
+
execute_process(
|
| 39 |
+
COMMAND
|
| 40 |
+
"${_PYTHON_EXECUTABLE}" "-c" "${EXPR}"
|
| 41 |
+
OUTPUT_VARIABLE PYTHON_OUT
|
| 42 |
+
RESULT_VARIABLE PYTHON_ERROR_CODE
|
| 43 |
+
ERROR_VARIABLE PYTHON_STDERR
|
| 44 |
+
OUTPUT_STRIP_TRAILING_WHITESPACE)
|
| 45 |
+
|
| 46 |
+
if(NOT PYTHON_ERROR_CODE EQUAL 0)
|
| 47 |
+
message(FATAL_ERROR "${ERR_MSG}: ${PYTHON_STDERR}")
|
| 48 |
+
endif()
|
| 49 |
+
set(${OUT} ${PYTHON_OUT} PARENT_SCOPE)
|
| 50 |
+
endfunction()
|
| 51 |
+
|
| 52 |
+
#
|
| 53 |
+
# Run `SCRIPT_PATH` in Python. The standard output of Python is stored in
|
| 54 |
+
# `OUT` and has trailing whitespace stripped. If the script exits with a
|
| 55 |
+
# non-zero code, a fatal message `ERR_MSG` is issued.
|
| 56 |
+
#
|
| 57 |
+
function (run_python_script OUT SCRIPT_PATH ERR_MSG)
|
| 58 |
+
if(Python3_EXECUTABLE)
|
| 59 |
+
set(_PYTHON_EXECUTABLE "${Python3_EXECUTABLE}")
|
| 60 |
+
elseif(Python_EXECUTABLE)
|
| 61 |
+
set(_PYTHON_EXECUTABLE "${Python_EXECUTABLE}")
|
| 62 |
+
else()
|
| 63 |
+
message(FATAL_ERROR "No Python executable found. Set Python3_EXECUTABLE or Python_EXECUTABLE.")
|
| 64 |
+
endif()
|
| 65 |
+
execute_process(
|
| 66 |
+
COMMAND
|
| 67 |
+
"${_PYTHON_EXECUTABLE}" "${SCRIPT_PATH}"
|
| 68 |
+
OUTPUT_VARIABLE PYTHON_OUT
|
| 69 |
+
RESULT_VARIABLE PYTHON_ERROR_CODE
|
| 70 |
+
ERROR_VARIABLE PYTHON_STDERR
|
| 71 |
+
OUTPUT_STRIP_TRAILING_WHITESPACE)
|
| 72 |
+
|
| 73 |
+
if(NOT PYTHON_ERROR_CODE EQUAL 0)
|
| 74 |
+
message(FATAL_ERROR "${ERR_MSG}: ${PYTHON_STDERR}")
|
| 75 |
+
endif()
|
| 76 |
+
set(${OUT} ${PYTHON_OUT} PARENT_SCOPE)
|
| 77 |
+
endfunction()
|
| 78 |
+
|
| 79 |
+
#
|
| 80 |
+
# Run `EXPR` in python. The standard output of python is stored in `OUT` and
|
| 81 |
+
# has trailing whitespace stripped. If an error is encountered when running
|
| 82 |
+
# python, `SUCCESS` is set to FALSE. If successful, `SUCCESS` is set to TRUE.
|
| 83 |
+
#
|
| 84 |
+
function (try_run_python OUT SUCCESS EXPR)
|
| 85 |
+
if(Python3_EXECUTABLE)
|
| 86 |
+
set(_PYTHON_EXECUTABLE "${Python3_EXECUTABLE}")
|
| 87 |
+
elseif(Python_EXECUTABLE)
|
| 88 |
+
set(_PYTHON_EXECUTABLE "${Python_EXECUTABLE}")
|
| 89 |
+
else()
|
| 90 |
+
message(FATAL_ERROR "No Python executable found. Set Python3_EXECUTABLE or Python_EXECUTABLE.")
|
| 91 |
+
endif()
|
| 92 |
+
execute_process(
|
| 93 |
+
COMMAND
|
| 94 |
+
"${_PYTHON_EXECUTABLE}" "-c" "${EXPR}"
|
| 95 |
+
OUTPUT_VARIABLE PYTHON_OUT
|
| 96 |
+
RESULT_VARIABLE PYTHON_ERROR_CODE
|
| 97 |
+
ERROR_QUIET
|
| 98 |
+
OUTPUT_STRIP_TRAILING_WHITESPACE)
|
| 99 |
+
|
| 100 |
+
if(NOT PYTHON_ERROR_CODE EQUAL 0)
|
| 101 |
+
set(${SUCCESS} FALSE PARENT_SCOPE)
|
| 102 |
+
set(${OUT} "" PARENT_SCOPE)
|
| 103 |
+
else()
|
| 104 |
+
set(${SUCCESS} TRUE PARENT_SCOPE)
|
| 105 |
+
set(${OUT} ${PYTHON_OUT} PARENT_SCOPE)
|
| 106 |
+
endif()
|
| 107 |
+
endfunction()
|
| 108 |
+
|
| 109 |
+
# Run `EXPR` in python after importing `PKG`. Use the result of this to extend
|
| 110 |
+
# `CMAKE_PREFIX_PATH` so the torch cmake configuration can be imported.
|
| 111 |
+
macro (append_cmake_prefix_path PKG EXPR)
|
| 112 |
+
run_python(_PREFIX_PATH
|
| 113 |
+
"import ${PKG}; print(${EXPR})" "Failed to locate ${PKG} path")
|
| 114 |
+
list(APPEND CMAKE_PREFIX_PATH ${_PREFIX_PATH})
|
| 115 |
+
endmacro()
|
| 116 |
+
|
| 117 |
+
#
|
| 118 |
+
# Add a target named `hipify${NAME}` that runs the hipify preprocessor on a set
|
| 119 |
+
# of CUDA source files. The names of the corresponding "hipified" sources are
|
| 120 |
+
# stored in `OUT_SRCS`.
|
| 121 |
+
#
|
| 122 |
+
function (hipify_sources_target OUT_SRCS NAME ORIG_SRCS)
|
| 123 |
+
#
|
| 124 |
+
# Split into C++ and non-C++ (i.e. CUDA) sources.
|
| 125 |
+
#
|
| 126 |
+
set(NODUP_SRCS ${ORIG_SRCS})
|
| 127 |
+
list(REMOVE_DUPLICATES NODUP_SRCS)
|
| 128 |
+
set(SRCS ${NODUP_SRCS})
|
| 129 |
+
set(CXX_SRCS ${NODUP_SRCS})
|
| 130 |
+
list(FILTER SRCS INCLUDE REGEX "\.cu$")
|
| 131 |
+
list(FILTER CXX_SRCS EXCLUDE REGEX "\.cu$")
|
| 132 |
+
|
| 133 |
+
#
|
| 134 |
+
# Generate ROCm/HIP source file names from CUDA file names.
|
| 135 |
+
# Since HIP files are generated code, they will appear in the build area
|
| 136 |
+
# `CMAKE_CURRENT_BINARY_DIR` directory rather than the original csrc dir.
|
| 137 |
+
#
|
| 138 |
+
set(HIP_SRCS)
|
| 139 |
+
foreach (SRC ${SRCS})
|
| 140 |
+
get_source_file_property(include_dirs "${SRC}" INCLUDE_DIRECTORIES)
|
| 141 |
+
get_source_file_property(compile_options "${SRC}" COMPILE_OPTIONS)
|
| 142 |
+
string(REGEX REPLACE "\.cu$" "\.hip" SRC ${SRC})
|
| 143 |
+
string(REGEX REPLACE "cuda" "hip" SRC ${SRC})
|
| 144 |
+
|
| 145 |
+
if(include_dirs)
|
| 146 |
+
# Copy over include directories from the original CUDA file.
|
| 147 |
+
set_source_files_properties(
|
| 148 |
+
${SRC}
|
| 149 |
+
PROPERTIES INCLUDE_DIRECTORIES "${include_dirs}")
|
| 150 |
+
endif()
|
| 151 |
+
|
| 152 |
+
if(compile_options)
|
| 153 |
+
set_source_files_properties(
|
| 154 |
+
${SRC}
|
| 155 |
+
PROPERTIES COMPILE_OPTIONS "${compile_options}")
|
| 156 |
+
endif()
|
| 157 |
+
|
| 158 |
+
list(APPEND HIP_SRCS "${CMAKE_CURRENT_BINARY_DIR}/${SRC}")
|
| 159 |
+
endforeach()
|
| 160 |
+
|
| 161 |
+
add_custom_target(
|
| 162 |
+
hipify${NAME}
|
| 163 |
+
COMMAND "${Python3_EXECUTABLE}" ${CMAKE_SOURCE_DIR}/cmake/hipify.py -p ${CMAKE_SOURCE_DIR} -o ${CMAKE_CURRENT_BINARY_DIR} ${SRCS}
|
| 164 |
+
DEPENDS ${CMAKE_SOURCE_DIR}/cmake/hipify.py ${SRCS}
|
| 165 |
+
BYPRODUCTS ${HIP_SRCS}
|
| 166 |
+
COMMENT "Running hipify on ${NAME} extension source files.")
|
| 167 |
+
|
| 168 |
+
# Swap out original extension sources with hipified sources.
|
| 169 |
+
list(APPEND HIP_SRCS ${CXX_SRCS})
|
| 170 |
+
set(${OUT_SRCS} ${HIP_SRCS} PARENT_SCOPE)
|
| 171 |
+
endfunction()
|
| 172 |
+
|
| 173 |
+
#
|
| 174 |
+
# Get additional GPU compiler flags from torch.
|
| 175 |
+
#
|
| 176 |
+
function (get_torch_gpu_compiler_flags OUT_GPU_FLAGS GPU_LANG)
|
| 177 |
+
if (${GPU_LANG} STREQUAL "CUDA")
|
| 178 |
+
#
|
| 179 |
+
# Get common NVCC flags from torch.
|
| 180 |
+
#
|
| 181 |
+
run_python(GPU_FLAGS
|
| 182 |
+
"from torch.utils.cpp_extension import COMMON_NVCC_FLAGS; print(';'.join(COMMON_NVCC_FLAGS))"
|
| 183 |
+
"Failed to determine torch nvcc compiler flags")
|
| 184 |
+
|
| 185 |
+
if (CUDA_VERSION VERSION_GREATER_EQUAL 11.8)
|
| 186 |
+
list(APPEND GPU_FLAGS "-DENABLE_FP8")
|
| 187 |
+
list(REMOVE_ITEM GPU_FLAGS
|
| 188 |
+
"-D__CUDA_NO_HALF_OPERATORS__"
|
| 189 |
+
"-D__CUDA_NO_HALF_CONVERSIONS__"
|
| 190 |
+
"-D__CUDA_NO_BFLOAT16_CONVERSIONS__"
|
| 191 |
+
"-D__CUDA_NO_HALF2_OPERATORS__")
|
| 192 |
+
endif()
|
| 193 |
+
|
| 194 |
+
elseif(${GPU_LANG} STREQUAL "HIP")
|
| 195 |
+
#
|
| 196 |
+
# Get common HIP/HIPCC flags from torch.
|
| 197 |
+
#
|
| 198 |
+
run_python(GPU_FLAGS
|
| 199 |
+
"import torch.utils.cpp_extension as t; print(';'.join(t.COMMON_HIP_FLAGS + t.COMMON_HIPCC_FLAGS))"
|
| 200 |
+
"Failed to determine torch nvcc compiler flags")
|
| 201 |
+
|
| 202 |
+
list(APPEND GPU_FLAGS
|
| 203 |
+
"-DUSE_ROCM"
|
| 204 |
+
"-DENABLE_FP8"
|
| 205 |
+
"-U__HIP_NO_HALF_CONVERSIONS__"
|
| 206 |
+
"-U__HIP_NO_HALF_OPERATORS__"
|
| 207 |
+
"-fno-gpu-rdc")
|
| 208 |
+
|
| 209 |
+
endif()
|
| 210 |
+
set(${OUT_GPU_FLAGS} ${GPU_FLAGS} PARENT_SCOPE)
|
| 211 |
+
endfunction()
|
| 212 |
+
|
| 213 |
+
# Macro for converting a `gencode` version number to a cmake version number.
|
| 214 |
+
macro(string_to_ver OUT_VER IN_STR)
|
| 215 |
+
string(REGEX REPLACE "\([0-9]+\)\([0-9]\)" "\\1.\\2" ${OUT_VER} ${IN_STR})
|
| 216 |
+
endmacro()
|
| 217 |
+
|
| 218 |
+
#
|
| 219 |
+
# Clear all `-gencode` flags from `CMAKE_CUDA_FLAGS`.
|
| 220 |
+
#
|
| 221 |
+
# Example:
|
| 222 |
+
# CMAKE_CUDA_FLAGS="-Wall -gencode arch=compute_70,code=sm_70 -gencode arch=compute_75,code=sm_75"
|
| 223 |
+
# clear_gencode_flags()
|
| 224 |
+
# CMAKE_CUDA_FLAGS="-Wall"
|
| 225 |
+
#
|
| 226 |
+
macro(clear_gencode_flags)
|
| 227 |
+
# Remove all `-gencode` flags from `CMAKE_CUDA_FLAGS` since they will be modified
|
| 228 |
+
# and passed back via the `CUDA_ARCHITECTURES` property.
|
| 229 |
+
string(REGEX REPLACE "-gencode arch=[^ ]+ *" "" CMAKE_CUDA_FLAGS
|
| 230 |
+
${CMAKE_CUDA_FLAGS})
|
| 231 |
+
endmacro()
|
| 232 |
+
|
| 233 |
+
#
|
| 234 |
+
# Extract unique CUDA architectures from a list of compute capabilities codes in
|
| 235 |
+
# the form `<major><minor>[<letter>]`, convert them to the form sort
|
| 236 |
+
# `<major>.<minor>`, dedupes them and then sorts them in ascending order and
|
| 237 |
+
# stores them in `OUT_ARCHES`.
|
| 238 |
+
#
|
| 239 |
+
# Example:
|
| 240 |
+
# CUDA_ARCH_FLAGS="-gencode arch=compute_75,code=sm_75;...;-gencode arch=compute_90a,code=sm_90a"
|
| 241 |
+
# extract_unique_cuda_archs_ascending(OUT_ARCHES CUDA_ARCH_FLAGS)
|
| 242 |
+
# OUT_ARCHES="7.5;...;9.0"
|
| 243 |
+
function(extract_unique_cuda_archs_ascending OUT_ARCHES CUDA_ARCH_FLAGS)
|
| 244 |
+
set(_CUDA_ARCHES)
|
| 245 |
+
foreach(_ARCH ${CUDA_ARCH_FLAGS})
|
| 246 |
+
string(REGEX MATCH "arch=compute_\([0-9]+a?\)" _COMPUTE ${_ARCH})
|
| 247 |
+
if (_COMPUTE)
|
| 248 |
+
set(_COMPUTE ${CMAKE_MATCH_1})
|
| 249 |
+
endif()
|
| 250 |
+
|
| 251 |
+
string_to_ver(_COMPUTE_VER ${_COMPUTE})
|
| 252 |
+
list(APPEND _CUDA_ARCHES ${_COMPUTE_VER})
|
| 253 |
+
endforeach()
|
| 254 |
+
|
| 255 |
+
list(REMOVE_DUPLICATES _CUDA_ARCHES)
|
| 256 |
+
list(SORT _CUDA_ARCHES COMPARE NATURAL ORDER ASCENDING)
|
| 257 |
+
set(${OUT_ARCHES} ${_CUDA_ARCHES} PARENT_SCOPE)
|
| 258 |
+
endfunction()
|
| 259 |
+
|
| 260 |
+
#
|
| 261 |
+
# For a specific file set the `-gencode` flag in compile options conditionally
|
| 262 |
+
# for the CUDA language.
|
| 263 |
+
#
|
| 264 |
+
# Example:
|
| 265 |
+
# set_gencode_flag_for_srcs(
|
| 266 |
+
# SRCS "foo.cu"
|
| 267 |
+
# ARCH "compute_75"
|
| 268 |
+
# CODE "sm_75")
|
| 269 |
+
# adds: "-gencode arch=compute_75,code=sm_75" to the compile options for
|
| 270 |
+
# `foo.cu` (only for the CUDA language).
|
| 271 |
+
#
|
| 272 |
+
macro(set_gencode_flag_for_srcs)
|
| 273 |
+
set(options)
|
| 274 |
+
set(oneValueArgs ARCH CODE)
|
| 275 |
+
set(multiValueArgs SRCS)
|
| 276 |
+
cmake_parse_arguments(arg "${options}" "${oneValueArgs}"
|
| 277 |
+
"${multiValueArgs}" ${ARGN} )
|
| 278 |
+
set(_FLAG -gencode arch=${arg_ARCH},code=${arg_CODE})
|
| 279 |
+
set_property(
|
| 280 |
+
SOURCE ${arg_SRCS}
|
| 281 |
+
APPEND PROPERTY
|
| 282 |
+
COMPILE_OPTIONS "$<$<COMPILE_LANGUAGE:CUDA>:${_FLAG}>"
|
| 283 |
+
)
|
| 284 |
+
|
| 285 |
+
message(DEBUG "Setting gencode flag for ${arg_SRCS}: ${_FLAG}")
|
| 286 |
+
endmacro(set_gencode_flag_for_srcs)
|
| 287 |
+
|
| 288 |
+
#
|
| 289 |
+
# For a list of source files set the `-gencode` flags in the files specific
|
| 290 |
+
# compile options (specifically for the CUDA language).
|
| 291 |
+
#
|
| 292 |
+
# arguments are:
|
| 293 |
+
# SRCS: list of source files
|
| 294 |
+
# CUDA_ARCHS: list of CUDA architectures in the form `<major>.<minor>[letter]`
|
| 295 |
+
# BUILD_PTX_FOR_ARCH: if set to true, then the PTX code will be built
|
| 296 |
+
# for architecture `BUILD_PTX_FOR_ARCH` if there is a CUDA_ARCH in CUDA_ARCHS
|
| 297 |
+
# that is larger than BUILD_PTX_FOR_ARCH.
|
| 298 |
+
#
|
| 299 |
+
macro(set_gencode_flags_for_srcs)
|
| 300 |
+
set(options)
|
| 301 |
+
set(oneValueArgs BUILD_PTX_FOR_ARCH)
|
| 302 |
+
set(multiValueArgs SRCS CUDA_ARCHS)
|
| 303 |
+
cmake_parse_arguments(arg "${options}" "${oneValueArgs}"
|
| 304 |
+
"${multiValueArgs}" ${ARGN} )
|
| 305 |
+
|
| 306 |
+
foreach(_ARCH ${arg_CUDA_ARCHS})
|
| 307 |
+
# handle +PTX suffix: generate both sm and ptx codes if requested
|
| 308 |
+
string(FIND "${_ARCH}" "+PTX" _HAS_PTX)
|
| 309 |
+
if(NOT _HAS_PTX EQUAL -1)
|
| 310 |
+
string(REPLACE "+PTX" "" _BASE_ARCH "${_ARCH}")
|
| 311 |
+
string(REPLACE "." "" _STRIPPED_ARCH "${_BASE_ARCH}")
|
| 312 |
+
set_gencode_flag_for_srcs(
|
| 313 |
+
SRCS ${arg_SRCS}
|
| 314 |
+
ARCH "compute_${_STRIPPED_ARCH}"
|
| 315 |
+
CODE "sm_${_STRIPPED_ARCH}")
|
| 316 |
+
set_gencode_flag_for_srcs(
|
| 317 |
+
SRCS ${arg_SRCS}
|
| 318 |
+
ARCH "compute_${_STRIPPED_ARCH}"
|
| 319 |
+
CODE "compute_${_STRIPPED_ARCH}")
|
| 320 |
+
else()
|
| 321 |
+
string(REPLACE "." "" _STRIPPED_ARCH "${_ARCH}")
|
| 322 |
+
set_gencode_flag_for_srcs(
|
| 323 |
+
SRCS ${arg_SRCS}
|
| 324 |
+
ARCH "compute_${_STRIPPED_ARCH}"
|
| 325 |
+
CODE "sm_${_STRIPPED_ARCH}")
|
| 326 |
+
endif()
|
| 327 |
+
endforeach()
|
| 328 |
+
|
| 329 |
+
if (${arg_BUILD_PTX_FOR_ARCH})
|
| 330 |
+
list(SORT arg_CUDA_ARCHS COMPARE NATURAL ORDER ASCENDING)
|
| 331 |
+
list(GET arg_CUDA_ARCHS -1 _HIGHEST_ARCH)
|
| 332 |
+
if (_HIGHEST_ARCH VERSION_GREATER_EQUAL ${arg_BUILD_PTX_FOR_ARCH})
|
| 333 |
+
string(REPLACE "." "" _PTX_ARCH "${arg_BUILD_PTX_FOR_ARCH}")
|
| 334 |
+
set_gencode_flag_for_srcs(
|
| 335 |
+
SRCS ${arg_SRCS}
|
| 336 |
+
ARCH "compute_${_PTX_ARCH}"
|
| 337 |
+
CODE "compute_${_PTX_ARCH}")
|
| 338 |
+
endif()
|
| 339 |
+
endif()
|
| 340 |
+
endmacro()
|
| 341 |
+
|
| 342 |
+
#
|
| 343 |
+
# For the given `SRC_CUDA_ARCHS` list of gencode versions in the form
|
| 344 |
+
# `<major>.<minor>[letter]` compute the "loose intersection" with the
|
| 345 |
+
# `TGT_CUDA_ARCHS` list of gencodes. We also support the `+PTX` suffix in
|
| 346 |
+
# `SRC_CUDA_ARCHS` which indicates that the PTX code should be built when there
|
| 347 |
+
# is a CUDA_ARCH in `TGT_CUDA_ARCHS` that is equal to or larger than the
|
| 348 |
+
# architecture in `SRC_CUDA_ARCHS`.
|
| 349 |
+
# The loose intersection is defined as:
|
| 350 |
+
# { max{ x \in tgt | x <= y } | y \in src, { x \in tgt | x <= y } != {} }
|
| 351 |
+
# where `<=` is the version comparison operator.
|
| 352 |
+
# In other words, for each version in `TGT_CUDA_ARCHS` find the highest version
|
| 353 |
+
# in `SRC_CUDA_ARCHS` that is less or equal to the version in `TGT_CUDA_ARCHS`.
|
| 354 |
+
# We have special handling for x.0a, if x.0a is in `SRC_CUDA_ARCHS` and x.0 is
|
| 355 |
+
# in `TGT_CUDA_ARCHS` then we should remove x.0a from `SRC_CUDA_ARCHS` and add
|
| 356 |
+
# x.0a to the result (and remove x.0 from TGT_CUDA_ARCHS).
|
| 357 |
+
# The result is stored in `OUT_CUDA_ARCHS`.
|
| 358 |
+
#
|
| 359 |
+
# Example:
|
| 360 |
+
# SRC_CUDA_ARCHS="7.5;8.0;8.6;9.0;9.0a"
|
| 361 |
+
# TGT_CUDA_ARCHS="8.0;8.9;9.0"
|
| 362 |
+
# cuda_archs_loose_intersection(OUT_CUDA_ARCHS SRC_CUDA_ARCHS TGT_CUDA_ARCHS)
|
| 363 |
+
# OUT_CUDA_ARCHS="8.0;8.6;9.0;9.0a"
|
| 364 |
+
#
|
| 365 |
+
# Example With PTX:
|
| 366 |
+
# SRC_CUDA_ARCHS="8.0+PTX"
|
| 367 |
+
# TGT_CUDA_ARCHS="9.0"
|
| 368 |
+
# cuda_archs_loose_intersection(OUT_CUDA_ARCHS SRC_CUDA_ARCHS TGT_CUDA_ARCHS)
|
| 369 |
+
# OUT_CUDA_ARCHS="8.0+PTX"
|
| 370 |
+
#
|
| 371 |
+
function(cuda_archs_loose_intersection OUT_CUDA_ARCHS SRC_CUDA_ARCHS TGT_CUDA_ARCHS)
|
| 372 |
+
set(_SRC_CUDA_ARCHS "${SRC_CUDA_ARCHS}")
|
| 373 |
+
set(_TGT_CUDA_ARCHS ${TGT_CUDA_ARCHS})
|
| 374 |
+
|
| 375 |
+
# handle +PTX suffix: separate base arch for matching, record PTX requests
|
| 376 |
+
set(_PTX_ARCHS)
|
| 377 |
+
foreach(_arch ${_SRC_CUDA_ARCHS})
|
| 378 |
+
if(_arch MATCHES "\\+PTX$")
|
| 379 |
+
string(REPLACE "+PTX" "" _base "${_arch}")
|
| 380 |
+
list(APPEND _PTX_ARCHS "${_base}")
|
| 381 |
+
list(REMOVE_ITEM _SRC_CUDA_ARCHS "${_arch}")
|
| 382 |
+
list(APPEND _SRC_CUDA_ARCHS "${_base}")
|
| 383 |
+
endif()
|
| 384 |
+
endforeach()
|
| 385 |
+
list(REMOVE_DUPLICATES _PTX_ARCHS)
|
| 386 |
+
list(REMOVE_DUPLICATES _SRC_CUDA_ARCHS)
|
| 387 |
+
|
| 388 |
+
# If x.0a or x.0f is in SRC_CUDA_ARCHS and x.0 is in CUDA_ARCHS then we should
|
| 389 |
+
# remove x.0a or x.0f from SRC_CUDA_ARCHS and add x.0a or x.0f to _CUDA_ARCHS
|
| 390 |
+
set(_CUDA_ARCHS)
|
| 391 |
+
foreach(_arch ${_SRC_CUDA_ARCHS})
|
| 392 |
+
if(_arch MATCHES "[af]$")
|
| 393 |
+
list(REMOVE_ITEM _SRC_CUDA_ARCHS "${_arch}")
|
| 394 |
+
string(REGEX REPLACE "[af]$" "" _base "${_arch}")
|
| 395 |
+
if ("${_base}" IN_LIST TGT_CUDA_ARCHS)
|
| 396 |
+
list(REMOVE_ITEM _TGT_CUDA_ARCHS "${_base}")
|
| 397 |
+
list(APPEND _CUDA_ARCHS "${_arch}")
|
| 398 |
+
endif()
|
| 399 |
+
endif()
|
| 400 |
+
endforeach()
|
| 401 |
+
|
| 402 |
+
list(SORT _SRC_CUDA_ARCHS COMPARE NATURAL ORDER ASCENDING)
|
| 403 |
+
|
| 404 |
+
# for each ARCH in TGT_CUDA_ARCHS find the highest arch in SRC_CUDA_ARCHS that
|
| 405 |
+
# is less or equal to ARCH (but has the same major version since SASS binary
|
| 406 |
+
# compatibility is only forward compatible within the same major version).
|
| 407 |
+
foreach(_ARCH ${_TGT_CUDA_ARCHS})
|
| 408 |
+
set(_TMP_ARCH)
|
| 409 |
+
# Extract the major version of the target arch
|
| 410 |
+
string(REGEX REPLACE "^([0-9]+)\\..*$" "\\1" TGT_ARCH_MAJOR "${_ARCH}")
|
| 411 |
+
foreach(_SRC_ARCH ${_SRC_CUDA_ARCHS})
|
| 412 |
+
# Extract the major version of the source arch
|
| 413 |
+
string(REGEX REPLACE "^([0-9]+)\\..*$" "\\1" SRC_ARCH_MAJOR "${_SRC_ARCH}")
|
| 414 |
+
# Check version-less-or-equal, and allow PTX arches to match across majors
|
| 415 |
+
if (_SRC_ARCH VERSION_LESS_EQUAL _ARCH)
|
| 416 |
+
if (_SRC_ARCH IN_LIST _PTX_ARCHS OR SRC_ARCH_MAJOR STREQUAL TGT_ARCH_MAJOR)
|
| 417 |
+
set(_TMP_ARCH "${_SRC_ARCH}")
|
| 418 |
+
endif()
|
| 419 |
+
else()
|
| 420 |
+
# If we hit a version greater than the target, we can break
|
| 421 |
+
break()
|
| 422 |
+
endif()
|
| 423 |
+
endforeach()
|
| 424 |
+
|
| 425 |
+
# If we found a matching _TMP_ARCH, append it to _CUDA_ARCHS
|
| 426 |
+
if (_TMP_ARCH)
|
| 427 |
+
list(APPEND _CUDA_ARCHS "${_TMP_ARCH}")
|
| 428 |
+
endif()
|
| 429 |
+
endforeach()
|
| 430 |
+
|
| 431 |
+
list(REMOVE_DUPLICATES _CUDA_ARCHS)
|
| 432 |
+
|
| 433 |
+
# reapply +PTX suffix to architectures that requested PTX
|
| 434 |
+
set(_FINAL_ARCHS)
|
| 435 |
+
foreach(_arch ${_CUDA_ARCHS})
|
| 436 |
+
if(_arch IN_LIST _PTX_ARCHS)
|
| 437 |
+
list(APPEND _FINAL_ARCHS "${_arch}+PTX")
|
| 438 |
+
else()
|
| 439 |
+
list(APPEND _FINAL_ARCHS "${_arch}")
|
| 440 |
+
endif()
|
| 441 |
+
endforeach()
|
| 442 |
+
set(_CUDA_ARCHS ${_FINAL_ARCHS})
|
| 443 |
+
|
| 444 |
+
list(SORT _CUDA_ARCHS COMPARE NATURAL ORDER ASCENDING)
|
| 445 |
+
|
| 446 |
+
set(${OUT_CUDA_ARCHS} ${_CUDA_ARCHS} PARENT_SCOPE)
|
| 447 |
+
endfunction()
|
| 448 |
+
|
| 449 |
+
#
|
| 450 |
+
# For the given `SRC_ROCM_ARCHS` list of architecture versions in the form
|
| 451 |
+
# `<name>` compute the "loose intersection" with the `TGT_ROCM_ARCHS` list.
|
| 452 |
+
# The loose intersection is defined as:
|
| 453 |
+
# { max{ x \in tgt | x <= y } | y \in src, { x \in tgt | x <= y } != {} }
|
| 454 |
+
# where `<=` is the version comparison operator.
|
| 455 |
+
# In other words, for each version in `TGT_ROCM_ARCHS` find the highest version
|
| 456 |
+
# in `SRC_ROCM_ARCHS` that is less or equal to the version in `TGT_ROCM_ARCHS`.
|
| 457 |
+
# The result is stored in `OUT_ROCM_ARCHS`.
|
| 458 |
+
#
|
| 459 |
+
# Example:
|
| 460 |
+
# SRC_ROCM_ARCHS="gfx900;gfx906;gfx908;gfx90a"
|
| 461 |
+
# TGT_ROCM_ARCHS="gfx906;gfx908;gfx1030"
|
| 462 |
+
# hip_archs_loose_intersection(OUT_ROCM_ARCHS SRC_ROCM_ARCHS TGT_ROCM_ARCHS)
|
| 463 |
+
# OUT_ROCM_ARCHS="gfx906;gfx908"
|
| 464 |
+
#
|
| 465 |
+
function(hip_archs_loose_intersection OUT_ROCM_ARCHS SRC_ROCM_ARCHS TGT_ROCM_ARCHS)
|
| 466 |
+
list(REMOVE_DUPLICATES SRC_ROCM_ARCHS)
|
| 467 |
+
|
| 468 |
+
# ROCm architectures are typically in format gfxNNN or gfxNNNx where N is a digit
|
| 469 |
+
# and x is a letter. We can sort them by string comparison which works for this format.
|
| 470 |
+
list(SORT SRC_ROCM_ARCHS COMPARE STRING ORDER ASCENDING)
|
| 471 |
+
|
| 472 |
+
set(_ROCM_ARCHS)
|
| 473 |
+
|
| 474 |
+
# Find the intersection of supported architectures
|
| 475 |
+
foreach(_SRC_ARCH ${SRC_ROCM_ARCHS})
|
| 476 |
+
if(_SRC_ARCH IN_LIST TGT_ROCM_ARCHS)
|
| 477 |
+
list(APPEND _ROCM_ARCHS ${_SRC_ARCH})
|
| 478 |
+
endif()
|
| 479 |
+
endforeach()
|
| 480 |
+
|
| 481 |
+
list(REMOVE_DUPLICATES _ROCM_ARCHS)
|
| 482 |
+
set(${OUT_ROCM_ARCHS} ${_ROCM_ARCHS} PARENT_SCOPE)
|
| 483 |
+
endfunction()
|
| 484 |
+
|
| 485 |
+
function(cuda_remove_ptx_suffixes OUT_CUDA_ARCHS CUDA_ARCHS)
|
| 486 |
+
set(_CUDA_ARCHS "${CUDA_ARCHS}")
|
| 487 |
+
|
| 488 |
+
# handle +PTX suffix: separate base arch for matching, record PTX requests
|
| 489 |
+
foreach(_arch ${CUDA_ARCHS})
|
| 490 |
+
if(_arch MATCHES "\\+PTX$")
|
| 491 |
+
string(REPLACE "+PTX" "" _base "${_arch}")
|
| 492 |
+
list(REMOVE_ITEM _CUDA_ARCHS "${_arch}")
|
| 493 |
+
list(APPEND _CUDA_ARCHS "${_base}")
|
| 494 |
+
endif()
|
| 495 |
+
endforeach()
|
| 496 |
+
|
| 497 |
+
list(REMOVE_DUPLICATES _CUDA_ARCHS)
|
| 498 |
+
list(SORT _CUDA_ARCHS COMPARE NATURAL ORDER ASCENDING)
|
| 499 |
+
|
| 500 |
+
set(${OUT_CUDA_ARCHS} ${_CUDA_ARCHS} PARENT_SCOPE)
|
| 501 |
+
endfunction()
|
| 502 |
+
|
| 503 |
+
#
|
| 504 |
+
# Define a target named `GPU_MOD_NAME` for a single extension. The
|
| 505 |
+
# arguments are:
|
| 506 |
+
#
|
| 507 |
+
# DESTINATION <dest> - Module destination directory.
|
| 508 |
+
# LANGUAGE <lang> - The GPU language for this module, e.g CUDA, HIP,
|
| 509 |
+
# etc.
|
| 510 |
+
# SOURCES <sources> - List of source files relative to CMakeLists.txt
|
| 511 |
+
# directory.
|
| 512 |
+
#
|
| 513 |
+
# Optional arguments:
|
| 514 |
+
#
|
| 515 |
+
# ARCHITECTURES <arches> - A list of target GPU architectures in cmake
|
| 516 |
+
# format.
|
| 517 |
+
# Refer `CMAKE_CUDA_ARCHITECTURES` documentation
|
| 518 |
+
# and `CMAKE_HIP_ARCHITECTURES` for more info.
|
| 519 |
+
# ARCHITECTURES will use cmake's defaults if
|
| 520 |
+
# not provided.
|
| 521 |
+
# COMPILE_FLAGS <flags> - Extra compiler flags passed to NVCC/hip.
|
| 522 |
+
# INCLUDE_DIRECTORIES <dirs> - Extra include directories.
|
| 523 |
+
# LIBRARIES <libraries> - Extra link libraries.
|
| 524 |
+
# WITH_SOABI - Generate library with python SOABI suffix name.
|
| 525 |
+
# USE_SABI <version> - Use python stable api <version>
|
| 526 |
+
#
|
| 527 |
+
# Note: optimization level/debug info is set via cmake build type.
|
| 528 |
+
#
|
| 529 |
+
function (define_gpu_extension_target GPU_MOD_NAME)
|
| 530 |
+
cmake_parse_arguments(PARSE_ARGV 1
|
| 531 |
+
GPU
|
| 532 |
+
"WITH_SOABI"
|
| 533 |
+
"DESTINATION;LANGUAGE;USE_SABI"
|
| 534 |
+
"SOURCES;COMPILE_FLAGS;INCLUDE_DIRECTORIES;LIBRARIES")
|
| 535 |
+
|
| 536 |
+
# Add hipify preprocessing step when building with HIP/ROCm.
|
| 537 |
+
if (GPU_LANGUAGE STREQUAL "HIP")
|
| 538 |
+
hipify_sources_target(GPU_SOURCES ${GPU_MOD_NAME} "${GPU_SOURCES}")
|
| 539 |
+
endif()
|
| 540 |
+
|
| 541 |
+
if (GPU_WITH_SOABI)
|
| 542 |
+
set(GPU_WITH_SOABI WITH_SOABI)
|
| 543 |
+
else()
|
| 544 |
+
set(GPU_WITH_SOABI)
|
| 545 |
+
endif()
|
| 546 |
+
|
| 547 |
+
if (GPU_USE_SABI)
|
| 548 |
+
Python3_add_library(${GPU_MOD_NAME} MODULE USE_SABI ${GPU_USE_SABI} ${GPU_WITH_SOABI} "${GPU_SOURCES}")
|
| 549 |
+
else()
|
| 550 |
+
Python3_add_library(${GPU_MOD_NAME} MODULE ${GPU_WITH_SOABI} "${GPU_SOURCES}")
|
| 551 |
+
endif()
|
| 552 |
+
|
| 553 |
+
if (GPU_LANGUAGE STREQUAL "HIP")
|
| 554 |
+
# Make this target dependent on the hipify preprocessor step.
|
| 555 |
+
add_dependencies(${GPU_MOD_NAME} hipify${GPU_MOD_NAME})
|
| 556 |
+
|
| 557 |
+
# Clear target architectures, we are passing arch flags per source file.
|
| 558 |
+
set_property(TARGET ${GPU_MOD_NAME} PROPERTY HIP_ARCHITECTURES off)
|
| 559 |
+
endif()
|
| 560 |
+
|
| 561 |
+
if (TORCH_VERSION VERSION_LESS 2.12.0)
|
| 562 |
+
set_property(TARGET ${GPU_MOD_NAME} PROPERTY CXX_STANDARD 17)
|
| 563 |
+
else()
|
| 564 |
+
set_property(TARGET ${GPU_MOD_NAME} PROPERTY CXX_STANDARD 20)
|
| 565 |
+
endif()
|
| 566 |
+
|
| 567 |
+
target_compile_options(${GPU_MOD_NAME} PRIVATE
|
| 568 |
+
$<$<COMPILE_LANGUAGE:${GPU_LANGUAGE}>:${GPU_COMPILE_FLAGS}>)
|
| 569 |
+
|
| 570 |
+
target_compile_definitions(${GPU_MOD_NAME} PRIVATE
|
| 571 |
+
"-DTORCH_EXTENSION_NAME=${GPU_MOD_NAME}")
|
| 572 |
+
|
| 573 |
+
target_include_directories(${GPU_MOD_NAME} PRIVATE csrc
|
| 574 |
+
${GPU_INCLUDE_DIRECTORIES})
|
| 575 |
+
|
| 576 |
+
target_link_libraries(${GPU_MOD_NAME} PRIVATE torch ${GPU_LIBRARIES})
|
| 577 |
+
|
| 578 |
+
# Don't use `TORCH_LIBRARIES` for CUDA since it pulls in a bunch of
|
| 579 |
+
# dependencies that are not necessary and may not be installed.
|
| 580 |
+
if (GPU_LANGUAGE STREQUAL "CUDA")
|
| 581 |
+
target_link_libraries(${GPU_MOD_NAME} PRIVATE CUDA::cudart)
|
| 582 |
+
else()
|
| 583 |
+
target_link_libraries(${GPU_MOD_NAME} PRIVATE ${TORCH_LIBRARIES})
|
| 584 |
+
endif()
|
| 585 |
+
|
| 586 |
+
install(TARGETS ${GPU_MOD_NAME} LIBRARY DESTINATION ${GPU_DESTINATION} COMPONENT ${GPU_MOD_NAME})
|
| 587 |
+
endfunction()
|
| 588 |
+
|
| 589 |
+
# Map a GPU language to its backend name.
|
| 590 |
+
#
|
| 591 |
+
# Arguments:
|
| 592 |
+
# OUT_BACKEND - Output variable name for the backend string
|
| 593 |
+
# GPU_LANG - The GPU language (CPU, CUDA, HIP, METAL, SYCL)
|
| 594 |
+
#
|
| 595 |
+
function(gpu_lang_to_backend OUT_BACKEND GPU_LANG)
|
| 596 |
+
if (${GPU_LANG} STREQUAL "CPU")
|
| 597 |
+
set(_BACKEND "cpu")
|
| 598 |
+
elseif (${GPU_LANG} STREQUAL "CUDA")
|
| 599 |
+
set(_BACKEND "cuda")
|
| 600 |
+
elseif (${GPU_LANG} STREQUAL "HIP")
|
| 601 |
+
set(_BACKEND "rocm")
|
| 602 |
+
elseif (${GPU_LANG} STREQUAL "METAL")
|
| 603 |
+
set(_BACKEND "metal")
|
| 604 |
+
elseif (${GPU_LANG} STREQUAL "SYCL")
|
| 605 |
+
set(_BACKEND "xpu")
|
| 606 |
+
else()
|
| 607 |
+
message(FATAL_ERROR "Unsupported GPU_LANG: ${GPU_LANG}")
|
| 608 |
+
endif()
|
| 609 |
+
|
| 610 |
+
set(${OUT_BACKEND} "${_BACKEND}" PARENT_SCOPE)
|
| 611 |
+
endfunction()
|
src/kernel-source/rms-quant/compat.py
ADDED
|
@@ -0,0 +1,26 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import ctypes
|
| 2 |
+
import importlib.util
|
| 3 |
+
import sys
|
| 4 |
+
from pathlib import Path
|
| 5 |
+
from types import ModuleType
|
| 6 |
+
|
| 7 |
+
|
| 8 |
+
def _import_from_path(file_path: Path) -> ModuleType:
|
| 9 |
+
# We cannot use the module name as-is, after adding it to `sys.modules`,
|
| 10 |
+
# it would also be used for other imports. So, we make a module name that
|
| 11 |
+
# depends on the path for it to be unique using the hex-encoded hash of
|
| 12 |
+
# the path.
|
| 13 |
+
path_hash = "{:x}".format(ctypes.c_size_t(hash(file_path.absolute())).value)
|
| 14 |
+
module_name = path_hash
|
| 15 |
+
spec = importlib.util.spec_from_file_location(module_name, file_path)
|
| 16 |
+
if spec is None:
|
| 17 |
+
raise ImportError(f"Cannot load spec for {module_name} from {file_path}")
|
| 18 |
+
module = importlib.util.module_from_spec(spec)
|
| 19 |
+
if module is None:
|
| 20 |
+
raise ImportError(f"Cannot load module {module_name} from spec")
|
| 21 |
+
sys.modules[module_name] = module
|
| 22 |
+
spec.loader.exec_module(module) # type: ignore
|
| 23 |
+
return module
|
| 24 |
+
|
| 25 |
+
|
| 26 |
+
globals().update(vars(_import_from_path(Path(__file__).parent.parent / "__init__.py")))
|
src/kernel-source/rms-quant/flake.nix
ADDED
|
@@ -0,0 +1,11 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
inputs = {
|
| 3 |
+
kernel-builder.url = "github:huggingface/kernels";
|
| 4 |
+
};
|
| 5 |
+
outputs =
|
| 6 |
+
{ self, kernel-builder, ... }:
|
| 7 |
+
kernel-builder.lib.genKernelFlakeOutputs {
|
| 8 |
+
inherit self;
|
| 9 |
+
path = ./.;
|
| 10 |
+
};
|
| 11 |
+
}
|
src/kernel-source/rms-quant/metadata-cann.json
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"name": "yue2-qkv-fused",
|
| 3 |
+
"id": "_yue2_qkv_fused_cann_cf0db58",
|
| 4 |
+
"version": 1,
|
| 5 |
+
"license": "Apache-2.0",
|
| 6 |
+
"python-depends": [],
|
| 7 |
+
"backend": {
|
| 8 |
+
"type": "cann"
|
| 9 |
+
},
|
| 10 |
+
"provenance": {
|
| 11 |
+
"kernel-builder": {
|
| 12 |
+
"version": "0.17.0-dev0",
|
| 13 |
+
"sha": "05f9a9517021bff08000f178f8c4d789b04b1320",
|
| 14 |
+
"dirty": false
|
| 15 |
+
},
|
| 16 |
+
"kernel": {
|
| 17 |
+
"sha": "92dbd686220262d8d24b09c27a6d6703e19f9465",
|
| 18 |
+
"dirty": false
|
| 19 |
+
}
|
| 20 |
+
}
|
| 21 |
+
}
|
src/kernel-source/rms-quant/metadata-cpu.json
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"name": "yue2-qkv-fused",
|
| 3 |
+
"id": "_yue2_qkv_fused_cpu_cf0db58",
|
| 4 |
+
"version": 1,
|
| 5 |
+
"license": "Apache-2.0",
|
| 6 |
+
"python-depends": [],
|
| 7 |
+
"backend": {
|
| 8 |
+
"type": "cpu"
|
| 9 |
+
},
|
| 10 |
+
"provenance": {
|
| 11 |
+
"kernel-builder": {
|
| 12 |
+
"version": "0.17.0-dev0",
|
| 13 |
+
"sha": "05f9a9517021bff08000f178f8c4d789b04b1320",
|
| 14 |
+
"dirty": false
|
| 15 |
+
},
|
| 16 |
+
"kernel": {
|
| 17 |
+
"sha": "92dbd686220262d8d24b09c27a6d6703e19f9465",
|
| 18 |
+
"dirty": false
|
| 19 |
+
}
|
| 20 |
+
}
|
| 21 |
+
}
|
src/kernel-source/rms-quant/metadata-cuda.json
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"name": "yue2-qkv-fused",
|
| 3 |
+
"id": "_yue2_qkv_fused_cuda_cf0db58",
|
| 4 |
+
"version": 1,
|
| 5 |
+
"license": "Apache-2.0",
|
| 6 |
+
"python-depends": [],
|
| 7 |
+
"backend": {
|
| 8 |
+
"type": "cuda"
|
| 9 |
+
},
|
| 10 |
+
"provenance": {
|
| 11 |
+
"kernel-builder": {
|
| 12 |
+
"version": "0.17.0-dev0",
|
| 13 |
+
"sha": "05f9a9517021bff08000f178f8c4d789b04b1320",
|
| 14 |
+
"dirty": false
|
| 15 |
+
},
|
| 16 |
+
"kernel": {
|
| 17 |
+
"sha": "92dbd686220262d8d24b09c27a6d6703e19f9465",
|
| 18 |
+
"dirty": false
|
| 19 |
+
}
|
| 20 |
+
}
|
| 21 |
+
}
|
src/kernel-source/rms-quant/metadata-metal.json
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"name": "yue2-qkv-fused",
|
| 3 |
+
"id": "_yue2_qkv_fused_metal_cf0db58",
|
| 4 |
+
"version": 1,
|
| 5 |
+
"license": "Apache-2.0",
|
| 6 |
+
"python-depends": [],
|
| 7 |
+
"backend": {
|
| 8 |
+
"type": "metal"
|
| 9 |
+
},
|
| 10 |
+
"provenance": {
|
| 11 |
+
"kernel-builder": {
|
| 12 |
+
"version": "0.17.0-dev0",
|
| 13 |
+
"sha": "05f9a9517021bff08000f178f8c4d789b04b1320",
|
| 14 |
+
"dirty": false
|
| 15 |
+
},
|
| 16 |
+
"kernel": {
|
| 17 |
+
"sha": "92dbd686220262d8d24b09c27a6d6703e19f9465",
|
| 18 |
+
"dirty": false
|
| 19 |
+
}
|
| 20 |
+
}
|
| 21 |
+
}
|
src/kernel-source/rms-quant/metadata-neuron.json
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"name": "yue2-qkv-fused",
|
| 3 |
+
"id": "_yue2_qkv_fused_neuron_cf0db58",
|
| 4 |
+
"version": 1,
|
| 5 |
+
"license": "Apache-2.0",
|
| 6 |
+
"python-depends": [],
|
| 7 |
+
"backend": {
|
| 8 |
+
"type": "neuron"
|
| 9 |
+
},
|
| 10 |
+
"provenance": {
|
| 11 |
+
"kernel-builder": {
|
| 12 |
+
"version": "0.17.0-dev0",
|
| 13 |
+
"sha": "05f9a9517021bff08000f178f8c4d789b04b1320",
|
| 14 |
+
"dirty": false
|
| 15 |
+
},
|
| 16 |
+
"kernel": {
|
| 17 |
+
"sha": "92dbd686220262d8d24b09c27a6d6703e19f9465",
|
| 18 |
+
"dirty": false
|
| 19 |
+
}
|
| 20 |
+
}
|
| 21 |
+
}
|
src/kernel-source/rms-quant/metadata-rocm.json
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"name": "yue2-qkv-fused",
|
| 3 |
+
"id": "_yue2_qkv_fused_rocm_cf0db58",
|
| 4 |
+
"version": 1,
|
| 5 |
+
"license": "Apache-2.0",
|
| 6 |
+
"python-depends": [],
|
| 7 |
+
"backend": {
|
| 8 |
+
"type": "rocm"
|
| 9 |
+
},
|
| 10 |
+
"provenance": {
|
| 11 |
+
"kernel-builder": {
|
| 12 |
+
"version": "0.17.0-dev0",
|
| 13 |
+
"sha": "05f9a9517021bff08000f178f8c4d789b04b1320",
|
| 14 |
+
"dirty": false
|
| 15 |
+
},
|
| 16 |
+
"kernel": {
|
| 17 |
+
"sha": "92dbd686220262d8d24b09c27a6d6703e19f9465",
|
| 18 |
+
"dirty": false
|
| 19 |
+
}
|
| 20 |
+
}
|
| 21 |
+
}
|
src/kernel-source/rms-quant/metadata-xpu.json
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"name": "yue2-qkv-fused",
|
| 3 |
+
"id": "_yue2_qkv_fused_xpu_cf0db58",
|
| 4 |
+
"version": 1,
|
| 5 |
+
"license": "Apache-2.0",
|
| 6 |
+
"python-depends": [],
|
| 7 |
+
"backend": {
|
| 8 |
+
"type": "xpu"
|
| 9 |
+
},
|
| 10 |
+
"provenance": {
|
| 11 |
+
"kernel-builder": {
|
| 12 |
+
"version": "0.17.0-dev0",
|
| 13 |
+
"sha": "05f9a9517021bff08000f178f8c4d789b04b1320",
|
| 14 |
+
"dirty": false
|
| 15 |
+
},
|
| 16 |
+
"kernel": {
|
| 17 |
+
"sha": "92dbd686220262d8d24b09c27a6d6703e19f9465",
|
| 18 |
+
"dirty": false
|
| 19 |
+
}
|
| 20 |
+
}
|
| 21 |
+
}
|
src/kernel-source/rms-quant/orbitquant_gemv_cuda/fused_ops.cu
ADDED
|
@@ -0,0 +1,67 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#include <ATen/cuda/CUDAContext.h>
|
| 2 |
+
#include <c10/cuda/CUDAGuard.h>
|
| 3 |
+
#include <c10/cuda/CUDAException.h>
|
| 4 |
+
#include <torch/torch.h>
|
| 5 |
+
|
| 6 |
+
using BF = c10::BFloat16;
|
| 7 |
+
__global__ void rms(BF* out,const BF* x,const BF* w,int dim,float eps) {
|
| 8 |
+
int row=blockIdx.x,t=threadIdx.x;
|
| 9 |
+
float sum=0.f;
|
| 10 |
+
for(int j=t;j<dim;j+=blockDim.x){float v=float(x[row*dim+j]);sum=__fadd_rn(sum,__fmul_rn(v,v));}
|
| 11 |
+
for(int d=16;d;d>>=1)sum=__fadd_rn(sum,__shfl_down_sync(0xffffffff,sum,d));
|
| 12 |
+
__shared__ float warps[8];
|
| 13 |
+
if((t&31)==0)warps[t/32]=sum;
|
| 14 |
+
__syncthreads();
|
| 15 |
+
if(t<32){sum=t<8?warps[t]:0.f;for(int d=16;d;d>>=1)sum=__fadd_rn(sum,__shfl_down_sync(0xffffffff,sum,d));if(t==0)warps[0]=rsqrtf(__fadd_rn(sum/float(dim),eps));}
|
| 16 |
+
__syncthreads();
|
| 17 |
+
BF scale=BF(warps[0]);
|
| 18 |
+
for(int j=t;j<dim;j+=blockDim.x){BF first=BF(__fmul_rn(float(x[row*dim+j]),float(scale)));out[row*dim+j]=BF(__fmul_rn(float(first),float(w[j])));}
|
| 19 |
+
}
|
| 20 |
+
__global__ void rope_kernel(BF* out,const BF* x,const float* c,const float* s,int rows,int dim,int heads){
|
| 21 |
+
int id=blockIdx.x*blockDim.x+threadIdx.x;int half=dim/2;if(id>=rows*half)return;
|
| 22 |
+
int row=id/half,d=id%half,time=row/heads;BF cv=BF(c[time*half+d]),sv=BF(s[time*half+d]);BF a=x[row*dim+d],b=x[row*dim+half+d];
|
| 23 |
+
BF ac=BF(__fmul_rn(float(a),float(cv))),bs=BF(__fmul_rn(float(b),float(sv))),bc=BF(__fmul_rn(float(b),float(cv))),as=BF(__fmul_rn(float(a),float(sv)));
|
| 24 |
+
out[row*dim+d]=BF(__fsub_rn(float(ac),float(bs)));out[row*dim+half+d]=BF(__fadd_rn(float(bc),float(as)));
|
| 25 |
+
}
|
| 26 |
+
__global__ void swiglu_kernel(BF* out,const BF* x,int rows,int dim){
|
| 27 |
+
int id=blockIdx.x*blockDim.x+threadIdx.x;if(id>=rows*dim)return;int row=id/dim,d=id%dim;float g=float(x[row*dim*2+d]);
|
| 28 |
+
BF gate=BF(g/(1.f+expf(-g)));out[id]=BF(__fmul_rn(float(gate),float(x[row*dim*2+dim+d])));
|
| 29 |
+
}
|
| 30 |
+
void common(torch::Tensor const& out,torch::Tensor const& x){
|
| 31 |
+
TORCH_CHECK(x.is_cuda() && x.scalar_type()==torch::kBFloat16 && x.is_contiguous(),"contiguous CUDA BF16 input required");
|
| 32 |
+
TORCH_CHECK(out.device()==x.device() && out.scalar_type()==x.scalar_type() && out.is_contiguous(),"matching contiguous output required");
|
| 33 |
+
}
|
| 34 |
+
void yue2_rmsnorm(torch::Tensor &out,torch::Tensor const &x,torch::Tensor const &weight,double eps){
|
| 35 |
+
common(out,x);TORCH_CHECK(x.dim()>=1 && x.size(-1)>0 && x.size(-1)<=8192 && x.numel()==out.numel(),"invalid RMS shape");
|
| 36 |
+
TORCH_CHECK(weight.device()==x.device() && weight.is_contiguous() && weight.scalar_type()==x.scalar_type() && weight.numel()==x.size(-1),"invalid RMS weight");
|
| 37 |
+
const at::cuda::OptionalCUDAGuard guard(device_of(x));int dim=x.size(-1),rows=x.numel()/dim;if(!rows)return;
|
| 38 |
+
rms<<<rows,256,0,at::cuda::getCurrentCUDAStream()>>>(out.data_ptr<BF>(),x.data_ptr<BF>(),weight.data_ptr<BF>(),dim,eps);C10_CUDA_KERNEL_LAUNCH_CHECK();
|
| 39 |
+
}
|
| 40 |
+
void yue2_rope(torch::Tensor &out,torch::Tensor const &x,torch::Tensor const &cos,torch::Tensor const &sin,int64_t heads){
|
| 41 |
+
common(out,x);TORCH_CHECK(x.dim()>=2 && x.size(-1)%2==0 && x.size(-1)>0 && heads>0 && out.numel()==x.numel(),"invalid RoPE shape");
|
| 42 |
+
for(const auto* t:{&cos,&sin})TORCH_CHECK(t->device()==x.device() && t->scalar_type()==torch::kFloat && t->is_contiguous() && t->numel()==x.numel()/heads/2,"invalid RoPE table");
|
| 43 |
+
const at::cuda::OptionalCUDAGuard guard(device_of(x));int dim=x.size(-1),rows=x.numel()/dim;if(!rows)return;
|
| 44 |
+
rope_kernel<<<(rows*dim/2+255)/256,256,0,at::cuda::getCurrentCUDAStream()>>>(out.data_ptr<BF>(),x.data_ptr<BF>(),cos.data_ptr<float>(),sin.data_ptr<float>(),rows,dim,heads);C10_CUDA_KERNEL_LAUNCH_CHECK();
|
| 45 |
+
}
|
| 46 |
+
void yue2_swiglu(torch::Tensor &out,torch::Tensor const &x){
|
| 47 |
+
common(out,x);TORCH_CHECK(x.dim()>=1 && x.size(-1)>0 && x.size(-1)%2==0 && out.numel()*2==x.numel(),"invalid SwiGLU shape");
|
| 48 |
+
const at::cuda::OptionalCUDAGuard guard(device_of(x));int dim=x.size(-1)/2,rows=x.numel()/dim/2;if(!rows)return;
|
| 49 |
+
swiglu_kernel<<<(out.numel()+255)/256,256,0,at::cuda::getCurrentCUDAStream()>>>(out.data_ptr<BF>(),x.data_ptr<BF>(),rows,dim);C10_CUDA_KERNEL_LAUNCH_CHECK();
|
| 50 |
+
}
|
| 51 |
+
|
| 52 |
+
__global__ void square_bf16(float* out,const BF* x,int64_t count){
|
| 53 |
+
int64_t i=int64_t(blockIdx.x)*blockDim.x+threadIdx.x;if(i<count){float v=float(x[i]);out[i]=__fmul_rn(v,v);}
|
| 54 |
+
}
|
| 55 |
+
__global__ void rms_apply(BF* out,const BF* x,const BF* w,const float* means,int dim,int64_t count,float eps){
|
| 56 |
+
int64_t i=int64_t(blockIdx.x)*blockDim.x+threadIdx.x;if(i>=count)return;
|
| 57 |
+
BF scale=BF(rsqrtf(__fadd_rn(means[i/dim],eps)));BF first=BF(__fmul_rn(float(x[i]),float(scale)));out[i]=BF(__fmul_rn(float(first),float(w[i%dim])));
|
| 58 |
+
}
|
| 59 |
+
void yue2_rmsnorm_exact(torch::Tensor &out,torch::Tensor const &x,torch::Tensor const &weight,double eps){
|
| 60 |
+
common(out,x);TORCH_CHECK(x.dim()>=1 && x.size(-1)>0 && x.numel()==out.numel(),"invalid RMS shape");
|
| 61 |
+
TORCH_CHECK(weight.device()==x.device() && weight.is_contiguous() && weight.scalar_type()==x.scalar_type() && weight.numel()==x.size(-1),"invalid RMS weight");
|
| 62 |
+
const at::cuda::OptionalCUDAGuard guard(device_of(x));if(!x.numel())return;
|
| 63 |
+
auto squared=torch::empty(x.sizes(),x.options().dtype(torch::kFloat));
|
| 64 |
+
square_bf16<<<(x.numel()+255)/256,256,0,at::cuda::getCurrentCUDAStream()>>>(squared.data_ptr<float>(),x.data_ptr<BF>(),x.numel());C10_CUDA_KERNEL_LAUNCH_CHECK();
|
| 65 |
+
auto means=squared.mean(-1,true);
|
| 66 |
+
rms_apply<<<(x.numel()+255)/256,256,0,at::cuda::getCurrentCUDAStream()>>>(out.data_ptr<BF>(),x.data_ptr<BF>(),weight.data_ptr<BF>(),means.data_ptr<float>(),x.size(-1),x.numel(),eps);C10_CUDA_KERNEL_LAUNCH_CHECK();
|
| 67 |
+
}
|
src/kernel-source/rms-quant/orbitquant_gemv_cuda/orbitquant_gemv.cu
ADDED
|
@@ -0,0 +1,74 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#include <ATen/cuda/CUDAContext.h>
|
| 2 |
+
#include <c10/cuda/CUDAGuard.h>
|
| 3 |
+
#include <c10/cuda/CUDAException.h>
|
| 4 |
+
#include <torch/torch.h>
|
| 5 |
+
|
| 6 |
+
template<class T>
|
| 7 |
+
__global__ void packed_gemv(T* out, const uint8_t* x, const uint8_t* w,
|
| 8 |
+
const float* xn, const c10::BFloat16* wn, const int8_t* ac, const int8_t* wc,
|
| 9 |
+
const T* bias, int n, int packed_k, float as, float ws) {
|
| 10 |
+
const int lane = threadIdx.x & 31;
|
| 11 |
+
const int col = blockIdx.x * 4 + threadIdx.x / 32;
|
| 12 |
+
const int row = blockIdx.y;
|
| 13 |
+
__shared__ int a_codes[16], w_codes[16];
|
| 14 |
+
if (threadIdx.x < 16) { a_codes[threadIdx.x] = ac[threadIdx.x]; w_codes[threadIdx.x] = wc[threadIdx.x]; }
|
| 15 |
+
__syncthreads();
|
| 16 |
+
if (col >= n) return;
|
| 17 |
+
int sum = 0;
|
| 18 |
+
if (packed_k % 4 == 0 && (reinterpret_cast<uintptr_t>(x) & 3) == 0 && (reinterpret_cast<uintptr_t>(w) & 3) == 0) {
|
| 19 |
+
const uint32_t* xp = reinterpret_cast<const uint32_t*>(x + row * packed_k);
|
| 20 |
+
const uint32_t* wp = reinterpret_cast<const uint32_t*>(w + col * packed_k);
|
| 21 |
+
for (int k = lane; k < packed_k / 4; k += 32) {
|
| 22 |
+
const uint32_t av = xp[k], bv = wp[k];
|
| 23 |
+
uint32_t al = 0, ah = 0, bl = 0, bh = 0;
|
| 24 |
+
#pragma unroll
|
| 25 |
+
for (int j = 0; j < 4; ++j) {
|
| 26 |
+
al |= uint32_t(uint8_t(a_codes[(av >> (j * 8)) & 15])) << (j * 8);
|
| 27 |
+
ah |= uint32_t(uint8_t(a_codes[(av >> (j * 8 + 4)) & 15])) << (j * 8);
|
| 28 |
+
bl |= uint32_t(uint8_t(w_codes[(bv >> (j * 8)) & 15])) << (j * 8);
|
| 29 |
+
bh |= uint32_t(uint8_t(w_codes[(bv >> (j * 8 + 4)) & 15])) << (j * 8);
|
| 30 |
+
}
|
| 31 |
+
sum = __dp4a(static_cast<int>(al), static_cast<int>(bl), sum);
|
| 32 |
+
sum = __dp4a(static_cast<int>(ah), static_cast<int>(bh), sum);
|
| 33 |
+
}
|
| 34 |
+
} else {
|
| 35 |
+
for (int k = lane; k < packed_k; k += 32) {
|
| 36 |
+
const uint8_t a = x[row * packed_k + k];
|
| 37 |
+
const uint8_t b = w[col * packed_k + k];
|
| 38 |
+
sum += a_codes[a & 15] * w_codes[b & 15];
|
| 39 |
+
sum += a_codes[a >> 4] * w_codes[b >> 4];
|
| 40 |
+
}
|
| 41 |
+
}
|
| 42 |
+
#pragma unroll
|
| 43 |
+
for (int offset = 16; offset; offset >>= 1) sum += __shfl_down_sync(0xffffffff, sum, offset);
|
| 44 |
+
if (lane == 0) {
|
| 45 |
+
float value = static_cast<float>(sum);
|
| 46 |
+
value *= xn[row] * static_cast<float>(wn[col]) * (as * ws);
|
| 47 |
+
if (bias) value += static_cast<float>(bias[col]);
|
| 48 |
+
out[row * n + col] = static_cast<T>(value);
|
| 49 |
+
}
|
| 50 |
+
}
|
| 51 |
+
|
| 52 |
+
void orbitquant_gemv(torch::Tensor &out, torch::Tensor const &x, torch::Tensor const &w, torch::Tensor const &xn, torch::Tensor const &wn, torch::Tensor const &ac, torch::Tensor const &wc, torch::Tensor const &bias, bool has_bias, double activation_scale, double weight_scale) {
|
| 53 |
+
TORCH_CHECK(x.is_cuda() && x.scalar_type() == torch::kUInt8 && x.dim() == 2, "x must be rank-2 packed CUDA uint8");
|
| 54 |
+
const torch::Tensor* tensors[] = {&out, &w, &xn, &wn, &ac, &wc};
|
| 55 |
+
for (const auto* t : tensors) {
|
| 56 |
+
TORCH_CHECK(t->device() == x.device() && t->is_contiguous(), "tensors must be contiguous on the input CUDA device");
|
| 57 |
+
}
|
| 58 |
+
TORCH_CHECK(x.is_contiguous() && w.scalar_type() == torch::kUInt8, "packed inputs must be contiguous uint8");
|
| 59 |
+
TORCH_CHECK(x.size(0) >= 1 && x.size(0) <= 8 && x.size(1) > 0 && x.size(1) <= 8192, "supported rows 1..8 and packed K 1..8192");
|
| 60 |
+
TORCH_CHECK(wn.scalar_type() == torch::kBFloat16 && xn.scalar_type() == torch::kFloat, "norm dtypes must be BF16 weights and FP32 activations");
|
| 61 |
+
TORCH_CHECK(ac.scalar_type() == torch::kChar && wc.scalar_type() == torch::kChar && ac.numel() == 16 && wc.numel() == 16, "codebooks must contain 16 INT8 entries");
|
| 62 |
+
TORCH_CHECK(out.dim() == 2 && out.size(0) == x.size(0) && out.size(1) == wn.numel() && xn.numel() == x.size(0), "output and norm shapes mismatch");
|
| 63 |
+
TORCH_CHECK(w.numel() == wn.numel() * x.size(1), "packed weight size mismatch");
|
| 64 |
+
TORCH_CHECK(out.scalar_type() == torch::kBFloat16 || out.scalar_type() == torch::kHalf, "output must be BF16 or FP16");
|
| 65 |
+
if (has_bias) TORCH_CHECK(bias.device() == x.device() && bias.is_contiguous() && bias.scalar_type() == out.scalar_type() && bias.numel() == wn.numel(), "invalid bias");
|
| 66 |
+
if (wn.numel() == 0) return;
|
| 67 |
+
const at::cuda::OptionalCUDAGuard guard(device_of(x));
|
| 68 |
+
const dim3 grid((wn.numel() + 3) / 4, x.size(0));
|
| 69 |
+
const auto stream = at::cuda::getCurrentCUDAStream();
|
| 70 |
+
#define LAUNCH(T) packed_gemv<T><<<grid, 128, 0, stream>>>(out.data_ptr<T>(), x.data_ptr<uint8_t>(), w.data_ptr<uint8_t>(), xn.data_ptr<float>(), wn.data_ptr<c10::BFloat16>(), ac.data_ptr<int8_t>(), wc.data_ptr<int8_t>(), has_bias ? bias.data_ptr<T>() : nullptr, wn.numel(), x.size(1), activation_scale, weight_scale)
|
| 71 |
+
if (out.scalar_type() == torch::kBFloat16) { LAUNCH(c10::BFloat16); } else { LAUNCH(c10::Half); }
|
| 72 |
+
#undef LAUNCH
|
| 73 |
+
C10_CUDA_KERNEL_LAUNCH_CHECK();
|
| 74 |
+
}
|
src/kernel-source/rms-quant/orbitquant_gemv_cuda/qkv_decode.cu
ADDED
|
@@ -0,0 +1,31 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#include <ATen/cuda/CUDAContext.h>
|
| 2 |
+
#include <c10/cuda/CUDAGuard.h>
|
| 3 |
+
#include <c10/cuda/CUDAException.h>
|
| 4 |
+
#include <torch/torch.h>
|
| 5 |
+
using BF=c10::BFloat16;
|
| 6 |
+
__global__ void qkv_decode_kernel(BF* q,BF* keys,BF* values,const BF* x,const BF* qw,const BF* kw,const float* cos,const float* sin,const int64_t* positions,int hq,int hk,int capacity,float eps){
|
| 7 |
+
int h=blockIdx.x,b=blockIdx.y,t=threadIdx.x;bool is_q=h<hq;int head=is_q?h:h-hq;int width=(hq+2*hk)*128;
|
| 8 |
+
int64_t pos=positions[b];CUDA_KERNEL_ASSERT(pos>=0 && pos<capacity);
|
| 9 |
+
BF v=x[b*width+h*128+t];float fv=float(v),sum=__fmul_rn(fv,fv);
|
| 10 |
+
for(int d=16;d;d>>=1)sum=__fadd_rn(sum,__shfl_down_sync(0xffffffff,sum,d));
|
| 11 |
+
__shared__ float warp[4];__shared__ BF normalized[128];
|
| 12 |
+
if((t&31)==0)warp[t/32]=sum;__syncthreads();
|
| 13 |
+
if(t<32){sum=t<4?warp[t]:0.f;for(int d=16;d;d>>=1)sum=__fadd_rn(sum,__shfl_down_sync(0xffffffff,sum,d));if(t==0)warp[0]=rsqrtf(__fadd_rn(sum/128.f,eps));}
|
| 14 |
+
__syncthreads();BF scale=BF(warp[0]);BF first=BF(__fmul_rn(fv,float(scale)));normalized[t]=BF(__fmul_rn(float(first),float((is_q?qw:kw)[t])));
|
| 15 |
+
__syncthreads();
|
| 16 |
+
if(t<64){BF c=BF(cos[b*64+t]),s=BF(sin[b*64+t]);float a=float(normalized[t]),z=float(normalized[t+64]);BF ac=BF(__fmul_rn(a,float(c))),zs=BF(__fmul_rn(z,float(s))),zc=BF(__fmul_rn(z,float(c))),as=BF(__fmul_rn(a,float(s)));
|
| 17 |
+
int64_t index=is_q?(int64_t(b)*hq+head)*128:(int64_t(b)*capacity+pos)*hk*128+head*128;BF* target=is_q?q:keys;target[index+t]=BF(__fsub_rn(float(ac),float(zs)));target[index+t+64]=BF(__fadd_rn(float(zc),float(as)));
|
| 18 |
+
}
|
| 19 |
+
if(!is_q)values[(int64_t(b)*capacity+pos)*hk*128+head*128+t]=x[b*width+(hq+hk+head)*128+t];
|
| 20 |
+
}
|
| 21 |
+
void yue2_qkv_decode(torch::Tensor &q,torch::Tensor &keys,torch::Tensor &values,torch::Tensor const &x,torch::Tensor const &qw,torch::Tensor const &kw,torch::Tensor const &cos,torch::Tensor const &sin,torch::Tensor const &positions,double eps){
|
| 22 |
+
TORCH_CHECK(x.is_cuda() && x.dim()==3 && x.size(1)==1 && x.scalar_type()==torch::kBFloat16 && x.is_contiguous(),"QKV needs contiguous CUDA BF16 [B,1,D]");
|
| 23 |
+
const torch::Tensor* tensors[]={&q,&keys,&values,&qw,&kw};
|
| 24 |
+
for(const auto* t:tensors)TORCH_CHECK(t->device()==x.device() && t->scalar_type()==torch::kBFloat16 && t->is_contiguous(),"QKV output/cache/weights must match input");
|
| 25 |
+
TORCH_CHECK(q.dim()==4 && q.size(0)==x.size(0) && q.size(1)==1 && q.size(3)==128 && keys.dim()==4 && keys.size(0)==x.size(0) && keys.size(3)==128 && values.sizes()==keys.sizes(),"invalid QKV cache shape");
|
| 26 |
+
int b=x.size(0),hq=q.size(2),hk=keys.size(2);TORCH_CHECK(hq>0 && hk>0 && x.size(2)==(hq+2*hk)*128 && qw.numel()==128 && kw.numel()==128,"invalid QKV widths");
|
| 27 |
+
for(const auto* t:{&cos,&sin})TORCH_CHECK(t->device()==x.device() && t->scalar_type()==torch::kFloat && t->is_contiguous() && t->numel()==b*64,"invalid QKV rotary tables");
|
| 28 |
+
TORCH_CHECK(positions.device()==x.device() && positions.scalar_type()==torch::kLong && positions.is_contiguous() && positions.numel()==b,"invalid QKV positions");
|
| 29 |
+
const at::cuda::OptionalCUDAGuard guard(device_of(x));if(!b)return;
|
| 30 |
+
qkv_decode_kernel<<<dim3(hq+hk,b),128,0,at::cuda::getCurrentCUDAStream()>>>(q.data_ptr<BF>(),keys.data_ptr<BF>(),values.data_ptr<BF>(),x.data_ptr<BF>(),qw.data_ptr<BF>(),kw.data_ptr<BF>(),cos.data_ptr<float>(),sin.data_ptr<float>(),positions.data_ptr<int64_t>(),hq,hk,keys.size(1),eps);C10_CUDA_KERNEL_LAUNCH_CHECK();
|
| 31 |
+
}
|
src/kernel-source/rms-quant/orbitquant_gemv_cuda/rmsquant.cu
ADDED
|
@@ -0,0 +1,126 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#include <ATen/cuda/CUDAContext.h>
|
| 2 |
+
#include <c10/cuda/CUDAGuard.h>
|
| 3 |
+
#include <c10/cuda/CUDAException.h>
|
| 4 |
+
#include <torch/torch.h>
|
| 5 |
+
#include <cmath>
|
| 6 |
+
|
| 7 |
+
using BF = c10::BFloat16;
|
| 8 |
+
|
| 9 |
+
// The positive RMS scale cancels in quantization except for quant_eps.
|
| 10 |
+
// Keep it in the returned token norm; deliberately omit BF16 intermediates.
|
| 11 |
+
__global__ void rmsquant_kernel(uint8_t* packed, float* norms, const BF* x,
|
| 12 |
+
const BF* weight, const int32_t* permutation, const int8_t* signs,
|
| 13 |
+
const float* boundaries, float rms_eps, float quant_eps) {
|
| 14 |
+
constexpr int dim=2048;
|
| 15 |
+
__shared__ float values[dim], raw_sum[256], weighted_sum[256], bounds[15];
|
| 16 |
+
const int t=threadIdx.x, row=blockIdx.x;
|
| 17 |
+
float sx=0, sy=0;
|
| 18 |
+
for(int j=t;j<dim;j+=256){
|
| 19 |
+
int source=permutation[j];
|
| 20 |
+
float raw=float(x[row*dim+source]);
|
| 21 |
+
float weighted=raw*float(weight[source]);
|
| 22 |
+
values[j]=weighted*float(signs[j]);
|
| 23 |
+
sx=fmaf(raw,raw,sx);sy=fmaf(weighted,weighted,sy);
|
| 24 |
+
}
|
| 25 |
+
raw_sum[t]=sx;weighted_sum[t]=sy;
|
| 26 |
+
if(t<15)bounds[t]=boundaries[t];
|
| 27 |
+
__syncthreads();
|
| 28 |
+
for(int stride=128;stride>=32;stride>>=1){
|
| 29 |
+
if(t<stride){raw_sum[t]+=raw_sum[t+stride];weighted_sum[t]+=weighted_sum[t+stride];}
|
| 30 |
+
__syncthreads();
|
| 31 |
+
}
|
| 32 |
+
// Preserve the original reduction tree, but finish inside the first warp.
|
| 33 |
+
if(t<32){
|
| 34 |
+
float raw=raw_sum[t],weighted=weighted_sum[t];
|
| 35 |
+
#pragma unroll
|
| 36 |
+
for(int stride=16;stride;stride>>=1){
|
| 37 |
+
raw+=__shfl_down_sync(0xffffffff,raw,stride);
|
| 38 |
+
weighted+=__shfl_down_sync(0xffffffff,weighted,stride);
|
| 39 |
+
}
|
| 40 |
+
if(t==0){raw_sum[0]=raw;weighted_sum[0]=weighted;}
|
| 41 |
+
}
|
| 42 |
+
__syncthreads();
|
| 43 |
+
float scale=rsqrtf(raw_sum[0]/float(dim)+rms_eps);
|
| 44 |
+
float norm=sqrtf(weighted_sum[0]);
|
| 45 |
+
if(t==0)norms[row]=norm*scale;
|
| 46 |
+
float inv=1.f/(norm+quant_eps/scale);
|
| 47 |
+
float local[8];
|
| 48 |
+
#pragma unroll
|
| 49 |
+
for(int i=0;i<8;i++)local[i]=values[t+256*i]*inv;
|
| 50 |
+
// The first five Hadamard stages stay inside each warp.
|
| 51 |
+
#pragma unroll
|
| 52 |
+
for(int width=1;width<32;width<<=1){
|
| 53 |
+
#pragma unroll
|
| 54 |
+
for(int i=0;i<8;i++){
|
| 55 |
+
float other=__shfl_xor_sync(0xffffffff,local[i],width);
|
| 56 |
+
local[i]=(t&width)?other-local[i]:local[i]+other;
|
| 57 |
+
}
|
| 58 |
+
}
|
| 59 |
+
#pragma unroll
|
| 60 |
+
for(int i=0;i<8;i++)values[t+256*i]=local[i];
|
| 61 |
+
__syncthreads();
|
| 62 |
+
// Only three stages exchange values across warps.
|
| 63 |
+
for(int width=32;width<=128;width<<=1){
|
| 64 |
+
for(int b=t;b<dim/2;b+=256){
|
| 65 |
+
int group=b/width,offset=b-group*width;
|
| 66 |
+
int left=group*width*2+offset,right=left+width;
|
| 67 |
+
float a=values[left],v=values[right];values[left]=a+v;values[right]=a-v;
|
| 68 |
+
}
|
| 69 |
+
__syncthreads();
|
| 70 |
+
}
|
| 71 |
+
#pragma unroll
|
| 72 |
+
for(int i=0;i<8;i++)local[i]=values[t+256*i];
|
| 73 |
+
// Dimensions256/512/1024 are eight register values owned by one thread.
|
| 74 |
+
#pragma unroll
|
| 75 |
+
for(int width=1;width<8;width<<=1){
|
| 76 |
+
#pragma unroll
|
| 77 |
+
for(int b=0;b<4;b++){
|
| 78 |
+
int group=b/width,offset=b-group*width;
|
| 79 |
+
int left=group*width*2+offset,right=left+width;
|
| 80 |
+
float a=local[left],v=local[right];local[left]=a+v;local[right]=a-v;
|
| 81 |
+
}
|
| 82 |
+
}
|
| 83 |
+
// All reads finished before shared storage is reused for packing.
|
| 84 |
+
__syncthreads();
|
| 85 |
+
#pragma unroll
|
| 86 |
+
for(int i=0;i<8;i++)values[t+256*i]=local[i];
|
| 87 |
+
__syncthreads();
|
| 88 |
+
for(int j=t;j<dim/2;j+=256){
|
| 89 |
+
uint8_t codes[2];
|
| 90 |
+
#pragma unroll
|
| 91 |
+
for(int k=0;k<2;k++){
|
| 92 |
+
float v=values[2*j+k]*0.02209708691207961f;
|
| 93 |
+
int index=v>bounds[7]?8:0;
|
| 94 |
+
index+=v>bounds[index+3]?4:0;
|
| 95 |
+
index+=v>bounds[index+1]?2:0;
|
| 96 |
+
index+=v>bounds[index]?1:0;
|
| 97 |
+
codes[k]=index;
|
| 98 |
+
}
|
| 99 |
+
packed[row*(dim/2)+j]=codes[0]|(codes[1]<<4);
|
| 100 |
+
}
|
| 101 |
+
}
|
| 102 |
+
|
| 103 |
+
void rmsquant(torch::Tensor &packed, torch::Tensor &norms, torch::Tensor const &x,
|
| 104 |
+
torch::Tensor const &weight, torch::Tensor const &permutation,
|
| 105 |
+
torch::Tensor const &signs, torch::Tensor const &boundaries,
|
| 106 |
+
double rms_eps, double quant_eps) {
|
| 107 |
+
TORCH_CHECK(x.is_cuda() && x.is_contiguous() && x.scalar_type()==torch::kBFloat16
|
| 108 |
+
&& x.dim()==2 && x.size(1)==2048,"requires contiguous CUDA BF16 [rows,2048]");
|
| 109 |
+
for(auto* t:{&weight,&permutation,&signs,&boundaries})
|
| 110 |
+
TORCH_CHECK(t->device()==x.device() && t->is_contiguous() && t->dim()==1,"invalid constant device/layout");
|
| 111 |
+
TORCH_CHECK(weight.scalar_type()==torch::kBFloat16 && weight.numel()==2048,"invalid RMS weight");
|
| 112 |
+
TORCH_CHECK(permutation.scalar_type()==torch::kInt && permutation.numel()==2048,"invalid permutation");
|
| 113 |
+
TORCH_CHECK(signs.scalar_type()==torch::kChar && signs.numel()==2048,"invalid signs");
|
| 114 |
+
TORCH_CHECK(boundaries.scalar_type()==torch::kFloat && boundaries.numel()==15,"invalid boundaries");
|
| 115 |
+
TORCH_CHECK(packed.device()==x.device() && packed.is_contiguous() && packed.scalar_type()==torch::kUInt8
|
| 116 |
+
&& packed.dim()==2 && packed.size(0)==x.size(0) && packed.size(1)==1024,"invalid packed output");
|
| 117 |
+
TORCH_CHECK(norms.device()==x.device() && norms.is_contiguous() && norms.scalar_type()==torch::kFloat
|
| 118 |
+
&& norms.dim()==1 && norms.numel()==x.size(0),"invalid norm output");
|
| 119 |
+
TORCH_CHECK(std::isfinite(rms_eps) && rms_eps>0 && std::isfinite(quant_eps) && quant_eps>0,"epsilons must be finite and positive");
|
| 120 |
+
const at::cuda::OptionalCUDAGuard guard(device_of(x));
|
| 121 |
+
if(!x.size(0))return;
|
| 122 |
+
rmsquant_kernel<<<x.size(0),256,0,at::cuda::getCurrentCUDAStream()>>>(packed.data_ptr<uint8_t>(),norms.data_ptr<float>(),
|
| 123 |
+
x.data_ptr<BF>(),weight.data_ptr<BF>(),permutation.data_ptr<int32_t>(),signs.data_ptr<int8_t>(),
|
| 124 |
+
boundaries.data_ptr<float>(),rms_eps,quant_eps);
|
| 125 |
+
C10_CUDA_KERNEL_LAUNCH_CHECK();
|
| 126 |
+
}
|
src/kernel-source/rms-quant/tests/__init__.py
ADDED
|
File without changes
|
src/kernel-source/rms-quant/tests/test_fused_ops.py
ADDED
|
@@ -0,0 +1,23 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import torch,pytest,json
|
| 2 |
+
from pathlib import Path
|
| 3 |
+
from kernels import get_local_kernel
|
| 4 |
+
kernel=get_local_kernel(Path(__file__).resolve().parents[1]/'build','yue2_qkv_fused')
|
| 5 |
+
@pytest.mark.parametrize('rows,dim',[(1,2048),(2,2048),(16,128),(32,128),(1,6144),(67,2048)])
|
| 6 |
+
def test_rmsnorm(rows,dim):
|
| 7 |
+
torch.manual_seed(12)
|
| 8 |
+
x=(torch.randn(rows,dim,device='cuda')*torch.logspace(-3,3,rows,device='cuda')[:,None]).bfloat16();w=torch.randn(dim,device='cuda',dtype=torch.bfloat16)
|
| 9 |
+
ref=x*torch.rsqrt(x.float().pow(2).mean(-1,keepdim=True)+1e-6).to(x.dtype)*w
|
| 10 |
+
out=kernel.rmsnorm(x,w,1e-6)
|
| 11 |
+
torch.testing.assert_close(out,ref,rtol=.008,atol=1e-5)
|
| 12 |
+
print('rmsnorm exact fraction',rows,dim,float((out==ref).float().mean()))
|
| 13 |
+
@pytest.mark.parametrize('b,t,h',[(1,1,16),(1,1,8),(2,1,16),(1,17,8)])
|
| 14 |
+
def test_rope(b,t,h):
|
| 15 |
+
torch.manual_seed(13);x=torch.randn(b,t,h,128,device='cuda',dtype=torch.bfloat16);a=torch.randn(b,t,1,64,device='cuda');c=a.cos();s=a.sin();cb=c.bfloat16();sb=s.bfloat16();x1,x2=x.chunk(2,-1)
|
| 16 |
+
ref=torch.cat([x1*cb-x2*sb,x2*cb+x1*sb],-1)
|
| 17 |
+
torch.testing.assert_close(kernel.rope(x,c,s),ref,rtol=0,atol=0)
|
| 18 |
+
@pytest.mark.parametrize('rows',[1,2,16])
|
| 19 |
+
def test_swiglu(rows):
|
| 20 |
+
torch.manual_seed(14);x=torch.randn(rows,12288,device='cuda',dtype=torch.bfloat16);g,u=x.chunk(2,-1)
|
| 21 |
+
torch.testing.assert_close(kernel.swiglu(x),torch.nn.functional.silu(g)*u,rtol=0,atol=0)
|
| 22 |
+
def test_invalid_norm_weight():
|
| 23 |
+
with pytest.raises(RuntimeError,match='weight'):kernel.rmsnorm(torch.ones(1,128,device='cuda',dtype=torch.bfloat16),torch.ones(64,device='cuda',dtype=torch.bfloat16),1e-6)
|
src/kernel-source/rms-quant/tests/test_orbitquant_gemv.py
ADDED
|
@@ -0,0 +1,54 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import json
|
| 2 |
+
from pathlib import Path
|
| 3 |
+
import pytest
|
| 4 |
+
import torch
|
| 5 |
+
from kernels import get_local_kernel
|
| 6 |
+
from orbitquant.kernels.native_packed_matmul import matmul_packed_w4a4_int8_with_native_kernel as original
|
| 7 |
+
|
| 8 |
+
kernel=get_local_kernel(Path(__file__).resolve().parents[1] / 'build', 'yue2_qkv_fused')
|
| 9 |
+
|
| 10 |
+
@pytest.mark.parametrize('rows,n,k',[(1,2048,2048),(2,6144,2048),(8,2048,6144),(1,129,128),(2,1024,2048),(1,129,130),(1,129,16)])
|
| 11 |
+
@pytest.mark.parametrize('dtype',[torch.bfloat16,torch.float16])
|
| 12 |
+
def test_exact_integer_gemv(rows,n,k,dtype):
|
| 13 |
+
torch.manual_seed(123)
|
| 14 |
+
x=torch.randint(0,256,(rows,k//2),device='cuda',dtype=torch.uint8)
|
| 15 |
+
w=torch.randint(0,256,(n*k//2,),device='cuda',dtype=torch.uint8)
|
| 16 |
+
xn=torch.rand(rows,device='cuda');wn=torch.rand(n,device='cuda',dtype=torch.bfloat16)
|
| 17 |
+
ac=torch.randint(-127,128,(16,),device='cuda',dtype=torch.int8);wc=ac.flip(0).contiguous()
|
| 18 |
+
bias=None
|
| 19 |
+
kw=dict(activation_scale=.03125,weight_scale=.0625,bias=bias,output_dtype=dtype)
|
| 20 |
+
if k % 128 == 0:
|
| 21 |
+
ref=original(x,w,xn,wn,ac,wc,out_features=n,in_features=k,**kw)
|
| 22 |
+
else:
|
| 23 |
+
def unpack(z):return torch.stack([z & 15,z >> 4],dim=-1).reshape(z.shape[0],-1).long().cpu()
|
| 24 |
+
aa=ac.cpu().long()[unpack(x)];ww=wc.cpu().long()[unpack(w.reshape(n,k//2))]
|
| 25 |
+
sums=(aa @ ww.T).to(device='cuda',dtype=torch.float32)
|
| 26 |
+
ref=(sums*(xn[:,None]*wn.float()[None,:]*(kw['activation_scale']*kw['weight_scale']))).to(dtype)
|
| 27 |
+
out=kernel.gemv(x,w,xn,wn,ac,wc,**kw)
|
| 28 |
+
torch.testing.assert_close(out,ref,rtol=0,atol=0)
|
| 29 |
+
stream=torch.cuda.Stream();stream.wait_stream(torch.cuda.current_stream())
|
| 30 |
+
with torch.cuda.stream(stream):
|
| 31 |
+
for _ in range(3):kernel.gemv(x,w,xn,wn,ac,wc,**kw)
|
| 32 |
+
torch.cuda.current_stream().wait_stream(stream)
|
| 33 |
+
graph=torch.cuda.CUDAGraph()
|
| 34 |
+
with torch.cuda.graph(graph): captured=kernel.gemv(x,w,xn,wn,ac,wc,**kw)
|
| 35 |
+
graph.replay();torch.cuda.synchronize()
|
| 36 |
+
torch.testing.assert_close(captured,ref,rtol=0,atol=0)
|
| 37 |
+
|
| 38 |
+
def test_rejects_bad_weight_shape():
|
| 39 |
+
x=torch.zeros(1,64,device='cuda',dtype=torch.uint8)
|
| 40 |
+
with pytest.raises(RuntimeError,match='packed weight size'):
|
| 41 |
+
kernel.gemv(x,x.flatten(),torch.ones(1,device='cuda'),torch.ones(2,device='cuda',dtype=torch.bfloat16),torch.zeros(16,device='cuda',dtype=torch.int8),torch.zeros(16,device='cuda',dtype=torch.int8),activation_scale=1,weight_scale=1)
|
| 42 |
+
|
| 43 |
+
@pytest.mark.parametrize('offset',[1,2,3])
|
| 44 |
+
def test_misaligned_contiguous_packed_inputs(offset):
|
| 45 |
+
rows,n,k=1,129,128
|
| 46 |
+
torch.manual_seed(10)
|
| 47 |
+
x=torch.randint(0,256,(rows*k//2+offset,),device='cuda',dtype=torch.uint8)[offset:].view(rows,k//2)
|
| 48 |
+
w=torch.randint(0,256,(n*k//2+offset,),device='cuda',dtype=torch.uint8)[offset:]
|
| 49 |
+
xn=torch.rand(rows,device='cuda');wn=torch.rand(n,device='cuda',dtype=torch.bfloat16)
|
| 50 |
+
ac=torch.randint(-127,128,(16,),device='cuda',dtype=torch.int8);wc=ac.flip(0).contiguous()
|
| 51 |
+
kw=dict(activation_scale=.03125,weight_scale=.0625,bias=None,output_dtype=torch.bfloat16)
|
| 52 |
+
ref=original(x.contiguous().clone(),w.contiguous().clone(),xn,wn,ac,wc,out_features=n,in_features=k,**kw)
|
| 53 |
+
out=kernel.gemv(x,w,xn,wn,ac,wc,**kw)
|
| 54 |
+
torch.testing.assert_close(out,ref,rtol=0,atol=0)
|
src/kernel-source/rms-quant/tests/test_qkv_decode.py
ADDED
|
@@ -0,0 +1,20 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import torch,pytest
|
| 2 |
+
from pathlib import Path
|
| 3 |
+
from kernels import get_local_kernel
|
| 4 |
+
kernel=get_local_kernel(Path(__file__).resolve().parents[1]/'build','yue2_qkv_fused')
|
| 5 |
+
@pytest.mark.parametrize('b,capacity',[(1,32),(2,257)])
|
| 6 |
+
def test_decode_qkv_cache(b,capacity):
|
| 7 |
+
torch.manual_seed(42);hq,hk,d=16,8,128;x=torch.randn(b,1,(hq+2*hk)*d,device='cuda',dtype=torch.bfloat16);qw=torch.randn(d,device='cuda',dtype=torch.bfloat16);kw=torch.randn(d,device='cuda',dtype=torch.bfloat16)
|
| 8 |
+
a=torch.randn(b,1,d//2,device='cuda');c=a.cos();s=a.sin();positions=torch.tensor([capacity-1-i for i in range(b)],device='cuda');keys=torch.randn(b,capacity,hk,d,device='cuda',dtype=torch.bfloat16);values=keys.clone();expected_k=keys.clone();expected_v=values.clone()
|
| 9 |
+
q,k,v=x.split((hq*d,hk*d,hk*d),-1);q=q.reshape(b,1,hq,d);k=k.reshape(b,1,hk,d);v=v.reshape(b,1,hk,d)
|
| 10 |
+
def norm(z,w):return z*torch.rsqrt(z.float().pow(2).mean(-1,keepdim=True)+1e-6).bfloat16()*w
|
| 11 |
+
def rope(z):
|
| 12 |
+
x1,x2=z.chunk(2,-1);cb=c.unsqueeze(2).bfloat16();sb=s.unsqueeze(2).bfloat16();return torch.cat([x1*cb-x2*sb,x2*cb+x1*sb],-1)
|
| 13 |
+
q=rope(norm(q,qw));k=rope(norm(k,kw));slots=positions[:,None,None,None].expand(b,1,hk,d);expected_k.scatter_(1,slots,k);expected_v.scatter_(1,slots,v)
|
| 14 |
+
actual=kernel.qkv_decode(x,qw,kw,c,s,positions,keys,values,hq,1e-6)
|
| 15 |
+
torch.testing.assert_close(actual,q,rtol=0,atol=0);torch.testing.assert_close(keys,expected_k,rtol=0,atol=0);torch.testing.assert_close(values,expected_v,rtol=0,atol=0)
|
| 16 |
+
@pytest.mark.parametrize('rows,dim',[(1,2048),(16,128),(2,2048),(32,128),(67,2048),(1,6144)])
|
| 17 |
+
def test_exact_rmsnorm(rows,dim):
|
| 18 |
+
torch.manual_seed(456);x=torch.randn(rows,dim,device='cuda',dtype=torch.bfloat16);w=torch.randn(dim,device='cuda',dtype=torch.bfloat16)
|
| 19 |
+
ref=x*torch.rsqrt(x.float().pow(2).mean(-1,keepdim=True)+1e-6).bfloat16()*w
|
| 20 |
+
torch.testing.assert_close(kernel.rmsnorm_exact(x,w,1e-6),ref,rtol=0,atol=0)
|
src/kernel-source/rms-quant/tests/test_rmsquant.py
ADDED
|
@@ -0,0 +1,56 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from pathlib import Path
|
| 2 |
+
import pytest
|
| 3 |
+
import torch
|
| 4 |
+
from kernels import get_local_kernel
|
| 5 |
+
|
| 6 |
+
kernel=get_local_kernel(Path(__file__).resolve().parents[1]/'build','yue2_qkv_fused')
|
| 7 |
+
|
| 8 |
+
def inputs(rows,scale=1):
|
| 9 |
+
torch.manual_seed(51)
|
| 10 |
+
x=(torch.randn(rows,2048,device='cuda')*scale).bfloat16()
|
| 11 |
+
w=torch.randn(2048,device='cuda').bfloat16()
|
| 12 |
+
p=torch.randperm(2048,device='cuda').int()
|
| 13 |
+
s=(torch.randint(0,2,(2048,),device='cuda')*2-1).to(torch.int8)
|
| 14 |
+
bounds=torch.linspace(-.065,.065,15,device='cuda')
|
| 15 |
+
return x,w,p,s,bounds
|
| 16 |
+
|
| 17 |
+
@pytest.mark.parametrize('rows',[0,1,2,8,32])
|
| 18 |
+
@pytest.mark.parametrize('scale',[0,1e-9,1,100])
|
| 19 |
+
def test_continuous_formula(rows,scale):
|
| 20 |
+
x,w,p,s,b=inputs(rows,scale)
|
| 21 |
+
packed,norms=kernel.rmsquant(x,w,p,s,b)
|
| 22 |
+
y=x.float()*torch.rsqrt(x.float().square().mean(-1,keepdim=True)+1e-6)*w.float()
|
| 23 |
+
expected_norm=y.norm(dim=-1)
|
| 24 |
+
y=y/(expected_norm[:,None]+1e-8)
|
| 25 |
+
y=y[:,p.long()]*s.float()
|
| 26 |
+
width=1
|
| 27 |
+
while width<2048:
|
| 28 |
+
z=y.reshape(rows,2048//(2*width),2,width)
|
| 29 |
+
a,c=z[:,:,0],z[:,:,1]
|
| 30 |
+
y=torch.stack([a+c,a-c],dim=2).reshape(rows,2048)
|
| 31 |
+
width*=2
|
| 32 |
+
codes=torch.bucketize(y/(2048**.5),b)
|
| 33 |
+
expected=(codes[:,0::2]|(codes[:,1::2]<<4)).byte()
|
| 34 |
+
torch.testing.assert_close(norms,expected_norm,rtol=2e-6,atol=1e-9)
|
| 35 |
+
# Only boundaries within FP32 reduction noise may differ.
|
| 36 |
+
if rows:
|
| 37 |
+
assert (expected!=packed).float().mean().item()<.001
|
| 38 |
+
assert torch.isfinite(norms).all()
|
| 39 |
+
|
| 40 |
+
def test_graph_and_invalid_contract():
|
| 41 |
+
args=inputs(2)
|
| 42 |
+
expected=kernel.rmsquant(*args)
|
| 43 |
+
stream=torch.cuda.Stream();stream.wait_stream(torch.cuda.current_stream())
|
| 44 |
+
with torch.cuda.stream(stream):
|
| 45 |
+
for _ in range(3):kernel.rmsquant(*args)
|
| 46 |
+
torch.cuda.current_stream().wait_stream(stream)
|
| 47 |
+
graph=torch.cuda.CUDAGraph()
|
| 48 |
+
with torch.cuda.graph(graph):result=kernel.rmsquant(*args)
|
| 49 |
+
graph.replay();torch.cuda.synchronize()
|
| 50 |
+
for a,b in zip(expected,result):torch.testing.assert_close(a,b,rtol=0,atol=0)
|
| 51 |
+
with pytest.raises(RuntimeError,match='BF16'):
|
| 52 |
+
kernel.rmsquant(args[0].float(),*args[1:])
|
| 53 |
+
with pytest.raises(RuntimeError,match='positive'):
|
| 54 |
+
kernel.rmsquant(*args,rms_eps=0)
|
| 55 |
+
with pytest.raises(RuntimeError,match='permutation'):
|
| 56 |
+
kernel.rmsquant(args[0],args[1],args[2].long(),*args[3:])
|
src/kernel-source/rms-quant/torch-ext/registration.h
ADDED
|
@@ -0,0 +1,39 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
// Registration macros from vLLM:
|
| 2 |
+
// https://github.com/vllm-project/vllm/blob/main/csrc/core/registration.h
|
| 3 |
+
|
| 4 |
+
#pragma once
|
| 5 |
+
|
| 6 |
+
#include <Python.h>
|
| 7 |
+
|
| 8 |
+
#define _CONCAT(A, B) A##B
|
| 9 |
+
#define CONCAT(A, B) _CONCAT(A, B)
|
| 10 |
+
|
| 11 |
+
#define _STRINGIFY(A) #A
|
| 12 |
+
#define STRINGIFY(A) _STRINGIFY(A)
|
| 13 |
+
|
| 14 |
+
// A version of the TORCH_LIBRARY macro that expands the NAME, i.e. so NAME
|
| 15 |
+
// could be a macro instead of a literal token.
|
| 16 |
+
#define TORCH_LIBRARY_EXPAND(NAME, MODULE) TORCH_LIBRARY(NAME, MODULE)
|
| 17 |
+
|
| 18 |
+
// A version of the STABLE_TORCH_LIBRARY macro that expands the NAME, i.e. so NAME
|
| 19 |
+
// could be a macro instead of a literal token.
|
| 20 |
+
#define STABLE_TORCH_LIBRARY_EXPAND(NAME, MODULE) STABLE_TORCH_LIBRARY(NAME, MODULE)
|
| 21 |
+
|
| 22 |
+
// A version of the TORCH_LIBRARY_IMPL macro that expands the NAME, i.e. so NAME
|
| 23 |
+
// could be a macro instead of a literal token.
|
| 24 |
+
#define TORCH_LIBRARY_IMPL_EXPAND(NAME, DEVICE, MODULE) \
|
| 25 |
+
TORCH_LIBRARY_IMPL(NAME, DEVICE, MODULE)
|
| 26 |
+
|
| 27 |
+
// A version of the STABLE_TORCH_LIBRARY_IMPL macro that expands the NAME, i.e. so NAME
|
| 28 |
+
// could be a macro instead of a literal token.
|
| 29 |
+
#define STABLE_TORCH_LIBRARY_IMPL_EXPAND(NAME, DEVICE, MODULE) \
|
| 30 |
+
STABLE_TORCH_LIBRARY_IMPL(NAME, DEVICE, MODULE)
|
| 31 |
+
|
| 32 |
+
// REGISTER_EXTENSION allows the shared library to be loaded and initialized
|
| 33 |
+
// via python's import statement.
|
| 34 |
+
#define REGISTER_EXTENSION(NAME) \
|
| 35 |
+
PyMODINIT_FUNC CONCAT(PyInit_, NAME)() { \
|
| 36 |
+
static struct PyModuleDef module = {PyModuleDef_HEAD_INIT, \
|
| 37 |
+
STRINGIFY(NAME), nullptr, 0, nullptr}; \
|
| 38 |
+
return PyModule_Create(&module); \
|
| 39 |
+
}
|
src/kernel-source/rms-quant/torch-ext/torch_binding.cpp
ADDED
|
@@ -0,0 +1,20 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#include <torch/library.h>
|
| 2 |
+
#include "registration.h"
|
| 3 |
+
#include "torch_binding.h"
|
| 4 |
+
TORCH_LIBRARY_EXPAND(TORCH_EXTENSION_NAME, ops) {
|
| 5 |
+
ops.def("rmsquant(Tensor! packed, Tensor! norms, Tensor x, Tensor weight, Tensor permutation, Tensor signs, Tensor boundaries, float rms_eps, float quant_eps) -> ()");
|
| 6 |
+
ops.impl("rmsquant", torch::kCUDA, &rmsquant);
|
| 7 |
+
ops.def("orbitquant_gemv(Tensor! out, Tensor x, Tensor w, Tensor xn, Tensor wn, Tensor ac, Tensor wc, Tensor bias, bool has_bias, float activation_scale, float weight_scale) -> ()");
|
| 8 |
+
ops.impl("orbitquant_gemv", torch::kCUDA, &orbitquant_gemv);
|
| 9 |
+
ops.def("yue2_rmsnorm(Tensor! out, Tensor x, Tensor weight, float eps) -> ()");
|
| 10 |
+
ops.impl("yue2_rmsnorm", torch::kCUDA, &yue2_rmsnorm);
|
| 11 |
+
ops.def("yue2_rope(Tensor! out, Tensor x, Tensor cos, Tensor sin, int heads) -> ()");
|
| 12 |
+
ops.impl("yue2_rope", torch::kCUDA, &yue2_rope);
|
| 13 |
+
ops.def("yue2_swiglu(Tensor! out, Tensor x) -> ()");
|
| 14 |
+
ops.impl("yue2_swiglu", torch::kCUDA, &yue2_swiglu);
|
| 15 |
+
ops.def("yue2_qkv_decode(Tensor! q, Tensor! keys, Tensor! values, Tensor x, Tensor qw, Tensor kw, Tensor cos, Tensor sin, Tensor positions, float eps) -> ()");
|
| 16 |
+
ops.impl("yue2_qkv_decode", torch::kCUDA, &yue2_qkv_decode);
|
| 17 |
+
ops.def("yue2_rmsnorm_exact(Tensor! out, Tensor x, Tensor weight, float eps) -> ()");
|
| 18 |
+
ops.impl("yue2_rmsnorm_exact", torch::kCUDA, &yue2_rmsnorm_exact);
|
| 19 |
+
}
|
| 20 |
+
REGISTER_EXTENSION(TORCH_EXTENSION_NAME)
|
src/kernel-source/rms-quant/torch-ext/torch_binding.h
ADDED
|
@@ -0,0 +1,13 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#pragma once
|
| 2 |
+
#include <torch/torch.h>
|
| 3 |
+
void orbitquant_gemv(torch::Tensor &out, torch::Tensor const &x, torch::Tensor const &w, torch::Tensor const &xn, torch::Tensor const &wn, torch::Tensor const &ac, torch::Tensor const &wc, torch::Tensor const &bias, bool has_bias, double activation_scale, double weight_scale);
|
| 4 |
+
|
| 5 |
+
void yue2_rmsnorm(torch::Tensor &out, torch::Tensor const &x, torch::Tensor const &weight, double eps);
|
| 6 |
+
void yue2_rope(torch::Tensor &out, torch::Tensor const &x, torch::Tensor const &cos, torch::Tensor const &sin, int64_t heads);
|
| 7 |
+
void yue2_swiglu(torch::Tensor &out, torch::Tensor const &x);
|
| 8 |
+
|
| 9 |
+
void yue2_qkv_decode(torch::Tensor &q, torch::Tensor &keys, torch::Tensor &values, torch::Tensor const &x, torch::Tensor const &qw, torch::Tensor const &kw, torch::Tensor const &cos, torch::Tensor const &sin, torch::Tensor const &positions, double eps);
|
| 10 |
+
|
| 11 |
+
void yue2_rmsnorm_exact(torch::Tensor &out, torch::Tensor const &x, torch::Tensor const &weight, double eps);
|
| 12 |
+
|
| 13 |
+
void rmsquant(torch::Tensor &packed, torch::Tensor &norms, torch::Tensor const &x, torch::Tensor const &weight, torch::Tensor const &permutation, torch::Tensor const &signs, torch::Tensor const &boundaries, double rms_eps, double quant_eps);
|
src/kernel-source/rms-quant/torch-ext/yue2_qkv_fused/__init__.py
ADDED
|
@@ -0,0 +1,34 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import torch
|
| 2 |
+
from ._ops import ops
|
| 3 |
+
|
| 4 |
+
def gemv(packed_x, packed_w, token_norms, row_norms, activation_codes, weight_codes, *, activation_scale, weight_scale, bias=None, output_dtype=torch.bfloat16):
|
| 5 |
+
out = torch.empty((packed_x.shape[0], row_norms.numel()), device=packed_x.device, dtype=output_dtype)
|
| 6 |
+
empty = out if bias is None else bias
|
| 7 |
+
ops.orbitquant_gemv(out, packed_x, packed_w, token_norms, row_norms, activation_codes, weight_codes, empty, bias is not None, activation_scale, weight_scale)
|
| 8 |
+
return out
|
| 9 |
+
|
| 10 |
+
def rmsnorm(x, weight, eps):
|
| 11 |
+
x=x.contiguous();out=torch.empty_like(x);ops.yue2_rmsnorm(out,x,weight,eps);return out
|
| 12 |
+
|
| 13 |
+
def rope(x, cos, sin):
|
| 14 |
+
x=x.contiguous();out=torch.empty_like(x);ops.yue2_rope(out,x,cos.contiguous(),sin.contiguous(),x.shape[-2]);return out
|
| 15 |
+
|
| 16 |
+
def swiglu(x):
|
| 17 |
+
x=x.contiguous();out=torch.empty((*x.shape[:-1],x.shape[-1]//2),device=x.device,dtype=x.dtype);ops.yue2_swiglu(out,x);return out
|
| 18 |
+
|
| 19 |
+
def qkv_decode(x, qw, kw, cos, sin, positions, keys, values, query_heads, eps):
|
| 20 |
+
q=torch.empty((x.shape[0],1,query_heads,128),device=x.device,dtype=x.dtype)
|
| 21 |
+
ops.yue2_qkv_decode(q,keys,values,x,qw,kw,cos,sin,positions,eps)
|
| 22 |
+
return q
|
| 23 |
+
|
| 24 |
+
def rmsnorm_exact(x, weight, eps):
|
| 25 |
+
x=x.contiguous();out=torch.empty_like(x);ops.yue2_rmsnorm_exact(out,x,weight,eps);return out
|
| 26 |
+
|
| 27 |
+
|
| 28 |
+
def rmsquant(x, weight, permutation, signs, boundaries, rms_eps=1e-6, quant_eps=1e-8):
|
| 29 |
+
shape=x.shape
|
| 30 |
+
x=x.reshape(-1,shape[-1]).contiguous()
|
| 31 |
+
packed=torch.empty((x.shape[0],x.shape[1]//2),device=x.device,dtype=torch.uint8)
|
| 32 |
+
norms=torch.empty(x.shape[0],device=x.device,dtype=torch.float32)
|
| 33 |
+
ops.rmsquant(packed,norms,x,weight,permutation,signs,boundaries,rms_eps,quant_eps)
|
| 34 |
+
return packed,norms
|
src/requirements.txt
CHANGED
|
@@ -7,4 +7,4 @@ numpy==2.2.6
|
|
| 7 |
soundfile==0.13.1
|
| 8 |
accelerate==1.13.0
|
| 9 |
kernels==0.12.2
|
| 10 |
-
./wheels/orbitquant-0.9.
|
|
|
|
| 7 |
soundfile==0.13.1
|
| 8 |
accelerate==1.13.0
|
| 9 |
kernels==0.12.2
|
| 10 |
+
./wheels/orbitquant-0.9.5-py3-none-any.whl
|
src/run.py
CHANGED
|
@@ -17,13 +17,15 @@ PROFILES = ("reference", "fused", "fast", "memory")
|
|
| 17 |
|
| 18 |
|
| 19 |
def load_pipeline(*, vae=None, low_memory=False, offload_ar=False, progress=True,
|
| 20 |
-
profile=None, ode_steps=None, vae_precision=None):
|
| 21 |
import torch
|
| 22 |
profile = profile or ("reference" if low_memory else "fast")
|
| 23 |
if profile not in PROFILES:
|
| 24 |
raise ValueError(f"Unknown profile {profile!r}")
|
| 25 |
if low_memory and profile != "reference":
|
| 26 |
raise ValueError("Unfused --low-memory requires profile=reference; use profile=memory for staged experts")
|
|
|
|
|
|
|
| 27 |
if not torch.cuda.is_available():
|
| 28 |
raise RuntimeError("This release requires an NVIDIA CUDA GPU.")
|
| 29 |
if torch.cuda.get_device_capability() != (12, 0):
|
|
@@ -44,6 +46,7 @@ def load_pipeline(*, vae=None, low_memory=False, offload_ar=False, progress=True
|
|
| 44 |
pipe.fused = not low_memory
|
| 45 |
pipe.trim_cache = True
|
| 46 |
pipe.profile = profile
|
|
|
|
| 47 |
fast = profile in {"fast", "memory"}
|
| 48 |
pipe.generation_config = dataclasses.replace(pipe.generation_config,
|
| 49 |
ode_steps=ode_steps if ode_steps is not None else 8 if fast else 32)
|
|
@@ -69,6 +72,8 @@ def main():
|
|
| 69 |
parser.add_argument("--allocator", choices=("native", "expandable"))
|
| 70 |
parser.add_argument("--low-memory", action="store_true")
|
| 71 |
parser.add_argument("--offload-ar", action="store_true")
|
|
|
|
|
|
|
| 72 |
args = parser.parse_args()
|
| 73 |
allocator = args.allocator or ("expandable" if (args.profile or "fast") in {"fast", "memory"} and not args.low_memory else None)
|
| 74 |
if allocator is not None:
|
|
@@ -77,7 +82,7 @@ def main():
|
|
| 77 |
out = Path(args.output)
|
| 78 |
out.mkdir(parents=True, exist_ok=True)
|
| 79 |
with load_pipeline(vae=args.vae, low_memory=args.low_memory, offload_ar=args.offload_ar,
|
| 80 |
-
profile=args.profile, ode_steps=args.ode_steps, vae_precision=args.vae_precision) as pipe:
|
| 81 |
start = time.perf_counter()
|
| 82 |
song = pipe(**{k: request[k] for k in ("style", "lyrics", "seed", "cot") if k in request})
|
| 83 |
song.save_artifacts(out)
|
|
|
|
| 17 |
|
| 18 |
|
| 19 |
def load_pipeline(*, vae=None, low_memory=False, offload_ar=False, progress=True,
|
| 20 |
+
profile=None, ode_steps=None, vae_precision=None, fuse_rms_quant=False):
|
| 21 |
import torch
|
| 22 |
profile = profile or ("reference" if low_memory else "fast")
|
| 23 |
if profile not in PROFILES:
|
| 24 |
raise ValueError(f"Unknown profile {profile!r}")
|
| 25 |
if low_memory and profile != "reference":
|
| 26 |
raise ValueError("Unfused --low-memory requires profile=reference; use profile=memory for staged experts")
|
| 27 |
+
if fuse_rms_quant and profile not in {"fast", "memory"}:
|
| 28 |
+
raise ValueError("RMS/activation fusion requires profile=fast or profile=memory")
|
| 29 |
if not torch.cuda.is_available():
|
| 30 |
raise RuntimeError("This release requires an NVIDIA CUDA GPU.")
|
| 31 |
if torch.cuda.get_device_capability() != (12, 0):
|
|
|
|
| 46 |
pipe.fused = not low_memory
|
| 47 |
pipe.trim_cache = True
|
| 48 |
pipe.profile = profile
|
| 49 |
+
pipe.fuse_rms_quant = bool(fuse_rms_quant)
|
| 50 |
fast = profile in {"fast", "memory"}
|
| 51 |
pipe.generation_config = dataclasses.replace(pipe.generation_config,
|
| 52 |
ode_steps=ode_steps if ode_steps is not None else 8 if fast else 32)
|
|
|
|
| 72 |
parser.add_argument("--allocator", choices=("native", "expandable"))
|
| 73 |
parser.add_argument("--low-memory", action="store_true")
|
| 74 |
parser.add_argument("--offload-ar", action="store_true")
|
| 75 |
+
parser.add_argument("--fuse-rms-quant", action="store_true",
|
| 76 |
+
help="Fuse RMSNorm into activation quantization; faster experimental arithmetic")
|
| 77 |
args = parser.parse_args()
|
| 78 |
allocator = args.allocator or ("expandable" if (args.profile or "fast") in {"fast", "memory"} and not args.low_memory else None)
|
| 79 |
if allocator is not None:
|
|
|
|
| 82 |
out = Path(args.output)
|
| 83 |
out.mkdir(parents=True, exist_ok=True)
|
| 84 |
with load_pipeline(vae=args.vae, low_memory=args.low_memory, offload_ar=args.offload_ar,
|
| 85 |
+
profile=args.profile, ode_steps=args.ode_steps, vae_precision=args.vae_precision, fuse_rms_quant=args.fuse_rms_quant) as pipe:
|
| 86 |
start = time.perf_counter()
|
| 87 |
song = pipe(**{k: request[k] for k in ("style", "lyrics", "seed", "cot") if k in request})
|
| 88 |
song.save_artifacts(out)
|