WaveCut commited on
Commit
b35ab1f
·
verified ·
1 Parent(s): a7fd5a2

Publish OrbitQuant 0.9.5 byte-pair decode and optional RMS activation fusion

Browse files
This view is limited to 50 files because it contains too many changes.   See raw diff
Files changed (50) hide show
  1. .gitattributes +3 -0
  2. README.md +24 -4
  3. SHA256SUMS +60 -14
  4. evaluation/runtime-v3/contract-checks.json +1 -0
  5. evaluation/runtime-v3/fast-rms0-s8.json +304 -0
  6. evaluation/runtime-v3/fast-rms1-s8.json +304 -0
  7. evaluation/runtime-v3/memory-rms1-s8.json +304 -0
  8. examples/runtime-v3/fast-rms.wav +3 -0
  9. src/OPTIMIZATION_IDEAS.md +11 -0
  10. src/kernel-source/orbitquant-packed-matmul/CARD.md +10 -0
  11. src/kernel-source/orbitquant-packed-matmul/orbitquant_packed_matmul_cuda/packed_matmul.cu +58 -4
  12. src/kernel-source/orbitquant-packed-matmul/tests/test_decode_gemv.py +15 -2
  13. src/kernel-source/rms-quant/.gitattributes +3 -0
  14. src/kernel-source/rms-quant/.gitignore +1 -0
  15. src/kernel-source/rms-quant/CARD.md +15 -0
  16. src/kernel-source/rms-quant/build.toml +22 -0
  17. src/kernel-source/rms-quant/cmake/_ops.py.in +9 -0
  18. src/kernel-source/rms-quant/cmake/add_build_metadata.py +56 -0
  19. src/kernel-source/rms-quant/cmake/build-variants.cmake +328 -0
  20. src/kernel-source/rms-quant/cmake/compile-metal.cmake +104 -0
  21. src/kernel-source/rms-quant/cmake/get_gpu_lang.cmake +9 -0
  22. src/kernel-source/rms-quant/cmake/get_gpu_lang.py +20 -0
  23. src/kernel-source/rms-quant/cmake/hipify.py +76 -0
  24. src/kernel-source/rms-quant/cmake/kernel.cmake +317 -0
  25. src/kernel-source/rms-quant/cmake/metallib_to_header.py +73 -0
  26. src/kernel-source/rms-quant/cmake/utils.cmake +611 -0
  27. src/kernel-source/rms-quant/compat.py +26 -0
  28. src/kernel-source/rms-quant/flake.nix +11 -0
  29. src/kernel-source/rms-quant/metadata-cann.json +21 -0
  30. src/kernel-source/rms-quant/metadata-cpu.json +21 -0
  31. src/kernel-source/rms-quant/metadata-cuda.json +21 -0
  32. src/kernel-source/rms-quant/metadata-metal.json +21 -0
  33. src/kernel-source/rms-quant/metadata-neuron.json +21 -0
  34. src/kernel-source/rms-quant/metadata-rocm.json +21 -0
  35. src/kernel-source/rms-quant/metadata-xpu.json +21 -0
  36. src/kernel-source/rms-quant/orbitquant_gemv_cuda/fused_ops.cu +67 -0
  37. src/kernel-source/rms-quant/orbitquant_gemv_cuda/orbitquant_gemv.cu +74 -0
  38. src/kernel-source/rms-quant/orbitquant_gemv_cuda/qkv_decode.cu +31 -0
  39. src/kernel-source/rms-quant/orbitquant_gemv_cuda/rmsquant.cu +126 -0
  40. src/kernel-source/rms-quant/tests/__init__.py +0 -0
  41. src/kernel-source/rms-quant/tests/test_fused_ops.py +23 -0
  42. src/kernel-source/rms-quant/tests/test_orbitquant_gemv.py +54 -0
  43. src/kernel-source/rms-quant/tests/test_qkv_decode.py +20 -0
  44. src/kernel-source/rms-quant/tests/test_rmsquant.py +56 -0
  45. src/kernel-source/rms-quant/torch-ext/registration.h +39 -0
  46. src/kernel-source/rms-quant/torch-ext/torch_binding.cpp +20 -0
  47. src/kernel-source/rms-quant/torch-ext/torch_binding.h +13 -0
  48. src/kernel-source/rms-quant/torch-ext/yue2_qkv_fused/__init__.py +34 -0
  49. src/requirements.txt +1 -1
  50. src/run.py +7 -2
.gitattributes CHANGED
@@ -44,3 +44,6 @@ src/wheels/orbitquant-0.9.2-py3-none-any.whl filter=lfs diff=lfs merge=lfs -text
44
  src/runtime/kernels/fast/torch210-cxx11-cu128-x86_64-linux/_yue2_qkv_fused_cuda_46ceaaf_dirty.abi3.so filter=lfs diff=lfs merge=lfs -text
45
  src/wheels/orbitquant-0.9.3-py3-none-any.whl filter=lfs diff=lfs merge=lfs -text
46
  src/wheels/orbitquant-0.9.4-py3-none-any.whl filter=lfs diff=lfs merge=lfs -text
 
 
 
 
44
  src/runtime/kernels/fast/torch210-cxx11-cu128-x86_64-linux/_yue2_qkv_fused_cuda_46ceaaf_dirty.abi3.so filter=lfs diff=lfs merge=lfs -text
45
  src/wheels/orbitquant-0.9.3-py3-none-any.whl filter=lfs diff=lfs merge=lfs -text
46
  src/wheels/orbitquant-0.9.4-py3-none-any.whl filter=lfs diff=lfs merge=lfs -text
47
+ examples/runtime-v3/fast-rms.wav filter=lfs diff=lfs merge=lfs -text
48
+ src/runtime/kernels/rms/torch210-cxx11-cu128-x86_64-linux/_yue2_qkv_fused_cuda_cf0db58.abi3.so filter=lfs diff=lfs merge=lfs -text
49
+ src/wheels/orbitquant-0.9.5-py3-none-any.whl filter=lfs diff=lfs merge=lfs -text
README.md CHANGED
@@ -29,7 +29,7 @@ The autoregressive branch plans an ABC score and generates semantic audio tokens
29
 
30
  2,818,572,288 transformer weights occupy 1,409,286,144 packed bytes. The complete model checkpoint is **3,035,926,840 bytes**, versus **7,261,441,640 bytes** upstream: 2.39× smaller (58.2% reduction). The separate 530,512,720-byte VAE checkpoint is unchanged and downloaded at a pinned revision.
31
 
32
- Compatible Q/K/V and gate/up projections are grouped: 224 physical packed modules represent 392 logical projections without duplicated packed storage. OrbitQuant 0.9.4 with native kernels 1.0.1 provides a vectorized DP4A W4A4 GEMV for 1–8 rows; larger inputs retain the packed Tensor Core path. No persistent decoded BF16 or INT8 weight caches are used. Original AR CUDA graphs and prefix caching were already present upstream and are not credited as new optimizations.
33
 
34
  The default VAE core is 512 frames with the original 16-frame halo. On the recorded Russian song this was waveform-identical to 1024 frames, while lowering the decoder's isolated memory peak. This is a tested example, not a universal bitwise-equivalence guarantee.
35
 
@@ -40,11 +40,11 @@ The default VAE core is 512 frames with the original 16-frame halo. On the recor
40
  - `examples/`: Russian WAV comparisons and the default prompt in `examples/prompts/`.
41
  - `evaluation/`: recorded metrics and validation evidence.
42
 
43
- Use `src/run.py` for packed loading. The inherited `auto_map` for the original dense model has been removed: it cannot load these packed weights. Earlier clean-load reports retain their original pinned revision and paths; they describe the earlier release. Runtime v2 changes numerical execution and sampling defaults; weights are unchanged.
44
 
45
  ## Supported runtime
46
 
47
- **Runtime v2 validated on NVIDIA RTX PRO 6000 Blackwell Server Edition (SM120); the earlier runtime was also validated on RTX 5090. Linux x86_64, glibc ≥ 2.34, Python 3.12, PyTorch 2.10.0 / CUDA 12.8.** Packaged kernels use Python ABI3. Other GPUs and Torch/CUDA combinations require rebuilding and validation; the runner rejects unsupported GPU capability. This is not a generic `AutoModel.from_pretrained` integration.
48
 
49
  ```bash
50
  hf download WaveCut/YuE2-3B-OrbitQuant-W4A4 --local-dir YuE2-W4A4
@@ -72,12 +72,32 @@ The output includes `audio.wav` and generation artifacts. `--prompt` accepts JSO
72
  ```bash
73
  python src/run.py --profile fused --output song-fused
74
  python src/run.py --profile memory --output song-memory
 
 
75
  python src/run.py --profile fast --allocator expandable --output song-fast-expandable
76
  python src/run.py --profile reference --output song-reference
77
  ```
78
 
 
 
79
  `--ode-steps` and `--vae-precision fp32|fp16|bf16` override the profile. VAE checkpoint weights remain FP32 even with FP16 compute. `--low-memory` retains the older unfused reference alternative; it cannot be combined with fast/fused/memory. `--offload-ar` is also available independently. In Python, set `PYTORCH_ALLOC_CONF=expandable_segments:True` **before importing torch** to reproduce the memory CLI allocator. Offloading is sequential; one pipeline instance is not a concurrent request server.
80
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
81
  ## Runtime v2.1: model-lifetime fix
82
 
83
  OrbitQuant 0.9.4 removes strong references that retained discarded quantized layers. In the YuE2 load/fuse/close probe, retained layers fell from 504 to zero and allocated tensor memory after close fell from 2.36 GB to zero. Arithmetic and weights are unchanged. CPU regression and fullgraph checks passed (574 tests / 85 skips in the full local suite); the PyPI release workflow passed.
@@ -146,6 +166,6 @@ Earlier unoptimized packed execution was substantially slower than BF16, illustr
146
 
147
  ## Provenance and license
148
 
149
- Source pins are in `source-lock.json`; runtime changes are in `src/runtime/OPTIMIZATIONS.md`; standalone kernel sources and tests are in `src/kernel-source/`. The runtime includes the verified OrbitQuant 0.9.4 wheel and native kernels 1.0.1; `src/runtime-lock.json` records their provenance. Weights were generated with OrbitQuant 0.9.2, so the original weight source lock and quantization manifest intentionally retain that version. Original licenses and third-party notices are retained.
150
 
151
  Model weights are **CC BY-NC 4.0**, inherited from YuE2. This release does not grant commercial rights. Runtime and kernel components retain their respective code licenses. Credit M-A-P / YuE2 and OrbitQuant when using this derivative.
 
29
 
30
  2,818,572,288 transformer weights occupy 1,409,286,144 packed bytes. The complete model checkpoint is **3,035,926,840 bytes**, versus **7,261,441,640 bytes** upstream: 2.39× smaller (58.2% reduction). The separate 530,512,720-byte VAE checkpoint is unchanged and downloaded at a pinned revision.
31
 
32
+ Compatible Q/K/V and gate/up projections are grouped: 224 physical packed modules represent 392 logical projections without duplicated packed storage. OrbitQuant 0.9.5 with native kernels 1.0.2 provides a byte-pair lookup DP4A W4A4 GEMV for 1–8 rows at input widths 1024–16384 (smaller inputs retain the previous GEMV); larger inputs retain the packed Tensor Core path. No persistent decoded BF16 or INT8 weight caches are used. Original AR CUDA graphs and prefix caching were already present upstream and are not credited as new optimizations.
33
 
34
  The default VAE core is 512 frames with the original 16-frame halo. On the recorded Russian song this was waveform-identical to 1024 frames, while lowering the decoder's isolated memory peak. This is a tested example, not a universal bitwise-equivalence guarantee.
35
 
 
40
  - `examples/`: Russian WAV comparisons and the default prompt in `examples/prompts/`.
41
  - `evaluation/`: recorded metrics and validation evidence.
42
 
43
+ Use `src/run.py` for packed loading. The inherited `auto_map` for the original dense model has been removed: it cannot load these packed weights. Earlier clean-load reports retain their original pinned revision and paths; they describe the earlier release. Runtime v2 changed numerical execution and sampling defaults; v3 adds the byte-pair kernel and optional RMS/activation fusion. Weights are unchanged.
44
 
45
  ## Supported runtime
46
 
47
+ **Runtime v3 validated on NVIDIA RTX PRO 4500 Blackwell (SM120); earlier runtimes were validated on RTX PRO 6000 Blackwell Server Edition and RTX 5090. Linux x86_64, glibc ≥ 2.34, Python 3.12, PyTorch 2.10.0 / CUDA 12.8.** Packaged kernels use Python ABI3. Other GPUs and Torch/CUDA combinations require rebuilding and validation; the runner rejects unsupported GPU capability. This is not a generic `AutoModel.from_pretrained` integration.
48
 
49
  ```bash
50
  hf download WaveCut/YuE2-3B-OrbitQuant-W4A4 --local-dir YuE2-W4A4
 
72
  ```bash
73
  python src/run.py --profile fused --output song-fused
74
  python src/run.py --profile memory --output song-memory
75
+ python src/run.py --profile fast --fuse-rms-quant --output song-rms
76
+ python src/run.py --profile memory --fuse-rms-quant --output song-rms-memory
77
  python src/run.py --profile fast --allocator expandable --output song-fast-expandable
78
  python src/run.py --profile reference --output song-reference
79
  ```
80
 
81
+ `--fuse-rms-quant` is an experimental fast/memory option: it merges RMSNorm into activation quantization, keeps intermediate products in FP32 and can change the sampled arrangement. The flag is recorded in saved configuration and runtime reports. In Python pass `fuse_rms_quant=True` to `load_pipeline`.
82
+
83
  `--ode-steps` and `--vae-precision fp32|fp16|bf16` override the profile. VAE checkpoint weights remain FP32 even with FP16 compute. `--low-memory` retains the older unfused reference alternative; it cannot be combined with fast/fused/memory. `--offload-ar` is also available independently. In Python, set `PYTORCH_ALLOC_CONF=expandable_segments:True` **before importing torch** to reproduce the memory CLI allocator. Offloading is sequential; one pipeline instance is not a concurrent request server.
84
 
85
+ ## Runtime v3: byte-pair decode and optional RMS fusion
86
+
87
+ Native 1.0.2 replaces two nibble lookups with one byte-pair lookup in eligible decode projections. `--fuse-rms-quant` additionally folds RMSNorm into activation quantization, including its epsilon handling. The option is implemented directly in the runtime and supports fast/memory profiles, two-branch CFG, cancellation, and expert staging. It is disabled by default because the changed intermediate rounding can produce a different arrangement.
88
+
89
+ RTX PRO 4500 Blackwell, OrbitQuant 0.9.5, 8 midpoint steps, FP16 VAE, expandable allocator. Two requests per fresh process, warm request shown; loading excluded. Fast with and without RMS fusion generate different arrangements. The two RMS-fused rows have identical semantic tokens, latents, and decoded audio.
90
+
91
+ | Profile | Audio seconds | Warm generation seconds | NVML peak GiB |
92
+ |---|---:|---:|---:|
93
+ | fast | 187.64 | 30.49 | 5.55 |
94
+ | fast + RMS/activation fusion | 183.48 | 28.50 | 5.41 |
95
+ | memory + RMS/activation fusion | 183.48 | 29.82 | 5.33 |
96
+
97
+ The independent interleaved kernel comparison reduced identical 187.64 s audio from 31.82 to 30.34 s. Adding the byte-pair kernel to the same RMS-fused 183.48 s song reduced 29.81 to 28.28 s. Those are separate timing experiments; their pooled-process memory is not compared with this table. The saved 128-position native decode comparison was exact and improved 3.148 to 2.943 ms per token. RMS fusion itself changes rounding: its earlier 128-position probe versus unfused fast had mean KL 0.0266 nats and top-1 agreement 84.38%; whole-song normalized log-band energy differed by 2.76 dB. These are diagnostics, not perceptual scores.
98
+
99
+ Measurements and positive/negative runtime contract checks are in `evaluation/runtime-v3/`. The Russian RMS-fused example is [available here](examples/runtime-v3/fast-rms.wav). Earlier listening milestones and numerical policies remain available below.
100
+
101
  ## Runtime v2.1: model-lifetime fix
102
 
103
  OrbitQuant 0.9.4 removes strong references that retained discarded quantized layers. In the YuE2 load/fuse/close probe, retained layers fell from 504 to zero and allocated tensor memory after close fell from 2.36 GB to zero. Arithmetic and weights are unchanged. CPU regression and fullgraph checks passed (574 tests / 85 skips in the full local suite); the PyPI release workflow passed.
 
166
 
167
  ## Provenance and license
168
 
169
+ Source pins are in `source-lock.json`; runtime changes are in `src/runtime/OPTIMIZATIONS.md`; standalone kernel sources and tests are in `src/kernel-source/`. The runtime includes the verified OrbitQuant 0.9.5 wheel and native kernels 1.0.2; `src/runtime-lock.json` records their provenance. Weights were generated with OrbitQuant 0.9.2, so the original weight source lock and quantization manifest intentionally retain that version. Original licenses and third-party notices are retained.
170
 
171
  Model weights are **CC BY-NC 4.0**, inherited from YuE2. This release does not grant commercial rights. Runtime and kernel components retain their respective code licenses. Credit M-A-P / YuE2 and OrbitQuant when using this derivative.
SHA256SUMS CHANGED
@@ -1,5 +1,5 @@
1
  060985741d20e70613b4c189c7de106cabd3fb2109fbbfb6d705c9d619417dd0 LICENSE
2
- cfabef496dbcf8c10d2518cd218672291decb669d736e072b8987eac930fad8a README.md
3
  14d3fd9f6fee86b4260b69b0979735b99ffec8c6b4db254567047a4215cd9af3 THIRD_PARTY_NOTICES.md
4
  e479ab4f807b93d4c945ae9a14c5269dafb34751ee059d29f9883a5e812f2e56 config.json
5
  5e375ad03c84d8118f3b37e56a3d01cf137b5aa1526b5725d91b17857839616d evaluation/ar-fused-fixed.json
@@ -37,10 +37,15 @@ baa19f27eca20bc767b67057910e10d78f066e9f6e5ed45df639f0ec9678552d evaluation/run
37
  142bb2808102bbd36a5266d7d28b2173f3522a514081957600be06d842339530 evaluation/runtime-v2/pro4500-reference.json
38
  96a39130f0b58355a872edb608713bc57f65e7b0ef921081b022f14b27bc4df3 evaluation/runtime-v2/reference.json
39
  a1badc0e42720b32b8259a55cbfee8a9e2d9537296ee517ed6c575b432291d29 evaluation/runtime-v2/spectra.json
 
 
 
 
40
  00da89c0d8a7242efb498fc30f221fab25b0becc127a61079a528d333be229df evaluation/vae-chunk-benchmark.json
41
  6daa417ffd2972c8a7f681083ca59f131a03b9f5caf62871a9f37ec30905179a examples/orbitquant-ru.wav
42
  4879336415f868fe92e0d0d21733dea194c09f8deb10484c27add434950b4bb6 examples/original-ru.wav
43
  a67f3becd1edcc3c42c2602735b47d455bf76cf1258862a3fe5978cf87fb16b0 examples/prompts/default-ru.json
 
44
  ccc842772f029049fadf9b288393bcdd9f79271b361c0f1875ec1218e4ba5170 generation_config.json
45
  da9858d516047d82096d01c112a61bd67f26d289039464d668a1d45f91738ecc licenses/SnakeBeta-NVIDIA-MIT.txt
46
  a1fac33b7bcd791b74fb33aeb439f825e7277e239fc119fb7d2ab6f084a0c101 licenses/stable-audio-tools-MIT.txt
@@ -48,7 +53,7 @@ e169774b8e79f423e6e8586221558acd28a35bacc67aae912d9a8a905231134a model.safetens
48
  c9fe5177b33b8727cdf97d946ba173645a490e8f30181c55a3f172b9ed852979 orbitquant_manifest.json
49
  b2b1b8dfb5cc5f024bafc373121c6aba3f66f9a5a0269e243470a1de16a33186 qwen.tiktoken
50
  bd90ffcd3768e450363a5b583cb6153966417f818b08427d0e8f180b1d33944a source-lock.json
51
- c3ab9fb9bbbe770da9a96e661bc3f1f64d79d0924eb0d7de142fde89ab6e10e1 src/OPTIMIZATION_IDEAS.md
52
  49a3cfbb640f0c2dc098630907db27f5e60b7fb810ba3d1d31dcff98ac5579c8 src/evaluation_harness/ar_bench.py
53
  7faa379989e0a77c28be169a1ea91a0f4b960c7f1d949479a8b18511083179c4 src/evaluation_harness/ar_quality.py
54
  863e3137fb6a429c7439208484aa79dcfc1fc81fa3286f4dfe39301bcd165fd0 src/evaluation_harness/asr_evaluate.py
@@ -94,7 +99,7 @@ d0193d3790e87ca7e4bd7b3afa2eb399a6507150273dbce54e548829520d811c src/kernel-sou
94
  4de6806c3d18b3b6421ed8a4b3e50e1308e9963721a4b93ea30a8419a84c0e79 src/kernel-source/orbitquant-gemv/torch-ext/registration.h
95
  4228ded5775db7e8c4071a412e8b87575bd02c5bc2197009fcb08734cbffd134 src/kernel-source/orbitquant-gemv/torch-ext/torch_binding.cpp
96
  ad6d0f6601ee0e2e126164fad6e4ac572fc5820544ce71f79e2ba152bb08e7a1 src/kernel-source/orbitquant-gemv/torch-ext/torch_binding.h
97
- 5da634084d0d7d67de0a6228b4750bbea6416ec2cd2d031dca46e40a84ce9efa src/kernel-source/orbitquant-packed-matmul/CARD.md
98
  7a9ffddcd684ec325c9ef87b3ebdb2fdbbb0c2747dfc6e1d63144a920252b565 src/kernel-source/orbitquant-packed-matmul/benchmarks/benchmark.py
99
  1e51ee864385d2116ca5329c8b31091c8bdb81f9446094e9a92121e0b2390842 src/kernel-source/orbitquant-packed-matmul/benchmarks/benchmark_decode.py
100
  59f8eafa81f046711b70b2340b48c6b9be1775353feae95e4310f62331dee2be src/kernel-source/orbitquant-packed-matmul/build.toml
@@ -116,16 +121,52 @@ dcf9baa1b69fbf4bf3621ee4037bab9d8002fdabf3802f8a22c3636dcb9d4c3d src/kernel-sou
116
  bf91916d187fc1c6f387029a354d57379297384b8e414deab074858a284b274d src/kernel-source/orbitquant-packed-matmul/orbitquant_packed_matmul_cpu/packed_matmul_x86.cpp
117
  a90525ad3c2c30fe12cd4266d50c3ac391b2861ea37e885be0da3cb599df7eb0 src/kernel-source/orbitquant-packed-matmul/orbitquant_packed_matmul_cpu/packed_matmul_x86_avx512.cpp
118
  0781a49f03f263d384fc1e66a3eb5ed196c25373d2811b7880b4496662ea048c src/kernel-source/orbitquant-packed-matmul/orbitquant_packed_matmul_cpu/quantize_activations_cpu.cpp
119
- 6cf3c0a692a436fb69556a43571de6fd61c467d60536fffb91daf8417db4b54e src/kernel-source/orbitquant-packed-matmul/orbitquant_packed_matmul_cuda/packed_matmul.cu
120
  02d02f3b8c271507fd63de99609b155e201a089f476a3e4a4338bc568cebdae8 src/kernel-source/orbitquant-packed-matmul/orbitquant_packed_matmul_metal/packed_matmul.metal
121
  4e600bb69569f831f0cab0e4298ec82a19e9a4406dc3d77bb1b6d11413cdb5dc src/kernel-source/orbitquant-packed-matmul/orbitquant_packed_matmul_metal/packed_matmul.mm
122
  33b4a2c75b1d34f0810274230455e137ff920e0f90a95009a3d0ce5bd681126d src/kernel-source/orbitquant-packed-matmul/scripts/prepare_wheel_project.py
123
  32aa8cb8f2c2cdd29c68e70413227d96d15df6688c3e8bf8028c60c99e20d33b src/kernel-source/orbitquant-packed-matmul/tests/__init__.py
124
- ef7d7f87dca9a4d94850a3ff5c918a34d644fd66e5b4d360c72cf53918d12cc4 src/kernel-source/orbitquant-packed-matmul/tests/test_decode_gemv.py
125
  7e192d41a6a93f1b63f395c21383c7722a0c69d90c696629a57a2c5aeaf6794f src/kernel-source/orbitquant-packed-matmul/tests/test_packed_matmul.py
126
  414ecd0cbfc1a6bc66e0a6e2aeaa1bb3ac433d0172099af7071806bb41234950 src/kernel-source/orbitquant-packed-matmul/torch-ext/orbitquant_packed_matmul/__init__.py
127
  8782bd8bef9d52708606053d1e85bf44c032bec14cf22beeed420e26dcd110bf src/kernel-source/orbitquant-packed-matmul/torch-ext/torch_binding.cpp
128
  8af7352374d8694a4870c89d6998ae8df4fe34ccb2a08b261fe4e24913da251f src/kernel-source/orbitquant-packed-matmul/torch-ext/torch_binding.h
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
129
  9dad1cdeac2b3861bf7cce889ec8ae50133999eb76a93a7bfd8528ca7a5364c9 src/kernel-source/yue2-fast/.gitattributes
130
  44575cf5b28512d75644bf54a517dcef304ff809fd511747621b4d64f19aac66 src/kernel-source/yue2-fast/.gitignore
131
  89de5c024ac81e58a2f9e2a2484f424d6823fee075c4403e29e279e04e07c88c src/kernel-source/yue2-fast/CARD.md
@@ -164,20 +205,25 @@ da90c2dc55ecca5ffd2f7b15d1968180d5744a5b31ade45a7d221041096bdebe src/kernel-sou
164
  ac54a84ce2bb7a0742a1cd46df398aeb23d83ebde9ed8c7cc0f7d914b59837dd src/kernel-source/yue2-fast/torch-ext/torch_binding.h
165
  a368cb32d65829479d15c19451fe3199cf7a2d6510891a9596aa08fc4ed746dd src/kernel-source/yue2-fast/torch-ext/yue2_qkv_fused/__init__.py
166
  62d2bd758c8fff1a0d64584a699343076fca9a7880db5d8e19263bc30ae79d7c src/modeling_yue2.py
167
- c320d02effd7dcd8f2eb24cb368b12e05af2330776d3a9283f16d30d47ecd726 src/requirements.txt
168
- 19df91c2f001546a9b5a60b4ba66bf4de07b3aef2a9fce9fdd20513fae4f3112 src/run.py
169
- 1762d6192e76afcd9379dc22f965bbd0e4618f89858a92f311ba6b1d44ba26d1 src/runtime-lock.json
170
- 13ec94ceac874a44049edc45fb36090381546f66fb616b4e77ab994d0e055b84 src/runtime/OPTIMIZATIONS.md
171
- cf3d652addc2688f530f7dd8ac769d52674f502a7ef1219ec9e792a4f3d5bbcd src/runtime/fast_ops.py
172
  026e36da4a42cf2095b75ebacb250cf8e20f5011763b2f0e44247afabdb9d112 src/runtime/fuse_packed.py
173
  414ecd0cbfc1a6bc66e0a6e2aeaa1bb3ac433d0172099af7071806bb41234950 src/runtime/kernels/base/orbitquant_packed_matmul/__init__.py
174
  c1df0cc119cdfe0471754469458fa83fc718c6f42feca0b79a0d07cb2c068ce5 src/runtime/kernels/base/orbitquant_packed_matmul/_ops.py
175
- 780a552256adc3363c73124a6ffff969666559eb2266d99c6ec72b2495af6d5b src/runtime/kernels/base/orbitquant_packed_matmul/_orbitquant_packed_matmul_cuda_rel_cu128_t210.abi3.so
176
  a368cb32d65829479d15c19451fe3199cf7a2d6510891a9596aa08fc4ed746dd src/runtime/kernels/fast/torch210-cxx11-cu128-x86_64-linux/__init__.py
177
  44aec945e18569118863f1eda191d273eabea41e5b6589771e5104eab3936d5c src/runtime/kernels/fast/torch210-cxx11-cu128-x86_64-linux/_ops.py
178
  b0a4d667dc1e0b1701f0bf9371254c293e30e974892e5681341980c9fe4d00ef src/runtime/kernels/fast/torch210-cxx11-cu128-x86_64-linux/_yue2_qkv_fused_cuda_46ceaaf_dirty.abi3.so
179
  0a68364ee298078b690fc2360e714d05da28fb932bf3ad6eb0c9c5b3e1bb7b24 src/runtime/kernels/fast/torch210-cxx11-cu128-x86_64-linux/metadata.json
180
  0c560f96b857c188c4a8297ff27d1299619957c3459a5e433e132329fbfdf066 src/runtime/kernels/fast/torch210-cxx11-cu128-x86_64-linux/yue2_qkv_fused/__init__.py
 
 
 
 
 
181
  060985741d20e70613b4c189c7de106cabd3fb2109fbbfb6d705c9d619417dd0 src/runtime/licenses/LICENSE
182
  14d3fd9f6fee86b4260b69b0979735b99ffec8c6b4db254567047a4215cd9af3 src/runtime/licenses/THIRD_PARTY_NOTICES.md
183
  da9858d516047d82096d01c112a61bd67f26d289039464d668a1d45f91738ecc src/runtime/licenses/licenses/SnakeBeta-NVIDIA-MIT.txt
@@ -185,7 +231,7 @@ a1fac33b7bcd791b74fb33aeb439f825e7277e239fc119fb7d2ab6f084a0c101 src/runtime/li
185
  e6a6b3c798f697ee3011e6b817d0620ca79824c823e56c6c0748889c0a5b4407 src/runtime/yue2/__init__.py
186
  3a25793081ed10325225e4e7bd9303243998abecd3a1a5b7a15139d664b70604 src/runtime/yue2/cli.py
187
  617ee6914d534a234f1ff5d5f4685391eac5dc9f6be667429949eabf1e541517 src/runtime/yue2/compact_sampling.py
188
- bc88a5a3a56ac614704f2359754157cc809d5ad60a5d85c990ca9e5dd1715263 src/runtime/yue2/cuda_graph.py
189
  e1722959ad0b0bf27eeaaf019cb9f109212a2450f03dd145866b1a0239151f38 src/runtime/yue2/fast.py
190
  e6523f113132df6c04878103bca1a8b3de519fefcbc0117e97455f0dd49d1fba src/runtime/yue2/modeling_vae.py
191
  62d2bd758c8fff1a0d64584a699343076fca9a7880db5d8e19263bc30ae79d7c src/runtime/yue2/modeling_yue2.py
@@ -197,7 +243,7 @@ e00d32bf22baac46c7d4dafaeecd4f3d5a4753d651b3dc8f4164b9f3774980ad src/runtime/yu
197
  63731de35665ac426130c781f3ba7ff1159c58d34a9b3e6ee2e78ff0c5881ff9 src/runtime/yue2/sampling.py
198
  a5bcc181b16412b45bc560e5264ba0b303667b5fd81957991ac0266c87fe2a5d src/runtime/yue2/storage.py
199
  fc88387f96e5db2020a0c3f5374b82726206026d83159b68b83e4136bb741acc src/runtime/yue2/tokenization_yue2.py
200
- 382ca21b78d2777b13a91769d2dd9a666b7de1ee60bd226397059c354e6cc815 src/runtime/yue2_orbit.py
201
- 7c14da5108938656a1d8a6143a32647f35a899b754355215282365c0a2532733 src/wheels/orbitquant-0.9.4-py3-none-any.whl
202
  46ce45128d77c97565aac511de8f000c098cc978b2694304c8475cea8b01a1d1 weights_manifest.json
203
  203830cebde6e3644eb291925362990d198e66c3b6006cd11f1aaa21904bcc61 yue2_generation_config.json
 
1
  060985741d20e70613b4c189c7de106cabd3fb2109fbbfb6d705c9d619417dd0 LICENSE
2
+ 09ebd5b472d0ee3b2e4ceda1bbb8169dee42196ea31a84a39dc4a0c6b5e54105 README.md
3
  14d3fd9f6fee86b4260b69b0979735b99ffec8c6b4db254567047a4215cd9af3 THIRD_PARTY_NOTICES.md
4
  e479ab4f807b93d4c945ae9a14c5269dafb34751ee059d29f9883a5e812f2e56 config.json
5
  5e375ad03c84d8118f3b37e56a3d01cf137b5aa1526b5725d91b17857839616d evaluation/ar-fused-fixed.json
 
37
  142bb2808102bbd36a5266d7d28b2173f3522a514081957600be06d842339530 evaluation/runtime-v2/pro4500-reference.json
38
  96a39130f0b58355a872edb608713bc57f65e7b0ef921081b022f14b27bc4df3 evaluation/runtime-v2/reference.json
39
  a1badc0e42720b32b8259a55cbfee8a9e2d9537296ee517ed6c575b432291d29 evaluation/runtime-v2/spectra.json
40
+ d27cd1f8f8c5b8dc667e5bedb7c5a4ac1c72aae8cac83c25b794caba544b58c9 evaluation/runtime-v3/contract-checks.json
41
+ 04f7ce93db3d265c80fe5302f8cd7bf8151deb02e51ff9c7096cbeb7273ab64f evaluation/runtime-v3/fast-rms0-s8.json
42
+ a9a2dc4587cf5c842b4eb48f1ef1abe655109426c0b9ae042e30871a0d49048f evaluation/runtime-v3/fast-rms1-s8.json
43
+ 370cc04e8caf6fb345197e9e76161452abe62d077a1388dde6385472b5935276 evaluation/runtime-v3/memory-rms1-s8.json
44
  00da89c0d8a7242efb498fc30f221fab25b0becc127a61079a528d333be229df evaluation/vae-chunk-benchmark.json
45
  6daa417ffd2972c8a7f681083ca59f131a03b9f5caf62871a9f37ec30905179a examples/orbitquant-ru.wav
46
  4879336415f868fe92e0d0d21733dea194c09f8deb10484c27add434950b4bb6 examples/original-ru.wav
47
  a67f3becd1edcc3c42c2602735b47d455bf76cf1258862a3fe5978cf87fb16b0 examples/prompts/default-ru.json
48
+ 30a70e2224b008e08f10ba0e0f9b8243b06b32a4fcbd7381e980d68cd5a7974e examples/runtime-v3/fast-rms.wav
49
  ccc842772f029049fadf9b288393bcdd9f79271b361c0f1875ec1218e4ba5170 generation_config.json
50
  da9858d516047d82096d01c112a61bd67f26d289039464d668a1d45f91738ecc licenses/SnakeBeta-NVIDIA-MIT.txt
51
  a1fac33b7bcd791b74fb33aeb439f825e7277e239fc119fb7d2ab6f084a0c101 licenses/stable-audio-tools-MIT.txt
 
53
  c9fe5177b33b8727cdf97d946ba173645a490e8f30181c55a3f172b9ed852979 orbitquant_manifest.json
54
  b2b1b8dfb5cc5f024bafc373121c6aba3f66f9a5a0269e243470a1de16a33186 qwen.tiktoken
55
  bd90ffcd3768e450363a5b583cb6153966417f818b08427d0e8f180b1d33944a source-lock.json
56
+ 331c06a248811f778f9f5ed280101e86bc0dd580e80db85619893148ff885720 src/OPTIMIZATION_IDEAS.md
57
  49a3cfbb640f0c2dc098630907db27f5e60b7fb810ba3d1d31dcff98ac5579c8 src/evaluation_harness/ar_bench.py
58
  7faa379989e0a77c28be169a1ea91a0f4b960c7f1d949479a8b18511083179c4 src/evaluation_harness/ar_quality.py
59
  863e3137fb6a429c7439208484aa79dcfc1fc81fa3286f4dfe39301bcd165fd0 src/evaluation_harness/asr_evaluate.py
 
99
  4de6806c3d18b3b6421ed8a4b3e50e1308e9963721a4b93ea30a8419a84c0e79 src/kernel-source/orbitquant-gemv/torch-ext/registration.h
100
  4228ded5775db7e8c4071a412e8b87575bd02c5bc2197009fcb08734cbffd134 src/kernel-source/orbitquant-gemv/torch-ext/torch_binding.cpp
101
  ad6d0f6601ee0e2e126164fad6e4ac572fc5820544ce71f79e2ba152bb08e7a1 src/kernel-source/orbitquant-gemv/torch-ext/torch_binding.h
102
+ ed46fd7deb251b44cda8b238e0b4dc44b9b0f84c692043e82cd5e4b58cd5a6ca src/kernel-source/orbitquant-packed-matmul/CARD.md
103
  7a9ffddcd684ec325c9ef87b3ebdb2fdbbb0c2747dfc6e1d63144a920252b565 src/kernel-source/orbitquant-packed-matmul/benchmarks/benchmark.py
104
  1e51ee864385d2116ca5329c8b31091c8bdb81f9446094e9a92121e0b2390842 src/kernel-source/orbitquant-packed-matmul/benchmarks/benchmark_decode.py
105
  59f8eafa81f046711b70b2340b48c6b9be1775353feae95e4310f62331dee2be src/kernel-source/orbitquant-packed-matmul/build.toml
 
121
  bf91916d187fc1c6f387029a354d57379297384b8e414deab074858a284b274d src/kernel-source/orbitquant-packed-matmul/orbitquant_packed_matmul_cpu/packed_matmul_x86.cpp
122
  a90525ad3c2c30fe12cd4266d50c3ac391b2861ea37e885be0da3cb599df7eb0 src/kernel-source/orbitquant-packed-matmul/orbitquant_packed_matmul_cpu/packed_matmul_x86_avx512.cpp
123
  0781a49f03f263d384fc1e66a3eb5ed196c25373d2811b7880b4496662ea048c src/kernel-source/orbitquant-packed-matmul/orbitquant_packed_matmul_cpu/quantize_activations_cpu.cpp
124
+ 4036bdf8916a85aa623e90522cb54833c32a0ee25312fd023679d9802beb77cc src/kernel-source/orbitquant-packed-matmul/orbitquant_packed_matmul_cuda/packed_matmul.cu
125
  02d02f3b8c271507fd63de99609b155e201a089f476a3e4a4338bc568cebdae8 src/kernel-source/orbitquant-packed-matmul/orbitquant_packed_matmul_metal/packed_matmul.metal
126
  4e600bb69569f831f0cab0e4298ec82a19e9a4406dc3d77bb1b6d11413cdb5dc src/kernel-source/orbitquant-packed-matmul/orbitquant_packed_matmul_metal/packed_matmul.mm
127
  33b4a2c75b1d34f0810274230455e137ff920e0f90a95009a3d0ce5bd681126d src/kernel-source/orbitquant-packed-matmul/scripts/prepare_wheel_project.py
128
  32aa8cb8f2c2cdd29c68e70413227d96d15df6688c3e8bf8028c60c99e20d33b src/kernel-source/orbitquant-packed-matmul/tests/__init__.py
129
+ 2ec1604258719eb1388496249b1c7cb59f6a53e8501189f10d4374c3e9700043 src/kernel-source/orbitquant-packed-matmul/tests/test_decode_gemv.py
130
  7e192d41a6a93f1b63f395c21383c7722a0c69d90c696629a57a2c5aeaf6794f src/kernel-source/orbitquant-packed-matmul/tests/test_packed_matmul.py
131
  414ecd0cbfc1a6bc66e0a6e2aeaa1bb3ac433d0172099af7071806bb41234950 src/kernel-source/orbitquant-packed-matmul/torch-ext/orbitquant_packed_matmul/__init__.py
132
  8782bd8bef9d52708606053d1e85bf44c032bec14cf22beeed420e26dcd110bf src/kernel-source/orbitquant-packed-matmul/torch-ext/torch_binding.cpp
133
  8af7352374d8694a4870c89d6998ae8df4fe34ccb2a08b261fe4e24913da251f src/kernel-source/orbitquant-packed-matmul/torch-ext/torch_binding.h
134
+ 9dad1cdeac2b3861bf7cce889ec8ae50133999eb76a93a7bfd8528ca7a5364c9 src/kernel-source/rms-quant/.gitattributes
135
+ 44575cf5b28512d75644bf54a517dcef304ff809fd511747621b4d64f19aac66 src/kernel-source/rms-quant/.gitignore
136
+ 163420273229ab3481f39d2f35984e0e93ca88848bb09595ab46404e4f8edfcb src/kernel-source/rms-quant/CARD.md
137
+ 85ed8c1d9e3e9e5b0b61e638d18898fb71a5624d34be61b72f49a64020f6e959 src/kernel-source/rms-quant/build.toml
138
+ 933e544bcbad82151547591748831765cc21b6d55564055989859ccbe905a0d7 src/kernel-source/rms-quant/cmake/_ops.py.in
139
+ f2ac9825685c35b0c96cd47f9b9bc7790754f16e3236ad558f97d0d1f9ff74de src/kernel-source/rms-quant/cmake/add_build_metadata.py
140
+ c6a0ea1116893886ab52b4559821d4e42c8830ed644980e58f22ac58b3cce80a src/kernel-source/rms-quant/cmake/build-variants.cmake
141
+ 3c4a3ab628be03ff2be0771b43281e4f8e0c01bfd1530c7ca18967cbd22ac285 src/kernel-source/rms-quant/cmake/compile-metal.cmake
142
+ 28f11b0b855fe9ea47b2a5b46bf50556388380b91396e1007a2302c4908b01be src/kernel-source/rms-quant/cmake/get_gpu_lang.cmake
143
+ 2f8fb8e6d51d8265b34ce54a630bd9154839ccc3ba331e57634b996589bf4739 src/kernel-source/rms-quant/cmake/get_gpu_lang.py
144
+ 5cfc2220d20f0587327f37ca27e0d74cdbdea2540ce6ae2032569fb995b79ac8 src/kernel-source/rms-quant/cmake/hipify.py
145
+ 20d526820f9e4867bae370c55280bc8e853908658fe6caff29dab7101e94f46c src/kernel-source/rms-quant/cmake/kernel.cmake
146
+ 0defa64521b9093e4509eadd3059e609cc04dfab4b085426d7e7f1770971f0eb src/kernel-source/rms-quant/cmake/metallib_to_header.py
147
+ 757019d6d92eb0c09c26d43b9a83b1c8838ce5392b78f47388cfd81c235fca08 src/kernel-source/rms-quant/cmake/utils.cmake
148
+ 0c560f96b857c188c4a8297ff27d1299619957c3459a5e433e132329fbfdf066 src/kernel-source/rms-quant/compat.py
149
+ bc6a259827061020c2f5ae9a5dc1e947bb0c81abe81dec1abf507b15270dfa16 src/kernel-source/rms-quant/flake.nix
150
+ 1d1d6efe831924e14fd729ec59035c9bcc9b65ce9497f2c99219ab9bea71ad3d src/kernel-source/rms-quant/metadata-cann.json
151
+ 03e00877b6f8b923c2162dbd564089f694047d9ebdb5dadea81bd8f5e0140617 src/kernel-source/rms-quant/metadata-cpu.json
152
+ 13e39897751174039c2f62a6013f9ddd605596e659f29d2547388bb79fdfb34b src/kernel-source/rms-quant/metadata-cuda.json
153
+ 7932dba1c62fbd904ad0dd2bd2ead57a03f050eaca5c0b5cbc9e1d76f6c9ae0d src/kernel-source/rms-quant/metadata-metal.json
154
+ 0acdd9dfef658f58dd087fbe95a0886ae4e8495ba0ba666eb5365e36518ad591 src/kernel-source/rms-quant/metadata-neuron.json
155
+ 99234c78d40650bfcc9c8e430b3857cc769f06473dee850a943baaa182dc5080 src/kernel-source/rms-quant/metadata-rocm.json
156
+ 314e0552e15b459f9ca3e31029ab73ccfa6efa4188cc0c35e8ec5aa8af7a2a4e src/kernel-source/rms-quant/metadata-xpu.json
157
+ 6128f79d7ec43624eff68f40f13037f4c261ae748ddb6882abf7a5b7841cd82f src/kernel-source/rms-quant/orbitquant_gemv_cuda/fused_ops.cu
158
+ b930e00662b3e0f2ad381f8769f86c750761c3ce476eb7ae42671c50eb43f473 src/kernel-source/rms-quant/orbitquant_gemv_cuda/orbitquant_gemv.cu
159
+ 062ff955c58c2efa111c611cbd733afd259e138f862a56255e5d5f8b45f90de5 src/kernel-source/rms-quant/orbitquant_gemv_cuda/qkv_decode.cu
160
+ 92e9a8e1c80976451255939b190cf8032a12e9ce36c2c2d224ddbda7edb2cd8d src/kernel-source/rms-quant/orbitquant_gemv_cuda/rmsquant.cu
161
+ e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 src/kernel-source/rms-quant/tests/__init__.py
162
+ 6760124522ecb74b0e7ae5a63d0d5094ff4c98f1de48fc69dfa6548f22f2eff1 src/kernel-source/rms-quant/tests/test_fused_ops.py
163
+ b2275e6926b515bebf8ae4de0adeaf10a84c40e1441b10430541a68323c51a9e src/kernel-source/rms-quant/tests/test_orbitquant_gemv.py
164
+ aac2795a2000382e177737078a5698abb9bafd83fbae2a56c858d6b2ca5d3f95 src/kernel-source/rms-quant/tests/test_qkv_decode.py
165
+ 608939f6f4cb359018d2ea49a1312c5f9dd979b8b5c348a341550347b0c756c8 src/kernel-source/rms-quant/tests/test_rmsquant.py
166
+ 4de6806c3d18b3b6421ed8a4b3e50e1308e9963721a4b93ea30a8419a84c0e79 src/kernel-source/rms-quant/torch-ext/registration.h
167
+ 5b24f09a78fa634db9963ba96178fb6f2b99085cfff1f6c35b5a6209e1a720d9 src/kernel-source/rms-quant/torch-ext/torch_binding.cpp
168
+ 0b95faa0b766fba75859b66c4121638a372df28807207b318cb4a5833506d5e4 src/kernel-source/rms-quant/torch-ext/torch_binding.h
169
+ 02e9eb8b6490f31c840fbaa70371189882303141f52d2e47043499dcf4f5be38 src/kernel-source/rms-quant/torch-ext/yue2_qkv_fused/__init__.py
170
  9dad1cdeac2b3861bf7cce889ec8ae50133999eb76a93a7bfd8528ca7a5364c9 src/kernel-source/yue2-fast/.gitattributes
171
  44575cf5b28512d75644bf54a517dcef304ff809fd511747621b4d64f19aac66 src/kernel-source/yue2-fast/.gitignore
172
  89de5c024ac81e58a2f9e2a2484f424d6823fee075c4403e29e279e04e07c88c src/kernel-source/yue2-fast/CARD.md
 
205
  ac54a84ce2bb7a0742a1cd46df398aeb23d83ebde9ed8c7cc0f7d914b59837dd src/kernel-source/yue2-fast/torch-ext/torch_binding.h
206
  a368cb32d65829479d15c19451fe3199cf7a2d6510891a9596aa08fc4ed746dd src/kernel-source/yue2-fast/torch-ext/yue2_qkv_fused/__init__.py
207
  62d2bd758c8fff1a0d64584a699343076fca9a7880db5d8e19263bc30ae79d7c src/modeling_yue2.py
208
+ 0c0e34419a8d8c3f95cfb3113f2453616d8a906c67ee919cad4f47be127ee939 src/requirements.txt
209
+ 8f9c3e42202c02e33dcaa25ef4090ef5d8f8e53f4747f3757104091960893c21 src/run.py
210
+ 919ee3f67cf15f6bc2cdb9b7e8eb2a9387fe1401dd452758b56d980fa5516c80 src/runtime-lock.json
211
+ fd1b581582dfe3e2503e7f7625d807129c90135692bddeb411bda559a05becee src/runtime/OPTIMIZATIONS.md
212
+ e70020c928aee0ba295d3aa71abb9d1331c9cc28898c88f66f1dbe7ffe41cb6b src/runtime/fast_ops.py
213
  026e36da4a42cf2095b75ebacb250cf8e20f5011763b2f0e44247afabdb9d112 src/runtime/fuse_packed.py
214
  414ecd0cbfc1a6bc66e0a6e2aeaa1bb3ac433d0172099af7071806bb41234950 src/runtime/kernels/base/orbitquant_packed_matmul/__init__.py
215
  c1df0cc119cdfe0471754469458fa83fc718c6f42feca0b79a0d07cb2c068ce5 src/runtime/kernels/base/orbitquant_packed_matmul/_ops.py
216
+ 96839947a05b59c11a781346184fc344ed6af96404081ba1e8b9ad5fa29d828a src/runtime/kernels/base/orbitquant_packed_matmul/_orbitquant_packed_matmul_cuda_rel_cu128_t210.abi3.so
217
  a368cb32d65829479d15c19451fe3199cf7a2d6510891a9596aa08fc4ed746dd src/runtime/kernels/fast/torch210-cxx11-cu128-x86_64-linux/__init__.py
218
  44aec945e18569118863f1eda191d273eabea41e5b6589771e5104eab3936d5c src/runtime/kernels/fast/torch210-cxx11-cu128-x86_64-linux/_ops.py
219
  b0a4d667dc1e0b1701f0bf9371254c293e30e974892e5681341980c9fe4d00ef src/runtime/kernels/fast/torch210-cxx11-cu128-x86_64-linux/_yue2_qkv_fused_cuda_46ceaaf_dirty.abi3.so
220
  0a68364ee298078b690fc2360e714d05da28fb932bf3ad6eb0c9c5b3e1bb7b24 src/runtime/kernels/fast/torch210-cxx11-cu128-x86_64-linux/metadata.json
221
  0c560f96b857c188c4a8297ff27d1299619957c3459a5e433e132329fbfdf066 src/runtime/kernels/fast/torch210-cxx11-cu128-x86_64-linux/yue2_qkv_fused/__init__.py
222
+ 02e9eb8b6490f31c840fbaa70371189882303141f52d2e47043499dcf4f5be38 src/runtime/kernels/rms/torch210-cxx11-cu128-x86_64-linux/__init__.py
223
+ 1c4350939f2e52581663e8694cb869926bf2dcc04beb80b9c09233571dd1695f src/runtime/kernels/rms/torch210-cxx11-cu128-x86_64-linux/_ops.py
224
+ 1ca79342c379a231d86fa5c8e94e45134c3624d7b85ca5e55997004ec69ff542 src/runtime/kernels/rms/torch210-cxx11-cu128-x86_64-linux/_yue2_qkv_fused_cuda_cf0db58.abi3.so
225
+ 40e4a06eb58f41ef286f95a434bfb06d10742387e9cb47219d2d6b088a59bf3d src/runtime/kernels/rms/torch210-cxx11-cu128-x86_64-linux/metadata.json
226
+ 0c560f96b857c188c4a8297ff27d1299619957c3459a5e433e132329fbfdf066 src/runtime/kernels/rms/torch210-cxx11-cu128-x86_64-linux/yue2_qkv_fused/__init__.py
227
  060985741d20e70613b4c189c7de106cabd3fb2109fbbfb6d705c9d619417dd0 src/runtime/licenses/LICENSE
228
  14d3fd9f6fee86b4260b69b0979735b99ffec8c6b4db254567047a4215cd9af3 src/runtime/licenses/THIRD_PARTY_NOTICES.md
229
  da9858d516047d82096d01c112a61bd67f26d289039464d668a1d45f91738ecc src/runtime/licenses/licenses/SnakeBeta-NVIDIA-MIT.txt
 
231
  e6a6b3c798f697ee3011e6b817d0620ca79824c823e56c6c0748889c0a5b4407 src/runtime/yue2/__init__.py
232
  3a25793081ed10325225e4e7bd9303243998abecd3a1a5b7a15139d664b70604 src/runtime/yue2/cli.py
233
  617ee6914d534a234f1ff5d5f4685391eac5dc9f6be667429949eabf1e541517 src/runtime/yue2/compact_sampling.py
234
+ 20f1764f1c9298fcff0411e755b99fc9a040d1819079461e527cfe1b20f03017 src/runtime/yue2/cuda_graph.py
235
  e1722959ad0b0bf27eeaaf019cb9f109212a2450f03dd145866b1a0239151f38 src/runtime/yue2/fast.py
236
  e6523f113132df6c04878103bca1a8b3de519fefcbc0117e97455f0dd49d1fba src/runtime/yue2/modeling_vae.py
237
  62d2bd758c8fff1a0d64584a699343076fca9a7880db5d8e19263bc30ae79d7c src/runtime/yue2/modeling_yue2.py
 
243
  63731de35665ac426130c781f3ba7ff1159c58d34a9b3e6ee2e78ff0c5881ff9 src/runtime/yue2/sampling.py
244
  a5bcc181b16412b45bc560e5264ba0b303667b5fd81957991ac0266c87fe2a5d src/runtime/yue2/storage.py
245
  fc88387f96e5db2020a0c3f5374b82726206026d83159b68b83e4136bb741acc src/runtime/yue2/tokenization_yue2.py
246
+ 5f314395707eff25911314e7d1dabac7deeec96097dbece8efd929043820836b src/runtime/yue2_orbit.py
247
+ fe3f929910881e2fe0d9cfad5329e2c904441aa947bfa8d4494df7cbc97ac187 src/wheels/orbitquant-0.9.5-py3-none-any.whl
248
  46ce45128d77c97565aac511de8f000c098cc978b2694304c8475cea8b01a1d1 weights_manifest.json
249
  203830cebde6e3644eb291925362990d198e66c3b6006cd11f1aaa21904bcc61 yue2_generation_config.json
evaluation/runtime-v3/contract-checks.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"status": "passed", "checks": ["invalid profiles and overrides", "CFG two branches", "greedy min/max budget", "compact stochastic legal vocabulary", "cancellation cleanup and next request", "expert staging exception restoration"]}
evaluation/runtime-v3/fast-rms0-s8.json ADDED
@@ -0,0 +1,304 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [
2
+ {
3
+ "profile": "fast",
4
+ "fuse_rms_quant": false,
5
+ "repeat": 0,
6
+ "wall_seconds": 32.138701032847166,
7
+ "audio_seconds": 187.63866666666667,
8
+ "nvml_peak": 5757927424,
9
+ "torch_allocated_peak": 4513181184,
10
+ "timing": {
11
+ "abc": {
12
+ "seconds": 7.440518390387297,
13
+ "prefill_seconds": 0.8290309868752956,
14
+ "ttft_seconds": 0.9952553361654282,
15
+ "output_tokens": 1629,
16
+ "content_tokens": 1628,
17
+ "output_tps": 218.93635826565125,
18
+ "prefix_tokens": 423,
19
+ "cfg_branches": 1,
20
+ "execution": "cuda_graph",
21
+ "attention": "flash"
22
+ },
23
+ "semantic": {
24
+ "seconds": 17.255344545468688,
25
+ "prefill_seconds": 0.17433544248342514,
26
+ "ttft_seconds": 0.17487437278032303,
27
+ "output_tokens": 4692,
28
+ "content_tokens": 4691,
29
+ "output_tps": 271.91575268962885,
30
+ "prefix_tokens": 2053,
31
+ "cfg_branches": 1,
32
+ "execution": "cuda_graph",
33
+ "attention": "flash"
34
+ },
35
+ "nar_seconds": 3.986683363094926,
36
+ "vae_seconds": 3.4521421883255243,
37
+ "load": {
38
+ "resolve_and_integrity_seconds": 2.3701207600533962,
39
+ "mot_load_seconds": 0.8900818899273872
40
+ },
41
+ "e2e_seconds": 32.137739622965455
42
+ },
43
+ "semantic_sha256": "2cadb2f7623adcfe95baf6a8febb0532bae42e8fe1fe716d885c4a36790bd4f6",
44
+ "waveform_sha256": "102fb2bd604da4cfa44320973ce7e02afd21782d87a32a5dcab2129e79c416dc",
45
+ "latent_sha256": "56b429ba02eda8d86408a06131950753b56eb056966e43b3fb305d2edb0645b5",
46
+ "config": {
47
+ "generation": {
48
+ "abc": {
49
+ "temperature": 0.7,
50
+ "top_p": 0.9,
51
+ "top_k": 30,
52
+ "repetition_penalty": 1.005,
53
+ "penalty_window": 100,
54
+ "min_tokens": 32,
55
+ "max_tokens": 4096
56
+ },
57
+ "semantic": {
58
+ "temperature": 1.0,
59
+ "top_p": 0.95,
60
+ "top_k": 100,
61
+ "repetition_penalty": 1.2,
62
+ "penalty_window": 50,
63
+ "min_tokens": 200,
64
+ "max_tokens": 9000
65
+ },
66
+ "ode_steps": 8,
67
+ "ode_method": "midpoint",
68
+ "context": 24576,
69
+ "version": "yue2-native-v1"
70
+ },
71
+ "overrides": {
72
+ "ode_steps": 8
73
+ },
74
+ "cot": "full",
75
+ "cfg_scale": 1.0,
76
+ "cfg_negative": "same_instruction_and_exact_abc",
77
+ "backend": "torch",
78
+ "quantization": "orbitquant",
79
+ "model_dtype": "bfloat16",
80
+ "vae_dtype": "float32",
81
+ "vae_decode": "halo_crop",
82
+ "vae_core_frames": 512,
83
+ "vae_halo_frames": 16,
84
+ "device": "cuda:0",
85
+ "memory_budget_gib": 30.0,
86
+ "offload_ar": false,
87
+ "runtime_sha256": "e48937c70a5ca16669e6e722b392614379112aeb8f3a6a617120b4cdf1cd5c6a",
88
+ "decoder_release": "standard",
89
+ "validation_status": "unvalidated",
90
+ "runtime_profile": "fast",
91
+ "fuse_rms_quant": false,
92
+ "runtime_bundle_sha256": "fa427b99412fb298a91661cad5254f0b9a2c649d569a495de04b65708cac9904",
93
+ "vae_compute_dtype": "torch.float16",
94
+ "expert_staging": false,
95
+ "orbitquant": {
96
+ "quant_method": "orbitquant",
97
+ "weight_bits": 4,
98
+ "activation_bits": 4,
99
+ "rotation": "rpbh",
100
+ "rotation_seed": 0,
101
+ "block_size": "paper",
102
+ "codebook": "lloyd_max",
103
+ "codebook_version": 2,
104
+ "codebook_dtype": "float32",
105
+ "row_norm_dtype": "bfloat16",
106
+ "activation_norm_dtype": "float32",
107
+ "activation_eps": 1e-10,
108
+ "weight_pack_dtype": "uint8",
109
+ "target_policy": "universal",
110
+ "adaln_policy": "int4_rtn",
111
+ "adaln_group_size": 64,
112
+ "modules_to_convert": [],
113
+ "modules_to_use_adaln": [],
114
+ "modules_to_not_convert": [],
115
+ "modules_dtype_dict": {},
116
+ "artifact_format_version": 1,
117
+ "runtime_mode": "auto_fused",
118
+ "activation_kernel_backend": "auto",
119
+ "packed_matmul_block_m": 64,
120
+ "packed_matmul_block_n": 64,
121
+ "packed_matmul_block_k": 128,
122
+ "packed_matmul_num_warps": 4,
123
+ "weight_row_tile_size": 256,
124
+ "w4a4_int8_weight_cache": false,
125
+ "lowbit_boundary_protection": "auto",
126
+ "lowbit_protected_blocks": 4,
127
+ "lowbit_protected_bits": 4,
128
+ "lowbit_interior_protection": "auto"
129
+ }
130
+ },
131
+ "runtime": {
132
+ "fuse_rms_quant": false,
133
+ "orbitquant_version": "0.9.5",
134
+ "profile": "fast",
135
+ "allocator_backend": "native",
136
+ "modules": 224,
137
+ "logical_projections": 392,
138
+ "fused": true,
139
+ "effective_modes": {
140
+ "native_packed_matmul": 224
141
+ },
142
+ "activation_backends": {
143
+ "native_cuda_int8_surrogate": 168,
144
+ "triton_cuda_packed_w4": 56
145
+ },
146
+ "decoded_weight_caches": 0,
147
+ "int8_weight_caches": 0,
148
+ "packed_weight_bytes": 1409286144
149
+ },
150
+ "allocator_config": "expandable_segments:True",
151
+ "expandable_segments": 8
152
+ },
153
+ {
154
+ "profile": "fast",
155
+ "fuse_rms_quant": false,
156
+ "repeat": 1,
157
+ "wall_seconds": 30.492892453446984,
158
+ "audio_seconds": 187.63866666666667,
159
+ "nvml_peak": 5956501504,
160
+ "torch_allocated_peak": 4796496896,
161
+ "timing": {
162
+ "abc": {
163
+ "seconds": 6.603233300149441,
164
+ "prefill_seconds": 0.15628607757389545,
165
+ "ttft_seconds": 0.15696118772029877,
166
+ "output_tokens": 1629,
167
+ "content_tokens": 1628,
168
+ "output_tps": 246.697326287583,
169
+ "prefix_tokens": 423,
170
+ "cfg_branches": 1,
171
+ "execution": "cuda_graph",
172
+ "attention": "flash"
173
+ },
174
+ "semantic": {
175
+ "seconds": 17.143573760986328,
176
+ "prefill_seconds": 0.14921116270124912,
177
+ "ttft_seconds": 0.14976098202168941,
178
+ "output_tokens": 4692,
179
+ "content_tokens": 4691,
180
+ "output_tps": 273.6885590726477,
181
+ "prefix_tokens": 2053,
182
+ "cfg_branches": 1,
183
+ "execution": "cuda_graph",
184
+ "attention": "flash"
185
+ },
186
+ "nar_seconds": 3.957493392750621,
187
+ "vae_seconds": 2.4542733766138554,
188
+ "load": {
189
+ "resolve_and_integrity_seconds": 2.3701207600533962,
190
+ "mot_load_seconds": 0.8900818899273872
191
+ },
192
+ "e2e_seconds": 30.487808683887124
193
+ },
194
+ "semantic_sha256": "2cadb2f7623adcfe95baf6a8febb0532bae42e8fe1fe716d885c4a36790bd4f6",
195
+ "waveform_sha256": "102fb2bd604da4cfa44320973ce7e02afd21782d87a32a5dcab2129e79c416dc",
196
+ "latent_sha256": "56b429ba02eda8d86408a06131950753b56eb056966e43b3fb305d2edb0645b5",
197
+ "config": {
198
+ "generation": {
199
+ "abc": {
200
+ "temperature": 0.7,
201
+ "top_p": 0.9,
202
+ "top_k": 30,
203
+ "repetition_penalty": 1.005,
204
+ "penalty_window": 100,
205
+ "min_tokens": 32,
206
+ "max_tokens": 4096
207
+ },
208
+ "semantic": {
209
+ "temperature": 1.0,
210
+ "top_p": 0.95,
211
+ "top_k": 100,
212
+ "repetition_penalty": 1.2,
213
+ "penalty_window": 50,
214
+ "min_tokens": 200,
215
+ "max_tokens": 9000
216
+ },
217
+ "ode_steps": 8,
218
+ "ode_method": "midpoint",
219
+ "context": 24576,
220
+ "version": "yue2-native-v1"
221
+ },
222
+ "overrides": {
223
+ "ode_steps": 8
224
+ },
225
+ "cot": "full",
226
+ "cfg_scale": 1.0,
227
+ "cfg_negative": "same_instruction_and_exact_abc",
228
+ "backend": "torch",
229
+ "quantization": "orbitquant",
230
+ "model_dtype": "bfloat16",
231
+ "vae_dtype": "float32",
232
+ "vae_decode": "halo_crop",
233
+ "vae_core_frames": 512,
234
+ "vae_halo_frames": 16,
235
+ "device": "cuda:0",
236
+ "memory_budget_gib": 30.0,
237
+ "offload_ar": false,
238
+ "runtime_sha256": "e48937c70a5ca16669e6e722b392614379112aeb8f3a6a617120b4cdf1cd5c6a",
239
+ "decoder_release": "standard",
240
+ "validation_status": "unvalidated",
241
+ "runtime_profile": "fast",
242
+ "fuse_rms_quant": false,
243
+ "runtime_bundle_sha256": "fa427b99412fb298a91661cad5254f0b9a2c649d569a495de04b65708cac9904",
244
+ "vae_compute_dtype": "torch.float16",
245
+ "expert_staging": false,
246
+ "orbitquant": {
247
+ "quant_method": "orbitquant",
248
+ "weight_bits": 4,
249
+ "activation_bits": 4,
250
+ "rotation": "rpbh",
251
+ "rotation_seed": 0,
252
+ "block_size": "paper",
253
+ "codebook": "lloyd_max",
254
+ "codebook_version": 2,
255
+ "codebook_dtype": "float32",
256
+ "row_norm_dtype": "bfloat16",
257
+ "activation_norm_dtype": "float32",
258
+ "activation_eps": 1e-10,
259
+ "weight_pack_dtype": "uint8",
260
+ "target_policy": "universal",
261
+ "adaln_policy": "int4_rtn",
262
+ "adaln_group_size": 64,
263
+ "modules_to_convert": [],
264
+ "modules_to_use_adaln": [],
265
+ "modules_to_not_convert": [],
266
+ "modules_dtype_dict": {},
267
+ "artifact_format_version": 1,
268
+ "runtime_mode": "auto_fused",
269
+ "activation_kernel_backend": "auto",
270
+ "packed_matmul_block_m": 64,
271
+ "packed_matmul_block_n": 64,
272
+ "packed_matmul_block_k": 128,
273
+ "packed_matmul_num_warps": 4,
274
+ "weight_row_tile_size": 256,
275
+ "w4a4_int8_weight_cache": false,
276
+ "lowbit_boundary_protection": "auto",
277
+ "lowbit_protected_blocks": 4,
278
+ "lowbit_protected_bits": 4,
279
+ "lowbit_interior_protection": "auto"
280
+ }
281
+ },
282
+ "runtime": {
283
+ "fuse_rms_quant": false,
284
+ "orbitquant_version": "0.9.5",
285
+ "profile": "fast",
286
+ "allocator_backend": "native",
287
+ "modules": 224,
288
+ "logical_projections": 392,
289
+ "fused": true,
290
+ "effective_modes": {
291
+ "native_packed_matmul": 224
292
+ },
293
+ "activation_backends": {
294
+ "native_cuda_int8_surrogate": 168,
295
+ "triton_cuda_packed_w4": 56
296
+ },
297
+ "decoded_weight_caches": 0,
298
+ "int8_weight_caches": 0,
299
+ "packed_weight_bytes": 1409286144
300
+ },
301
+ "allocator_config": "expandable_segments:True",
302
+ "expandable_segments": 11
303
+ }
304
+ ]
evaluation/runtime-v3/fast-rms1-s8.json ADDED
@@ -0,0 +1,304 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [
2
+ {
3
+ "profile": "fast",
4
+ "fuse_rms_quant": true,
5
+ "repeat": 0,
6
+ "wall_seconds": 29.99516544304788,
7
+ "audio_seconds": 183.47866666666667,
8
+ "nvml_peak": 5709692928,
9
+ "torch_allocated_peak": 4458537472,
10
+ "timing": {
11
+ "abc": {
12
+ "seconds": 6.863483566790819,
13
+ "prefill_seconds": 0.8106300234794617,
14
+ "ttft_seconds": 0.9729824792593718,
15
+ "output_tokens": 1573,
16
+ "content_tokens": 1572,
17
+ "output_tps": 229.18390999156898,
18
+ "prefix_tokens": 423,
19
+ "cfg_branches": 1,
20
+ "execution": "cuda_graph",
21
+ "attention": "flash"
22
+ },
23
+ "semantic": {
24
+ "seconds": 15.786963351070881,
25
+ "prefill_seconds": 0.12841831147670746,
26
+ "ttft_seconds": 0.1289493814110756,
27
+ "output_tokens": 4588,
28
+ "content_tokens": 4587,
29
+ "output_tps": 290.61953828433894,
30
+ "prefix_tokens": 1997,
31
+ "cfg_branches": 1,
32
+ "execution": "cuda_graph",
33
+ "attention": "flash"
34
+ },
35
+ "nar_seconds": 3.9035949781537056,
36
+ "vae_seconds": 3.437227487564087,
37
+ "load": {
38
+ "resolve_and_integrity_seconds": 2.3494524490088224,
39
+ "mot_load_seconds": 0.8739112019538879
40
+ },
41
+ "e2e_seconds": 29.99417370557785
42
+ },
43
+ "semantic_sha256": "b3a0fe62887f254806d06db9720f4ed69c5c1ce0c0431cedcc55b040b2325f34",
44
+ "waveform_sha256": "73844f9a945ace67ed8f58a53b80d4e39a1db9b8bff75f76de30efbfe8d72d15",
45
+ "latent_sha256": "90cb11118fa43a7f2b6ce99a5d4915dfc6b2dc7958fc18616fdc02ee19252aab",
46
+ "config": {
47
+ "generation": {
48
+ "abc": {
49
+ "temperature": 0.7,
50
+ "top_p": 0.9,
51
+ "top_k": 30,
52
+ "repetition_penalty": 1.005,
53
+ "penalty_window": 100,
54
+ "min_tokens": 32,
55
+ "max_tokens": 4096
56
+ },
57
+ "semantic": {
58
+ "temperature": 1.0,
59
+ "top_p": 0.95,
60
+ "top_k": 100,
61
+ "repetition_penalty": 1.2,
62
+ "penalty_window": 50,
63
+ "min_tokens": 200,
64
+ "max_tokens": 9000
65
+ },
66
+ "ode_steps": 8,
67
+ "ode_method": "midpoint",
68
+ "context": 24576,
69
+ "version": "yue2-native-v1"
70
+ },
71
+ "overrides": {
72
+ "ode_steps": 8
73
+ },
74
+ "cot": "full",
75
+ "cfg_scale": 1.0,
76
+ "cfg_negative": "same_instruction_and_exact_abc",
77
+ "backend": "torch",
78
+ "quantization": "orbitquant",
79
+ "model_dtype": "bfloat16",
80
+ "vae_dtype": "float32",
81
+ "vae_decode": "halo_crop",
82
+ "vae_core_frames": 512,
83
+ "vae_halo_frames": 16,
84
+ "device": "cuda:0",
85
+ "memory_budget_gib": 30.0,
86
+ "offload_ar": false,
87
+ "runtime_sha256": "e48937c70a5ca16669e6e722b392614379112aeb8f3a6a617120b4cdf1cd5c6a",
88
+ "decoder_release": "standard",
89
+ "validation_status": "unvalidated",
90
+ "runtime_profile": "fast",
91
+ "fuse_rms_quant": true,
92
+ "runtime_bundle_sha256": "fa427b99412fb298a91661cad5254f0b9a2c649d569a495de04b65708cac9904",
93
+ "vae_compute_dtype": "torch.float16",
94
+ "expert_staging": false,
95
+ "orbitquant": {
96
+ "quant_method": "orbitquant",
97
+ "weight_bits": 4,
98
+ "activation_bits": 4,
99
+ "rotation": "rpbh",
100
+ "rotation_seed": 0,
101
+ "block_size": "paper",
102
+ "codebook": "lloyd_max",
103
+ "codebook_version": 2,
104
+ "codebook_dtype": "float32",
105
+ "row_norm_dtype": "bfloat16",
106
+ "activation_norm_dtype": "float32",
107
+ "activation_eps": 1e-10,
108
+ "weight_pack_dtype": "uint8",
109
+ "target_policy": "universal",
110
+ "adaln_policy": "int4_rtn",
111
+ "adaln_group_size": 64,
112
+ "modules_to_convert": [],
113
+ "modules_to_use_adaln": [],
114
+ "modules_to_not_convert": [],
115
+ "modules_dtype_dict": {},
116
+ "artifact_format_version": 1,
117
+ "runtime_mode": "auto_fused",
118
+ "activation_kernel_backend": "auto",
119
+ "packed_matmul_block_m": 64,
120
+ "packed_matmul_block_n": 64,
121
+ "packed_matmul_block_k": 128,
122
+ "packed_matmul_num_warps": 4,
123
+ "weight_row_tile_size": 256,
124
+ "w4a4_int8_weight_cache": false,
125
+ "lowbit_boundary_protection": "auto",
126
+ "lowbit_protected_blocks": 4,
127
+ "lowbit_protected_bits": 4,
128
+ "lowbit_interior_protection": "auto"
129
+ }
130
+ },
131
+ "runtime": {
132
+ "fuse_rms_quant": true,
133
+ "orbitquant_version": "0.9.5",
134
+ "profile": "fast",
135
+ "allocator_backend": "native",
136
+ "modules": 224,
137
+ "logical_projections": 392,
138
+ "fused": true,
139
+ "effective_modes": {
140
+ "native_packed_matmul": 224
141
+ },
142
+ "activation_backends": {
143
+ "native_cuda_int8_surrogate": 168,
144
+ "triton_cuda_packed_w4": 56
145
+ },
146
+ "decoded_weight_caches": 0,
147
+ "int8_weight_caches": 0,
148
+ "packed_weight_bytes": 1409286144
149
+ },
150
+ "allocator_config": "expandable_segments:True",
151
+ "expandable_segments": 10
152
+ },
153
+ {
154
+ "profile": "fast",
155
+ "fuse_rms_quant": true,
156
+ "repeat": 1,
157
+ "wall_seconds": 28.4998283572495,
158
+ "audio_seconds": 183.47866666666667,
159
+ "nvml_peak": 5804064768,
160
+ "torch_allocated_peak": 4741853184,
161
+ "timing": {
162
+ "abc": {
163
+ "seconds": 6.031709000468254,
164
+ "prefill_seconds": 0.13977668620646,
165
+ "ttft_seconds": 0.14045371487736702,
166
+ "output_tokens": 1573,
167
+ "content_tokens": 1572,
168
+ "output_tps": 260.78844318880186,
169
+ "prefix_tokens": 423,
170
+ "cfg_branches": 1,
171
+ "execution": "cuda_graph",
172
+ "attention": "flash"
173
+ },
174
+ "semantic": {
175
+ "seconds": 15.868235632777214,
176
+ "prefill_seconds": 0.11507388576865196,
177
+ "ttft_seconds": 0.11560502462089062,
178
+ "output_tokens": 4588,
179
+ "content_tokens": 4587,
180
+ "output_tps": 289.13107330742486,
181
+ "prefix_tokens": 1997,
182
+ "cfg_branches": 1,
183
+ "execution": "cuda_graph",
184
+ "attention": "flash"
185
+ },
186
+ "nar_seconds": 3.8611518684774637,
187
+ "vae_seconds": 2.4071050081402063,
188
+ "load": {
189
+ "resolve_and_integrity_seconds": 2.3494524490088224,
190
+ "mot_load_seconds": 0.8739112019538879
191
+ },
192
+ "e2e_seconds": 28.494855623692274
193
+ },
194
+ "semantic_sha256": "b3a0fe62887f254806d06db9720f4ed69c5c1ce0c0431cedcc55b040b2325f34",
195
+ "waveform_sha256": "73844f9a945ace67ed8f58a53b80d4e39a1db9b8bff75f76de30efbfe8d72d15",
196
+ "latent_sha256": "90cb11118fa43a7f2b6ce99a5d4915dfc6b2dc7958fc18616fdc02ee19252aab",
197
+ "config": {
198
+ "generation": {
199
+ "abc": {
200
+ "temperature": 0.7,
201
+ "top_p": 0.9,
202
+ "top_k": 30,
203
+ "repetition_penalty": 1.005,
204
+ "penalty_window": 100,
205
+ "min_tokens": 32,
206
+ "max_tokens": 4096
207
+ },
208
+ "semantic": {
209
+ "temperature": 1.0,
210
+ "top_p": 0.95,
211
+ "top_k": 100,
212
+ "repetition_penalty": 1.2,
213
+ "penalty_window": 50,
214
+ "min_tokens": 200,
215
+ "max_tokens": 9000
216
+ },
217
+ "ode_steps": 8,
218
+ "ode_method": "midpoint",
219
+ "context": 24576,
220
+ "version": "yue2-native-v1"
221
+ },
222
+ "overrides": {
223
+ "ode_steps": 8
224
+ },
225
+ "cot": "full",
226
+ "cfg_scale": 1.0,
227
+ "cfg_negative": "same_instruction_and_exact_abc",
228
+ "backend": "torch",
229
+ "quantization": "orbitquant",
230
+ "model_dtype": "bfloat16",
231
+ "vae_dtype": "float32",
232
+ "vae_decode": "halo_crop",
233
+ "vae_core_frames": 512,
234
+ "vae_halo_frames": 16,
235
+ "device": "cuda:0",
236
+ "memory_budget_gib": 30.0,
237
+ "offload_ar": false,
238
+ "runtime_sha256": "e48937c70a5ca16669e6e722b392614379112aeb8f3a6a617120b4cdf1cd5c6a",
239
+ "decoder_release": "standard",
240
+ "validation_status": "unvalidated",
241
+ "runtime_profile": "fast",
242
+ "fuse_rms_quant": true,
243
+ "runtime_bundle_sha256": "fa427b99412fb298a91661cad5254f0b9a2c649d569a495de04b65708cac9904",
244
+ "vae_compute_dtype": "torch.float16",
245
+ "expert_staging": false,
246
+ "orbitquant": {
247
+ "quant_method": "orbitquant",
248
+ "weight_bits": 4,
249
+ "activation_bits": 4,
250
+ "rotation": "rpbh",
251
+ "rotation_seed": 0,
252
+ "block_size": "paper",
253
+ "codebook": "lloyd_max",
254
+ "codebook_version": 2,
255
+ "codebook_dtype": "float32",
256
+ "row_norm_dtype": "bfloat16",
257
+ "activation_norm_dtype": "float32",
258
+ "activation_eps": 1e-10,
259
+ "weight_pack_dtype": "uint8",
260
+ "target_policy": "universal",
261
+ "adaln_policy": "int4_rtn",
262
+ "adaln_group_size": 64,
263
+ "modules_to_convert": [],
264
+ "modules_to_use_adaln": [],
265
+ "modules_to_not_convert": [],
266
+ "modules_dtype_dict": {},
267
+ "artifact_format_version": 1,
268
+ "runtime_mode": "auto_fused",
269
+ "activation_kernel_backend": "auto",
270
+ "packed_matmul_block_m": 64,
271
+ "packed_matmul_block_n": 64,
272
+ "packed_matmul_block_k": 128,
273
+ "packed_matmul_num_warps": 4,
274
+ "weight_row_tile_size": 256,
275
+ "w4a4_int8_weight_cache": false,
276
+ "lowbit_boundary_protection": "auto",
277
+ "lowbit_protected_blocks": 4,
278
+ "lowbit_protected_bits": 4,
279
+ "lowbit_interior_protection": "auto"
280
+ }
281
+ },
282
+ "runtime": {
283
+ "fuse_rms_quant": true,
284
+ "orbitquant_version": "0.9.5",
285
+ "profile": "fast",
286
+ "allocator_backend": "native",
287
+ "modules": 224,
288
+ "logical_projections": 392,
289
+ "fused": true,
290
+ "effective_modes": {
291
+ "native_packed_matmul": 224
292
+ },
293
+ "activation_backends": {
294
+ "native_cuda_int8_surrogate": 168,
295
+ "triton_cuda_packed_w4": 56
296
+ },
297
+ "decoded_weight_caches": 0,
298
+ "int8_weight_caches": 0,
299
+ "packed_weight_bytes": 1409286144
300
+ },
301
+ "allocator_config": "expandable_segments:True",
302
+ "expandable_segments": 12
303
+ }
304
+ ]
evaluation/runtime-v3/memory-rms1-s8.json ADDED
@@ -0,0 +1,304 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [
2
+ {
3
+ "profile": "memory",
4
+ "fuse_rms_quant": true,
5
+ "repeat": 0,
6
+ "wall_seconds": 31.11274246312678,
7
+ "audio_seconds": 183.47866666666667,
8
+ "nvml_peak": 5692915712,
9
+ "torch_allocated_peak": 3750597120,
10
+ "timing": {
11
+ "abc": {
12
+ "seconds": 6.880559112876654,
13
+ "prefill_seconds": 0.8072306606918573,
14
+ "ttft_seconds": 0.9844317771494389,
15
+ "output_tokens": 1573,
16
+ "content_tokens": 1572,
17
+ "output_tps": 228.61514219915674,
18
+ "prefix_tokens": 423,
19
+ "cfg_branches": 1,
20
+ "execution": "cuda_graph",
21
+ "attention": "flash"
22
+ },
23
+ "semantic": {
24
+ "seconds": 15.86077463068068,
25
+ "prefill_seconds": 0.1298980787396431,
26
+ "ttft_seconds": 0.13040952757000923,
27
+ "output_tokens": 4588,
28
+ "content_tokens": 4587,
29
+ "output_tps": 289.2670822725826,
30
+ "prefix_tokens": 1997,
31
+ "cfg_branches": 1,
32
+ "execution": "cuda_graph",
33
+ "attention": "flash"
34
+ },
35
+ "nar_seconds": 5.559019308537245,
36
+ "vae_seconds": 2.8005002085119486,
37
+ "load": {
38
+ "resolve_and_integrity_seconds": 2.3766595162451267,
39
+ "mot_load_seconds": 0.9255613535642624
40
+ },
41
+ "e2e_seconds": 31.11171407625079
42
+ },
43
+ "semantic_sha256": "b3a0fe62887f254806d06db9720f4ed69c5c1ce0c0431cedcc55b040b2325f34",
44
+ "waveform_sha256": "73844f9a945ace67ed8f58a53b80d4e39a1db9b8bff75f76de30efbfe8d72d15",
45
+ "latent_sha256": "90cb11118fa43a7f2b6ce99a5d4915dfc6b2dc7958fc18616fdc02ee19252aab",
46
+ "config": {
47
+ "generation": {
48
+ "abc": {
49
+ "temperature": 0.7,
50
+ "top_p": 0.9,
51
+ "top_k": 30,
52
+ "repetition_penalty": 1.005,
53
+ "penalty_window": 100,
54
+ "min_tokens": 32,
55
+ "max_tokens": 4096
56
+ },
57
+ "semantic": {
58
+ "temperature": 1.0,
59
+ "top_p": 0.95,
60
+ "top_k": 100,
61
+ "repetition_penalty": 1.2,
62
+ "penalty_window": 50,
63
+ "min_tokens": 200,
64
+ "max_tokens": 9000
65
+ },
66
+ "ode_steps": 8,
67
+ "ode_method": "midpoint",
68
+ "context": 24576,
69
+ "version": "yue2-native-v1"
70
+ },
71
+ "overrides": {
72
+ "ode_steps": 8
73
+ },
74
+ "cot": "full",
75
+ "cfg_scale": 1.0,
76
+ "cfg_negative": "same_instruction_and_exact_abc",
77
+ "backend": "torch",
78
+ "quantization": "orbitquant",
79
+ "model_dtype": "bfloat16",
80
+ "vae_dtype": "float32",
81
+ "vae_decode": "halo_crop",
82
+ "vae_core_frames": 512,
83
+ "vae_halo_frames": 16,
84
+ "device": "cuda:0",
85
+ "memory_budget_gib": 30.0,
86
+ "offload_ar": true,
87
+ "runtime_sha256": "e48937c70a5ca16669e6e722b392614379112aeb8f3a6a617120b4cdf1cd5c6a",
88
+ "decoder_release": "standard",
89
+ "validation_status": "unvalidated",
90
+ "runtime_profile": "memory",
91
+ "fuse_rms_quant": true,
92
+ "runtime_bundle_sha256": "fa427b99412fb298a91661cad5254f0b9a2c649d569a495de04b65708cac9904",
93
+ "vae_compute_dtype": "torch.float16",
94
+ "expert_staging": true,
95
+ "orbitquant": {
96
+ "quant_method": "orbitquant",
97
+ "weight_bits": 4,
98
+ "activation_bits": 4,
99
+ "rotation": "rpbh",
100
+ "rotation_seed": 0,
101
+ "block_size": "paper",
102
+ "codebook": "lloyd_max",
103
+ "codebook_version": 2,
104
+ "codebook_dtype": "float32",
105
+ "row_norm_dtype": "bfloat16",
106
+ "activation_norm_dtype": "float32",
107
+ "activation_eps": 1e-10,
108
+ "weight_pack_dtype": "uint8",
109
+ "target_policy": "universal",
110
+ "adaln_policy": "int4_rtn",
111
+ "adaln_group_size": 64,
112
+ "modules_to_convert": [],
113
+ "modules_to_use_adaln": [],
114
+ "modules_to_not_convert": [],
115
+ "modules_dtype_dict": {},
116
+ "artifact_format_version": 1,
117
+ "runtime_mode": "auto_fused",
118
+ "activation_kernel_backend": "auto",
119
+ "packed_matmul_block_m": 64,
120
+ "packed_matmul_block_n": 64,
121
+ "packed_matmul_block_k": 128,
122
+ "packed_matmul_num_warps": 4,
123
+ "weight_row_tile_size": 256,
124
+ "w4a4_int8_weight_cache": false,
125
+ "lowbit_boundary_protection": "auto",
126
+ "lowbit_protected_blocks": 4,
127
+ "lowbit_protected_bits": 4,
128
+ "lowbit_interior_protection": "auto"
129
+ }
130
+ },
131
+ "runtime": {
132
+ "fuse_rms_quant": true,
133
+ "orbitquant_version": "0.9.5",
134
+ "profile": "memory",
135
+ "allocator_backend": "native",
136
+ "modules": 224,
137
+ "logical_projections": 392,
138
+ "fused": true,
139
+ "effective_modes": {
140
+ "native_packed_matmul": 224
141
+ },
142
+ "activation_backends": {
143
+ "native_cuda_int8_surrogate": 168,
144
+ "triton_cuda_packed_w4": 56
145
+ },
146
+ "decoded_weight_caches": 0,
147
+ "int8_weight_caches": 0,
148
+ "packed_weight_bytes": 1409286144
149
+ },
150
+ "allocator_config": "expandable_segments:True",
151
+ "expandable_segments": 9
152
+ },
153
+ {
154
+ "profile": "memory",
155
+ "fuse_rms_quant": true,
156
+ "repeat": 1,
157
+ "wall_seconds": 29.819518825039268,
158
+ "audio_seconds": 183.47866666666667,
159
+ "nvml_peak": 5726470144,
160
+ "torch_allocated_peak": 4033912832,
161
+ "timing": {
162
+ "abc": {
163
+ "seconds": 6.047573072835803,
164
+ "prefill_seconds": 0.13665939681231976,
165
+ "ttft_seconds": 0.13733276538550854,
166
+ "output_tokens": 1573,
167
+ "content_tokens": 1572,
168
+ "output_tps": 260.10433955160056,
169
+ "prefix_tokens": 423,
170
+ "cfg_branches": 1,
171
+ "execution": "cuda_graph",
172
+ "attention": "flash"
173
+ },
174
+ "semantic": {
175
+ "seconds": 15.967649232596159,
176
+ "prefill_seconds": 0.11747091263532639,
177
+ "ttft_seconds": 0.11801005341112614,
178
+ "output_tokens": 4588,
179
+ "content_tokens": 4587,
180
+ "output_tps": 287.3309610680897,
181
+ "prefix_tokens": 1997,
182
+ "cfg_branches": 1,
183
+ "execution": "cuda_graph",
184
+ "attention": "flash"
185
+ },
186
+ "nar_seconds": 5.237515918910503,
187
+ "vae_seconds": 2.0264065731316805,
188
+ "load": {
189
+ "resolve_and_integrity_seconds": 2.3766595162451267,
190
+ "mot_load_seconds": 0.9255613535642624
191
+ },
192
+ "e2e_seconds": 29.814160283654928
193
+ },
194
+ "semantic_sha256": "b3a0fe62887f254806d06db9720f4ed69c5c1ce0c0431cedcc55b040b2325f34",
195
+ "waveform_sha256": "73844f9a945ace67ed8f58a53b80d4e39a1db9b8bff75f76de30efbfe8d72d15",
196
+ "latent_sha256": "90cb11118fa43a7f2b6ce99a5d4915dfc6b2dc7958fc18616fdc02ee19252aab",
197
+ "config": {
198
+ "generation": {
199
+ "abc": {
200
+ "temperature": 0.7,
201
+ "top_p": 0.9,
202
+ "top_k": 30,
203
+ "repetition_penalty": 1.005,
204
+ "penalty_window": 100,
205
+ "min_tokens": 32,
206
+ "max_tokens": 4096
207
+ },
208
+ "semantic": {
209
+ "temperature": 1.0,
210
+ "top_p": 0.95,
211
+ "top_k": 100,
212
+ "repetition_penalty": 1.2,
213
+ "penalty_window": 50,
214
+ "min_tokens": 200,
215
+ "max_tokens": 9000
216
+ },
217
+ "ode_steps": 8,
218
+ "ode_method": "midpoint",
219
+ "context": 24576,
220
+ "version": "yue2-native-v1"
221
+ },
222
+ "overrides": {
223
+ "ode_steps": 8
224
+ },
225
+ "cot": "full",
226
+ "cfg_scale": 1.0,
227
+ "cfg_negative": "same_instruction_and_exact_abc",
228
+ "backend": "torch",
229
+ "quantization": "orbitquant",
230
+ "model_dtype": "bfloat16",
231
+ "vae_dtype": "float32",
232
+ "vae_decode": "halo_crop",
233
+ "vae_core_frames": 512,
234
+ "vae_halo_frames": 16,
235
+ "device": "cuda:0",
236
+ "memory_budget_gib": 30.0,
237
+ "offload_ar": true,
238
+ "runtime_sha256": "e48937c70a5ca16669e6e722b392614379112aeb8f3a6a617120b4cdf1cd5c6a",
239
+ "decoder_release": "standard",
240
+ "validation_status": "unvalidated",
241
+ "runtime_profile": "memory",
242
+ "fuse_rms_quant": true,
243
+ "runtime_bundle_sha256": "fa427b99412fb298a91661cad5254f0b9a2c649d569a495de04b65708cac9904",
244
+ "vae_compute_dtype": "torch.float16",
245
+ "expert_staging": true,
246
+ "orbitquant": {
247
+ "quant_method": "orbitquant",
248
+ "weight_bits": 4,
249
+ "activation_bits": 4,
250
+ "rotation": "rpbh",
251
+ "rotation_seed": 0,
252
+ "block_size": "paper",
253
+ "codebook": "lloyd_max",
254
+ "codebook_version": 2,
255
+ "codebook_dtype": "float32",
256
+ "row_norm_dtype": "bfloat16",
257
+ "activation_norm_dtype": "float32",
258
+ "activation_eps": 1e-10,
259
+ "weight_pack_dtype": "uint8",
260
+ "target_policy": "universal",
261
+ "adaln_policy": "int4_rtn",
262
+ "adaln_group_size": 64,
263
+ "modules_to_convert": [],
264
+ "modules_to_use_adaln": [],
265
+ "modules_to_not_convert": [],
266
+ "modules_dtype_dict": {},
267
+ "artifact_format_version": 1,
268
+ "runtime_mode": "auto_fused",
269
+ "activation_kernel_backend": "auto",
270
+ "packed_matmul_block_m": 64,
271
+ "packed_matmul_block_n": 64,
272
+ "packed_matmul_block_k": 128,
273
+ "packed_matmul_num_warps": 4,
274
+ "weight_row_tile_size": 256,
275
+ "w4a4_int8_weight_cache": false,
276
+ "lowbit_boundary_protection": "auto",
277
+ "lowbit_protected_blocks": 4,
278
+ "lowbit_protected_bits": 4,
279
+ "lowbit_interior_protection": "auto"
280
+ }
281
+ },
282
+ "runtime": {
283
+ "fuse_rms_quant": true,
284
+ "orbitquant_version": "0.9.5",
285
+ "profile": "memory",
286
+ "allocator_backend": "native",
287
+ "modules": 224,
288
+ "logical_projections": 392,
289
+ "fused": true,
290
+ "effective_modes": {
291
+ "native_packed_matmul": 224
292
+ },
293
+ "activation_backends": {
294
+ "native_cuda_int8_surrogate": 168,
295
+ "triton_cuda_packed_w4": 56
296
+ },
297
+ "decoded_weight_caches": 0,
298
+ "int8_weight_caches": 0,
299
+ "packed_weight_bytes": 1409286144
300
+ },
301
+ "allocator_config": "expandable_segments:True",
302
+ "expandable_segments": 15
303
+ }
304
+ ]
examples/runtime-v3/fast-rms.wav ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:30a70e2224b008e08f10ba0e0f9b8243b06b32a4fcbd7381e980d68cd5a7974e
3
+ size 70455896
src/OPTIMIZATION_IDEAS.md CHANGED
@@ -98,3 +98,14 @@ Observed initial microbench (RTX PRO 6000 Blackwell, same 256 forced tokens): ba
98
  - Autocast outside VAE alone was a no-op because decode disables it. Real inner autocast FP16 measured~.58s decoder vs~1.06sFP32; preserve FP32 weights and label compute precision.
99
  - NAR V-view backing-storage hypothesis is ALREADY covered by current runtime's contiguous cache materialization. Guard rejected stale-source patch before inference; no new optimization credited.
100
  - Bidirectional expert staging with deferred NAR restore after acoustic AR prefix improves some memory at transfer cost. Allocator experiments running to test fragmentation directly.
 
 
 
 
 
 
 
 
 
 
 
 
98
  - Autocast outside VAE alone was a no-op because decode disables it. Real inner autocast FP16 measured~.58s decoder vs~1.06sFP32; preserve FP32 weights and label compute precision.
99
  - NAR V-view backing-storage hypothesis is ALREADY covered by current runtime's contiguous cache materialization. Guard rejected stale-source patch before inference; no new optimization credited.
100
  - Bidirectional expert staging with deferred NAR restore after acoustic AR prefix improves some memory at transfer cost. Allocator experiments running to test fragmentation directly.
101
+
102
+
103
+ ## Measured follow-up: byte-pair lookup and RMS fusion
104
+
105
+ - Byte-pair LUT: selected. Native1.0.2 halves codebook lookups for K1024..16384; shorter inputs use the original kernel. Actual native decoder3.148->2.943ms, saved logits exact. Full interleaved song31.82->30.34s, identical PCM.
106
+ - RMSNorm plus packed activation quantization: selected as an explicit fast/memory option. FP32 intermediate products can change musical arrangement. Early Hadamard stages use warp shuffles and the final reduction stages run in one warp. Combined with byte-pair GEMV, the same fused song29.81->28.28s in interleaved measurements.
107
+ - Warp-resident16-entry lookup: only1.8% model gain; short-K full-mask shuffle bug repaired. Slower than byte-pair lookup; retained as research evidence.
108
+ - 8/16warps perblock: roughly2.99/3.00ms, slower than4warps at2.94ms. Larger blocks amortize setup but reduce useful scheduling flexibility.
109
+ - UINT16 pair-table storage:2.921ms in one fixed-token probe. About0.7% below the4warpUINT32 result; requires interleaved confirmation before selection.
110
+ - Generic quantizer warp rewrite: dynamic branching blocked unrolling; static specialization removed most regression but gave negligible model improvement. Not selected.
111
+ - Future investigations: fuse device sampling/history maintenance if profiling justifies it; use per-shape table-layout selection; test attention-cache compression separately with spectral/listening milestones.
src/kernel-source/orbitquant-packed-matmul/CARD.md CHANGED
@@ -45,6 +45,16 @@ The CUDA package also exports the operations used by OrbitQuant's W4A4 runtime:
45
  - `matmul_packed_w4a4_int8`: direct packed A4/W4 CUDA MMA with fused token norm,
46
  row norm, surrogate scales, and bias epilogue.
47
 
 
 
 
 
 
 
 
 
 
 
48
  On CUDA compute capability 8.0 or newer, OrbitQuant normally combines
49
  `quantize_activations_int8` with chunked packed-weight decode and Torch's
50
  CUTLASS-backed INT8 matmul. The direct packed MMA operation remains available
 
45
  - `matmul_packed_w4a4_int8`: direct packed A4/W4 CUDA MMA with fused token norm,
46
  row norm, surrogate scales, and bias epilogue.
47
 
48
+ For one to eight rows, row-major weights, and input width at most 16384,
49
+ `matmul_packed_w4a4_int8` dispatches a DP4A GEMV. Native release 1.0.2 uses two
50
+ 256-entry shared tables (2 KiB per block) for widths 1024–16384 to decode a packed byte into a pair
51
+ of signed INT8 values. Smaller widths retain the previous 16-entry tables.
52
+ Aligned inputs retain 32-bit vector loads; unaligned
53
+ contiguous views use the byte-load path. The packed format, output dtypes,
54
+ norms, and bias behavior are unchanged. Larger batches and K-major weights
55
+ continue to use the existing Tensor Core path. Set
56
+ `ORBITQUANT_W4A4_DISABLE_GEMV=1` before process startup to disable GEMV.
57
+
58
  On CUDA compute capability 8.0 or newer, OrbitQuant normally combines
59
  `quantize_activations_int8` with chunked packed-weight decode and Torch's
60
  CUTLASS-backed INT8 matmul. The direct packed MMA operation remains available
src/kernel-source/orbitquant-packed-matmul/orbitquant_packed_matmul_cuda/packed_matmul.cu CHANGED
@@ -65,6 +65,52 @@ __global__ void orbitquant_packed_w4a4_gemv(T* out, const uint8_t* x, const uint
65
  }
66
  }
67
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
68
  inline bool orbitquant_w4a4_gemv_disabled() {
69
  static const bool disabled = []() {
70
  const char *value = std::getenv("ORBITQUANT_W4A4_DISABLE_GEMV");
@@ -1873,17 +1919,25 @@ void matmul_packed_w4a4_int8(
1873
  !orbitquant_w4a4_gemv_disabled()) {
1874
  const dim3 gemv_grid((out_features + 3) / 4, rows);
1875
  const auto gemv_stream = at::cuda::getCurrentCUDAStream();
1876
- #define ORBITQUANT_LAUNCH_DECODE_GEMV(T) \
1877
- orbitquant_packed_w4a4_gemv<T><<<gemv_grid, 128, 0, gemv_stream>>>( \
1878
  out.data_ptr<T>(), packed_activations.data_ptr<uint8_t>(), \
1879
  packed_weight_indices.data_ptr<uint8_t>(), token_norms.data_ptr<float>(), \
1880
  row_norms.data_ptr<c10::BFloat16>(), activation_codes.data_ptr<int8_t>(), \
1881
  weight_codes.data_ptr<int8_t>(), has_bias ? bias.data_ptr<T>() : nullptr, \
1882
  out_features, in_features / 2, activation_scale, weight_scale)
1883
  if (out.scalar_type() == torch::kBFloat16) {
1884
- ORBITQUANT_LAUNCH_DECODE_GEMV(c10::BFloat16);
 
 
 
 
1885
  } else {
1886
- ORBITQUANT_LAUNCH_DECODE_GEMV(c10::Half);
 
 
 
 
1887
  }
1888
  #undef ORBITQUANT_LAUNCH_DECODE_GEMV
1889
  C10_CUDA_KERNEL_LAUNCH_CHECK();
 
65
  }
66
  }
67
 
68
+ // Pair tables amortize their initialization once K reaches 1024.
69
+ template<class T>
70
+ __global__ void orbitquant_packed_w4a4_gemv_byte_pairs(T* out, const uint8_t* x, const uint8_t* w,
71
+ const float* xn, const c10::BFloat16* wn, const int8_t* ac, const int8_t* wc,
72
+ const T* bias, int n, int packed_k, float as, float ws) {
73
+ const int lane = threadIdx.x & 31;
74
+ const int col = blockIdx.x * 4 + threadIdx.x / 32;
75
+ const int row = blockIdx.y;
76
+ __shared__ uint32_t a_pairs[256], w_pairs[256];
77
+ for (int byte = threadIdx.x; byte < 256; byte += blockDim.x) {
78
+ a_pairs[byte] = uint32_t(uint8_t(ac[byte & 15])) | (uint32_t(uint8_t(ac[byte >> 4])) << 8);
79
+ w_pairs[byte] = uint32_t(uint8_t(wc[byte & 15])) | (uint32_t(uint8_t(wc[byte >> 4])) << 8);
80
+ }
81
+ __syncthreads();
82
+ if (col >= n) return;
83
+ int sum = 0;
84
+ if (packed_k % 4 == 0 && (reinterpret_cast<uintptr_t>(x) & 3) == 0 && (reinterpret_cast<uintptr_t>(w) & 3) == 0) {
85
+ const uint32_t* xp = reinterpret_cast<const uint32_t*>(x + row * packed_k);
86
+ const uint32_t* wp = reinterpret_cast<const uint32_t*>(w + col * packed_k);
87
+ for (int k = lane; k < packed_k / 4; k += 32) {
88
+ const uint32_t av = xp[k], bv = wp[k];
89
+ // One table load decodes both nibbles of a byte to signed INT8 lanes.
90
+ const uint32_t al = a_pairs[av & 255] | (a_pairs[(av >> 8) & 255] << 16);
91
+ const uint32_t ah = a_pairs[(av >> 16) & 255] | (a_pairs[av >> 24] << 16);
92
+ const uint32_t bl = w_pairs[bv & 255] | (w_pairs[(bv >> 8) & 255] << 16);
93
+ const uint32_t bh = w_pairs[(bv >> 16) & 255] | (w_pairs[bv >> 24] << 16);
94
+ sum = __dp4a(static_cast<int>(al), static_cast<int>(bl), sum);
95
+ sum = __dp4a(static_cast<int>(ah), static_cast<int>(bh), sum);
96
+ }
97
+ } else {
98
+ for (int k = lane; k < packed_k; k += 32) {
99
+ const uint8_t a = x[row * packed_k + k];
100
+ const uint8_t b = w[col * packed_k + k];
101
+ sum = __dp4a(static_cast<int>(a_pairs[a]), static_cast<int>(w_pairs[b]), sum);
102
+ }
103
+ }
104
+ #pragma unroll
105
+ for (int offset = 16; offset; offset >>= 1) sum += __shfl_down_sync(0xffffffff, sum, offset);
106
+ if (lane == 0) {
107
+ float value = static_cast<float>(sum);
108
+ value *= xn[row] * static_cast<float>(wn[col]) * (as * ws);
109
+ if (bias) value += static_cast<float>(bias[col]);
110
+ out[row * n + col] = static_cast<T>(value);
111
+ }
112
+ }
113
+
114
  inline bool orbitquant_w4a4_gemv_disabled() {
115
  static const bool disabled = []() {
116
  const char *value = std::getenv("ORBITQUANT_W4A4_DISABLE_GEMV");
 
1919
  !orbitquant_w4a4_gemv_disabled()) {
1920
  const dim3 gemv_grid((out_features + 3) / 4, rows);
1921
  const auto gemv_stream = at::cuda::getCurrentCUDAStream();
1922
+ #define ORBITQUANT_LAUNCH_DECODE_GEMV(KERNEL, T) \
1923
+ KERNEL<T><<<gemv_grid, 128, 0, gemv_stream>>>( \
1924
  out.data_ptr<T>(), packed_activations.data_ptr<uint8_t>(), \
1925
  packed_weight_indices.data_ptr<uint8_t>(), token_norms.data_ptr<float>(), \
1926
  row_norms.data_ptr<c10::BFloat16>(), activation_codes.data_ptr<int8_t>(), \
1927
  weight_codes.data_ptr<int8_t>(), has_bias ? bias.data_ptr<T>() : nullptr, \
1928
  out_features, in_features / 2, activation_scale, weight_scale)
1929
  if (out.scalar_type() == torch::kBFloat16) {
1930
+ if (in_features >= 1024) {
1931
+ ORBITQUANT_LAUNCH_DECODE_GEMV(orbitquant_packed_w4a4_gemv_byte_pairs, c10::BFloat16);
1932
+ } else {
1933
+ ORBITQUANT_LAUNCH_DECODE_GEMV(orbitquant_packed_w4a4_gemv, c10::BFloat16);
1934
+ }
1935
  } else {
1936
+ if (in_features >= 1024) {
1937
+ ORBITQUANT_LAUNCH_DECODE_GEMV(orbitquant_packed_w4a4_gemv_byte_pairs, c10::Half);
1938
+ } else {
1939
+ ORBITQUANT_LAUNCH_DECODE_GEMV(orbitquant_packed_w4a4_gemv, c10::Half);
1940
+ }
1941
  }
1942
  #undef ORBITQUANT_LAUNCH_DECODE_GEMV
1943
  C10_CUDA_KERNEL_LAUNCH_CHECK();
src/kernel-source/orbitquant-packed-matmul/tests/test_decode_gemv.py CHANGED
@@ -7,7 +7,17 @@ from orbitquant_packed_matmul import matmul_packed_w4a4_int8, supports_device
7
 
8
  @pytest.mark.kernels_ci
9
  @pytest.mark.parametrize(
10
- "rows,n,k", [(1, 129, 64), (2, 1024, 2048), (8, 2048, 6144), (1, 257, 16384), (9, 129, 128)]
 
 
 
 
 
 
 
 
 
 
11
  )
12
  @pytest.mark.parametrize("dtype", [torch.bfloat16, torch.float16])
13
  @pytest.mark.parametrize(
@@ -20,7 +30,10 @@ def test_decode_against_integer_oracle(rows, n, k, dtype, offset, bias_enabled,
20
  torch.manual_seed(42)
21
  x = torch.randint(0, 256, (rows * k // 2 + offset,), dtype=torch.uint8)[offset:]
22
  w = torch.randint(0, 256, (n * k // 2 + offset,), dtype=torch.uint8)[offset:]
23
- ac = torch.randint(-127, 128, (16,), dtype=torch.int8)
 
 
 
24
  wc = ac.flip(0)
25
 
26
  def unpack(values, count):
 
7
 
8
  @pytest.mark.kernels_ci
9
  @pytest.mark.parametrize(
10
+ "rows,n,k",
11
+ [
12
+ (1, 129, 64),
13
+ (8, 3, 64),
14
+ (1, 129, 512),
15
+ (8, 129, 1024),
16
+ (2, 1024, 2048),
17
+ (8, 2048, 6144),
18
+ (1, 257, 16384),
19
+ (9, 129, 128),
20
+ ],
21
  )
22
  @pytest.mark.parametrize("dtype", [torch.bfloat16, torch.float16])
23
  @pytest.mark.parametrize(
 
30
  torch.manual_seed(42)
31
  x = torch.randint(0, 256, (rows * k // 2 + offset,), dtype=torch.uint8)[offset:]
32
  w = torch.randint(0, 256, (n * k // 2 + offset,), dtype=torch.uint8)[offset:]
33
+ # Exercise every code, including signed INT8 extremes and mixed nibble signs.
34
+ ac = torch.tensor(
35
+ [-128, 127, -1, 0, 1, -64, 64, -32, 32, -16, 16, -8, 8, -4, 4, 2], dtype=torch.int8
36
+ )
37
  wc = ac.flip(0)
38
 
39
  def unpack(values, count):
src/kernel-source/rms-quant/.gitattributes ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ *.so filter=lfs diff=lfs merge=lfs -text
2
+ *.pth filter=lfs diff=lfs merge=lfs -text
3
+ *.bin filter=lfs diff=lfs merge=lfs -text
src/kernel-source/rms-quant/.gitignore ADDED
@@ -0,0 +1 @@
 
 
1
+ build
src/kernel-source/rms-quant/CARD.md ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ library_name: kernels
4
+ ---
5
+ # OrbitQuant packed GEMV
6
+
7
+ Bias-free W4A4 integer-surrogate matrix-vector evaluation for 1-8 activation rows. Reads packed nibbles directly, accumulates exact INT32 dot products, and applies the existing OrbitQuant token/row norms and surrogate scales. Output is BF16 or FP16.
8
+
9
+ The selected YuE2 dispatch uses it only for bias-free row-major packed weights. Larger batches and other layouts retain the original OrbitQuant implementation. The build targets SM120; other architectures require a rebuild and validation.
10
+
11
+ On RTX 5090, CUDA-graph layer timings improved from 0.1355 to 0.0102 ms (2048x2048) and from 0.4617 to 0.0144 ms (2048x6144), including activation preparation. BF16 reference timings were 0.0083 and 0.0105 ms: these isolated small matrices are still slightly slower than BF16. Whole-model grouped projections are measured separately.
12
+
13
+ Tests compare exact results without bias to the existing packed native kernel and exercise CUDA graph capture and invalid weight dimensions. Do not infer musical quality or whole-model speed from the microbenchmarks.
14
+
15
+ Build with kernel-builder-generated project files: `python setup.py build_kernel`. Python ABI3 and manylinux_2_34 checks pass for the supplied binary; manylinux_2_28 does not, due to a GLIBC_2.32 symbol from the Ubuntu build environment.
src/kernel-source/rms-quant/build.toml ADDED
@@ -0,0 +1,22 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [general]
2
+ name = "yue2-qkv-fused"
3
+ license = "Apache-2.0"
4
+ backends = [
5
+ "cuda",
6
+ ]
7
+ version = 1
8
+ edition = 5
9
+
10
+ [general.hub]
11
+ repo-id = "WaveCut/orbitquant-gemv"
12
+
13
+ [torch]
14
+ src = [
15
+ "torch-ext/torch_binding.cpp",
16
+ "torch-ext/torch_binding.h",
17
+ ]
18
+
19
+ [kernel.orbitquant_gemv]
20
+ backend = "cuda"
21
+ depends = ["torch"]
22
+ src = ["orbitquant_gemv_cuda/orbitquant_gemv.cu", "orbitquant_gemv_cuda/fused_ops.cu", "orbitquant_gemv_cuda/qkv_decode.cu", "orbitquant_gemv_cuda/rmsquant.cu"]
src/kernel-source/rms-quant/cmake/_ops.py.in ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ import torch
2
+ from . import @OPS_NAME@
3
+ ops = torch.ops.@OPS_NAME@
4
+
5
+ def add_op_namespace_prefix(op_name: str):
6
+ """
7
+ Prefix op by namespace.
8
+ """
9
+ return f"@OPS_NAME@::{op_name}"
src/kernel-source/rms-quant/cmake/add_build_metadata.py ADDED
@@ -0,0 +1,56 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import argparse
2
+ import json
3
+ import sys
4
+
5
+
6
+ def main():
7
+ parser = argparse.ArgumentParser(
8
+ description="Write a metadata JSON file with GPU architecture information, "
9
+ "reading from a source file and writing to a destination."
10
+ )
11
+ parser.add_argument(
12
+ "input",
13
+ help="Path to the source metadata JSON file to read from.",
14
+ )
15
+ parser.add_argument(
16
+ "destination",
17
+ help="Path to write the output metadata JSON file to.",
18
+ )
19
+
20
+ parser.add_argument(
21
+ "--archs",
22
+ help="Semicolon-separated list of GPU architectures/capabilities.",
23
+ )
24
+
25
+ args = parser.parse_args()
26
+
27
+ archs = (
28
+ sorted(set(a for a in args.archs.split(";") if a))
29
+ if args.archs is not None
30
+ else None
31
+ )
32
+
33
+ try:
34
+ with open(args.input) as f:
35
+ data = json.load(f)
36
+ except FileNotFoundError:
37
+ print(f"Error: input metadata file not found: {args.input}", file=sys.stderr)
38
+ sys.exit(1)
39
+ except json.JSONDecodeError as e:
40
+ print(f"Error: failed to parse input metadata JSON: {e}", file=sys.stderr)
41
+ sys.exit(1)
42
+
43
+ if archs is not None:
44
+ data["backend"]["archs"] = archs
45
+
46
+ try:
47
+ with open(args.destination, "w") as f:
48
+ json.dump(data, f, indent=2)
49
+ f.write("\n")
50
+ except OSError as e:
51
+ print(f"Error: failed to write output metadata JSON: {e}", file=sys.stderr)
52
+ sys.exit(1)
53
+
54
+
55
+ if __name__ == "__main__":
56
+ main()
src/kernel-source/rms-quant/cmake/build-variants.cmake ADDED
@@ -0,0 +1,328 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Generate a standardized build variant name following the pattern:
2
+ # torch<VERSION>-[cxx11-]<COMPUTE>-<ARCH>-<OS>
3
+ # or, when compiled against the Torch stable ABI:
4
+ # torch-stable-abi<VERSION>-<COMPUTE>-<ARCH>-<OS>
5
+ #
6
+ # Arguments:
7
+ # OUT_BUILD_NAME - Output variable name
8
+ # TORCH_VERSION - PyTorch version (e.g., "2.7.1"); ignored when TORCH_STABLE_ABI is set
9
+ # COMPUTE_FRAMEWORK - One of: cuda, rocm, metal, xpu, cpu
10
+ # COMPUTE_VERSION - Version of compute framework (e.g., "12.4" for CUDA, "6.0" for ROCm)
11
+ # Optional for CPU-only builds (pass empty string or omit)
12
+ # Optional keyword arguments:
13
+ # TORCH_STABLE_ABI - Stable ABI version the extension was compiled against (e.g., "2.11");
14
+ # when set, TORCH_VERSION is ignored and the prefix becomes
15
+ # torch-stable-abi<VERSION> (e.g., "2.11" -> "torch-stable-abi211")
16
+ # Example output: torch27-cxx11-cu124-x86_64-linux (Linux)
17
+ # torch27-cu124-x86_64-windows (Windows)
18
+ # torch27-metal-aarch64-darwin (macOS)
19
+ # torch-stable-abi211-cu124-x86_64-linux (Linux, stable ABI)
20
+ #
21
+ function(generate_build_name OUT_BUILD_NAME TORCH_VERSION COMPUTE_FRAMEWORK COMPUTE_VERSION)
22
+ cmake_parse_arguments(ARG "" "TORCH_STABLE_ABI" "" ${ARGN})
23
+
24
+ if(ARG_TORCH_STABLE_ABI)
25
+ # Flatten the stable ABI version (e.g., "2.11" -> "211") and ignore TORCH_VERSION
26
+ string(REPLACE "." ";" VERSION_LIST "${ARG_TORCH_STABLE_ABI}")
27
+ list(LENGTH VERSION_LIST VERSION_COMPONENTS)
28
+ if(VERSION_COMPONENTS LESS 2)
29
+ list(APPEND VERSION_LIST "0")
30
+ endif()
31
+ list(GET VERSION_LIST 0 MAJOR)
32
+ list(GET VERSION_LIST 1 MINOR)
33
+ set(TORCH_PREFIX "torch-stable-abi${MAJOR}${MINOR}")
34
+ else()
35
+ # Flatten version by removing dots and padding to 2 components
36
+ string(REPLACE "." ";" VERSION_LIST "${TORCH_VERSION}")
37
+ list(LENGTH VERSION_LIST VERSION_COMPONENTS)
38
+
39
+ # Pad to at least 2 components
40
+ if(VERSION_COMPONENTS LESS 2)
41
+ list(APPEND VERSION_LIST "0")
42
+ endif()
43
+
44
+ # Take first 2 components and join without dots
45
+ list(GET VERSION_LIST 0 MAJOR)
46
+ list(GET VERSION_LIST 1 MINOR)
47
+ set(TORCH_PREFIX "torch${MAJOR}${MINOR}")
48
+ endif()
49
+
50
+ # Generate compute string
51
+ if(COMPUTE_FRAMEWORK STREQUAL "cuda")
52
+ # Flatten CUDA version (e.g., "12.4" -> "124")
53
+ string(REPLACE "." ";" COMPUTE_VERSION_LIST "${COMPUTE_VERSION}")
54
+ list(LENGTH COMPUTE_VERSION_LIST COMPUTE_COMPONENTS)
55
+ if(COMPUTE_COMPONENTS GREATER_EQUAL 2)
56
+ list(GET COMPUTE_VERSION_LIST 0 COMPUTE_MAJOR)
57
+ list(GET COMPUTE_VERSION_LIST 1 COMPUTE_MINOR)
58
+ set(COMPUTE_STRING "cu${COMPUTE_MAJOR}${COMPUTE_MINOR}")
59
+ else()
60
+ list(GET COMPUTE_VERSION_LIST 0 COMPUTE_MAJOR)
61
+ set(COMPUTE_STRING "cu${COMPUTE_MAJOR}0")
62
+ endif()
63
+ elseif(COMPUTE_FRAMEWORK STREQUAL "rocm")
64
+ # Flatten ROCm version (e.g., "6.0" -> "60")
65
+ string(REPLACE "." ";" COMPUTE_VERSION_LIST "${COMPUTE_VERSION}")
66
+ list(LENGTH COMPUTE_VERSION_LIST COMPUTE_COMPONENTS)
67
+ if(COMPUTE_COMPONENTS GREATER_EQUAL 2)
68
+ list(GET COMPUTE_VERSION_LIST 0 COMPUTE_MAJOR)
69
+ list(GET COMPUTE_VERSION_LIST 1 COMPUTE_MINOR)
70
+ set(COMPUTE_STRING "rocm${COMPUTE_MAJOR}${COMPUTE_MINOR}")
71
+ else()
72
+ list(GET COMPUTE_VERSION_LIST 0 COMPUTE_MAJOR)
73
+ set(COMPUTE_STRING "rocm${COMPUTE_MAJOR}0")
74
+ endif()
75
+ elseif(COMPUTE_FRAMEWORK STREQUAL "xpu")
76
+ # Flatten XPU version (e.g., "2025.2" -> "202552")
77
+ string(REPLACE "." ";" COMPUTE_VERSION_LIST "${COMPUTE_VERSION}")
78
+ list(LENGTH COMPUTE_VERSION_LIST COMPUTE_COMPONENTS)
79
+ if(COMPUTE_COMPONENTS GREATER_EQUAL 2)
80
+ list(GET COMPUTE_VERSION_LIST 0 COMPUTE_MAJOR)
81
+ list(GET COMPUTE_VERSION_LIST 1 COMPUTE_MINOR)
82
+ set(COMPUTE_STRING "xpu${COMPUTE_MAJOR}${COMPUTE_MINOR}")
83
+ else()
84
+ list(GET COMPUTE_VERSION_LIST 0 COMPUTE_MAJOR)
85
+ set(COMPUTE_STRING "xpu${COMPUTE_MAJOR}0")
86
+ endif()
87
+ elseif(COMPUTE_FRAMEWORK STREQUAL "metal")
88
+ set(COMPUTE_STRING "metal")
89
+ elseif(COMPUTE_FRAMEWORK STREQUAL "cpu")
90
+ set(COMPUTE_STRING "cpu")
91
+ else()
92
+ message(FATAL_ERROR "Unknown compute framework: ${COMPUTE_FRAMEWORK}")
93
+ endif()
94
+
95
+ # Detect from target system (CMAKE_SYSTEM_* variables refer to target, not host)
96
+ # Normalize architecture name
97
+ if(CMAKE_SYSTEM_PROCESSOR MATCHES "^(x86_64|amd64|AMD64)$")
98
+ set(CPU_ARCH "x86_64")
99
+ elseif(CMAKE_SYSTEM_PROCESSOR MATCHES "^(aarch64|arm64|ARM64)$")
100
+ set(CPU_ARCH "aarch64")
101
+ else()
102
+ message(FATAL_ERROR "Unsupported architecture: ${CMAKE_SYSTEM_PROCESSOR}")
103
+ endif()
104
+
105
+ # Normalize OS name
106
+ if(CMAKE_SYSTEM_NAME STREQUAL "Windows")
107
+ set(OS_NAME "windows")
108
+ elseif(CMAKE_SYSTEM_NAME STREQUAL "Linux")
109
+ set(OS_NAME "linux")
110
+ elseif(CMAKE_SYSTEM_NAME STREQUAL "Darwin")
111
+ set(OS_NAME "darwin")
112
+ else()
113
+ message(WARNING "Unknown OS ${CMAKE_SYSTEM_NAME}, using as-is")
114
+ string(TOLOWER "${CMAKE_SYSTEM_NAME}" OS_NAME)
115
+ endif()
116
+
117
+ set(ARCH_OS_STRING "${CPU_ARCH}-${OS_NAME}")
118
+
119
+ # Assemble the final build name
120
+ # For non-stable-ABI Linux builds, include cxx11 ABI indicator for compatibility
121
+ if(NOT ARG_TORCH_STABLE_ABI AND ARCH_OS_STRING MATCHES "-linux$")
122
+ set(BUILD_NAME "${TORCH_PREFIX}-cxx11-${COMPUTE_STRING}-${ARCH_OS_STRING}")
123
+ else()
124
+ set(BUILD_NAME "${TORCH_PREFIX}-${COMPUTE_STRING}-${ARCH_OS_STRING}")
125
+ endif()
126
+
127
+ set(${OUT_BUILD_NAME} "${BUILD_NAME}" PARENT_SCOPE)
128
+ message(STATUS "Generated build name: ${BUILD_NAME}")
129
+ endfunction()
130
+
131
+ #
132
+ # Create a custom install target for the huggingface/kernels library layout.
133
+ # This installs the extension into a directory structure suitable for kernel hub discovery:
134
+ # <PREFIX>/<BUILD_VARIANT_NAME>
135
+ #
136
+ # Arguments:
137
+ # TARGET_NAME - Name of the target to create the install rule for
138
+ # PACKAGE_NAME - Python package name (e.g., "activation")
139
+ # BUILD_VARIANT_NAME - Build variant name (e.g., "torch271-cxx11-cu124-x86_64-linux")
140
+ # INSTALL_PREFIX - Base installation directory (defaults to CMAKE_INSTALL_PREFIX)
141
+ # GPU_ARCHS - List of GPU architectures that were compiled
142
+ # (optional; when provided for CUDA/ROCm, metadata.json will include
143
+ # a "backend" key with the type and arch list)
144
+ #
145
+ function(add_kernels_install_target TARGET_NAME PACKAGE_NAME BUILD_VARIANT_NAME)
146
+ set(oneValueArgs INSTALL_PREFIX)
147
+ set(multiValueArgs DATA_EXTENSIONS GPU_ARCHS)
148
+ cmake_parse_arguments(ARG "" "${oneValueArgs}" "${multiValueArgs}" ${ARGN})
149
+
150
+ if(NOT ARG_INSTALL_PREFIX)
151
+ set(ARG_INSTALL_PREFIX "${CMAKE_INSTALL_PREFIX}")
152
+ endif()
153
+
154
+ gpu_lang_to_backend(_BACKEND ${GPU_LANG})
155
+
156
+ # Always include 'py' extension for Python files
157
+ set(ALL_EXTENSIONS ${ARG_DATA_EXTENSIONS})
158
+ list(APPEND ALL_EXTENSIONS "py")
159
+
160
+ # Set the installation directory
161
+ set(KERNEL_INSTALL_DIR "${ARG_INSTALL_PREFIX}/${BUILD_VARIANT_NAME}")
162
+
163
+ message(STATUS "Using PACKAGE_NAME: ${PACKAGE_NAME}")
164
+
165
+ # Install the compiled extension using CMake's install() command
166
+ # This will be triggered by the standard INSTALL target
167
+ install(TARGETS ${TARGET_NAME}
168
+ LIBRARY DESTINATION "${KERNEL_INSTALL_DIR}"
169
+ RUNTIME DESTINATION "${KERNEL_INSTALL_DIR}"
170
+ COMPONENT ${TARGET_NAME})
171
+
172
+ # Install data files with specified extensions
173
+ foreach(ext IN LISTS ALL_EXTENSIONS)
174
+ file(GLOB_RECURSE DATA_FILES RELATIVE "${CMAKE_SOURCE_DIR}/torch-ext/${PACKAGE_NAME}" "${CMAKE_SOURCE_DIR}/torch-ext/${PACKAGE_NAME}/*.${ext}")
175
+ foreach(data_file IN LISTS DATA_FILES)
176
+ get_filename_component(data_file_dir "${data_file}" DIRECTORY)
177
+ install(FILES "${CMAKE_SOURCE_DIR}/torch-ext/${PACKAGE_NAME}/${data_file}"
178
+ DESTINATION "${KERNEL_INSTALL_DIR}/${data_file_dir}"
179
+ COMPONENT ${TARGET_NAME})
180
+ endforeach()
181
+ endforeach()
182
+
183
+ message(STATUS "GPU archs: ${ARG_GPU_ARCHS}")
184
+
185
+ # Add the GPU archs to matadata.json when applicable.
186
+ if((GPU_LANG STREQUAL "CUDA" OR GPU_LANG STREQUAL "HIP") AND ARG_GPU_ARCHS)
187
+ list(JOIN ARG_GPU_ARCHS ";" _GPU_ARCHS_STR)
188
+ install(CODE "
189
+ file(MAKE_DIRECTORY \"${KERNEL_INSTALL_DIR}\")
190
+ execute_process(
191
+ COMMAND \"${Python3_EXECUTABLE}\"
192
+ \"${CMAKE_CURRENT_LIST_DIR}/cmake/add_build_metadata.py\"
193
+ \"${CMAKE_SOURCE_DIR}/metadata-${_BACKEND}.json\"
194
+ \"${KERNEL_INSTALL_DIR}/metadata.json\"
195
+ --archs \"${_GPU_ARCHS_STR}\"
196
+ RESULT_VARIABLE _METADATA_RESULT
197
+ ERROR_VARIABLE _METADATA_ERROR
198
+ )
199
+ if(NOT _METADATA_RESULT EQUAL 0)
200
+ message(WARNING \"Failed to add GPU arch metadata: \${_METADATA_ERROR}\")
201
+ endif()
202
+ " COMPONENT ${TARGET_NAME})
203
+ else()
204
+ install(CODE "
205
+ file(MAKE_DIRECTORY \"${KERNEL_INSTALL_DIR}\")
206
+ execute_process(
207
+ COMMAND \"${Python3_EXECUTABLE}\"
208
+ \"${CMAKE_CURRENT_LIST_DIR}/cmake/add_build_metadata.py\"
209
+ \"${CMAKE_SOURCE_DIR}/metadata-${_BACKEND}.json\"
210
+ \"${KERNEL_INSTALL_DIR}/metadata.json\"
211
+ RESULT_VARIABLE _METADATA_RESULT
212
+ ERROR_VARIABLE _METADATA_ERROR
213
+ )
214
+ if(NOT _METADATA_RESULT EQUAL 0)
215
+ message(WARNING \"Failed to write metadata: \${_METADATA_ERROR}\")
216
+ endif()
217
+ " COMPONENT ${TARGET_NAME})
218
+ endif()
219
+
220
+ # Compatibility with older kernels and direct Python imports.
221
+ install(FILES ${CMAKE_SOURCE_DIR}/compat.py
222
+ DESTINATION "${KERNEL_INSTALL_DIR}/${PACKAGE_NAME}"
223
+ RENAME "__init__.py"
224
+ COMPONENT ${TARGET_NAME})
225
+
226
+ message(STATUS "Added install rules for ${TARGET_NAME} -> ${BUILD_VARIANT_NAME}")
227
+ endfunction()
228
+
229
+ #
230
+ # Add install rules for local development with huggingface/kernels.
231
+ # This installs the extension into the layout expected by get_local_kernel():
232
+ # ${CMAKE_SOURCE_DIR}/build/<BUILD_VARIANT_NAME>/
233
+ #
234
+ # This allows developers to use get_local_kernel() from the kernels library to load
235
+ # locally built kernels without needing to publish to the hub.
236
+ #
237
+ # This uses the standard CMake install() command, so it works with the default
238
+ # "install" target that is always available.
239
+ #
240
+ # Arguments:
241
+ # TARGET_NAME - Name of the target to create the install rule for
242
+ # PACKAGE_NAME - Python package name (e.g., "activation")
243
+ # BUILD_VARIANT_NAME - Build variant name (e.g., "torch271-cxx11-cu124-x86_64-linux")
244
+ # GPU_ARCHS - List of GPU architectures that were compiled
245
+ # (optional; when provided for CUDA/ROCm, metadata.json will include
246
+ # a "backend" key with the type and arch list)
247
+ #
248
+ function(add_local_install_target TARGET_NAME PACKAGE_NAME BUILD_VARIANT_NAME)
249
+ set(oneValueArgs)
250
+ set(multiValueArgs DATA_EXTENSIONS GPU_ARCHS)
251
+ cmake_parse_arguments(ARG "" "${oneValueArgs}" "${multiValueArgs}" ${ARGN})
252
+
253
+ # Always include 'py' extension for Python files
254
+ set(ALL_EXTENSIONS ${ARG_DATA_EXTENSIONS})
255
+ list(APPEND ALL_EXTENSIONS "py")
256
+
257
+ # Define your local, folder based, installation directory
258
+ set(LOCAL_INSTALL_DIR "${CMAKE_SOURCE_DIR}/build/${BUILD_VARIANT_NAME}")
259
+ # Variant directory is where metadata.json should go (for kernels upload discovery)
260
+ set(VARIANT_DIR "${CMAKE_SOURCE_DIR}/build/${BUILD_VARIANT_NAME}")
261
+
262
+ # Create a custom target for local installation
263
+ add_custom_target(local_install
264
+ COMMENT "Installing files to local directory..."
265
+ )
266
+
267
+ gpu_lang_to_backend(_BACKEND ${GPU_LANG})
268
+
269
+ # Copy data files with specified extensions
270
+ foreach(ext IN LISTS ALL_EXTENSIONS)
271
+ file(GLOB_RECURSE DATA_FILES RELATIVE "${CMAKE_SOURCE_DIR}/torch-ext/${PACKAGE_NAME}" "${CMAKE_SOURCE_DIR}/torch-ext/${PACKAGE_NAME}/*.${ext}")
272
+ foreach(data_file IN LISTS DATA_FILES)
273
+ get_filename_component(data_file_dir "${data_file}" DIRECTORY)
274
+ add_custom_command(TARGET local_install POST_BUILD
275
+ COMMAND ${CMAKE_COMMAND} -E make_directory
276
+ ${LOCAL_INSTALL_DIR}/${data_file_dir}
277
+ COMMAND ${CMAKE_COMMAND} -E copy_if_different
278
+ ${CMAKE_SOURCE_DIR}/torch-ext/${PACKAGE_NAME}/${data_file}
279
+ ${LOCAL_INSTALL_DIR}/${data_file_dir}/
280
+ COMMENT "Copying ${data_file} to ${LOCAL_INSTALL_DIR}/${data_file_dir}"
281
+ )
282
+ endforeach()
283
+ endforeach()
284
+
285
+
286
+ # Add the GPU archs to matadata.json when applicable.
287
+ if((GPU_LANG STREQUAL "CUDA" OR GPU_LANG STREQUAL "HIP") AND ARG_GPU_ARCHS)
288
+ list(JOIN ARG_GPU_ARCHS ";" _GPU_ARCHS_STR)
289
+ add_custom_command(TARGET local_install POST_BUILD
290
+ COMMAND ${CMAKE_COMMAND} -E make_directory ${VARIANT_DIR}
291
+ COMMAND ${Python3_EXECUTABLE}
292
+ ${CMAKE_CURRENT_LIST_DIR}/cmake/add_build_metadata.py
293
+ ${CMAKE_SOURCE_DIR}/metadata-${_BACKEND}.json
294
+ ${VARIANT_DIR}/metadata.json
295
+ --archs "${_GPU_ARCHS_STR}"
296
+ COMMENT "Writing metadata.json with GPU arch info to ${VARIANT_DIR}"
297
+ )
298
+ else()
299
+ add_custom_command(TARGET local_install POST_BUILD
300
+ COMMAND ${CMAKE_COMMAND} -E make_directory ${VARIANT_DIR}
301
+ COMMAND ${Python3_EXECUTABLE}
302
+ ${CMAKE_CURRENT_LIST_DIR}/cmake/add_build_metadata.py
303
+ ${CMAKE_SOURCE_DIR}/metadata-${_BACKEND}.json
304
+ ${VARIANT_DIR}/metadata.json
305
+ COMMENT "Writing metadata.json to ${VARIANT_DIR}"
306
+ )
307
+ endif()
308
+
309
+ add_custom_command(TARGET local_install POST_BUILD
310
+ # Copy the shared library
311
+ COMMAND ${CMAKE_COMMAND} -E copy_if_different
312
+ $<TARGET_FILE:${TARGET_NAME}>
313
+ ${LOCAL_INSTALL_DIR}/
314
+
315
+ # Compatibility with older kernels and direct Python imports.
316
+ COMMAND ${CMAKE_COMMAND} -E copy_if_different
317
+ ${CMAKE_SOURCE_DIR}/compat.py
318
+ ${VARIANT_DIR}/${PACKAGE_NAME}/__init__.py
319
+
320
+ COMMENT "Copying shared library and Python files to ${LOCAL_INSTALL_DIR}"
321
+ COMMAND_EXPAND_LISTS
322
+ )
323
+
324
+ # Create both directories: variant dir for metadata.json, package dir for binaries
325
+ file(MAKE_DIRECTORY ${VARIANT_DIR})
326
+ file(MAKE_DIRECTORY ${LOCAL_INSTALL_DIR})
327
+ message(STATUS "Added install rules for ${TARGET_NAME} -> build/${BUILD_VARIANT_NAME}")
328
+ endfunction()
src/kernel-source/rms-quant/cmake/compile-metal.cmake ADDED
@@ -0,0 +1,104 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Metal shader compilation function
2
+ function(compile_metal_shaders TARGET_NAME METAL_SOURCES EXTRA_INCLUDE_DIRS)
3
+ if(NOT DEFINED METAL_TOOLCHAIN)
4
+ execute_process(
5
+ COMMAND "xcodebuild" "-showComponent" "MetalToolchain"
6
+ OUTPUT_VARIABLE FIND_METAL_OUT
7
+ RESULT_VARIABLE FIND_METAL_ERROR_CODE
8
+ ERROR_VARIABLE FIND_METAL_STDERR
9
+ OUTPUT_STRIP_TRAILING_WHITESPACE)
10
+
11
+ if(NOT FIND_METAL_ERROR_CODE EQUAL 0)
12
+ message(FATAL_ERROR "${ERR_MSG}: ${FIND_METAL_STDERR}")
13
+ endif()
14
+
15
+ # Extract the Toolchain Search Path value and append Metal.xctoolchain
16
+ string(REGEX MATCH "Toolchain Search Path: ([^\n]+)" MATCH_RESULT "${FIND_METAL_OUT}")
17
+ set(METAL_TOOLCHAIN "${CMAKE_MATCH_1}/Metal.xctoolchain")
18
+ endif()
19
+
20
+ # Set Metal compiler flags
21
+ set(METAL_FLAGS "-std=metal4.0" "-O2")
22
+
23
+ # Output directory for compiled metallib
24
+ set(METALLIB_OUTPUT_DIR "${CMAKE_BINARY_DIR}/metallib")
25
+ file(MAKE_DIRECTORY ${METALLIB_OUTPUT_DIR})
26
+
27
+ foreach(INC ${EXTRA_INCLUDE_DIRS})
28
+ list(APPEND METAL_FLAGS "-I${INC}")
29
+ endforeach()
30
+
31
+ # Separate .metal files from .h files and compile .metal files to .air
32
+ set(AIR_FILES)
33
+ set(METAL_FILES)
34
+ set(HEADER_FILES)
35
+
36
+ foreach(SOURCE_FILE ${METAL_SOURCES})
37
+ if(SOURCE_FILE MATCHES "\\.metal$")
38
+ list(APPEND METAL_FILES ${SOURCE_FILE})
39
+ elseif(SOURCE_FILE MATCHES "\\.h$")
40
+ list(APPEND HEADER_FILES ${SOURCE_FILE})
41
+ endif()
42
+ endforeach()
43
+
44
+ foreach(METAL_FILE ${METAL_FILES})
45
+ get_filename_component(METAL_NAME ${METAL_FILE} NAME_WE)
46
+ set(AIR_FILE "${CMAKE_BINARY_DIR}/${METAL_NAME}.air")
47
+
48
+ # Include header files as dependencies
49
+ set(ALL_DEPENDENCIES ${CMAKE_CURRENT_SOURCE_DIR}/${METAL_FILE})
50
+ foreach(HEADER_FILE ${HEADER_FILES})
51
+ list(APPEND ALL_DEPENDENCIES ${CMAKE_CURRENT_SOURCE_DIR}/${HEADER_FILE})
52
+ endforeach()
53
+
54
+ add_custom_command(
55
+ OUTPUT ${AIR_FILE}
56
+ COMMAND "${METAL_TOOLCHAIN}/usr/bin/metal" ${METAL_FLAGS}
57
+ -c ${CMAKE_CURRENT_SOURCE_DIR}/${METAL_FILE}
58
+ -o ${AIR_FILE}
59
+ DEPENDS ${ALL_DEPENDENCIES}
60
+ COMMENT "Compiling Metal shader ${METAL_FILE} to ${AIR_FILE}"
61
+ VERBATIM
62
+ )
63
+
64
+ list(APPEND AIR_FILES ${AIR_FILE})
65
+ endforeach()
66
+
67
+ # Link all .air files into a single .metallib
68
+ set(METALLIB_FILE "${METALLIB_OUTPUT_DIR}/${TARGET_NAME}.metallib")
69
+ add_custom_command(
70
+ OUTPUT ${METALLIB_FILE}
71
+ COMMAND "${METAL_TOOLCHAIN}/usr/bin/metallib" ${AIR_FILES}
72
+ -o ${METALLIB_FILE}
73
+ DEPENDS ${AIR_FILES}
74
+ COMMENT "Linking Metal library ${METALLIB_FILE}"
75
+ VERBATIM
76
+ )
77
+
78
+ # Generate C++ header with embedded metallib data
79
+ set(METALLIB_HEADER "${CMAKE_BINARY_DIR}/${TARGET_NAME}_metallib.h")
80
+ set(METALLIB_TO_HEADER_SCRIPT "${CMAKE_CURRENT_SOURCE_DIR}/cmake/metallib_to_header.py")
81
+
82
+ add_custom_command(
83
+ OUTPUT ${METALLIB_HEADER}
84
+ COMMAND ${Python3_EXECUTABLE} ${METALLIB_TO_HEADER_SCRIPT} ${METALLIB_FILE} ${METALLIB_HEADER} ${TARGET_NAME}
85
+ DEPENDS ${METALLIB_FILE} ${METALLIB_TO_HEADER_SCRIPT}
86
+ COMMENT "Generating embedded Metal library header ${METALLIB_HEADER}"
87
+ VERBATIM
88
+ )
89
+
90
+ # Create a custom target for the metallib
91
+ add_custom_target(${TARGET_NAME}_metallib ALL DEPENDS ${METALLIB_FILE} ${METALLIB_HEADER})
92
+
93
+ # Add dependency to main target
94
+ add_dependencies(${TARGET_NAME} ${TARGET_NAME}_metallib)
95
+
96
+ # Add the generated header to include directories
97
+ target_include_directories(${TARGET_NAME} PRIVATE ${CMAKE_BINARY_DIR})
98
+
99
+ # Pass the metallib header and namespace as compile definitions
100
+ target_compile_definitions(${TARGET_NAME} PRIVATE
101
+ EMBEDDED_METALLIB_HEADER="${TARGET_NAME}_metallib.h"
102
+ EMBEDDED_METALLIB_NAMESPACE=${TARGET_NAME}_metal
103
+ )
104
+ endfunction()
src/kernel-source/rms-quant/cmake/get_gpu_lang.cmake ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ #
2
+ # Get the GPU language from Torch.
3
+ #
4
+ function(get_gpu_lang OUT)
5
+ run_python_script(PYTHON_OUT
6
+ "${CMAKE_CURRENT_SOURCE_DIR}/cmake/get_gpu_lang.py"
7
+ "Cannot detect GPU language")
8
+ set(${OUT} ${PYTHON_OUT} PARENT_SCOPE)
9
+ endfunction()
src/kernel-source/rms-quant/cmake/get_gpu_lang.py ADDED
@@ -0,0 +1,20 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+
3
+ import sys
4
+
5
+ try:
6
+ import torch
7
+ except ImportError:
8
+ print("Torch is required for configuring a kernel build.", file=sys.stderr)
9
+ sys.exit(1)
10
+
11
+ if torch.version.cuda is not None:
12
+ print("CUDA")
13
+ elif torch.version.hip is not None:
14
+ print("HIP")
15
+ elif torch.backends.mps.is_available():
16
+ print("METAL")
17
+ elif hasattr(torch.version, "xpu") and torch.version.xpu is not None:
18
+ print("SYCL")
19
+ else:
20
+ print("CPU")
src/kernel-source/rms-quant/cmake/hipify.py ADDED
@@ -0,0 +1,76 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ # SPDX-License-Identifier: Apache-2.0
3
+
4
+ # From vLLM: https://github.com/vllm-project/vllm/blob/main/cmake/hipify.py
5
+
6
+ #
7
+ # A command line tool for running pytorch's hipify preprocessor on CUDA
8
+ # source files.
9
+ #
10
+ # See https://github.com/ROCm/hipify_torch
11
+ # and <torch install dir>/utils/hipify/hipify_python.py
12
+ #
13
+
14
+ import argparse
15
+ import os
16
+ import shutil
17
+
18
+ from torch.utils.hipify.hipify_python import hipify
19
+
20
+ if __name__ == '__main__':
21
+ parser = argparse.ArgumentParser()
22
+
23
+ # Project directory where all the source + include files live.
24
+ parser.add_argument(
25
+ "-p",
26
+ "--project_dir",
27
+ help="The project directory.",
28
+ )
29
+
30
+ # Directory where hipified files are written.
31
+ parser.add_argument(
32
+ "-o",
33
+ "--output_dir",
34
+ help="The output directory.",
35
+ )
36
+
37
+ # Source files to convert.
38
+ parser.add_argument("sources",
39
+ help="Source files to hipify.",
40
+ nargs="*",
41
+ default=[])
42
+
43
+ args = parser.parse_args()
44
+
45
+ # Limit include scope to project_dir only
46
+ includes = [os.path.join(args.project_dir, '*')]
47
+
48
+ # Get absolute path for all source files.
49
+ extra_files = [os.path.abspath(s) for s in args.sources]
50
+
51
+ # Copy sources from project directory to output directory.
52
+ # The directory might already exist to hold object files so we ignore that.
53
+ shutil.copytree(args.project_dir, args.output_dir, dirs_exist_ok=True)
54
+
55
+ hipify_result = hipify(project_directory=args.project_dir,
56
+ output_directory=args.output_dir,
57
+ header_include_dirs=[],
58
+ includes=includes,
59
+ extra_files=extra_files,
60
+ show_detailed=True,
61
+ is_pytorch_extension=True,
62
+ hipify_extra_files_only=True)
63
+
64
+ hipified_sources = []
65
+ for source in args.sources:
66
+ s_abs = os.path.abspath(source)
67
+ hipified_s_abs = (hipify_result[s_abs].hipified_path if
68
+ (s_abs in hipify_result
69
+ and hipify_result[s_abs].hipified_path is not None)
70
+ else s_abs)
71
+ hipified_sources.append(hipified_s_abs)
72
+
73
+ assert (len(hipified_sources) == len(args.sources))
74
+
75
+ # Print hipified source files.
76
+ print("\n".join(hipified_sources))
src/kernel-source/rms-quant/cmake/kernel.cmake ADDED
@@ -0,0 +1,317 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ function(accumulate_gpu_archs OUT_ACC ACC EXTRA_ARCHS)
2
+ list(APPEND ACC ${EXTRA_ARCHS})
3
+ list(REMOVE_DUPLICATES ACC)
4
+ list(SORT ACC)
5
+ set(${OUT_ACC} ${ACC} PARENT_SCOPE)
6
+ endfunction()
7
+
8
+ function(cuda_kernel_component SRC_VAR)
9
+ set(oneValueArgs CUDA_MINVER NAME)
10
+ set(multiValueArgs SOURCES INCLUDES CUDA_CAPABILITIES CUDA_FLAGS CXX_FLAGS)
11
+ cmake_parse_arguments(KERNEL "${options}" "${oneValueArgs}" "${multiValueArgs}" ${ARGN})
12
+
13
+ if(NOT KERNEL_SOURCES)
14
+ message(FATAL_ERROR "cuda_kernel_component: SOURCES argument is required")
15
+ endif()
16
+
17
+ # Bail out if this component is not supported by the CUDA version.
18
+ if(KERNEL_CUDA_MINVER)
19
+ if(CUDA_VERSION VERSION_LESS ${KERNEL_CUDA_MINVER})
20
+ return()
21
+ endif()
22
+ endif()
23
+
24
+ set(_KERNEL_SRC ${KERNEL_SOURCES})
25
+
26
+ if(KERNEL_INCLUDES)
27
+ # TODO: check if CLion support this:
28
+ # https://youtrack.jetbrains.com/issue/CPP-16510/CLion-does-not-handle-per-file-include-directories
29
+ set_source_files_properties(
30
+ ${_KERNEL_SRC}
31
+ PROPERTIES INCLUDE_DIRECTORIES "${KERNEL_INCLUDES}")
32
+ endif()
33
+
34
+ # Determine CUDA architectures
35
+ if(KERNEL_CUDA_CAPABILITIES)
36
+ cuda_archs_loose_intersection(_KERNEL_ARCHS "${KERNEL_CUDA_CAPABILITIES}" "${CUDA_ARCHS}")
37
+ if(NOT _KERNEL_ARCHS)
38
+ message(FATAL_ERROR "CUDA kernel: ${KERNEL_NAME}, empty set of capabilities after intersection (kernel: ${KERNEL_CUDA_CAPABILITIES}, supported: ${CUDA_ARCHS})")
39
+ endif()
40
+ else()
41
+ set(_KERNEL_ARCHS "${CUDA_KERNEL_ARCHS}")
42
+ endif()
43
+ message(STATUS "CUDA kernel: ${KERNEL_NAME}, capabilities: ${_KERNEL_ARCHS}")
44
+ set_gencode_flags_for_srcs(SRCS "${_KERNEL_SRC}" CUDA_ARCHS "${_KERNEL_ARCHS}")
45
+
46
+ accumulate_gpu_archs(_ALL_GPU_ARCHS "${ALL_GPU_ARCHS}" "${_KERNEL_ARCHS}")
47
+ set(ALL_GPU_ARCHS ${_ALL_GPU_ARCHS} PARENT_SCOPE)
48
+
49
+ # Apply CUDA-specific compile flags
50
+ if(KERNEL_CUDA_FLAGS)
51
+ set(_CUDA_FLAGS "${KERNEL_CUDA_FLAGS}")
52
+ # -static-global-template-stub is not supported on CUDA < 12.8. Remove this
53
+ # once we don't support CUDA 12.6 anymore.
54
+ if(CUDA_VERSION VERSION_LESS 12.8)
55
+ string(REGEX REPLACE "-static-global-template-stub=(true|false)" "" _CUDA_FLAGS "${_CUDA_FLAGS}")
56
+ endif()
57
+
58
+ foreach(_SRC ${_KERNEL_SRC})
59
+ if(_SRC MATCHES ".*\\.cu$")
60
+ set_property(
61
+ SOURCE ${_SRC}
62
+ APPEND PROPERTY
63
+ COMPILE_OPTIONS "$<$<COMPILE_LANGUAGE:CUDA>:${_CUDA_FLAGS}>"
64
+ )
65
+ endif()
66
+ endforeach()
67
+ endif()
68
+
69
+ # Apply CXX-specific compile flags
70
+ if(KERNEL_CXX_FLAGS)
71
+ foreach(_SRC ${_KERNEL_SRC})
72
+ set_property(
73
+ SOURCE ${_SRC}
74
+ APPEND PROPERTY
75
+ COMPILE_OPTIONS "$<$<COMPILE_LANGUAGE:CXX>:${KERNEL_CXX_FLAGS}>"
76
+ )
77
+ endforeach()
78
+ endif()
79
+
80
+ set(_TMP_SRC ${${SRC_VAR}})
81
+ list(APPEND _TMP_SRC ${_KERNEL_SRC})
82
+ set(${SRC_VAR} ${_TMP_SRC} PARENT_SCOPE)
83
+ endfunction()
84
+
85
+ function(hip_kernel_component SRC_VAR)
86
+ set(options SUPPORTS_HIPIFY)
87
+ set(oneValueArgs CUDA_MINVER NAME)
88
+ set(multiValueArgs SOURCES INCLUDES CXX_FLAGS HIP_FLAGS ROCM_ARCHS)
89
+ cmake_parse_arguments(KERNEL "${options}" "${oneValueArgs}" "${multiValueArgs}" ${ARGN})
90
+
91
+ if(NOT KERNEL_SOURCES)
92
+ message(FATAL_ERROR "hip_kernel_component: SOURCES argument is required")
93
+ endif()
94
+
95
+ set(_KERNEL_SRC ${KERNEL_SOURCES})
96
+
97
+ if(KERNEL_INCLUDES)
98
+ # TODO: check if CLion support this:
99
+ # https://youtrack.jetbrains.com/issue/CPP-16510/CLion-does-not-handle-per-file-include-directories
100
+ set_source_files_properties(
101
+ ${_KERNEL_SRC}
102
+ PROPERTIES INCLUDE_DIRECTORIES "${KERNEL_INCLUDES}")
103
+ endif()
104
+
105
+ # Apply HIP-specific compile flags
106
+ if(KERNEL_HIP_FLAGS)
107
+ foreach(_SRC ${_KERNEL_SRC})
108
+ if(_SRC MATCHES ".*\\.(cu|hip)$")
109
+ set_property(
110
+ SOURCE ${_SRC}
111
+ APPEND PROPERTY
112
+ COMPILE_OPTIONS "$<$<COMPILE_LANGUAGE:HIP>:${KERNEL_HIP_FLAGS}>"
113
+ )
114
+ endif()
115
+ endforeach()
116
+ endif()
117
+
118
+ # Determine ROCm architectures
119
+ if(KERNEL_ROCM_ARCHS)
120
+ hip_archs_loose_intersection(_KERNEL_ARCHS "${KERNEL_ROCM_ARCHS}" "${ROCM_ARCHS}")
121
+ if(NOT _KERNEL_ARCHS)
122
+ message(FATAL_ERROR "ROCm kernel: ${KERNEL_NAME}, empty set of architectures after intersection (kernel: ${KERNEL_ROCM_ARCHS}, supported: ${ROCM_ARCHS})")
123
+ endif()
124
+ else()
125
+ set(_KERNEL_ARCHS "${ROCM_ARCHS}")
126
+ endif()
127
+ message(STATUS "ROCm kernel: ${KERNEL_NAME}, archs: ${_KERNEL_ARCHS}")
128
+
129
+ accumulate_gpu_archs(_ALL_GPU_ARCHS "${ALL_GPU_ARCHS}" "${_KERNEL_ARCHS}")
130
+ set(ALL_GPU_ARCHS ${_ALL_GPU_ARCHS} PARENT_SCOPE)
131
+
132
+ foreach(_SRC ${_KERNEL_SRC})
133
+ if(_SRC MATCHES ".*\\.(cu|hip)$")
134
+ foreach(_ARCH ${_KERNEL_ARCHS})
135
+ set_property(
136
+ SOURCE ${_SRC}
137
+ APPEND PROPERTY
138
+ COMPILE_OPTIONS "$<$<COMPILE_LANGUAGE:HIP>:--offload-arch=${_ARCH}>"
139
+ )
140
+ endforeach()
141
+ endif()
142
+ endforeach()
143
+
144
+ set(_TMP_SRC ${${SRC_VAR}})
145
+ list(APPEND _TMP_SRC ${_KERNEL_SRC})
146
+ set(${SRC_VAR} ${_TMP_SRC} PARENT_SCOPE)
147
+ endfunction()
148
+
149
+
150
+ function(xpu_kernel_component SRC_VAR)
151
+ set(options)
152
+ set(oneValueArgs)
153
+ set(multiValueArgs SOURCES INCLUDES CXX_FLAGS SYCL_FLAGS)
154
+ cmake_parse_arguments(KERNEL "${options}" "${oneValueArgs}" "${multiValueArgs}" ${ARGN})
155
+
156
+ if(NOT KERNEL_SOURCES)
157
+ message(FATAL_ERROR "xpu_kernel_component: SOURCES argument is required")
158
+ endif()
159
+
160
+ set(_KERNEL_SRC ${KERNEL_SOURCES})
161
+
162
+ # Handle per-file include directories if specified
163
+ if(KERNEL_INCLUDES)
164
+ # TODO: check if CLion support this:
165
+ # https://youtrack.jetbrains.com/issue/CPP-16510/CLion-does-not-handle-per-file-include-directories
166
+ set_source_files_properties(
167
+ ${_KERNEL_SRC}
168
+ PROPERTIES INCLUDE_DIRECTORIES "${KERNEL_INCLUDES}")
169
+ endif()
170
+
171
+ # Apply CXX-specific compile flags
172
+ if(KERNEL_CXX_FLAGS)
173
+ foreach(_SRC ${_KERNEL_SRC})
174
+ set_property(
175
+ SOURCE ${_SRC}
176
+ APPEND PROPERTY
177
+ COMPILE_OPTIONS "$<$<COMPILE_LANGUAGE:CXX>:${KERNEL_CXX_FLAGS}>"
178
+ )
179
+ endforeach()
180
+ endif()
181
+
182
+ # Add SYCL-specific compilation flags for XPU sources
183
+ if(KERNEL_SYCL_FLAGS)
184
+ # Use kernel-specific SYCL flags plus SYCL flags from the parent scope.
185
+ set(_SYCL_FLAGS ${sycl_flags})
186
+ list(APPEND _SYCL_FLAGS ${KERNEL_SYCL_FLAGS})
187
+ foreach(_SRC ${_KERNEL_SRC})
188
+ if(_SRC MATCHES ".*\\.(cpp|cxx|cc)$")
189
+ set_property(
190
+ SOURCE ${_SRC}
191
+ APPEND PROPERTY
192
+ COMPILE_OPTIONS "$<$<COMPILE_LANGUAGE:CXX>:${_SYCL_FLAGS}>"
193
+ )
194
+ endif()
195
+ endforeach()
196
+ else()
197
+ # Use default SYCL flags (from parent scope variable sycl_flags)
198
+ foreach(_SRC ${_KERNEL_SRC})
199
+ if(_SRC MATCHES ".*\\.(cpp|cxx|cc)$")
200
+ set_property(
201
+ SOURCE ${_SRC}
202
+ APPEND PROPERTY
203
+ COMPILE_OPTIONS "$<$<COMPILE_LANGUAGE:CXX>:${sycl_flags}>"
204
+ )
205
+ endif()
206
+ endforeach()
207
+ endif()
208
+
209
+ # Append to parent scope SRC variable
210
+ set(_TMP_SRC ${${SRC_VAR}})
211
+ list(APPEND _TMP_SRC ${_KERNEL_SRC})
212
+ set(${SRC_VAR} ${_TMP_SRC} PARENT_SCOPE)
213
+ endfunction()
214
+
215
+ function(cpu_kernel_component SRC_VAR)
216
+ set(options)
217
+ set(oneValueArgs)
218
+ set(multiValueArgs SOURCES INCLUDES CXX_FLAGS)
219
+ cmake_parse_arguments(KERNEL "${options}" "${oneValueArgs}" "${multiValueArgs}" ${ARGN})
220
+
221
+ if(NOT KERNEL_SOURCES)
222
+ message(FATAL_ERROR "cpu_kernel_component: SOURCES argument is required")
223
+ endif()
224
+
225
+ set(_KERNEL_SRC ${KERNEL_SOURCES})
226
+
227
+ # Handle per-file include directories if specified
228
+ if(KERNEL_INCLUDES)
229
+ # TODO: check if CLion support this:
230
+ # https://youtrack.jetbrains.com/issue/CPP-16510/CLion-does-not-handle-per-file-include-directories
231
+ set_source_files_properties(
232
+ ${_KERNEL_SRC}
233
+ PROPERTIES INCLUDE_DIRECTORIES "${KERNEL_INCLUDES}")
234
+ endif()
235
+
236
+ # Apply CXX-specific compile flags
237
+ if(KERNEL_CXX_FLAGS)
238
+ foreach(_SRC ${_KERNEL_SRC})
239
+ set_property(
240
+ SOURCE ${_SRC}
241
+ APPEND PROPERTY
242
+ COMPILE_OPTIONS "$<$<COMPILE_LANGUAGE:CXX>:${KERNEL_CXX_FLAGS}>"
243
+ )
244
+ endforeach()
245
+ endif()
246
+
247
+ # Append to parent scope SRC variable
248
+ set(_TMP_SRC ${${SRC_VAR}})
249
+ list(APPEND _TMP_SRC ${_KERNEL_SRC})
250
+ set(${SRC_VAR} ${_TMP_SRC} PARENT_SCOPE)
251
+ endfunction()
252
+
253
+ function(metal_kernel_component SRC_VAR)
254
+ set(options)
255
+ set(oneValueArgs)
256
+ set(multiValueArgs SOURCES INCLUDES CXX_FLAGS)
257
+ cmake_parse_arguments(KERNEL "${options}" "${oneValueArgs}" "${multiValueArgs}" ${ARGN})
258
+
259
+ if(NOT KERNEL_SOURCES)
260
+ message(FATAL_ERROR "metal_kernel_component: SOURCES argument is required")
261
+ endif()
262
+
263
+ set(_KERNEL_SRC ${KERNEL_SOURCES})
264
+
265
+ # Separate Metal shader files from other sources
266
+ set(_METAL_SRC)
267
+ set(_CPP_SRC)
268
+
269
+ foreach(_SRC_FILE IN LISTS _KERNEL_SRC)
270
+ if(_SRC_FILE MATCHES "\\.(metal|h)$")
271
+ list(APPEND _METAL_SRC ${_SRC_FILE})
272
+ else()
273
+ list(APPEND _CPP_SRC ${_SRC_FILE})
274
+ endif()
275
+ endforeach()
276
+
277
+ # Handle per-file include directories if specified (for C++ sources only)
278
+ if(KERNEL_INCLUDES AND _CPP_SRC)
279
+ # TODO: check if CLion support this:
280
+ # https://youtrack.jetbrains.com/issue/CPP-16510/CLion-does-not-handle-per-file-include-directories
281
+ set_source_files_properties(
282
+ ${_CPP_SRC}
283
+ PROPERTIES INCLUDE_DIRECTORIES "${KERNEL_INCLUDES}")
284
+ endif()
285
+
286
+ # Apply CXX-specific compile flags
287
+ if(KERNEL_CXX_FLAGS AND _CPP_SRC)
288
+ foreach(_SRC ${_CPP_SRC})
289
+ set_property(
290
+ SOURCE ${_SRC}
291
+ APPEND PROPERTY
292
+ COMPILE_OPTIONS "$<$<COMPILE_LANGUAGE:CXX>:${KERNEL_CXX_FLAGS}>"
293
+ )
294
+ endforeach()
295
+ endif()
296
+
297
+ # Add C++ sources to main source list
298
+ if(_CPP_SRC)
299
+ set(_TMP_SRC ${${SRC_VAR}})
300
+ list(APPEND _TMP_SRC ${_CPP_SRC})
301
+ set(${SRC_VAR} ${_TMP_SRC} PARENT_SCOPE)
302
+ endif()
303
+
304
+ # Keep track of Metal sources for later compilation
305
+ if(_METAL_SRC)
306
+ set(_TMP_METAL ${ALL_METAL_SOURCES})
307
+ list(APPEND _TMP_METAL ${_METAL_SRC})
308
+ set(ALL_METAL_SOURCES ${_TMP_METAL} PARENT_SCOPE)
309
+ endif()
310
+
311
+ # Keep the includes directory for the Metal sources
312
+ if(KERNEL_INCLUDES AND _METAL_SRC)
313
+ set(_TMP_METAL_INCLUDES ${METAL_INCLUDE_DIRS})
314
+ list(APPEND _TMP_METAL_INCLUDES ${KERNEL_INCLUDES})
315
+ set(METAL_INCLUDE_DIRS ${_TMP_METAL_INCLUDES} PARENT_SCOPE)
316
+ endif()
317
+ endfunction()
src/kernel-source/rms-quant/cmake/metallib_to_header.py ADDED
@@ -0,0 +1,73 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ import sys
3
+ import os
4
+
5
+ def convert_metallib_to_header(metallib_path: str, header_path: str, target_name: str) -> None:
6
+ """Convert a metallib binary file to a C++ header with embedded data."""
7
+
8
+ # Read the metallib binary data
9
+ with open(metallib_path, 'rb') as f:
10
+ data: bytes = f.read()
11
+
12
+ # Generate the header content
13
+ header_content: str = """// Auto-generated file containing embedded Metal library
14
+ #pragma once
15
+ #include <cstddef>
16
+ #include <Metal/Metal.h>
17
+
18
+ namespace """ + target_name + """_metal {
19
+ static const unsigned char metallib_data[] = {
20
+ """
21
+
22
+ # Convert binary data to C array format
23
+ bytes_per_line: int = 16
24
+ for i in range(0, len(data), bytes_per_line):
25
+ chunk: bytes = data[i:i + bytes_per_line]
26
+ hex_values: str = ', '.join('0x{:02x}'.format(b) for b in chunk)
27
+ header_content += " " + hex_values + ","
28
+ if i + bytes_per_line < len(data):
29
+ header_content += "\n"
30
+
31
+ header_content += """
32
+ };
33
+ static const size_t metallib_data_len = """ + str(len(data)) + """;
34
+
35
+ // Convenience function to create Metal library from embedded data
36
+ inline id<MTLLibrary> createLibrary(id<MTLDevice> device, NSError** error = nullptr) {
37
+ dispatch_data_t libraryData = dispatch_data_create(
38
+ metallib_data,
39
+ metallib_data_len,
40
+ dispatch_get_main_queue(),
41
+ ^{ /* No cleanup needed for static data */ });
42
+
43
+ NSError* localError = nil;
44
+ id<MTLLibrary> library = [device newLibraryWithData:libraryData error:&localError];
45
+
46
+ if (error) {
47
+ *error = localError;
48
+ }
49
+
50
+ return library;
51
+ }
52
+ } // namespace """ + target_name + """_metal
53
+ """
54
+
55
+ # Write the header file
56
+ dir_path: str = os.path.dirname(header_path)
57
+ if dir_path:
58
+ os.makedirs(dir_path, exist_ok=True)
59
+ with open(header_path, 'w') as f:
60
+ f.write(header_content)
61
+
62
+ print("Generated {} ({} bytes)".format(header_path, len(data)))
63
+
64
+ if __name__ == "__main__":
65
+ if len(sys.argv) != 4:
66
+ print("Usage: metallib_to_header.py <metallib_path> <header_path> <target_name>")
67
+ sys.exit(1)
68
+
69
+ metallib_path: str = sys.argv[1]
70
+ header_path: str = sys.argv[2]
71
+ target_name: str = sys.argv[3]
72
+
73
+ convert_metallib_to_header(metallib_path, header_path, target_name)
src/kernel-source/rms-quant/cmake/utils.cmake ADDED
@@ -0,0 +1,611 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Vendored from vLLM:
2
+ #
3
+ # https://github.com/vllm-project/vllm/blob/main/cmake/utils.cmake
4
+ #
5
+ # Attempt to find the python package that uses the same python executable as
6
+ # `EXECUTABLE` and is one of the `SUPPORTED_VERSIONS`.
7
+ #
8
+ macro (find_python_from_executable EXECUTABLE SUPPORTED_VERSIONS)
9
+ file(REAL_PATH ${EXECUTABLE} EXECUTABLE)
10
+ set(Python3_EXECUTABLE ${EXECUTABLE})
11
+ find_package(Python3 COMPONENTS Interpreter Development.Module Development.SABIModule)
12
+ if (NOT Python3_FOUND)
13
+ message(FATAL_ERROR "Unable to find python matching: ${EXECUTABLE}.")
14
+ endif()
15
+ set(_VER "${Python3_VERSION_MAJOR}.${Python3_VERSION_MINOR}")
16
+ set(_SUPPORTED_VERSIONS_LIST ${SUPPORTED_VERSIONS} ${ARGN})
17
+ if (NOT _VER IN_LIST _SUPPORTED_VERSIONS_LIST)
18
+ message(FATAL_ERROR
19
+ "Python version (${_VER}) is not one of the supported versions: "
20
+ "${_SUPPORTED_VERSIONS_LIST}.")
21
+ endif()
22
+ message(STATUS "Found python matching: ${EXECUTABLE}.")
23
+ endmacro()
24
+
25
+ #
26
+ # Run `EXPR` in python. The standard output of python is stored in `OUT` and
27
+ # has trailing whitespace stripped. If an error is encountered when running
28
+ # python, a fatal message `ERR_MSG` is issued.
29
+ #
30
+ function (run_python OUT EXPR ERR_MSG)
31
+ if(Python3_EXECUTABLE)
32
+ set(_PYTHON_EXECUTABLE "${Python3_EXECUTABLE}")
33
+ elseif(Python_EXECUTABLE)
34
+ set(_PYTHON_EXECUTABLE "${Python_EXECUTABLE}")
35
+ else()
36
+ message(FATAL_ERROR "No Python executable found. Set Python3_EXECUTABLE or Python_EXECUTABLE.")
37
+ endif()
38
+ execute_process(
39
+ COMMAND
40
+ "${_PYTHON_EXECUTABLE}" "-c" "${EXPR}"
41
+ OUTPUT_VARIABLE PYTHON_OUT
42
+ RESULT_VARIABLE PYTHON_ERROR_CODE
43
+ ERROR_VARIABLE PYTHON_STDERR
44
+ OUTPUT_STRIP_TRAILING_WHITESPACE)
45
+
46
+ if(NOT PYTHON_ERROR_CODE EQUAL 0)
47
+ message(FATAL_ERROR "${ERR_MSG}: ${PYTHON_STDERR}")
48
+ endif()
49
+ set(${OUT} ${PYTHON_OUT} PARENT_SCOPE)
50
+ endfunction()
51
+
52
+ #
53
+ # Run `SCRIPT_PATH` in Python. The standard output of Python is stored in
54
+ # `OUT` and has trailing whitespace stripped. If the script exits with a
55
+ # non-zero code, a fatal message `ERR_MSG` is issued.
56
+ #
57
+ function (run_python_script OUT SCRIPT_PATH ERR_MSG)
58
+ if(Python3_EXECUTABLE)
59
+ set(_PYTHON_EXECUTABLE "${Python3_EXECUTABLE}")
60
+ elseif(Python_EXECUTABLE)
61
+ set(_PYTHON_EXECUTABLE "${Python_EXECUTABLE}")
62
+ else()
63
+ message(FATAL_ERROR "No Python executable found. Set Python3_EXECUTABLE or Python_EXECUTABLE.")
64
+ endif()
65
+ execute_process(
66
+ COMMAND
67
+ "${_PYTHON_EXECUTABLE}" "${SCRIPT_PATH}"
68
+ OUTPUT_VARIABLE PYTHON_OUT
69
+ RESULT_VARIABLE PYTHON_ERROR_CODE
70
+ ERROR_VARIABLE PYTHON_STDERR
71
+ OUTPUT_STRIP_TRAILING_WHITESPACE)
72
+
73
+ if(NOT PYTHON_ERROR_CODE EQUAL 0)
74
+ message(FATAL_ERROR "${ERR_MSG}: ${PYTHON_STDERR}")
75
+ endif()
76
+ set(${OUT} ${PYTHON_OUT} PARENT_SCOPE)
77
+ endfunction()
78
+
79
+ #
80
+ # Run `EXPR` in python. The standard output of python is stored in `OUT` and
81
+ # has trailing whitespace stripped. If an error is encountered when running
82
+ # python, `SUCCESS` is set to FALSE. If successful, `SUCCESS` is set to TRUE.
83
+ #
84
+ function (try_run_python OUT SUCCESS EXPR)
85
+ if(Python3_EXECUTABLE)
86
+ set(_PYTHON_EXECUTABLE "${Python3_EXECUTABLE}")
87
+ elseif(Python_EXECUTABLE)
88
+ set(_PYTHON_EXECUTABLE "${Python_EXECUTABLE}")
89
+ else()
90
+ message(FATAL_ERROR "No Python executable found. Set Python3_EXECUTABLE or Python_EXECUTABLE.")
91
+ endif()
92
+ execute_process(
93
+ COMMAND
94
+ "${_PYTHON_EXECUTABLE}" "-c" "${EXPR}"
95
+ OUTPUT_VARIABLE PYTHON_OUT
96
+ RESULT_VARIABLE PYTHON_ERROR_CODE
97
+ ERROR_QUIET
98
+ OUTPUT_STRIP_TRAILING_WHITESPACE)
99
+
100
+ if(NOT PYTHON_ERROR_CODE EQUAL 0)
101
+ set(${SUCCESS} FALSE PARENT_SCOPE)
102
+ set(${OUT} "" PARENT_SCOPE)
103
+ else()
104
+ set(${SUCCESS} TRUE PARENT_SCOPE)
105
+ set(${OUT} ${PYTHON_OUT} PARENT_SCOPE)
106
+ endif()
107
+ endfunction()
108
+
109
+ # Run `EXPR` in python after importing `PKG`. Use the result of this to extend
110
+ # `CMAKE_PREFIX_PATH` so the torch cmake configuration can be imported.
111
+ macro (append_cmake_prefix_path PKG EXPR)
112
+ run_python(_PREFIX_PATH
113
+ "import ${PKG}; print(${EXPR})" "Failed to locate ${PKG} path")
114
+ list(APPEND CMAKE_PREFIX_PATH ${_PREFIX_PATH})
115
+ endmacro()
116
+
117
+ #
118
+ # Add a target named `hipify${NAME}` that runs the hipify preprocessor on a set
119
+ # of CUDA source files. The names of the corresponding "hipified" sources are
120
+ # stored in `OUT_SRCS`.
121
+ #
122
+ function (hipify_sources_target OUT_SRCS NAME ORIG_SRCS)
123
+ #
124
+ # Split into C++ and non-C++ (i.e. CUDA) sources.
125
+ #
126
+ set(NODUP_SRCS ${ORIG_SRCS})
127
+ list(REMOVE_DUPLICATES NODUP_SRCS)
128
+ set(SRCS ${NODUP_SRCS})
129
+ set(CXX_SRCS ${NODUP_SRCS})
130
+ list(FILTER SRCS INCLUDE REGEX "\.cu$")
131
+ list(FILTER CXX_SRCS EXCLUDE REGEX "\.cu$")
132
+
133
+ #
134
+ # Generate ROCm/HIP source file names from CUDA file names.
135
+ # Since HIP files are generated code, they will appear in the build area
136
+ # `CMAKE_CURRENT_BINARY_DIR` directory rather than the original csrc dir.
137
+ #
138
+ set(HIP_SRCS)
139
+ foreach (SRC ${SRCS})
140
+ get_source_file_property(include_dirs "${SRC}" INCLUDE_DIRECTORIES)
141
+ get_source_file_property(compile_options "${SRC}" COMPILE_OPTIONS)
142
+ string(REGEX REPLACE "\.cu$" "\.hip" SRC ${SRC})
143
+ string(REGEX REPLACE "cuda" "hip" SRC ${SRC})
144
+
145
+ if(include_dirs)
146
+ # Copy over include directories from the original CUDA file.
147
+ set_source_files_properties(
148
+ ${SRC}
149
+ PROPERTIES INCLUDE_DIRECTORIES "${include_dirs}")
150
+ endif()
151
+
152
+ if(compile_options)
153
+ set_source_files_properties(
154
+ ${SRC}
155
+ PROPERTIES COMPILE_OPTIONS "${compile_options}")
156
+ endif()
157
+
158
+ list(APPEND HIP_SRCS "${CMAKE_CURRENT_BINARY_DIR}/${SRC}")
159
+ endforeach()
160
+
161
+ add_custom_target(
162
+ hipify${NAME}
163
+ COMMAND "${Python3_EXECUTABLE}" ${CMAKE_SOURCE_DIR}/cmake/hipify.py -p ${CMAKE_SOURCE_DIR} -o ${CMAKE_CURRENT_BINARY_DIR} ${SRCS}
164
+ DEPENDS ${CMAKE_SOURCE_DIR}/cmake/hipify.py ${SRCS}
165
+ BYPRODUCTS ${HIP_SRCS}
166
+ COMMENT "Running hipify on ${NAME} extension source files.")
167
+
168
+ # Swap out original extension sources with hipified sources.
169
+ list(APPEND HIP_SRCS ${CXX_SRCS})
170
+ set(${OUT_SRCS} ${HIP_SRCS} PARENT_SCOPE)
171
+ endfunction()
172
+
173
+ #
174
+ # Get additional GPU compiler flags from torch.
175
+ #
176
+ function (get_torch_gpu_compiler_flags OUT_GPU_FLAGS GPU_LANG)
177
+ if (${GPU_LANG} STREQUAL "CUDA")
178
+ #
179
+ # Get common NVCC flags from torch.
180
+ #
181
+ run_python(GPU_FLAGS
182
+ "from torch.utils.cpp_extension import COMMON_NVCC_FLAGS; print(';'.join(COMMON_NVCC_FLAGS))"
183
+ "Failed to determine torch nvcc compiler flags")
184
+
185
+ if (CUDA_VERSION VERSION_GREATER_EQUAL 11.8)
186
+ list(APPEND GPU_FLAGS "-DENABLE_FP8")
187
+ list(REMOVE_ITEM GPU_FLAGS
188
+ "-D__CUDA_NO_HALF_OPERATORS__"
189
+ "-D__CUDA_NO_HALF_CONVERSIONS__"
190
+ "-D__CUDA_NO_BFLOAT16_CONVERSIONS__"
191
+ "-D__CUDA_NO_HALF2_OPERATORS__")
192
+ endif()
193
+
194
+ elseif(${GPU_LANG} STREQUAL "HIP")
195
+ #
196
+ # Get common HIP/HIPCC flags from torch.
197
+ #
198
+ run_python(GPU_FLAGS
199
+ "import torch.utils.cpp_extension as t; print(';'.join(t.COMMON_HIP_FLAGS + t.COMMON_HIPCC_FLAGS))"
200
+ "Failed to determine torch nvcc compiler flags")
201
+
202
+ list(APPEND GPU_FLAGS
203
+ "-DUSE_ROCM"
204
+ "-DENABLE_FP8"
205
+ "-U__HIP_NO_HALF_CONVERSIONS__"
206
+ "-U__HIP_NO_HALF_OPERATORS__"
207
+ "-fno-gpu-rdc")
208
+
209
+ endif()
210
+ set(${OUT_GPU_FLAGS} ${GPU_FLAGS} PARENT_SCOPE)
211
+ endfunction()
212
+
213
+ # Macro for converting a `gencode` version number to a cmake version number.
214
+ macro(string_to_ver OUT_VER IN_STR)
215
+ string(REGEX REPLACE "\([0-9]+\)\([0-9]\)" "\\1.\\2" ${OUT_VER} ${IN_STR})
216
+ endmacro()
217
+
218
+ #
219
+ # Clear all `-gencode` flags from `CMAKE_CUDA_FLAGS`.
220
+ #
221
+ # Example:
222
+ # CMAKE_CUDA_FLAGS="-Wall -gencode arch=compute_70,code=sm_70 -gencode arch=compute_75,code=sm_75"
223
+ # clear_gencode_flags()
224
+ # CMAKE_CUDA_FLAGS="-Wall"
225
+ #
226
+ macro(clear_gencode_flags)
227
+ # Remove all `-gencode` flags from `CMAKE_CUDA_FLAGS` since they will be modified
228
+ # and passed back via the `CUDA_ARCHITECTURES` property.
229
+ string(REGEX REPLACE "-gencode arch=[^ ]+ *" "" CMAKE_CUDA_FLAGS
230
+ ${CMAKE_CUDA_FLAGS})
231
+ endmacro()
232
+
233
+ #
234
+ # Extract unique CUDA architectures from a list of compute capabilities codes in
235
+ # the form `<major><minor>[<letter>]`, convert them to the form sort
236
+ # `<major>.<minor>`, dedupes them and then sorts them in ascending order and
237
+ # stores them in `OUT_ARCHES`.
238
+ #
239
+ # Example:
240
+ # CUDA_ARCH_FLAGS="-gencode arch=compute_75,code=sm_75;...;-gencode arch=compute_90a,code=sm_90a"
241
+ # extract_unique_cuda_archs_ascending(OUT_ARCHES CUDA_ARCH_FLAGS)
242
+ # OUT_ARCHES="7.5;...;9.0"
243
+ function(extract_unique_cuda_archs_ascending OUT_ARCHES CUDA_ARCH_FLAGS)
244
+ set(_CUDA_ARCHES)
245
+ foreach(_ARCH ${CUDA_ARCH_FLAGS})
246
+ string(REGEX MATCH "arch=compute_\([0-9]+a?\)" _COMPUTE ${_ARCH})
247
+ if (_COMPUTE)
248
+ set(_COMPUTE ${CMAKE_MATCH_1})
249
+ endif()
250
+
251
+ string_to_ver(_COMPUTE_VER ${_COMPUTE})
252
+ list(APPEND _CUDA_ARCHES ${_COMPUTE_VER})
253
+ endforeach()
254
+
255
+ list(REMOVE_DUPLICATES _CUDA_ARCHES)
256
+ list(SORT _CUDA_ARCHES COMPARE NATURAL ORDER ASCENDING)
257
+ set(${OUT_ARCHES} ${_CUDA_ARCHES} PARENT_SCOPE)
258
+ endfunction()
259
+
260
+ #
261
+ # For a specific file set the `-gencode` flag in compile options conditionally
262
+ # for the CUDA language.
263
+ #
264
+ # Example:
265
+ # set_gencode_flag_for_srcs(
266
+ # SRCS "foo.cu"
267
+ # ARCH "compute_75"
268
+ # CODE "sm_75")
269
+ # adds: "-gencode arch=compute_75,code=sm_75" to the compile options for
270
+ # `foo.cu` (only for the CUDA language).
271
+ #
272
+ macro(set_gencode_flag_for_srcs)
273
+ set(options)
274
+ set(oneValueArgs ARCH CODE)
275
+ set(multiValueArgs SRCS)
276
+ cmake_parse_arguments(arg "${options}" "${oneValueArgs}"
277
+ "${multiValueArgs}" ${ARGN} )
278
+ set(_FLAG -gencode arch=${arg_ARCH},code=${arg_CODE})
279
+ set_property(
280
+ SOURCE ${arg_SRCS}
281
+ APPEND PROPERTY
282
+ COMPILE_OPTIONS "$<$<COMPILE_LANGUAGE:CUDA>:${_FLAG}>"
283
+ )
284
+
285
+ message(DEBUG "Setting gencode flag for ${arg_SRCS}: ${_FLAG}")
286
+ endmacro(set_gencode_flag_for_srcs)
287
+
288
+ #
289
+ # For a list of source files set the `-gencode` flags in the files specific
290
+ # compile options (specifically for the CUDA language).
291
+ #
292
+ # arguments are:
293
+ # SRCS: list of source files
294
+ # CUDA_ARCHS: list of CUDA architectures in the form `<major>.<minor>[letter]`
295
+ # BUILD_PTX_FOR_ARCH: if set to true, then the PTX code will be built
296
+ # for architecture `BUILD_PTX_FOR_ARCH` if there is a CUDA_ARCH in CUDA_ARCHS
297
+ # that is larger than BUILD_PTX_FOR_ARCH.
298
+ #
299
+ macro(set_gencode_flags_for_srcs)
300
+ set(options)
301
+ set(oneValueArgs BUILD_PTX_FOR_ARCH)
302
+ set(multiValueArgs SRCS CUDA_ARCHS)
303
+ cmake_parse_arguments(arg "${options}" "${oneValueArgs}"
304
+ "${multiValueArgs}" ${ARGN} )
305
+
306
+ foreach(_ARCH ${arg_CUDA_ARCHS})
307
+ # handle +PTX suffix: generate both sm and ptx codes if requested
308
+ string(FIND "${_ARCH}" "+PTX" _HAS_PTX)
309
+ if(NOT _HAS_PTX EQUAL -1)
310
+ string(REPLACE "+PTX" "" _BASE_ARCH "${_ARCH}")
311
+ string(REPLACE "." "" _STRIPPED_ARCH "${_BASE_ARCH}")
312
+ set_gencode_flag_for_srcs(
313
+ SRCS ${arg_SRCS}
314
+ ARCH "compute_${_STRIPPED_ARCH}"
315
+ CODE "sm_${_STRIPPED_ARCH}")
316
+ set_gencode_flag_for_srcs(
317
+ SRCS ${arg_SRCS}
318
+ ARCH "compute_${_STRIPPED_ARCH}"
319
+ CODE "compute_${_STRIPPED_ARCH}")
320
+ else()
321
+ string(REPLACE "." "" _STRIPPED_ARCH "${_ARCH}")
322
+ set_gencode_flag_for_srcs(
323
+ SRCS ${arg_SRCS}
324
+ ARCH "compute_${_STRIPPED_ARCH}"
325
+ CODE "sm_${_STRIPPED_ARCH}")
326
+ endif()
327
+ endforeach()
328
+
329
+ if (${arg_BUILD_PTX_FOR_ARCH})
330
+ list(SORT arg_CUDA_ARCHS COMPARE NATURAL ORDER ASCENDING)
331
+ list(GET arg_CUDA_ARCHS -1 _HIGHEST_ARCH)
332
+ if (_HIGHEST_ARCH VERSION_GREATER_EQUAL ${arg_BUILD_PTX_FOR_ARCH})
333
+ string(REPLACE "." "" _PTX_ARCH "${arg_BUILD_PTX_FOR_ARCH}")
334
+ set_gencode_flag_for_srcs(
335
+ SRCS ${arg_SRCS}
336
+ ARCH "compute_${_PTX_ARCH}"
337
+ CODE "compute_${_PTX_ARCH}")
338
+ endif()
339
+ endif()
340
+ endmacro()
341
+
342
+ #
343
+ # For the given `SRC_CUDA_ARCHS` list of gencode versions in the form
344
+ # `<major>.<minor>[letter]` compute the "loose intersection" with the
345
+ # `TGT_CUDA_ARCHS` list of gencodes. We also support the `+PTX` suffix in
346
+ # `SRC_CUDA_ARCHS` which indicates that the PTX code should be built when there
347
+ # is a CUDA_ARCH in `TGT_CUDA_ARCHS` that is equal to or larger than the
348
+ # architecture in `SRC_CUDA_ARCHS`.
349
+ # The loose intersection is defined as:
350
+ # { max{ x \in tgt | x <= y } | y \in src, { x \in tgt | x <= y } != {} }
351
+ # where `<=` is the version comparison operator.
352
+ # In other words, for each version in `TGT_CUDA_ARCHS` find the highest version
353
+ # in `SRC_CUDA_ARCHS` that is less or equal to the version in `TGT_CUDA_ARCHS`.
354
+ # We have special handling for x.0a, if x.0a is in `SRC_CUDA_ARCHS` and x.0 is
355
+ # in `TGT_CUDA_ARCHS` then we should remove x.0a from `SRC_CUDA_ARCHS` and add
356
+ # x.0a to the result (and remove x.0 from TGT_CUDA_ARCHS).
357
+ # The result is stored in `OUT_CUDA_ARCHS`.
358
+ #
359
+ # Example:
360
+ # SRC_CUDA_ARCHS="7.5;8.0;8.6;9.0;9.0a"
361
+ # TGT_CUDA_ARCHS="8.0;8.9;9.0"
362
+ # cuda_archs_loose_intersection(OUT_CUDA_ARCHS SRC_CUDA_ARCHS TGT_CUDA_ARCHS)
363
+ # OUT_CUDA_ARCHS="8.0;8.6;9.0;9.0a"
364
+ #
365
+ # Example With PTX:
366
+ # SRC_CUDA_ARCHS="8.0+PTX"
367
+ # TGT_CUDA_ARCHS="9.0"
368
+ # cuda_archs_loose_intersection(OUT_CUDA_ARCHS SRC_CUDA_ARCHS TGT_CUDA_ARCHS)
369
+ # OUT_CUDA_ARCHS="8.0+PTX"
370
+ #
371
+ function(cuda_archs_loose_intersection OUT_CUDA_ARCHS SRC_CUDA_ARCHS TGT_CUDA_ARCHS)
372
+ set(_SRC_CUDA_ARCHS "${SRC_CUDA_ARCHS}")
373
+ set(_TGT_CUDA_ARCHS ${TGT_CUDA_ARCHS})
374
+
375
+ # handle +PTX suffix: separate base arch for matching, record PTX requests
376
+ set(_PTX_ARCHS)
377
+ foreach(_arch ${_SRC_CUDA_ARCHS})
378
+ if(_arch MATCHES "\\+PTX$")
379
+ string(REPLACE "+PTX" "" _base "${_arch}")
380
+ list(APPEND _PTX_ARCHS "${_base}")
381
+ list(REMOVE_ITEM _SRC_CUDA_ARCHS "${_arch}")
382
+ list(APPEND _SRC_CUDA_ARCHS "${_base}")
383
+ endif()
384
+ endforeach()
385
+ list(REMOVE_DUPLICATES _PTX_ARCHS)
386
+ list(REMOVE_DUPLICATES _SRC_CUDA_ARCHS)
387
+
388
+ # If x.0a or x.0f is in SRC_CUDA_ARCHS and x.0 is in CUDA_ARCHS then we should
389
+ # remove x.0a or x.0f from SRC_CUDA_ARCHS and add x.0a or x.0f to _CUDA_ARCHS
390
+ set(_CUDA_ARCHS)
391
+ foreach(_arch ${_SRC_CUDA_ARCHS})
392
+ if(_arch MATCHES "[af]$")
393
+ list(REMOVE_ITEM _SRC_CUDA_ARCHS "${_arch}")
394
+ string(REGEX REPLACE "[af]$" "" _base "${_arch}")
395
+ if ("${_base}" IN_LIST TGT_CUDA_ARCHS)
396
+ list(REMOVE_ITEM _TGT_CUDA_ARCHS "${_base}")
397
+ list(APPEND _CUDA_ARCHS "${_arch}")
398
+ endif()
399
+ endif()
400
+ endforeach()
401
+
402
+ list(SORT _SRC_CUDA_ARCHS COMPARE NATURAL ORDER ASCENDING)
403
+
404
+ # for each ARCH in TGT_CUDA_ARCHS find the highest arch in SRC_CUDA_ARCHS that
405
+ # is less or equal to ARCH (but has the same major version since SASS binary
406
+ # compatibility is only forward compatible within the same major version).
407
+ foreach(_ARCH ${_TGT_CUDA_ARCHS})
408
+ set(_TMP_ARCH)
409
+ # Extract the major version of the target arch
410
+ string(REGEX REPLACE "^([0-9]+)\\..*$" "\\1" TGT_ARCH_MAJOR "${_ARCH}")
411
+ foreach(_SRC_ARCH ${_SRC_CUDA_ARCHS})
412
+ # Extract the major version of the source arch
413
+ string(REGEX REPLACE "^([0-9]+)\\..*$" "\\1" SRC_ARCH_MAJOR "${_SRC_ARCH}")
414
+ # Check version-less-or-equal, and allow PTX arches to match across majors
415
+ if (_SRC_ARCH VERSION_LESS_EQUAL _ARCH)
416
+ if (_SRC_ARCH IN_LIST _PTX_ARCHS OR SRC_ARCH_MAJOR STREQUAL TGT_ARCH_MAJOR)
417
+ set(_TMP_ARCH "${_SRC_ARCH}")
418
+ endif()
419
+ else()
420
+ # If we hit a version greater than the target, we can break
421
+ break()
422
+ endif()
423
+ endforeach()
424
+
425
+ # If we found a matching _TMP_ARCH, append it to _CUDA_ARCHS
426
+ if (_TMP_ARCH)
427
+ list(APPEND _CUDA_ARCHS "${_TMP_ARCH}")
428
+ endif()
429
+ endforeach()
430
+
431
+ list(REMOVE_DUPLICATES _CUDA_ARCHS)
432
+
433
+ # reapply +PTX suffix to architectures that requested PTX
434
+ set(_FINAL_ARCHS)
435
+ foreach(_arch ${_CUDA_ARCHS})
436
+ if(_arch IN_LIST _PTX_ARCHS)
437
+ list(APPEND _FINAL_ARCHS "${_arch}+PTX")
438
+ else()
439
+ list(APPEND _FINAL_ARCHS "${_arch}")
440
+ endif()
441
+ endforeach()
442
+ set(_CUDA_ARCHS ${_FINAL_ARCHS})
443
+
444
+ list(SORT _CUDA_ARCHS COMPARE NATURAL ORDER ASCENDING)
445
+
446
+ set(${OUT_CUDA_ARCHS} ${_CUDA_ARCHS} PARENT_SCOPE)
447
+ endfunction()
448
+
449
+ #
450
+ # For the given `SRC_ROCM_ARCHS` list of architecture versions in the form
451
+ # `<name>` compute the "loose intersection" with the `TGT_ROCM_ARCHS` list.
452
+ # The loose intersection is defined as:
453
+ # { max{ x \in tgt | x <= y } | y \in src, { x \in tgt | x <= y } != {} }
454
+ # where `<=` is the version comparison operator.
455
+ # In other words, for each version in `TGT_ROCM_ARCHS` find the highest version
456
+ # in `SRC_ROCM_ARCHS` that is less or equal to the version in `TGT_ROCM_ARCHS`.
457
+ # The result is stored in `OUT_ROCM_ARCHS`.
458
+ #
459
+ # Example:
460
+ # SRC_ROCM_ARCHS="gfx900;gfx906;gfx908;gfx90a"
461
+ # TGT_ROCM_ARCHS="gfx906;gfx908;gfx1030"
462
+ # hip_archs_loose_intersection(OUT_ROCM_ARCHS SRC_ROCM_ARCHS TGT_ROCM_ARCHS)
463
+ # OUT_ROCM_ARCHS="gfx906;gfx908"
464
+ #
465
+ function(hip_archs_loose_intersection OUT_ROCM_ARCHS SRC_ROCM_ARCHS TGT_ROCM_ARCHS)
466
+ list(REMOVE_DUPLICATES SRC_ROCM_ARCHS)
467
+
468
+ # ROCm architectures are typically in format gfxNNN or gfxNNNx where N is a digit
469
+ # and x is a letter. We can sort them by string comparison which works for this format.
470
+ list(SORT SRC_ROCM_ARCHS COMPARE STRING ORDER ASCENDING)
471
+
472
+ set(_ROCM_ARCHS)
473
+
474
+ # Find the intersection of supported architectures
475
+ foreach(_SRC_ARCH ${SRC_ROCM_ARCHS})
476
+ if(_SRC_ARCH IN_LIST TGT_ROCM_ARCHS)
477
+ list(APPEND _ROCM_ARCHS ${_SRC_ARCH})
478
+ endif()
479
+ endforeach()
480
+
481
+ list(REMOVE_DUPLICATES _ROCM_ARCHS)
482
+ set(${OUT_ROCM_ARCHS} ${_ROCM_ARCHS} PARENT_SCOPE)
483
+ endfunction()
484
+
485
+ function(cuda_remove_ptx_suffixes OUT_CUDA_ARCHS CUDA_ARCHS)
486
+ set(_CUDA_ARCHS "${CUDA_ARCHS}")
487
+
488
+ # handle +PTX suffix: separate base arch for matching, record PTX requests
489
+ foreach(_arch ${CUDA_ARCHS})
490
+ if(_arch MATCHES "\\+PTX$")
491
+ string(REPLACE "+PTX" "" _base "${_arch}")
492
+ list(REMOVE_ITEM _CUDA_ARCHS "${_arch}")
493
+ list(APPEND _CUDA_ARCHS "${_base}")
494
+ endif()
495
+ endforeach()
496
+
497
+ list(REMOVE_DUPLICATES _CUDA_ARCHS)
498
+ list(SORT _CUDA_ARCHS COMPARE NATURAL ORDER ASCENDING)
499
+
500
+ set(${OUT_CUDA_ARCHS} ${_CUDA_ARCHS} PARENT_SCOPE)
501
+ endfunction()
502
+
503
+ #
504
+ # Define a target named `GPU_MOD_NAME` for a single extension. The
505
+ # arguments are:
506
+ #
507
+ # DESTINATION <dest> - Module destination directory.
508
+ # LANGUAGE <lang> - The GPU language for this module, e.g CUDA, HIP,
509
+ # etc.
510
+ # SOURCES <sources> - List of source files relative to CMakeLists.txt
511
+ # directory.
512
+ #
513
+ # Optional arguments:
514
+ #
515
+ # ARCHITECTURES <arches> - A list of target GPU architectures in cmake
516
+ # format.
517
+ # Refer `CMAKE_CUDA_ARCHITECTURES` documentation
518
+ # and `CMAKE_HIP_ARCHITECTURES` for more info.
519
+ # ARCHITECTURES will use cmake's defaults if
520
+ # not provided.
521
+ # COMPILE_FLAGS <flags> - Extra compiler flags passed to NVCC/hip.
522
+ # INCLUDE_DIRECTORIES <dirs> - Extra include directories.
523
+ # LIBRARIES <libraries> - Extra link libraries.
524
+ # WITH_SOABI - Generate library with python SOABI suffix name.
525
+ # USE_SABI <version> - Use python stable api <version>
526
+ #
527
+ # Note: optimization level/debug info is set via cmake build type.
528
+ #
529
+ function (define_gpu_extension_target GPU_MOD_NAME)
530
+ cmake_parse_arguments(PARSE_ARGV 1
531
+ GPU
532
+ "WITH_SOABI"
533
+ "DESTINATION;LANGUAGE;USE_SABI"
534
+ "SOURCES;COMPILE_FLAGS;INCLUDE_DIRECTORIES;LIBRARIES")
535
+
536
+ # Add hipify preprocessing step when building with HIP/ROCm.
537
+ if (GPU_LANGUAGE STREQUAL "HIP")
538
+ hipify_sources_target(GPU_SOURCES ${GPU_MOD_NAME} "${GPU_SOURCES}")
539
+ endif()
540
+
541
+ if (GPU_WITH_SOABI)
542
+ set(GPU_WITH_SOABI WITH_SOABI)
543
+ else()
544
+ set(GPU_WITH_SOABI)
545
+ endif()
546
+
547
+ if (GPU_USE_SABI)
548
+ Python3_add_library(${GPU_MOD_NAME} MODULE USE_SABI ${GPU_USE_SABI} ${GPU_WITH_SOABI} "${GPU_SOURCES}")
549
+ else()
550
+ Python3_add_library(${GPU_MOD_NAME} MODULE ${GPU_WITH_SOABI} "${GPU_SOURCES}")
551
+ endif()
552
+
553
+ if (GPU_LANGUAGE STREQUAL "HIP")
554
+ # Make this target dependent on the hipify preprocessor step.
555
+ add_dependencies(${GPU_MOD_NAME} hipify${GPU_MOD_NAME})
556
+
557
+ # Clear target architectures, we are passing arch flags per source file.
558
+ set_property(TARGET ${GPU_MOD_NAME} PROPERTY HIP_ARCHITECTURES off)
559
+ endif()
560
+
561
+ if (TORCH_VERSION VERSION_LESS 2.12.0)
562
+ set_property(TARGET ${GPU_MOD_NAME} PROPERTY CXX_STANDARD 17)
563
+ else()
564
+ set_property(TARGET ${GPU_MOD_NAME} PROPERTY CXX_STANDARD 20)
565
+ endif()
566
+
567
+ target_compile_options(${GPU_MOD_NAME} PRIVATE
568
+ $<$<COMPILE_LANGUAGE:${GPU_LANGUAGE}>:${GPU_COMPILE_FLAGS}>)
569
+
570
+ target_compile_definitions(${GPU_MOD_NAME} PRIVATE
571
+ "-DTORCH_EXTENSION_NAME=${GPU_MOD_NAME}")
572
+
573
+ target_include_directories(${GPU_MOD_NAME} PRIVATE csrc
574
+ ${GPU_INCLUDE_DIRECTORIES})
575
+
576
+ target_link_libraries(${GPU_MOD_NAME} PRIVATE torch ${GPU_LIBRARIES})
577
+
578
+ # Don't use `TORCH_LIBRARIES` for CUDA since it pulls in a bunch of
579
+ # dependencies that are not necessary and may not be installed.
580
+ if (GPU_LANGUAGE STREQUAL "CUDA")
581
+ target_link_libraries(${GPU_MOD_NAME} PRIVATE CUDA::cudart)
582
+ else()
583
+ target_link_libraries(${GPU_MOD_NAME} PRIVATE ${TORCH_LIBRARIES})
584
+ endif()
585
+
586
+ install(TARGETS ${GPU_MOD_NAME} LIBRARY DESTINATION ${GPU_DESTINATION} COMPONENT ${GPU_MOD_NAME})
587
+ endfunction()
588
+
589
+ # Map a GPU language to its backend name.
590
+ #
591
+ # Arguments:
592
+ # OUT_BACKEND - Output variable name for the backend string
593
+ # GPU_LANG - The GPU language (CPU, CUDA, HIP, METAL, SYCL)
594
+ #
595
+ function(gpu_lang_to_backend OUT_BACKEND GPU_LANG)
596
+ if (${GPU_LANG} STREQUAL "CPU")
597
+ set(_BACKEND "cpu")
598
+ elseif (${GPU_LANG} STREQUAL "CUDA")
599
+ set(_BACKEND "cuda")
600
+ elseif (${GPU_LANG} STREQUAL "HIP")
601
+ set(_BACKEND "rocm")
602
+ elseif (${GPU_LANG} STREQUAL "METAL")
603
+ set(_BACKEND "metal")
604
+ elseif (${GPU_LANG} STREQUAL "SYCL")
605
+ set(_BACKEND "xpu")
606
+ else()
607
+ message(FATAL_ERROR "Unsupported GPU_LANG: ${GPU_LANG}")
608
+ endif()
609
+
610
+ set(${OUT_BACKEND} "${_BACKEND}" PARENT_SCOPE)
611
+ endfunction()
src/kernel-source/rms-quant/compat.py ADDED
@@ -0,0 +1,26 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import ctypes
2
+ import importlib.util
3
+ import sys
4
+ from pathlib import Path
5
+ from types import ModuleType
6
+
7
+
8
+ def _import_from_path(file_path: Path) -> ModuleType:
9
+ # We cannot use the module name as-is, after adding it to `sys.modules`,
10
+ # it would also be used for other imports. So, we make a module name that
11
+ # depends on the path for it to be unique using the hex-encoded hash of
12
+ # the path.
13
+ path_hash = "{:x}".format(ctypes.c_size_t(hash(file_path.absolute())).value)
14
+ module_name = path_hash
15
+ spec = importlib.util.spec_from_file_location(module_name, file_path)
16
+ if spec is None:
17
+ raise ImportError(f"Cannot load spec for {module_name} from {file_path}")
18
+ module = importlib.util.module_from_spec(spec)
19
+ if module is None:
20
+ raise ImportError(f"Cannot load module {module_name} from spec")
21
+ sys.modules[module_name] = module
22
+ spec.loader.exec_module(module) # type: ignore
23
+ return module
24
+
25
+
26
+ globals().update(vars(_import_from_path(Path(__file__).parent.parent / "__init__.py")))
src/kernel-source/rms-quant/flake.nix ADDED
@@ -0,0 +1,11 @@
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ inputs = {
3
+ kernel-builder.url = "github:huggingface/kernels";
4
+ };
5
+ outputs =
6
+ { self, kernel-builder, ... }:
7
+ kernel-builder.lib.genKernelFlakeOutputs {
8
+ inherit self;
9
+ path = ./.;
10
+ };
11
+ }
src/kernel-source/rms-quant/metadata-cann.json ADDED
@@ -0,0 +1,21 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "name": "yue2-qkv-fused",
3
+ "id": "_yue2_qkv_fused_cann_cf0db58",
4
+ "version": 1,
5
+ "license": "Apache-2.0",
6
+ "python-depends": [],
7
+ "backend": {
8
+ "type": "cann"
9
+ },
10
+ "provenance": {
11
+ "kernel-builder": {
12
+ "version": "0.17.0-dev0",
13
+ "sha": "05f9a9517021bff08000f178f8c4d789b04b1320",
14
+ "dirty": false
15
+ },
16
+ "kernel": {
17
+ "sha": "92dbd686220262d8d24b09c27a6d6703e19f9465",
18
+ "dirty": false
19
+ }
20
+ }
21
+ }
src/kernel-source/rms-quant/metadata-cpu.json ADDED
@@ -0,0 +1,21 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "name": "yue2-qkv-fused",
3
+ "id": "_yue2_qkv_fused_cpu_cf0db58",
4
+ "version": 1,
5
+ "license": "Apache-2.0",
6
+ "python-depends": [],
7
+ "backend": {
8
+ "type": "cpu"
9
+ },
10
+ "provenance": {
11
+ "kernel-builder": {
12
+ "version": "0.17.0-dev0",
13
+ "sha": "05f9a9517021bff08000f178f8c4d789b04b1320",
14
+ "dirty": false
15
+ },
16
+ "kernel": {
17
+ "sha": "92dbd686220262d8d24b09c27a6d6703e19f9465",
18
+ "dirty": false
19
+ }
20
+ }
21
+ }
src/kernel-source/rms-quant/metadata-cuda.json ADDED
@@ -0,0 +1,21 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "name": "yue2-qkv-fused",
3
+ "id": "_yue2_qkv_fused_cuda_cf0db58",
4
+ "version": 1,
5
+ "license": "Apache-2.0",
6
+ "python-depends": [],
7
+ "backend": {
8
+ "type": "cuda"
9
+ },
10
+ "provenance": {
11
+ "kernel-builder": {
12
+ "version": "0.17.0-dev0",
13
+ "sha": "05f9a9517021bff08000f178f8c4d789b04b1320",
14
+ "dirty": false
15
+ },
16
+ "kernel": {
17
+ "sha": "92dbd686220262d8d24b09c27a6d6703e19f9465",
18
+ "dirty": false
19
+ }
20
+ }
21
+ }
src/kernel-source/rms-quant/metadata-metal.json ADDED
@@ -0,0 +1,21 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "name": "yue2-qkv-fused",
3
+ "id": "_yue2_qkv_fused_metal_cf0db58",
4
+ "version": 1,
5
+ "license": "Apache-2.0",
6
+ "python-depends": [],
7
+ "backend": {
8
+ "type": "metal"
9
+ },
10
+ "provenance": {
11
+ "kernel-builder": {
12
+ "version": "0.17.0-dev0",
13
+ "sha": "05f9a9517021bff08000f178f8c4d789b04b1320",
14
+ "dirty": false
15
+ },
16
+ "kernel": {
17
+ "sha": "92dbd686220262d8d24b09c27a6d6703e19f9465",
18
+ "dirty": false
19
+ }
20
+ }
21
+ }
src/kernel-source/rms-quant/metadata-neuron.json ADDED
@@ -0,0 +1,21 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "name": "yue2-qkv-fused",
3
+ "id": "_yue2_qkv_fused_neuron_cf0db58",
4
+ "version": 1,
5
+ "license": "Apache-2.0",
6
+ "python-depends": [],
7
+ "backend": {
8
+ "type": "neuron"
9
+ },
10
+ "provenance": {
11
+ "kernel-builder": {
12
+ "version": "0.17.0-dev0",
13
+ "sha": "05f9a9517021bff08000f178f8c4d789b04b1320",
14
+ "dirty": false
15
+ },
16
+ "kernel": {
17
+ "sha": "92dbd686220262d8d24b09c27a6d6703e19f9465",
18
+ "dirty": false
19
+ }
20
+ }
21
+ }
src/kernel-source/rms-quant/metadata-rocm.json ADDED
@@ -0,0 +1,21 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "name": "yue2-qkv-fused",
3
+ "id": "_yue2_qkv_fused_rocm_cf0db58",
4
+ "version": 1,
5
+ "license": "Apache-2.0",
6
+ "python-depends": [],
7
+ "backend": {
8
+ "type": "rocm"
9
+ },
10
+ "provenance": {
11
+ "kernel-builder": {
12
+ "version": "0.17.0-dev0",
13
+ "sha": "05f9a9517021bff08000f178f8c4d789b04b1320",
14
+ "dirty": false
15
+ },
16
+ "kernel": {
17
+ "sha": "92dbd686220262d8d24b09c27a6d6703e19f9465",
18
+ "dirty": false
19
+ }
20
+ }
21
+ }
src/kernel-source/rms-quant/metadata-xpu.json ADDED
@@ -0,0 +1,21 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "name": "yue2-qkv-fused",
3
+ "id": "_yue2_qkv_fused_xpu_cf0db58",
4
+ "version": 1,
5
+ "license": "Apache-2.0",
6
+ "python-depends": [],
7
+ "backend": {
8
+ "type": "xpu"
9
+ },
10
+ "provenance": {
11
+ "kernel-builder": {
12
+ "version": "0.17.0-dev0",
13
+ "sha": "05f9a9517021bff08000f178f8c4d789b04b1320",
14
+ "dirty": false
15
+ },
16
+ "kernel": {
17
+ "sha": "92dbd686220262d8d24b09c27a6d6703e19f9465",
18
+ "dirty": false
19
+ }
20
+ }
21
+ }
src/kernel-source/rms-quant/orbitquant_gemv_cuda/fused_ops.cu ADDED
@@ -0,0 +1,67 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #include <ATen/cuda/CUDAContext.h>
2
+ #include <c10/cuda/CUDAGuard.h>
3
+ #include <c10/cuda/CUDAException.h>
4
+ #include <torch/torch.h>
5
+
6
+ using BF = c10::BFloat16;
7
+ __global__ void rms(BF* out,const BF* x,const BF* w,int dim,float eps) {
8
+ int row=blockIdx.x,t=threadIdx.x;
9
+ float sum=0.f;
10
+ for(int j=t;j<dim;j+=blockDim.x){float v=float(x[row*dim+j]);sum=__fadd_rn(sum,__fmul_rn(v,v));}
11
+ for(int d=16;d;d>>=1)sum=__fadd_rn(sum,__shfl_down_sync(0xffffffff,sum,d));
12
+ __shared__ float warps[8];
13
+ if((t&31)==0)warps[t/32]=sum;
14
+ __syncthreads();
15
+ if(t<32){sum=t<8?warps[t]:0.f;for(int d=16;d;d>>=1)sum=__fadd_rn(sum,__shfl_down_sync(0xffffffff,sum,d));if(t==0)warps[0]=rsqrtf(__fadd_rn(sum/float(dim),eps));}
16
+ __syncthreads();
17
+ BF scale=BF(warps[0]);
18
+ for(int j=t;j<dim;j+=blockDim.x){BF first=BF(__fmul_rn(float(x[row*dim+j]),float(scale)));out[row*dim+j]=BF(__fmul_rn(float(first),float(w[j])));}
19
+ }
20
+ __global__ void rope_kernel(BF* out,const BF* x,const float* c,const float* s,int rows,int dim,int heads){
21
+ int id=blockIdx.x*blockDim.x+threadIdx.x;int half=dim/2;if(id>=rows*half)return;
22
+ int row=id/half,d=id%half,time=row/heads;BF cv=BF(c[time*half+d]),sv=BF(s[time*half+d]);BF a=x[row*dim+d],b=x[row*dim+half+d];
23
+ BF ac=BF(__fmul_rn(float(a),float(cv))),bs=BF(__fmul_rn(float(b),float(sv))),bc=BF(__fmul_rn(float(b),float(cv))),as=BF(__fmul_rn(float(a),float(sv)));
24
+ out[row*dim+d]=BF(__fsub_rn(float(ac),float(bs)));out[row*dim+half+d]=BF(__fadd_rn(float(bc),float(as)));
25
+ }
26
+ __global__ void swiglu_kernel(BF* out,const BF* x,int rows,int dim){
27
+ int id=blockIdx.x*blockDim.x+threadIdx.x;if(id>=rows*dim)return;int row=id/dim,d=id%dim;float g=float(x[row*dim*2+d]);
28
+ BF gate=BF(g/(1.f+expf(-g)));out[id]=BF(__fmul_rn(float(gate),float(x[row*dim*2+dim+d])));
29
+ }
30
+ void common(torch::Tensor const& out,torch::Tensor const& x){
31
+ TORCH_CHECK(x.is_cuda() && x.scalar_type()==torch::kBFloat16 && x.is_contiguous(),"contiguous CUDA BF16 input required");
32
+ TORCH_CHECK(out.device()==x.device() && out.scalar_type()==x.scalar_type() && out.is_contiguous(),"matching contiguous output required");
33
+ }
34
+ void yue2_rmsnorm(torch::Tensor &out,torch::Tensor const &x,torch::Tensor const &weight,double eps){
35
+ common(out,x);TORCH_CHECK(x.dim()>=1 && x.size(-1)>0 && x.size(-1)<=8192 && x.numel()==out.numel(),"invalid RMS shape");
36
+ TORCH_CHECK(weight.device()==x.device() && weight.is_contiguous() && weight.scalar_type()==x.scalar_type() && weight.numel()==x.size(-1),"invalid RMS weight");
37
+ const at::cuda::OptionalCUDAGuard guard(device_of(x));int dim=x.size(-1),rows=x.numel()/dim;if(!rows)return;
38
+ rms<<<rows,256,0,at::cuda::getCurrentCUDAStream()>>>(out.data_ptr<BF>(),x.data_ptr<BF>(),weight.data_ptr<BF>(),dim,eps);C10_CUDA_KERNEL_LAUNCH_CHECK();
39
+ }
40
+ void yue2_rope(torch::Tensor &out,torch::Tensor const &x,torch::Tensor const &cos,torch::Tensor const &sin,int64_t heads){
41
+ common(out,x);TORCH_CHECK(x.dim()>=2 && x.size(-1)%2==0 && x.size(-1)>0 && heads>0 && out.numel()==x.numel(),"invalid RoPE shape");
42
+ for(const auto* t:{&cos,&sin})TORCH_CHECK(t->device()==x.device() && t->scalar_type()==torch::kFloat && t->is_contiguous() && t->numel()==x.numel()/heads/2,"invalid RoPE table");
43
+ const at::cuda::OptionalCUDAGuard guard(device_of(x));int dim=x.size(-1),rows=x.numel()/dim;if(!rows)return;
44
+ rope_kernel<<<(rows*dim/2+255)/256,256,0,at::cuda::getCurrentCUDAStream()>>>(out.data_ptr<BF>(),x.data_ptr<BF>(),cos.data_ptr<float>(),sin.data_ptr<float>(),rows,dim,heads);C10_CUDA_KERNEL_LAUNCH_CHECK();
45
+ }
46
+ void yue2_swiglu(torch::Tensor &out,torch::Tensor const &x){
47
+ common(out,x);TORCH_CHECK(x.dim()>=1 && x.size(-1)>0 && x.size(-1)%2==0 && out.numel()*2==x.numel(),"invalid SwiGLU shape");
48
+ const at::cuda::OptionalCUDAGuard guard(device_of(x));int dim=x.size(-1)/2,rows=x.numel()/dim/2;if(!rows)return;
49
+ swiglu_kernel<<<(out.numel()+255)/256,256,0,at::cuda::getCurrentCUDAStream()>>>(out.data_ptr<BF>(),x.data_ptr<BF>(),rows,dim);C10_CUDA_KERNEL_LAUNCH_CHECK();
50
+ }
51
+
52
+ __global__ void square_bf16(float* out,const BF* x,int64_t count){
53
+ int64_t i=int64_t(blockIdx.x)*blockDim.x+threadIdx.x;if(i<count){float v=float(x[i]);out[i]=__fmul_rn(v,v);}
54
+ }
55
+ __global__ void rms_apply(BF* out,const BF* x,const BF* w,const float* means,int dim,int64_t count,float eps){
56
+ int64_t i=int64_t(blockIdx.x)*blockDim.x+threadIdx.x;if(i>=count)return;
57
+ BF scale=BF(rsqrtf(__fadd_rn(means[i/dim],eps)));BF first=BF(__fmul_rn(float(x[i]),float(scale)));out[i]=BF(__fmul_rn(float(first),float(w[i%dim])));
58
+ }
59
+ void yue2_rmsnorm_exact(torch::Tensor &out,torch::Tensor const &x,torch::Tensor const &weight,double eps){
60
+ common(out,x);TORCH_CHECK(x.dim()>=1 && x.size(-1)>0 && x.numel()==out.numel(),"invalid RMS shape");
61
+ TORCH_CHECK(weight.device()==x.device() && weight.is_contiguous() && weight.scalar_type()==x.scalar_type() && weight.numel()==x.size(-1),"invalid RMS weight");
62
+ const at::cuda::OptionalCUDAGuard guard(device_of(x));if(!x.numel())return;
63
+ auto squared=torch::empty(x.sizes(),x.options().dtype(torch::kFloat));
64
+ square_bf16<<<(x.numel()+255)/256,256,0,at::cuda::getCurrentCUDAStream()>>>(squared.data_ptr<float>(),x.data_ptr<BF>(),x.numel());C10_CUDA_KERNEL_LAUNCH_CHECK();
65
+ auto means=squared.mean(-1,true);
66
+ rms_apply<<<(x.numel()+255)/256,256,0,at::cuda::getCurrentCUDAStream()>>>(out.data_ptr<BF>(),x.data_ptr<BF>(),weight.data_ptr<BF>(),means.data_ptr<float>(),x.size(-1),x.numel(),eps);C10_CUDA_KERNEL_LAUNCH_CHECK();
67
+ }
src/kernel-source/rms-quant/orbitquant_gemv_cuda/orbitquant_gemv.cu ADDED
@@ -0,0 +1,74 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #include <ATen/cuda/CUDAContext.h>
2
+ #include <c10/cuda/CUDAGuard.h>
3
+ #include <c10/cuda/CUDAException.h>
4
+ #include <torch/torch.h>
5
+
6
+ template<class T>
7
+ __global__ void packed_gemv(T* out, const uint8_t* x, const uint8_t* w,
8
+ const float* xn, const c10::BFloat16* wn, const int8_t* ac, const int8_t* wc,
9
+ const T* bias, int n, int packed_k, float as, float ws) {
10
+ const int lane = threadIdx.x & 31;
11
+ const int col = blockIdx.x * 4 + threadIdx.x / 32;
12
+ const int row = blockIdx.y;
13
+ __shared__ int a_codes[16], w_codes[16];
14
+ if (threadIdx.x < 16) { a_codes[threadIdx.x] = ac[threadIdx.x]; w_codes[threadIdx.x] = wc[threadIdx.x]; }
15
+ __syncthreads();
16
+ if (col >= n) return;
17
+ int sum = 0;
18
+ if (packed_k % 4 == 0 && (reinterpret_cast<uintptr_t>(x) & 3) == 0 && (reinterpret_cast<uintptr_t>(w) & 3) == 0) {
19
+ const uint32_t* xp = reinterpret_cast<const uint32_t*>(x + row * packed_k);
20
+ const uint32_t* wp = reinterpret_cast<const uint32_t*>(w + col * packed_k);
21
+ for (int k = lane; k < packed_k / 4; k += 32) {
22
+ const uint32_t av = xp[k], bv = wp[k];
23
+ uint32_t al = 0, ah = 0, bl = 0, bh = 0;
24
+ #pragma unroll
25
+ for (int j = 0; j < 4; ++j) {
26
+ al |= uint32_t(uint8_t(a_codes[(av >> (j * 8)) & 15])) << (j * 8);
27
+ ah |= uint32_t(uint8_t(a_codes[(av >> (j * 8 + 4)) & 15])) << (j * 8);
28
+ bl |= uint32_t(uint8_t(w_codes[(bv >> (j * 8)) & 15])) << (j * 8);
29
+ bh |= uint32_t(uint8_t(w_codes[(bv >> (j * 8 + 4)) & 15])) << (j * 8);
30
+ }
31
+ sum = __dp4a(static_cast<int>(al), static_cast<int>(bl), sum);
32
+ sum = __dp4a(static_cast<int>(ah), static_cast<int>(bh), sum);
33
+ }
34
+ } else {
35
+ for (int k = lane; k < packed_k; k += 32) {
36
+ const uint8_t a = x[row * packed_k + k];
37
+ const uint8_t b = w[col * packed_k + k];
38
+ sum += a_codes[a & 15] * w_codes[b & 15];
39
+ sum += a_codes[a >> 4] * w_codes[b >> 4];
40
+ }
41
+ }
42
+ #pragma unroll
43
+ for (int offset = 16; offset; offset >>= 1) sum += __shfl_down_sync(0xffffffff, sum, offset);
44
+ if (lane == 0) {
45
+ float value = static_cast<float>(sum);
46
+ value *= xn[row] * static_cast<float>(wn[col]) * (as * ws);
47
+ if (bias) value += static_cast<float>(bias[col]);
48
+ out[row * n + col] = static_cast<T>(value);
49
+ }
50
+ }
51
+
52
+ void orbitquant_gemv(torch::Tensor &out, torch::Tensor const &x, torch::Tensor const &w, torch::Tensor const &xn, torch::Tensor const &wn, torch::Tensor const &ac, torch::Tensor const &wc, torch::Tensor const &bias, bool has_bias, double activation_scale, double weight_scale) {
53
+ TORCH_CHECK(x.is_cuda() && x.scalar_type() == torch::kUInt8 && x.dim() == 2, "x must be rank-2 packed CUDA uint8");
54
+ const torch::Tensor* tensors[] = {&out, &w, &xn, &wn, &ac, &wc};
55
+ for (const auto* t : tensors) {
56
+ TORCH_CHECK(t->device() == x.device() && t->is_contiguous(), "tensors must be contiguous on the input CUDA device");
57
+ }
58
+ TORCH_CHECK(x.is_contiguous() && w.scalar_type() == torch::kUInt8, "packed inputs must be contiguous uint8");
59
+ TORCH_CHECK(x.size(0) >= 1 && x.size(0) <= 8 && x.size(1) > 0 && x.size(1) <= 8192, "supported rows 1..8 and packed K 1..8192");
60
+ TORCH_CHECK(wn.scalar_type() == torch::kBFloat16 && xn.scalar_type() == torch::kFloat, "norm dtypes must be BF16 weights and FP32 activations");
61
+ TORCH_CHECK(ac.scalar_type() == torch::kChar && wc.scalar_type() == torch::kChar && ac.numel() == 16 && wc.numel() == 16, "codebooks must contain 16 INT8 entries");
62
+ TORCH_CHECK(out.dim() == 2 && out.size(0) == x.size(0) && out.size(1) == wn.numel() && xn.numel() == x.size(0), "output and norm shapes mismatch");
63
+ TORCH_CHECK(w.numel() == wn.numel() * x.size(1), "packed weight size mismatch");
64
+ TORCH_CHECK(out.scalar_type() == torch::kBFloat16 || out.scalar_type() == torch::kHalf, "output must be BF16 or FP16");
65
+ if (has_bias) TORCH_CHECK(bias.device() == x.device() && bias.is_contiguous() && bias.scalar_type() == out.scalar_type() && bias.numel() == wn.numel(), "invalid bias");
66
+ if (wn.numel() == 0) return;
67
+ const at::cuda::OptionalCUDAGuard guard(device_of(x));
68
+ const dim3 grid((wn.numel() + 3) / 4, x.size(0));
69
+ const auto stream = at::cuda::getCurrentCUDAStream();
70
+ #define LAUNCH(T) packed_gemv<T><<<grid, 128, 0, stream>>>(out.data_ptr<T>(), x.data_ptr<uint8_t>(), w.data_ptr<uint8_t>(), xn.data_ptr<float>(), wn.data_ptr<c10::BFloat16>(), ac.data_ptr<int8_t>(), wc.data_ptr<int8_t>(), has_bias ? bias.data_ptr<T>() : nullptr, wn.numel(), x.size(1), activation_scale, weight_scale)
71
+ if (out.scalar_type() == torch::kBFloat16) { LAUNCH(c10::BFloat16); } else { LAUNCH(c10::Half); }
72
+ #undef LAUNCH
73
+ C10_CUDA_KERNEL_LAUNCH_CHECK();
74
+ }
src/kernel-source/rms-quant/orbitquant_gemv_cuda/qkv_decode.cu ADDED
@@ -0,0 +1,31 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #include <ATen/cuda/CUDAContext.h>
2
+ #include <c10/cuda/CUDAGuard.h>
3
+ #include <c10/cuda/CUDAException.h>
4
+ #include <torch/torch.h>
5
+ using BF=c10::BFloat16;
6
+ __global__ void qkv_decode_kernel(BF* q,BF* keys,BF* values,const BF* x,const BF* qw,const BF* kw,const float* cos,const float* sin,const int64_t* positions,int hq,int hk,int capacity,float eps){
7
+ int h=blockIdx.x,b=blockIdx.y,t=threadIdx.x;bool is_q=h<hq;int head=is_q?h:h-hq;int width=(hq+2*hk)*128;
8
+ int64_t pos=positions[b];CUDA_KERNEL_ASSERT(pos>=0 && pos<capacity);
9
+ BF v=x[b*width+h*128+t];float fv=float(v),sum=__fmul_rn(fv,fv);
10
+ for(int d=16;d;d>>=1)sum=__fadd_rn(sum,__shfl_down_sync(0xffffffff,sum,d));
11
+ __shared__ float warp[4];__shared__ BF normalized[128];
12
+ if((t&31)==0)warp[t/32]=sum;__syncthreads();
13
+ if(t<32){sum=t<4?warp[t]:0.f;for(int d=16;d;d>>=1)sum=__fadd_rn(sum,__shfl_down_sync(0xffffffff,sum,d));if(t==0)warp[0]=rsqrtf(__fadd_rn(sum/128.f,eps));}
14
+ __syncthreads();BF scale=BF(warp[0]);BF first=BF(__fmul_rn(fv,float(scale)));normalized[t]=BF(__fmul_rn(float(first),float((is_q?qw:kw)[t])));
15
+ __syncthreads();
16
+ if(t<64){BF c=BF(cos[b*64+t]),s=BF(sin[b*64+t]);float a=float(normalized[t]),z=float(normalized[t+64]);BF ac=BF(__fmul_rn(a,float(c))),zs=BF(__fmul_rn(z,float(s))),zc=BF(__fmul_rn(z,float(c))),as=BF(__fmul_rn(a,float(s)));
17
+ int64_t index=is_q?(int64_t(b)*hq+head)*128:(int64_t(b)*capacity+pos)*hk*128+head*128;BF* target=is_q?q:keys;target[index+t]=BF(__fsub_rn(float(ac),float(zs)));target[index+t+64]=BF(__fadd_rn(float(zc),float(as)));
18
+ }
19
+ if(!is_q)values[(int64_t(b)*capacity+pos)*hk*128+head*128+t]=x[b*width+(hq+hk+head)*128+t];
20
+ }
21
+ void yue2_qkv_decode(torch::Tensor &q,torch::Tensor &keys,torch::Tensor &values,torch::Tensor const &x,torch::Tensor const &qw,torch::Tensor const &kw,torch::Tensor const &cos,torch::Tensor const &sin,torch::Tensor const &positions,double eps){
22
+ TORCH_CHECK(x.is_cuda() && x.dim()==3 && x.size(1)==1 && x.scalar_type()==torch::kBFloat16 && x.is_contiguous(),"QKV needs contiguous CUDA BF16 [B,1,D]");
23
+ const torch::Tensor* tensors[]={&q,&keys,&values,&qw,&kw};
24
+ for(const auto* t:tensors)TORCH_CHECK(t->device()==x.device() && t->scalar_type()==torch::kBFloat16 && t->is_contiguous(),"QKV output/cache/weights must match input");
25
+ TORCH_CHECK(q.dim()==4 && q.size(0)==x.size(0) && q.size(1)==1 && q.size(3)==128 && keys.dim()==4 && keys.size(0)==x.size(0) && keys.size(3)==128 && values.sizes()==keys.sizes(),"invalid QKV cache shape");
26
+ int b=x.size(0),hq=q.size(2),hk=keys.size(2);TORCH_CHECK(hq>0 && hk>0 && x.size(2)==(hq+2*hk)*128 && qw.numel()==128 && kw.numel()==128,"invalid QKV widths");
27
+ for(const auto* t:{&cos,&sin})TORCH_CHECK(t->device()==x.device() && t->scalar_type()==torch::kFloat && t->is_contiguous() && t->numel()==b*64,"invalid QKV rotary tables");
28
+ TORCH_CHECK(positions.device()==x.device() && positions.scalar_type()==torch::kLong && positions.is_contiguous() && positions.numel()==b,"invalid QKV positions");
29
+ const at::cuda::OptionalCUDAGuard guard(device_of(x));if(!b)return;
30
+ qkv_decode_kernel<<<dim3(hq+hk,b),128,0,at::cuda::getCurrentCUDAStream()>>>(q.data_ptr<BF>(),keys.data_ptr<BF>(),values.data_ptr<BF>(),x.data_ptr<BF>(),qw.data_ptr<BF>(),kw.data_ptr<BF>(),cos.data_ptr<float>(),sin.data_ptr<float>(),positions.data_ptr<int64_t>(),hq,hk,keys.size(1),eps);C10_CUDA_KERNEL_LAUNCH_CHECK();
31
+ }
src/kernel-source/rms-quant/orbitquant_gemv_cuda/rmsquant.cu ADDED
@@ -0,0 +1,126 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #include <ATen/cuda/CUDAContext.h>
2
+ #include <c10/cuda/CUDAGuard.h>
3
+ #include <c10/cuda/CUDAException.h>
4
+ #include <torch/torch.h>
5
+ #include <cmath>
6
+
7
+ using BF = c10::BFloat16;
8
+
9
+ // The positive RMS scale cancels in quantization except for quant_eps.
10
+ // Keep it in the returned token norm; deliberately omit BF16 intermediates.
11
+ __global__ void rmsquant_kernel(uint8_t* packed, float* norms, const BF* x,
12
+ const BF* weight, const int32_t* permutation, const int8_t* signs,
13
+ const float* boundaries, float rms_eps, float quant_eps) {
14
+ constexpr int dim=2048;
15
+ __shared__ float values[dim], raw_sum[256], weighted_sum[256], bounds[15];
16
+ const int t=threadIdx.x, row=blockIdx.x;
17
+ float sx=0, sy=0;
18
+ for(int j=t;j<dim;j+=256){
19
+ int source=permutation[j];
20
+ float raw=float(x[row*dim+source]);
21
+ float weighted=raw*float(weight[source]);
22
+ values[j]=weighted*float(signs[j]);
23
+ sx=fmaf(raw,raw,sx);sy=fmaf(weighted,weighted,sy);
24
+ }
25
+ raw_sum[t]=sx;weighted_sum[t]=sy;
26
+ if(t<15)bounds[t]=boundaries[t];
27
+ __syncthreads();
28
+ for(int stride=128;stride>=32;stride>>=1){
29
+ if(t<stride){raw_sum[t]+=raw_sum[t+stride];weighted_sum[t]+=weighted_sum[t+stride];}
30
+ __syncthreads();
31
+ }
32
+ // Preserve the original reduction tree, but finish inside the first warp.
33
+ if(t<32){
34
+ float raw=raw_sum[t],weighted=weighted_sum[t];
35
+ #pragma unroll
36
+ for(int stride=16;stride;stride>>=1){
37
+ raw+=__shfl_down_sync(0xffffffff,raw,stride);
38
+ weighted+=__shfl_down_sync(0xffffffff,weighted,stride);
39
+ }
40
+ if(t==0){raw_sum[0]=raw;weighted_sum[0]=weighted;}
41
+ }
42
+ __syncthreads();
43
+ float scale=rsqrtf(raw_sum[0]/float(dim)+rms_eps);
44
+ float norm=sqrtf(weighted_sum[0]);
45
+ if(t==0)norms[row]=norm*scale;
46
+ float inv=1.f/(norm+quant_eps/scale);
47
+ float local[8];
48
+ #pragma unroll
49
+ for(int i=0;i<8;i++)local[i]=values[t+256*i]*inv;
50
+ // The first five Hadamard stages stay inside each warp.
51
+ #pragma unroll
52
+ for(int width=1;width<32;width<<=1){
53
+ #pragma unroll
54
+ for(int i=0;i<8;i++){
55
+ float other=__shfl_xor_sync(0xffffffff,local[i],width);
56
+ local[i]=(t&width)?other-local[i]:local[i]+other;
57
+ }
58
+ }
59
+ #pragma unroll
60
+ for(int i=0;i<8;i++)values[t+256*i]=local[i];
61
+ __syncthreads();
62
+ // Only three stages exchange values across warps.
63
+ for(int width=32;width<=128;width<<=1){
64
+ for(int b=t;b<dim/2;b+=256){
65
+ int group=b/width,offset=b-group*width;
66
+ int left=group*width*2+offset,right=left+width;
67
+ float a=values[left],v=values[right];values[left]=a+v;values[right]=a-v;
68
+ }
69
+ __syncthreads();
70
+ }
71
+ #pragma unroll
72
+ for(int i=0;i<8;i++)local[i]=values[t+256*i];
73
+ // Dimensions256/512/1024 are eight register values owned by one thread.
74
+ #pragma unroll
75
+ for(int width=1;width<8;width<<=1){
76
+ #pragma unroll
77
+ for(int b=0;b<4;b++){
78
+ int group=b/width,offset=b-group*width;
79
+ int left=group*width*2+offset,right=left+width;
80
+ float a=local[left],v=local[right];local[left]=a+v;local[right]=a-v;
81
+ }
82
+ }
83
+ // All reads finished before shared storage is reused for packing.
84
+ __syncthreads();
85
+ #pragma unroll
86
+ for(int i=0;i<8;i++)values[t+256*i]=local[i];
87
+ __syncthreads();
88
+ for(int j=t;j<dim/2;j+=256){
89
+ uint8_t codes[2];
90
+ #pragma unroll
91
+ for(int k=0;k<2;k++){
92
+ float v=values[2*j+k]*0.02209708691207961f;
93
+ int index=v>bounds[7]?8:0;
94
+ index+=v>bounds[index+3]?4:0;
95
+ index+=v>bounds[index+1]?2:0;
96
+ index+=v>bounds[index]?1:0;
97
+ codes[k]=index;
98
+ }
99
+ packed[row*(dim/2)+j]=codes[0]|(codes[1]<<4);
100
+ }
101
+ }
102
+
103
+ void rmsquant(torch::Tensor &packed, torch::Tensor &norms, torch::Tensor const &x,
104
+ torch::Tensor const &weight, torch::Tensor const &permutation,
105
+ torch::Tensor const &signs, torch::Tensor const &boundaries,
106
+ double rms_eps, double quant_eps) {
107
+ TORCH_CHECK(x.is_cuda() && x.is_contiguous() && x.scalar_type()==torch::kBFloat16
108
+ && x.dim()==2 && x.size(1)==2048,"requires contiguous CUDA BF16 [rows,2048]");
109
+ for(auto* t:{&weight,&permutation,&signs,&boundaries})
110
+ TORCH_CHECK(t->device()==x.device() && t->is_contiguous() && t->dim()==1,"invalid constant device/layout");
111
+ TORCH_CHECK(weight.scalar_type()==torch::kBFloat16 && weight.numel()==2048,"invalid RMS weight");
112
+ TORCH_CHECK(permutation.scalar_type()==torch::kInt && permutation.numel()==2048,"invalid permutation");
113
+ TORCH_CHECK(signs.scalar_type()==torch::kChar && signs.numel()==2048,"invalid signs");
114
+ TORCH_CHECK(boundaries.scalar_type()==torch::kFloat && boundaries.numel()==15,"invalid boundaries");
115
+ TORCH_CHECK(packed.device()==x.device() && packed.is_contiguous() && packed.scalar_type()==torch::kUInt8
116
+ && packed.dim()==2 && packed.size(0)==x.size(0) && packed.size(1)==1024,"invalid packed output");
117
+ TORCH_CHECK(norms.device()==x.device() && norms.is_contiguous() && norms.scalar_type()==torch::kFloat
118
+ && norms.dim()==1 && norms.numel()==x.size(0),"invalid norm output");
119
+ TORCH_CHECK(std::isfinite(rms_eps) && rms_eps>0 && std::isfinite(quant_eps) && quant_eps>0,"epsilons must be finite and positive");
120
+ const at::cuda::OptionalCUDAGuard guard(device_of(x));
121
+ if(!x.size(0))return;
122
+ rmsquant_kernel<<<x.size(0),256,0,at::cuda::getCurrentCUDAStream()>>>(packed.data_ptr<uint8_t>(),norms.data_ptr<float>(),
123
+ x.data_ptr<BF>(),weight.data_ptr<BF>(),permutation.data_ptr<int32_t>(),signs.data_ptr<int8_t>(),
124
+ boundaries.data_ptr<float>(),rms_eps,quant_eps);
125
+ C10_CUDA_KERNEL_LAUNCH_CHECK();
126
+ }
src/kernel-source/rms-quant/tests/__init__.py ADDED
File without changes
src/kernel-source/rms-quant/tests/test_fused_ops.py ADDED
@@ -0,0 +1,23 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import torch,pytest,json
2
+ from pathlib import Path
3
+ from kernels import get_local_kernel
4
+ kernel=get_local_kernel(Path(__file__).resolve().parents[1]/'build','yue2_qkv_fused')
5
+ @pytest.mark.parametrize('rows,dim',[(1,2048),(2,2048),(16,128),(32,128),(1,6144),(67,2048)])
6
+ def test_rmsnorm(rows,dim):
7
+ torch.manual_seed(12)
8
+ x=(torch.randn(rows,dim,device='cuda')*torch.logspace(-3,3,rows,device='cuda')[:,None]).bfloat16();w=torch.randn(dim,device='cuda',dtype=torch.bfloat16)
9
+ ref=x*torch.rsqrt(x.float().pow(2).mean(-1,keepdim=True)+1e-6).to(x.dtype)*w
10
+ out=kernel.rmsnorm(x,w,1e-6)
11
+ torch.testing.assert_close(out,ref,rtol=.008,atol=1e-5)
12
+ print('rmsnorm exact fraction',rows,dim,float((out==ref).float().mean()))
13
+ @pytest.mark.parametrize('b,t,h',[(1,1,16),(1,1,8),(2,1,16),(1,17,8)])
14
+ def test_rope(b,t,h):
15
+ torch.manual_seed(13);x=torch.randn(b,t,h,128,device='cuda',dtype=torch.bfloat16);a=torch.randn(b,t,1,64,device='cuda');c=a.cos();s=a.sin();cb=c.bfloat16();sb=s.bfloat16();x1,x2=x.chunk(2,-1)
16
+ ref=torch.cat([x1*cb-x2*sb,x2*cb+x1*sb],-1)
17
+ torch.testing.assert_close(kernel.rope(x,c,s),ref,rtol=0,atol=0)
18
+ @pytest.mark.parametrize('rows',[1,2,16])
19
+ def test_swiglu(rows):
20
+ torch.manual_seed(14);x=torch.randn(rows,12288,device='cuda',dtype=torch.bfloat16);g,u=x.chunk(2,-1)
21
+ torch.testing.assert_close(kernel.swiglu(x),torch.nn.functional.silu(g)*u,rtol=0,atol=0)
22
+ def test_invalid_norm_weight():
23
+ with pytest.raises(RuntimeError,match='weight'):kernel.rmsnorm(torch.ones(1,128,device='cuda',dtype=torch.bfloat16),torch.ones(64,device='cuda',dtype=torch.bfloat16),1e-6)
src/kernel-source/rms-quant/tests/test_orbitquant_gemv.py ADDED
@@ -0,0 +1,54 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import json
2
+ from pathlib import Path
3
+ import pytest
4
+ import torch
5
+ from kernels import get_local_kernel
6
+ from orbitquant.kernels.native_packed_matmul import matmul_packed_w4a4_int8_with_native_kernel as original
7
+
8
+ kernel=get_local_kernel(Path(__file__).resolve().parents[1] / 'build', 'yue2_qkv_fused')
9
+
10
+ @pytest.mark.parametrize('rows,n,k',[(1,2048,2048),(2,6144,2048),(8,2048,6144),(1,129,128),(2,1024,2048),(1,129,130),(1,129,16)])
11
+ @pytest.mark.parametrize('dtype',[torch.bfloat16,torch.float16])
12
+ def test_exact_integer_gemv(rows,n,k,dtype):
13
+ torch.manual_seed(123)
14
+ x=torch.randint(0,256,(rows,k//2),device='cuda',dtype=torch.uint8)
15
+ w=torch.randint(0,256,(n*k//2,),device='cuda',dtype=torch.uint8)
16
+ xn=torch.rand(rows,device='cuda');wn=torch.rand(n,device='cuda',dtype=torch.bfloat16)
17
+ ac=torch.randint(-127,128,(16,),device='cuda',dtype=torch.int8);wc=ac.flip(0).contiguous()
18
+ bias=None
19
+ kw=dict(activation_scale=.03125,weight_scale=.0625,bias=bias,output_dtype=dtype)
20
+ if k % 128 == 0:
21
+ ref=original(x,w,xn,wn,ac,wc,out_features=n,in_features=k,**kw)
22
+ else:
23
+ def unpack(z):return torch.stack([z & 15,z >> 4],dim=-1).reshape(z.shape[0],-1).long().cpu()
24
+ aa=ac.cpu().long()[unpack(x)];ww=wc.cpu().long()[unpack(w.reshape(n,k//2))]
25
+ sums=(aa @ ww.T).to(device='cuda',dtype=torch.float32)
26
+ ref=(sums*(xn[:,None]*wn.float()[None,:]*(kw['activation_scale']*kw['weight_scale']))).to(dtype)
27
+ out=kernel.gemv(x,w,xn,wn,ac,wc,**kw)
28
+ torch.testing.assert_close(out,ref,rtol=0,atol=0)
29
+ stream=torch.cuda.Stream();stream.wait_stream(torch.cuda.current_stream())
30
+ with torch.cuda.stream(stream):
31
+ for _ in range(3):kernel.gemv(x,w,xn,wn,ac,wc,**kw)
32
+ torch.cuda.current_stream().wait_stream(stream)
33
+ graph=torch.cuda.CUDAGraph()
34
+ with torch.cuda.graph(graph): captured=kernel.gemv(x,w,xn,wn,ac,wc,**kw)
35
+ graph.replay();torch.cuda.synchronize()
36
+ torch.testing.assert_close(captured,ref,rtol=0,atol=0)
37
+
38
+ def test_rejects_bad_weight_shape():
39
+ x=torch.zeros(1,64,device='cuda',dtype=torch.uint8)
40
+ with pytest.raises(RuntimeError,match='packed weight size'):
41
+ kernel.gemv(x,x.flatten(),torch.ones(1,device='cuda'),torch.ones(2,device='cuda',dtype=torch.bfloat16),torch.zeros(16,device='cuda',dtype=torch.int8),torch.zeros(16,device='cuda',dtype=torch.int8),activation_scale=1,weight_scale=1)
42
+
43
+ @pytest.mark.parametrize('offset',[1,2,3])
44
+ def test_misaligned_contiguous_packed_inputs(offset):
45
+ rows,n,k=1,129,128
46
+ torch.manual_seed(10)
47
+ x=torch.randint(0,256,(rows*k//2+offset,),device='cuda',dtype=torch.uint8)[offset:].view(rows,k//2)
48
+ w=torch.randint(0,256,(n*k//2+offset,),device='cuda',dtype=torch.uint8)[offset:]
49
+ xn=torch.rand(rows,device='cuda');wn=torch.rand(n,device='cuda',dtype=torch.bfloat16)
50
+ ac=torch.randint(-127,128,(16,),device='cuda',dtype=torch.int8);wc=ac.flip(0).contiguous()
51
+ kw=dict(activation_scale=.03125,weight_scale=.0625,bias=None,output_dtype=torch.bfloat16)
52
+ ref=original(x.contiguous().clone(),w.contiguous().clone(),xn,wn,ac,wc,out_features=n,in_features=k,**kw)
53
+ out=kernel.gemv(x,w,xn,wn,ac,wc,**kw)
54
+ torch.testing.assert_close(out,ref,rtol=0,atol=0)
src/kernel-source/rms-quant/tests/test_qkv_decode.py ADDED
@@ -0,0 +1,20 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import torch,pytest
2
+ from pathlib import Path
3
+ from kernels import get_local_kernel
4
+ kernel=get_local_kernel(Path(__file__).resolve().parents[1]/'build','yue2_qkv_fused')
5
+ @pytest.mark.parametrize('b,capacity',[(1,32),(2,257)])
6
+ def test_decode_qkv_cache(b,capacity):
7
+ torch.manual_seed(42);hq,hk,d=16,8,128;x=torch.randn(b,1,(hq+2*hk)*d,device='cuda',dtype=torch.bfloat16);qw=torch.randn(d,device='cuda',dtype=torch.bfloat16);kw=torch.randn(d,device='cuda',dtype=torch.bfloat16)
8
+ a=torch.randn(b,1,d//2,device='cuda');c=a.cos();s=a.sin();positions=torch.tensor([capacity-1-i for i in range(b)],device='cuda');keys=torch.randn(b,capacity,hk,d,device='cuda',dtype=torch.bfloat16);values=keys.clone();expected_k=keys.clone();expected_v=values.clone()
9
+ q,k,v=x.split((hq*d,hk*d,hk*d),-1);q=q.reshape(b,1,hq,d);k=k.reshape(b,1,hk,d);v=v.reshape(b,1,hk,d)
10
+ def norm(z,w):return z*torch.rsqrt(z.float().pow(2).mean(-1,keepdim=True)+1e-6).bfloat16()*w
11
+ def rope(z):
12
+ x1,x2=z.chunk(2,-1);cb=c.unsqueeze(2).bfloat16();sb=s.unsqueeze(2).bfloat16();return torch.cat([x1*cb-x2*sb,x2*cb+x1*sb],-1)
13
+ q=rope(norm(q,qw));k=rope(norm(k,kw));slots=positions[:,None,None,None].expand(b,1,hk,d);expected_k.scatter_(1,slots,k);expected_v.scatter_(1,slots,v)
14
+ actual=kernel.qkv_decode(x,qw,kw,c,s,positions,keys,values,hq,1e-6)
15
+ torch.testing.assert_close(actual,q,rtol=0,atol=0);torch.testing.assert_close(keys,expected_k,rtol=0,atol=0);torch.testing.assert_close(values,expected_v,rtol=0,atol=0)
16
+ @pytest.mark.parametrize('rows,dim',[(1,2048),(16,128),(2,2048),(32,128),(67,2048),(1,6144)])
17
+ def test_exact_rmsnorm(rows,dim):
18
+ torch.manual_seed(456);x=torch.randn(rows,dim,device='cuda',dtype=torch.bfloat16);w=torch.randn(dim,device='cuda',dtype=torch.bfloat16)
19
+ ref=x*torch.rsqrt(x.float().pow(2).mean(-1,keepdim=True)+1e-6).bfloat16()*w
20
+ torch.testing.assert_close(kernel.rmsnorm_exact(x,w,1e-6),ref,rtol=0,atol=0)
src/kernel-source/rms-quant/tests/test_rmsquant.py ADDED
@@ -0,0 +1,56 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from pathlib import Path
2
+ import pytest
3
+ import torch
4
+ from kernels import get_local_kernel
5
+
6
+ kernel=get_local_kernel(Path(__file__).resolve().parents[1]/'build','yue2_qkv_fused')
7
+
8
+ def inputs(rows,scale=1):
9
+ torch.manual_seed(51)
10
+ x=(torch.randn(rows,2048,device='cuda')*scale).bfloat16()
11
+ w=torch.randn(2048,device='cuda').bfloat16()
12
+ p=torch.randperm(2048,device='cuda').int()
13
+ s=(torch.randint(0,2,(2048,),device='cuda')*2-1).to(torch.int8)
14
+ bounds=torch.linspace(-.065,.065,15,device='cuda')
15
+ return x,w,p,s,bounds
16
+
17
+ @pytest.mark.parametrize('rows',[0,1,2,8,32])
18
+ @pytest.mark.parametrize('scale',[0,1e-9,1,100])
19
+ def test_continuous_formula(rows,scale):
20
+ x,w,p,s,b=inputs(rows,scale)
21
+ packed,norms=kernel.rmsquant(x,w,p,s,b)
22
+ y=x.float()*torch.rsqrt(x.float().square().mean(-1,keepdim=True)+1e-6)*w.float()
23
+ expected_norm=y.norm(dim=-1)
24
+ y=y/(expected_norm[:,None]+1e-8)
25
+ y=y[:,p.long()]*s.float()
26
+ width=1
27
+ while width<2048:
28
+ z=y.reshape(rows,2048//(2*width),2,width)
29
+ a,c=z[:,:,0],z[:,:,1]
30
+ y=torch.stack([a+c,a-c],dim=2).reshape(rows,2048)
31
+ width*=2
32
+ codes=torch.bucketize(y/(2048**.5),b)
33
+ expected=(codes[:,0::2]|(codes[:,1::2]<<4)).byte()
34
+ torch.testing.assert_close(norms,expected_norm,rtol=2e-6,atol=1e-9)
35
+ # Only boundaries within FP32 reduction noise may differ.
36
+ if rows:
37
+ assert (expected!=packed).float().mean().item()<.001
38
+ assert torch.isfinite(norms).all()
39
+
40
+ def test_graph_and_invalid_contract():
41
+ args=inputs(2)
42
+ expected=kernel.rmsquant(*args)
43
+ stream=torch.cuda.Stream();stream.wait_stream(torch.cuda.current_stream())
44
+ with torch.cuda.stream(stream):
45
+ for _ in range(3):kernel.rmsquant(*args)
46
+ torch.cuda.current_stream().wait_stream(stream)
47
+ graph=torch.cuda.CUDAGraph()
48
+ with torch.cuda.graph(graph):result=kernel.rmsquant(*args)
49
+ graph.replay();torch.cuda.synchronize()
50
+ for a,b in zip(expected,result):torch.testing.assert_close(a,b,rtol=0,atol=0)
51
+ with pytest.raises(RuntimeError,match='BF16'):
52
+ kernel.rmsquant(args[0].float(),*args[1:])
53
+ with pytest.raises(RuntimeError,match='positive'):
54
+ kernel.rmsquant(*args,rms_eps=0)
55
+ with pytest.raises(RuntimeError,match='permutation'):
56
+ kernel.rmsquant(args[0],args[1],args[2].long(),*args[3:])
src/kernel-source/rms-quant/torch-ext/registration.h ADDED
@@ -0,0 +1,39 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ // Registration macros from vLLM:
2
+ // https://github.com/vllm-project/vllm/blob/main/csrc/core/registration.h
3
+
4
+ #pragma once
5
+
6
+ #include <Python.h>
7
+
8
+ #define _CONCAT(A, B) A##B
9
+ #define CONCAT(A, B) _CONCAT(A, B)
10
+
11
+ #define _STRINGIFY(A) #A
12
+ #define STRINGIFY(A) _STRINGIFY(A)
13
+
14
+ // A version of the TORCH_LIBRARY macro that expands the NAME, i.e. so NAME
15
+ // could be a macro instead of a literal token.
16
+ #define TORCH_LIBRARY_EXPAND(NAME, MODULE) TORCH_LIBRARY(NAME, MODULE)
17
+
18
+ // A version of the STABLE_TORCH_LIBRARY macro that expands the NAME, i.e. so NAME
19
+ // could be a macro instead of a literal token.
20
+ #define STABLE_TORCH_LIBRARY_EXPAND(NAME, MODULE) STABLE_TORCH_LIBRARY(NAME, MODULE)
21
+
22
+ // A version of the TORCH_LIBRARY_IMPL macro that expands the NAME, i.e. so NAME
23
+ // could be a macro instead of a literal token.
24
+ #define TORCH_LIBRARY_IMPL_EXPAND(NAME, DEVICE, MODULE) \
25
+ TORCH_LIBRARY_IMPL(NAME, DEVICE, MODULE)
26
+
27
+ // A version of the STABLE_TORCH_LIBRARY_IMPL macro that expands the NAME, i.e. so NAME
28
+ // could be a macro instead of a literal token.
29
+ #define STABLE_TORCH_LIBRARY_IMPL_EXPAND(NAME, DEVICE, MODULE) \
30
+ STABLE_TORCH_LIBRARY_IMPL(NAME, DEVICE, MODULE)
31
+
32
+ // REGISTER_EXTENSION allows the shared library to be loaded and initialized
33
+ // via python's import statement.
34
+ #define REGISTER_EXTENSION(NAME) \
35
+ PyMODINIT_FUNC CONCAT(PyInit_, NAME)() { \
36
+ static struct PyModuleDef module = {PyModuleDef_HEAD_INIT, \
37
+ STRINGIFY(NAME), nullptr, 0, nullptr}; \
38
+ return PyModule_Create(&module); \
39
+ }
src/kernel-source/rms-quant/torch-ext/torch_binding.cpp ADDED
@@ -0,0 +1,20 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #include <torch/library.h>
2
+ #include "registration.h"
3
+ #include "torch_binding.h"
4
+ TORCH_LIBRARY_EXPAND(TORCH_EXTENSION_NAME, ops) {
5
+ ops.def("rmsquant(Tensor! packed, Tensor! norms, Tensor x, Tensor weight, Tensor permutation, Tensor signs, Tensor boundaries, float rms_eps, float quant_eps) -> ()");
6
+ ops.impl("rmsquant", torch::kCUDA, &rmsquant);
7
+ ops.def("orbitquant_gemv(Tensor! out, Tensor x, Tensor w, Tensor xn, Tensor wn, Tensor ac, Tensor wc, Tensor bias, bool has_bias, float activation_scale, float weight_scale) -> ()");
8
+ ops.impl("orbitquant_gemv", torch::kCUDA, &orbitquant_gemv);
9
+ ops.def("yue2_rmsnorm(Tensor! out, Tensor x, Tensor weight, float eps) -> ()");
10
+ ops.impl("yue2_rmsnorm", torch::kCUDA, &yue2_rmsnorm);
11
+ ops.def("yue2_rope(Tensor! out, Tensor x, Tensor cos, Tensor sin, int heads) -> ()");
12
+ ops.impl("yue2_rope", torch::kCUDA, &yue2_rope);
13
+ ops.def("yue2_swiglu(Tensor! out, Tensor x) -> ()");
14
+ ops.impl("yue2_swiglu", torch::kCUDA, &yue2_swiglu);
15
+ ops.def("yue2_qkv_decode(Tensor! q, Tensor! keys, Tensor! values, Tensor x, Tensor qw, Tensor kw, Tensor cos, Tensor sin, Tensor positions, float eps) -> ()");
16
+ ops.impl("yue2_qkv_decode", torch::kCUDA, &yue2_qkv_decode);
17
+ ops.def("yue2_rmsnorm_exact(Tensor! out, Tensor x, Tensor weight, float eps) -> ()");
18
+ ops.impl("yue2_rmsnorm_exact", torch::kCUDA, &yue2_rmsnorm_exact);
19
+ }
20
+ REGISTER_EXTENSION(TORCH_EXTENSION_NAME)
src/kernel-source/rms-quant/torch-ext/torch_binding.h ADDED
@@ -0,0 +1,13 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #pragma once
2
+ #include <torch/torch.h>
3
+ void orbitquant_gemv(torch::Tensor &out, torch::Tensor const &x, torch::Tensor const &w, torch::Tensor const &xn, torch::Tensor const &wn, torch::Tensor const &ac, torch::Tensor const &wc, torch::Tensor const &bias, bool has_bias, double activation_scale, double weight_scale);
4
+
5
+ void yue2_rmsnorm(torch::Tensor &out, torch::Tensor const &x, torch::Tensor const &weight, double eps);
6
+ void yue2_rope(torch::Tensor &out, torch::Tensor const &x, torch::Tensor const &cos, torch::Tensor const &sin, int64_t heads);
7
+ void yue2_swiglu(torch::Tensor &out, torch::Tensor const &x);
8
+
9
+ void yue2_qkv_decode(torch::Tensor &q, torch::Tensor &keys, torch::Tensor &values, torch::Tensor const &x, torch::Tensor const &qw, torch::Tensor const &kw, torch::Tensor const &cos, torch::Tensor const &sin, torch::Tensor const &positions, double eps);
10
+
11
+ void yue2_rmsnorm_exact(torch::Tensor &out, torch::Tensor const &x, torch::Tensor const &weight, double eps);
12
+
13
+ void rmsquant(torch::Tensor &packed, torch::Tensor &norms, torch::Tensor const &x, torch::Tensor const &weight, torch::Tensor const &permutation, torch::Tensor const &signs, torch::Tensor const &boundaries, double rms_eps, double quant_eps);
src/kernel-source/rms-quant/torch-ext/yue2_qkv_fused/__init__.py ADDED
@@ -0,0 +1,34 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import torch
2
+ from ._ops import ops
3
+
4
+ def gemv(packed_x, packed_w, token_norms, row_norms, activation_codes, weight_codes, *, activation_scale, weight_scale, bias=None, output_dtype=torch.bfloat16):
5
+ out = torch.empty((packed_x.shape[0], row_norms.numel()), device=packed_x.device, dtype=output_dtype)
6
+ empty = out if bias is None else bias
7
+ ops.orbitquant_gemv(out, packed_x, packed_w, token_norms, row_norms, activation_codes, weight_codes, empty, bias is not None, activation_scale, weight_scale)
8
+ return out
9
+
10
+ def rmsnorm(x, weight, eps):
11
+ x=x.contiguous();out=torch.empty_like(x);ops.yue2_rmsnorm(out,x,weight,eps);return out
12
+
13
+ def rope(x, cos, sin):
14
+ x=x.contiguous();out=torch.empty_like(x);ops.yue2_rope(out,x,cos.contiguous(),sin.contiguous(),x.shape[-2]);return out
15
+
16
+ def swiglu(x):
17
+ x=x.contiguous();out=torch.empty((*x.shape[:-1],x.shape[-1]//2),device=x.device,dtype=x.dtype);ops.yue2_swiglu(out,x);return out
18
+
19
+ def qkv_decode(x, qw, kw, cos, sin, positions, keys, values, query_heads, eps):
20
+ q=torch.empty((x.shape[0],1,query_heads,128),device=x.device,dtype=x.dtype)
21
+ ops.yue2_qkv_decode(q,keys,values,x,qw,kw,cos,sin,positions,eps)
22
+ return q
23
+
24
+ def rmsnorm_exact(x, weight, eps):
25
+ x=x.contiguous();out=torch.empty_like(x);ops.yue2_rmsnorm_exact(out,x,weight,eps);return out
26
+
27
+
28
+ def rmsquant(x, weight, permutation, signs, boundaries, rms_eps=1e-6, quant_eps=1e-8):
29
+ shape=x.shape
30
+ x=x.reshape(-1,shape[-1]).contiguous()
31
+ packed=torch.empty((x.shape[0],x.shape[1]//2),device=x.device,dtype=torch.uint8)
32
+ norms=torch.empty(x.shape[0],device=x.device,dtype=torch.float32)
33
+ ops.rmsquant(packed,norms,x,weight,permutation,signs,boundaries,rms_eps,quant_eps)
34
+ return packed,norms
src/requirements.txt CHANGED
@@ -7,4 +7,4 @@ numpy==2.2.6
7
  soundfile==0.13.1
8
  accelerate==1.13.0
9
  kernels==0.12.2
10
- ./wheels/orbitquant-0.9.4-py3-none-any.whl
 
7
  soundfile==0.13.1
8
  accelerate==1.13.0
9
  kernels==0.12.2
10
+ ./wheels/orbitquant-0.9.5-py3-none-any.whl
src/run.py CHANGED
@@ -17,13 +17,15 @@ PROFILES = ("reference", "fused", "fast", "memory")
17
 
18
 
19
  def load_pipeline(*, vae=None, low_memory=False, offload_ar=False, progress=True,
20
- profile=None, ode_steps=None, vae_precision=None):
21
  import torch
22
  profile = profile or ("reference" if low_memory else "fast")
23
  if profile not in PROFILES:
24
  raise ValueError(f"Unknown profile {profile!r}")
25
  if low_memory and profile != "reference":
26
  raise ValueError("Unfused --low-memory requires profile=reference; use profile=memory for staged experts")
 
 
27
  if not torch.cuda.is_available():
28
  raise RuntimeError("This release requires an NVIDIA CUDA GPU.")
29
  if torch.cuda.get_device_capability() != (12, 0):
@@ -44,6 +46,7 @@ def load_pipeline(*, vae=None, low_memory=False, offload_ar=False, progress=True
44
  pipe.fused = not low_memory
45
  pipe.trim_cache = True
46
  pipe.profile = profile
 
47
  fast = profile in {"fast", "memory"}
48
  pipe.generation_config = dataclasses.replace(pipe.generation_config,
49
  ode_steps=ode_steps if ode_steps is not None else 8 if fast else 32)
@@ -69,6 +72,8 @@ def main():
69
  parser.add_argument("--allocator", choices=("native", "expandable"))
70
  parser.add_argument("--low-memory", action="store_true")
71
  parser.add_argument("--offload-ar", action="store_true")
 
 
72
  args = parser.parse_args()
73
  allocator = args.allocator or ("expandable" if (args.profile or "fast") in {"fast", "memory"} and not args.low_memory else None)
74
  if allocator is not None:
@@ -77,7 +82,7 @@ def main():
77
  out = Path(args.output)
78
  out.mkdir(parents=True, exist_ok=True)
79
  with load_pipeline(vae=args.vae, low_memory=args.low_memory, offload_ar=args.offload_ar,
80
- profile=args.profile, ode_steps=args.ode_steps, vae_precision=args.vae_precision) as pipe:
81
  start = time.perf_counter()
82
  song = pipe(**{k: request[k] for k in ("style", "lyrics", "seed", "cot") if k in request})
83
  song.save_artifacts(out)
 
17
 
18
 
19
  def load_pipeline(*, vae=None, low_memory=False, offload_ar=False, progress=True,
20
+ profile=None, ode_steps=None, vae_precision=None, fuse_rms_quant=False):
21
  import torch
22
  profile = profile or ("reference" if low_memory else "fast")
23
  if profile not in PROFILES:
24
  raise ValueError(f"Unknown profile {profile!r}")
25
  if low_memory and profile != "reference":
26
  raise ValueError("Unfused --low-memory requires profile=reference; use profile=memory for staged experts")
27
+ if fuse_rms_quant and profile not in {"fast", "memory"}:
28
+ raise ValueError("RMS/activation fusion requires profile=fast or profile=memory")
29
  if not torch.cuda.is_available():
30
  raise RuntimeError("This release requires an NVIDIA CUDA GPU.")
31
  if torch.cuda.get_device_capability() != (12, 0):
 
46
  pipe.fused = not low_memory
47
  pipe.trim_cache = True
48
  pipe.profile = profile
49
+ pipe.fuse_rms_quant = bool(fuse_rms_quant)
50
  fast = profile in {"fast", "memory"}
51
  pipe.generation_config = dataclasses.replace(pipe.generation_config,
52
  ode_steps=ode_steps if ode_steps is not None else 8 if fast else 32)
 
72
  parser.add_argument("--allocator", choices=("native", "expandable"))
73
  parser.add_argument("--low-memory", action="store_true")
74
  parser.add_argument("--offload-ar", action="store_true")
75
+ parser.add_argument("--fuse-rms-quant", action="store_true",
76
+ help="Fuse RMSNorm into activation quantization; faster experimental arithmetic")
77
  args = parser.parse_args()
78
  allocator = args.allocator or ("expandable" if (args.profile or "fast") in {"fast", "memory"} and not args.low_memory else None)
79
  if allocator is not None:
 
82
  out = Path(args.output)
83
  out.mkdir(parents=True, exist_ok=True)
84
  with load_pipeline(vae=args.vae, low_memory=args.low_memory, offload_ar=args.offload_ar,
85
+ profile=args.profile, ode_steps=args.ode_steps, vae_precision=args.vae_precision, fuse_rms_quant=args.fuse_rms_quant) as pipe:
86
  start = time.perf_counter()
87
  song = pipe(**{k: request[k] for k in ("style", "lyrics", "seed", "cot") if k in request})
88
  song.save_artifacts(out)