llama-2-7b FP8-OC
Derived from unsloth/llama-2-7b at revision 93d41f48d9d232fd17620e20fc7d22b03cfdc83c.
LLM Compressor quantizes attention and MLP matrix weights to E4M3FN with one FP32 scale per output channel. HART stores those FP8 bytes and scales in the Bertha EVT0 packed weight format. Other model parameters remain BF16. This checkpoint requires the HyperAccel vLLM plugin.
- Downloads last month
- 13
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support
Model tree for hyper-accel/Llama-2-7b-hf-FP8-OC
Base model
unsloth/llama-2-7b