FP8 LLMs for vLLM - a neuralmagic Collection

neuralmagic 's Collections

FP8 LLMs for vLLM

Llama-3.1 Quantization

INT8 LLMs for vLLM

INT4 LLMs for vLLM

Sparse Foundational Llama 2 Models

Compression Papers

DeepSparse Sparse LLMs

Sparse Finetuning MPT

Compressed LLMs from the Community

FP8 LLMs for vLLM

updated 23 days ago

Accurate FP8 quantized models by Neural Magic, ready for use with vLLM!

neuralmagic/Meta-Llama-3.1-405B-Instruct-FP8-dynamic

Text Generation • Updated 28 days ago • 1.25k • 13
neuralmagic/Meta-Llama-3.1-405B-Instruct-FP8

Text Generation • Updated 28 days ago • 722 • 27
neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8

Text Generation • Updated 27 days ago • 26k • 27
neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8

Text Generation • Updated 27 days ago • 45.9k • 25
neuralmagic/Meta-Llama-3-8B-Instruct-FP8

Text Generation • Updated Jul 18 • 10.3k • 17
neuralmagic/Meta-Llama-3-70B-Instruct-FP8

Text Generation • Updated Jul 18 • 2.41k • 10
neuralmagic/Mixtral-8x7B-Instruct-v0.1-FP8

Text Generation • Updated Jul 18 • 857 • 2
neuralmagic/Mistral-Nemo-Instruct-2407-FP8

Text Generation • Updated Jul 19 • 2.52k • 13
neuralmagic/Meta-Llama-3-8B-Instruct-FP8-KV

Text Generation • Updated Jun 19 • 12.9k • 6
neuralmagic/Meta-Llama-3-70B-Instruct-FP8-KV

Text Generation • Updated Jun 26 • 245 • 2
neuralmagic/Mixtral-8x22B-Instruct-v0.1-FP8

Text Generation • Updated Aug 12 • 372
neuralmagic/Qwen2-72B-Instruct-FP8

Text Generation • Updated Jul 18 • 899 • 8
neuralmagic/Qwen2-7B-Instruct-FP8

Text Generation • Updated Jul 18 • 425 • 1
neuralmagic/Qwen2-1.5B-Instruct-FP8

Text Generation • Updated Jul 18 • 84
neuralmagic/Qwen2-0.5B-Instruct-FP8

Text Generation • Updated Jul 18 • 266 • 2
neuralmagic/Mistral-7B-Instruct-v0.3-FP8

Text Generation • Updated Jul 18 • 320 • 2
neuralmagic/Llama-2-7b-chat-hf-FP8

Text Generation • Updated Jul 18 • 219
neuralmagic/Phi-3-mini-128k-instruct-FP8

Text Generation • Updated Aug 12 • 320
neuralmagic/Phi-3-medium-128k-instruct-FP8

Text Generation • Updated Aug 12 • 34.4k • 4
neuralmagic/gemma-2-9b-it-FP8

Text Generation • Updated Jul 18 • 447 • 5
neuralmagic/Qwen2-57B-A14B-Instruct-FP8

Text Generation • Updated Jul 18 • 174 • 1
neuralmagic/DeepSeek-Coder-V2-Lite-Instruct-FP8

Text Generation • Updated Jul 18 • 2.07k • 4
neuralmagic/DeepSeek-Coder-V2-Lite-Base-FP8

Text Generation • Updated Jul 18 • 115
neuralmagic/DeepSeek-Coder-V2-Base-FP8

Text Generation • Updated Jul 22 • 308
neuralmagic/DeepSeek-Coder-V2-Instruct-FP8

Text Generation • Updated Jul 22 • 3.49k • 6
neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8-dynamic

Text Generation • Updated 27 days ago • 5.21k • 5
neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8-dynamic

Text Generation • Updated 27 days ago • 1.8k • 2
neuralmagic/Meta-Llama-3.1-8B-FP8

Text Generation • Updated Aug 13 • 1.57k • 5
neuralmagic/Meta-Llama-3.1-70B-FP8

Text Generation • Updated Aug 13 • 217
neuralmagic/starcoder2-15b-FP8

Text Generation • Updated Aug 1 • 55
neuralmagic/starcoder2-3b-FP8

Text Generation • Updated Aug 1 • 12
neuralmagic/starcoder2-7b-FP8

Text Generation • Updated Aug 1 • 6
neuralmagic/Meta-Llama-3.1-405B-FP8

Text Generation • Updated Aug 13 • 7
neuralmagic/gemma-2-2b-it-FP8

Updated Aug 13 • 380 • 1