File size: 1,078 Bytes
3958e85 aa9da76 3958e85 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 |
---
language:
- en
- de
- fr
- it
- pt
- hi
- es
- th
license: llama3.1
pipeline_tag: text-generation
tags:
- facebook
- meta
- pytorch
- llama
- llama-3
- gguf
- imatrix
base_model: meta-llama/Meta-Llama-3.1-70B-Instruct
---
## Still uploading and quantizing, quants will appear 1by1 as they become available.
# Quant Infos
- Requires latest master + [Rope Scaling PR](https://github.com/ggerganov/llama.cpp/pull/8676)
- Might not be perfect yet, but seems to mostly work.
- quants done with an importance matrix for improved quantization loss
- Quantized ggufs & imatrix from hf bf16, through bf16. `safetensors bf16 -> gguf bf16 -> quant` for *optimal* quant loss.
- Wide coverage of different gguf quant types from Q\_8\_0 down to IQ1\_S
- Imatrix generated with [this](https://gist.github.com/bartowski1182/eb213dccb3571f863da82e99418f81e8) multi-purpose dataset by [bartowski](https://huggingface.co/bartowski).
```
./imatrix -m $model_name-bf16.gguf -f calibration_datav3.txt -o $model_name.imatrix
```
# Original Model Card:
TODO |