Any chance you can do a 2.5bpw?
#1
by
justsumguy
- opened
Low VRAM user here. Any chance you can make a 2.5bpw? I unfortunately can't load a 4.0bpw but I know I can load 4x7b 2.5bpw so I've been hoping a 4x8b 2.5bpw would become available and hopefully not take too much more VRAM than a 4x7b.
How much VRAM do you have? I can probably make something.
I'm in the same situation as justsumguy. I'm working with 12gb.
Building a 2.5bpw h6 quaint right now, I'll test if it fits in a 12GB card on RunPod...
Edit: 2.5bpw h8 is up, h8 used a negligible amount more space then h6
https://huggingface.co/FuturisticVibes/L3-Arcania-4x8b-2.5bpw-h8-exl2
Thanks!
FuturisticVibes
changed discussion status to
closed