Go/No Go
#2
by
HighlandGNU
- opened
Thank you for creating and publishing this model. The 13b version is brilliant.
When announcing your models, I hope that you will consider accompanying them with a couple of brief statements:
- The minimum consumer grade hardware that would be required to run the model, with any suggested settings for that mimimum (e.g. which quantization etc) and the sort of inferencing rate to be expected.
- The relative strengths of the model, e.g. it is stronger at programming than story telling, how strongly compliant it is/alignment.
In the case of this model, for example, could you run it with 16GB VRAM and 64 GB RAM?
It was too large for me to benchmark so I can't say other than what huggingface leaderboard says, but it did have roleplaying data, so possibly better than most at it.
As for min requirement, 2x3090s or 4090s or an a6000 48gb is required to inference in 4bit