File size: 931 Bytes
6fe74ef 023f0ae 6fe74ef 957aaa7 023f0ae |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
---
license: apache-2.0
base_model: mistralai/Mistral-7B-v0.3
extra_gated_description: If you want to learn more about how we process your personal data, please read our <a href="https://mistral.ai/terms/">Privacy Policy</a>.
---
# Model Card for Mistral-7B-Instruct-v0.3 for inf2.xlarge
The Mistral-7B-Instruct-v0.3 Large Language Model (LLM) is an instruct fine-tuned version of the Mistral-7B-v0.3.
neuron compiled
these are the shapes currently cached.
don't ask me why 8196.. it was a typo.
```
python -m vllm.entrypoints.openai.api_server --model ./ --max-model-len 8196 --device neuron --tensor-parallel-size 2 --max-num-seqs 2
python -m vllm.entrypoints.openai.api_server --model ./ --max-model-len 8196 --device neuron --tensor-parallel-size 2 --max-num-seqs 4
python -m vllm.entrypoints.openai.api_server --model ./ --max-model-len 10240 --device neuron --tensor-parallel-size 2 --max-num-seqs 4
```
|