vllm says the requested model does not exist
#1
by
shivams101
- opened
I could not reproduce your steps for deploying the model.
vllm throws the error: requests.exceptions.HTTPError: The request model: neuralmagic/Meta-Llama-3.1-70B-Instruct-quantized.w8a8 does not exist!
I have not worked with vllm before. Do I need to manually add the model somewhere to make it work?
Can you share the command you used to launch the vLLM server?
Thank you for your prompt response.
The issue is resolved now. I had set the HF_TOKEN variable in my environment. Removing that and restarting shell fixed the issue.
shivams101
changed discussion status to
closed