ImportError: libcudart.so.12: cannot open shared object file: No such file or directory
Getting Error while downloading the Model
ImportError: libcudart.so.12: cannot open shared object file: No such file or directory
ImportError Traceback (most recent call last)
in <cell line: 6>()
4 # To use a different branch, change revision
5 # For example: revision="main"
----> 6 model = AutoModelForCausalLM.from_pretrained(model_name_or_path,
7 device_map="auto",
8 trust_remote_code=False,
6 frames
/usr/local/lib/python3.10/dist-packages/auto_gptq/nn_modules/qlinear/qlinear_exllama.py in
12
13 try:
---> 14 from exllama_kernels import make_q4, q4_matmul
15 except ImportError:
16 logger.error('exllama_kernels not installed.')
ImportError: libcudart.so.12: cannot open shared object file: No such file or directory
The default pre-built wheels for 0.5.0 now use PyTorch 2.1 and CUDA 12.1.
If you have CUDA 11.8, you can install AutoGPTQ for PyTorch 2.1 and CUDA 11.8 with:
pip3 uninstall -y auto-gptq
pip3 install auto-gptq --extra-index-url https://huggingface.github.io/autogptq-index/whl/cu118/
But you'll still need PyTorch 2.1. If you want to use PyTorch 2.0.1, then you can either build AutoGPTQ 0.5.0 from source, or downgrade to AutoGPTQ 0.4.2.
Thank You for Your Response .