Problem in loading falcon-7b-instruct locally
I have downloaded and saved the "falcon-7b-instruct" model files to your local machine (Windows 10 OS with 16 GB RAM and 1 TB SSD). But when i am trying to load model i am getting below error.
Traceback (most recent call last):
File "d:\qlora\models\bot.py", line 5, in
tokenizer = AutoTokenizer.from_pretrained(model_directory)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\yashwank\AppData\Local\Programs\Python\Python311\Lib\site-packages\transformers\models\auto\tokenization_auto.py", line 666, in from_pretrained
config = AutoConfig.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\yashwank\AppData\Local\Programs\Python\Python311\Lib\site-packages\transformers\models\auto\configuration_auto.py", line 958, in from_pretrained
trust_remote_code = resolve_trust_remote_code(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\yashwank\AppData\Local\Programs\Python\Python311\Lib\site-packages\transformers\dynamic_module_utils.py", line 535, in resolve_trust_remote_code
signal.signal(signal.SIGALRM, _raise_timeout_error)
^^^^^^^^^^^^^^
AttributeError: module 'signal' has no attribute 'SIGALRM'. Did you mean: 'SIGABRT'?
Code i am trying to load model**
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
model_directory = "path/to/your/model/directory"
tokenizer = AutoTokenizer.from_pretrained(model_directory)
model = AutoModelForCausalLM.from_pretrained(model_directory)
pipeline = transformers.pipeline(
"text-generation",
model=model,
tokenizer=tokenizer,
torch_dtype=torch.bfloat16,
trust_remote_code=True,
device_map="auto",
)
..............so on
Am i doing anything wrong? How to fix this?
I have the same problem when trying to use my fine-tuned Falcon. Are you using Windows?
Yes, i am trying this on Windows 10 OS. But here i am trying to load my base model and not the fine tuned.
however i found the issue is with my Model Weights and more precisely the issue is with model tokenizer files.. I tried to redownload the weights again and all files and then tried loading them. It worked.
It's trying to load remote code for the tokenizer, but trust_remote_code is False. Windows doesn't seem to handle signal(14). Adding trust_remote_code=True to the AutoTokenizer.from_pretrained call should allow it to download the appropriate tokenizer.