Text Generation
Transformers
Safetensors
English
falcon_mamba
conversational
Inference Endpoints

Transformers does not know model type 'falcon_mamba'

#3
by astrologos - opened

When attempting to run the sample code using transformers-4.44.0, huggingface-hub-0.24.5, tokenizers-0.19.1:

# pip install accelerate
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("tiiuae/falcon-mamba-7b-instruct")
model = AutoModelForCausalLM.from_pretrained("tiiuae/falcon-mamba-7b-instruct", device_map="auto")
# We use the tokenizer's chat template to format each message - see https://huggingface.co/docs/transformers/main/en/chat_templating
messages = [
{"role": "user", "content": "How many helicopters can a human eat in one sitting?"},
]
input_text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
input_ids = tokenizer(input_text, return_tensors="pt").input_ids.to("cuda")
outputs = model.generate(input_ids, max_new_tokens=30)
print(tokenizer.decode(outputs[0]))


I get the following output:
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
File c:\WBG\Anaconda3\envs\cuda\Lib\site-packages\transformers\models\auto\configuration_auto.py:993, in AutoConfig.from_pretrained(cls,
pretrained_model_name_or_path, **kwargs)
992 try:
--> 993 config_class = CONFIG_MAPPING[config_dict["model_type"]]
994 except KeyError:

File c:\WBG\Anaconda3\envs\cuda\Lib\site-packages\transformers\models\auto\configuration_auto.py:695, in _LazyConfigMapping.__getitem__(self,
key)
694 if key not in self._mapping:
--> 695 raise KeyError(key)
696 value = self._mapping[key]

KeyError: 'falcon_mamba'

During handling of the above exception, another exception occurred:

ValueError Traceback (most recent call last)
Cell In[1], line 5
2 from transformers import AutoTokenizer, AutoModelForCausalLM
4 tokenizer = AutoTokenizer.from_pretrained("tiiuae/falcon-mamba-7b-instruct")
----> 5 model = AutoModelForCausalLM.from_pretrained("tiiuae/falcon-mamba-7b-instruct", device_map="auto")
7 # We use the tokenizer's chat template to format each message - see https://huggingface.co/docs/transformers/main/en/chat_templating
8 messages = [
9 {"role": "user", "content": "How many helicopters can a human eat in one sitting?"},
10 ]


File c:\WBG\Anaconda3\envs\cuda\Lib\site-packages\transformers\models\auto\auto_factory.py:524, in _BaseAutoModelClass.from_pretrained(cls,
pretrained_model_name_or_path, *model_args, **kwargs)

521 if kwargs.get("quantization_config", None) is not None:

522 _ = kwargs.pop("quantization_config")

--> 524 config, kwargs = AutoConfig.from_pretrained(

525 pretrained_model_name_or_path,

526 return_unused_kwargs=True,

527 trust_remote_code=trust_remote_code,

528 code_revision=code_revision,

529 _commit_hash=commit_hash,

530 **hub_kwargs,

531 **kwargs,

532 )

534 # if torch_dtype=auto was passed here, ensure to pass it on

535 if kwargs_orig.get("torch_dtype", None) == "auto":


File c:\WBG\Anaconda3\envs\cuda\Lib\site-packages\transformers\models\auto\configuration_auto.py:995, in AutoConfig.from_pretrained(cls,
pretrained_model_name_or_path, **kwargs)

993 config_class = CONFIG_MAPPING[config_dict["model_type"]]

994 except KeyError:

--> 995 raise ValueError(

996 f"The checkpoint you are trying to load has model type {config_dict['model_type']} "

997 "but Transformers does not recognize this architecture. This could be because of an "

998 "issue with the checkpoint, or because your version of Transformers is out of date."

999 )

1000 return config_class.from_dict(config_dict, **unused_kwargs)

1001 else:

1002 # Fallback: use pattern matching on the string.

1003 # We go from longer names to shorter names to catch roberta before bert (for instance)



ValueError: The checkpoint you are trying to load has model type falcon_mamba but Transformers does not recognize this architecture. This could be
because of an issue with the checkpoint, or because your version of Transformers is out of date.

Can you please help?

Technology Innovation Institute org

Hi @astrologos
Sure yes, for now you need to install transformers from source

pip install -U git+https://github.com/huggingface/transformers.git

should solve your issue

hello How can we fine-tune the Falcon-Mamba 7B model for our own tasks?

this worked for me 3 weeks ago, but not today :/

Successfully installed transformers-4.45.0.dev0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable.It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
(textgen) [root@pve-m7330 text-generation-webui]# python server.py --api --listen --n-gpu-layers 32 --threads 8 --numa --tensorcores --trust-remote-code
โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ Traceback (most recent call last) โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ
โ”‚ /home/user/text-generation-webui/server.py:40 in โ”‚
โ”‚ โ”‚
โ”‚ 39 import modules.extensions as extensions_module โ”‚
โ”‚ โฑ 40 from modules import ( โ”‚
โ”‚ 41 chat, โ”‚
โ”‚ โ”‚
โ”‚ /home/user/text-generation-webui/modules/chat.py:26 in โ”‚
โ”‚ โ”‚
โ”‚ 25 from modules.logging_colors import logger โ”‚
โ”‚ โฑ 26 from modules.text_generation import ( โ”‚
โ”‚ 27 generate_reply, โ”‚
โ”‚ โ”‚
โ”‚ /home/user/text-generation-webui/modules/text_generation.py:19 in โ”‚
โ”‚ โ”‚
โ”‚ 18 import modules.shared as shared โ”‚
โ”‚ โฑ 19 from modules import models โ”‚
โ”‚ 20 from modules.cache_utils import process_llamacpp_cache โ”‚
โ”‚ โ”‚
โ”‚ /home/user/text-generation-webui/modules/models.py:59 in โ”‚
โ”‚ โ”‚
โ”‚ 58 โ”‚
โ”‚ โฑ 59 sampler_hijack.hijack_samplers() โ”‚
โ”‚ 60 โ”‚
โ”‚ โ”‚
โ”‚ /home/user/text-generation-webui/modules/sampler_hijack.py:554 in hijack_samplers โ”‚
โ”‚ โ”‚
โ”‚ 553 def hijack_samplers(): โ”‚
โ”‚ โฑ 554 transformers.GenerationMixin.get_logits_warper_old = transformers.GenerationMixin. โ”‚
โ”‚ 555 transformers.GenerationMixin._get_logits_warper = get_logits_warper_patch โ”‚
โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ
AttributeError: type object 'GenerationMixin' has no attribute '_get_logits_warper'

Sign up or log in to comment