Pad token
#4
by
Teja-Gollapudi
- opened
Hi what was the pad token and eos_token used to train this model?
For eos_token_id, i see it mentioned as 2 in the model config but 50256 in the custom tokenizer py file?
What values should i use for both?
Thank you
Hi what was the pad token and eos_token used to train this model?
For eos_token_id, i see it mentioned as 2 in the model config but 50256 in the custom tokenizer py file?
What values should i use for both?
Thank you
The eos_token_id is 50256. No pad token used in training.
rooa
changed discussion status to
closed