winglian's picture
better handling since xgen tokenizer breaks with convert_tokens_to_ids
2a428e8