test-charles-dickens / special_tokens_map.json
Dizzykong's picture
add tokenizer
279d959
raw
history blame contribute delete
No virus
114 Bytes
{"bos_token": "<|endoftext|>", "eos_token": "<|endoftext|>", "unk_token": "<|endoftext|>", "mask_token": "<mask>"}