xgen-7b-4k-base / tokenization_xgen.py

Commit History

add pad token and default eos token
fe79609

tianxie-sf commited on

.
f70089e

rooa commited on

.
d5c0667

rooa commited on

.
4fb6c92

rooa commited on