|
--- |
|
license: mit |
|
language: |
|
- ru |
|
library_name: transformers |
|
tags: |
|
- text-generation-inference |
|
--- |
|
# Model Card for maximxls/text-normalization-ru-terrible |
|
|
|
Normalization for Russian text. Couldn't find any existing solutions (besides algorithms, don't like those) so made this. |
|
|
|
## Model Details |
|
|
|
### Model Description |
|
|
|
Tiny T5 trained from scratch for normalizing Russian texts: |
|
- translating numbers into words |
|
- expanding abbreviations into phonetic letter combinations |
|
- transliterating english into russian letters |
|
- whatever else was in the dataset (see below) |
|
|
|
### Model Sources |
|
|
|
- **Training code repository:** https://github.com/maximxlss/text_normalization |
|
- **Main dataset:** https://www.kaggle.com/c/text-normalization-challenge-russian-language |
|
|
|
## Uses |
|
|
|
Useful in TTS, for example with Silero to make it read numbers and English words (even if not perfectly, it's at least not ignoring) |
|
|
|
### Quick Start |
|
|
|
```Python |
|
from transformers import ( |
|
T5ForConditionalGeneration, |
|
PreTrainedTokenizerFast, |
|
) |
|
|
|
|
|
model_path = "maximxls/text-normalization-ru-terrible" |
|
|
|
tokenizer = PreTrainedTokenizerFast.from_pretrained(model_path) |
|
model = T5ForConditionalGeneration.from_pretrained(model_path) |
|
|
|
|
|
example_text = "Я ходил в McDonald's 10 июля 2022 года." |
|
|
|
inp_ids = tokenizer( |
|
example_text, |
|
return_tensors="pt", |
|
).input_ids |
|
out_ids = model.generate(inp_ids, max_new_tokens=128)[0] |
|
out = tokenizer.decode(out_ids, skip_special_tokens=True) |
|
|
|
print(out) |
|
``` |
|
|
|
`я ходил в макдоналд'эс десятого июля две тысячи двадцать второго года.` |
|
|
|
## Bias, Risks, and Limitations |
|
|
|
**Very much unreliable:** |
|
- For some reason, sometimes skips over first couple of tokens. Might be benificial to add some extra padding or whatever so it would be more stable. Wasn't able to solve it in training. |
|
- Sometimes is pretty unstable with repeating or missing words (especially with transliteration) |
|
|
|
## Training Details |
|
|
|
### Training Data |
|
|
|
Data from [this Kaggle challenge](https://www.kaggle.com/c/text-normalization-challenge-russian-language) (761435 sentences) aswell as a bit of extra data written by me. |
|
|
|
### Training Procedure |
|
|
|
#### Preprocessing |
|
|
|
See [`preprocessing.py`](https://github.com/maximxlss/text_normalization/blob/master/preprocess.py) |
|
|
|
#### Training Hyperparameters |
|
|
|
See [`train.py`](https://github.com/maximxlss/text_normalization/blob/master/train.py) |
|
|
|
I have reset lr manually several times during training, see metrics. |
|
|
|
#### Details |
|
|
|
See [`README` on github](https://github.com/maximxlss/text_normalization) for a step-by-step overview of the training procedure. |
|
|
|
## Technical Specifications |
|
|
|
#### Hardware |
|
|
|
Couple tens of hours of RTX 3090 Ti compute on my personal PC (21.65 epochs) |
|
|
|
|
