ai-forever commited on
Commit
93abe5f
1 Parent(s): 0a760ff

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +34 -7
README.md CHANGED
@@ -19,18 +19,45 @@ Belorussian belongs to Indo-European language family. It's a very rhythmic langu
19
  2. It uses a variant of the Cyrillic script.
20
  3. It has a strong cultural connection with literature and folklore.
21
 
22
- ## Dataset
23
 
24
- TBD
25
 
26
- ## Technical details
 
 
 
 
 
 
27
 
28
- TBD
29
 
30
- ## Examples of usage
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
31
 
32
- Try different generation strategies to reach better results.
33
 
34
- TBD
35
 
36
  Model will be improved over time. Stay tuned!
 
19
  2. It uses a variant of the Cyrillic script.
20
  3. It has a strong cultural connection with literature and folklore.
21
 
22
+ ## Technical details
23
 
24
+ It's one of the models derived from the base [mGPT-XL (1.3B)](https://huggingface.co/ai-forever/mGPT) model (see the list below) which was originally trained on the 61 languages from 25 language families using Wikipedia and C4 corpus.
25
 
26
+ We've found additional data for 23 languages most of which are considered as minor and decided to further tune the base model. **Belorussian mGPT 1.3B** was trained for another 10000 steps with batch_size=4 and context window of **2048** tokens on 1 A100.
27
+
28
+ Final perplexity for this model on validation is **27.65**.
29
+
30
+ _Chart of the training loss and perplexity:_
31
+
32
+ ![](https://i.imgur.com/rMF4HYT.png)
33
 
34
+ ## Other mGPT-1.3B models
35
 
36
+ - [mGPT-1.3B-armenian](https://huggingface.co/ai-forever/mGPT-1.3B-armenian)
37
+ - [mGPT-1.3B-azerbaijan](https://huggingface.co/ai-forever/mGPT-1.3B-azerbaijan)
38
+ - [mGPT-1.3B-bashkir](https://huggingface.co/ai-forever/mGPT-1.3B-bashkir)
39
+ - [mGPT-1.3B-bulgarian](https://huggingface.co/ai-forever/mGPT-1.3B-bulgarian)
40
+ - [mGPT-1.3B-buryat](https://huggingface.co/ai-forever/mGPT-1.3B-buryat)
41
+ - [mGPT-1.3B-chuvash](https://huggingface.co/ai-forever/mGPT-1.3B-chuvash)
42
+ - [mGPT-1.3B-georgian](https://huggingface.co/ai-forever/mGPT-1.3B-georgian)
43
+ - [mGPT-1.3B-kalmyk](https://huggingface.co/ai-forever/mGPT-1.3B-kalmyk)
44
+ - [mGPT-1.3B-kazakh](https://huggingface.co/ai-forever/mGPT-1.3B-kazakh)
45
+ - [mGPT-1.3B-kirgiz](https://huggingface.co/ai-forever/mGPT-1.3B-kirgiz)
46
+ - [mGPT-1.3B-mari](https://huggingface.co/ai-forever/mGPT-1.3B-mari)
47
+ - [mGPT-1.3B-mongol](https://huggingface.co/ai-forever/mGPT-1.3B-mongol)
48
+ - [mGPT-1.3B-ossetian](https://huggingface.co/ai-forever/mGPT-1.3B-ossetian)
49
+ - [mGPT-1.3B-persian](https://huggingface.co/ai-forever/mGPT-1.3B-persian)
50
+ - [mGPT-1.3B-romanian](https://huggingface.co/ai-forever/mGPT-1.3B-romanian)
51
+ - [mGPT-1.3B-tajik](https://huggingface.co/ai-forever/mGPT-1.3B-tajik)
52
+ - [mGPT-1.3B-tatar](https://huggingface.co/ai-forever/mGPT-1.3B-tatar)
53
+ - [mGPT-1.3B-turkmen](https://huggingface.co/ai-forever/mGPT-1.3B-turkmen)
54
+ - [mGPT-1.3B-tuvan](https://huggingface.co/ai-forever/mGPT-1.3B-tuvan)
55
+ - [mGPT-1.3B-ukranian](https://huggingface.co/ai-forever/mGPT-1.3B-ukranian)
56
+ - [mGPT-1.3B-uzbek](https://huggingface.co/ai-forever/mGPT-1.3B-uzbek)
57
+ - [mGPT-1.3B-yakut](https://huggingface.co/ai-forever/mGPT-1.3B-yakut)
58
 
59
+ ## Feedback
60
 
61
+ If you'll found a bug of have additional data to train model on your language — please, give us feedback.
62
 
63
  Model will be improved over time. Stay tuned!