wgb14 commited on
Commit
21e4de8
1 Parent(s): d46019c

Upload data/lang_bpe_500/unigram_500.vocab

Browse files
Files changed (1) hide show
  1. data/lang_bpe_500/unigram_500.vocab +500 -0
data/lang_bpe_500/unigram_500.vocab ADDED
@@ -0,0 +1,500 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ <blk> 0
2
+ <sos/eos> 0
3
+ <unk> 0
4
+ S -2.91936
5
+ T -3.62226
6
+ ▁THE -3.63907
7
+ ▁A -3.91998
8
+ E -3.9774
9
+ ▁AND -4.03888
10
+ ▁TO -4.11528
11
+ N -4.13926
12
+ D -4.23828
13
+ ▁OF -4.26126
14
+ ' -4.32705
15
+ ING -4.36266
16
+ ▁I -4.43026
17
+ Y -4.47664
18
+ ▁IN -4.5039
19
+ ED -4.56388
20
+ ▁THAT -4.56493
21
+ ▁ -4.61161
22
+ P -4.68867
23
+ R -4.69194
24
+ ▁YOU -4.70013
25
+ M -4.73468
26
+ RE -4.77896
27
+ ER -4.79222
28
+ C -4.80441
29
+ O -4.84437
30
+ ▁IT -4.84599
31
+ L -4.92876
32
+ A -5.01304
33
+ U -5.04339
34
+ G -5.07981
35
+ ▁WE -5.15716
36
+ ▁IS -5.16592
37
+ ▁SO -5.20473
38
+ AL -5.20761
39
+ I -5.20892
40
+ ▁S -5.21941
41
+ ▁RE -5.25961
42
+ AR -5.26464
43
+ B -5.27126
44
+ ▁FOR -5.28788
45
+ ▁C -5.28918
46
+ ▁BE -5.30209
47
+ LE -5.3038
48
+ F -5.31727
49
+ W -5.3626
50
+ ▁E -5.39058
51
+ ▁HE -5.3984
52
+ LL -5.39936
53
+ ▁WAS -5.41947
54
+ LY -5.41982
55
+ OR -5.451
56
+ IN -5.51245
57
+ ▁F -5.51686
58
+ VE -5.54551
59
+ ▁THIS -5.54604
60
+ TH -5.55438
61
+ K -5.59782
62
+ ▁ON -5.60492
63
+ IT -5.63051
64
+ ▁B -5.64315
65
+ ▁WITH -5.6692
66
+ ▁BUT -5.69372
67
+ EN -5.69464
68
+ CE -5.70105
69
+ RI -5.74079
70
+ ▁DO -5.75023
71
+ UR -5.75089
72
+ ▁HAVE -5.75784
73
+ ▁DE -5.78198
74
+ ▁ME -5.78744
75
+ ▁T -5.79218
76
+ ENT -5.84937
77
+ CH -5.86931
78
+ ▁THEY -5.8741
79
+ ▁NOT -5.88022
80
+ ES -5.90245
81
+ V -5.90528
82
+ ▁AS -5.90745
83
+ RA -5.91436
84
+ ▁P -5.91784
85
+ ON -5.91788
86
+ TER -5.92587
87
+ ▁ARE -5.94781
88
+ ▁WHAT -5.95664
89
+ IC -5.96785
90
+ ▁ST -5.97911
91
+ ▁LIKE -6.02232
92
+ ATION -6.04962
93
+ ▁OR -6.05223
94
+ ▁CA -6.05876
95
+ ▁AT -6.0815
96
+ H -6.08965
97
+ ▁KNOW -6.09389
98
+ ▁G -6.09914
99
+ AN -6.10082
100
+ ▁CON -6.1086
101
+ IL -6.12259
102
+ ND -6.12748
103
+ RO -6.13926
104
+ ▁HIS -6.1401
105
+ ▁CAN -6.14445
106
+ ▁ALL -6.14626
107
+ TE -6.15654
108
+ ▁THERE -6.16263
109
+ ▁SU -6.18963
110
+ ▁MO -6.19822
111
+ ▁MA -6.21735
112
+ LI -6.24568
113
+ ▁ONE -6.25177
114
+ ▁ABOUT -6.253
115
+ LA -6.25309
116
+ ▁CO -6.25588
117
+ - -6.26085
118
+ ▁MY -6.27134
119
+ ▁HAD -6.2739
120
+ CK -6.27737
121
+ NG -6.2891
122
+ ▁NO -6.29472
123
+ MENT -6.30738
124
+ AD -6.31071
125
+ LO -6.31339
126
+ ME -6.3139
127
+ ▁AN -6.31782
128
+ ▁FROM -6.3267
129
+ NE -6.32822
130
+ ▁IF -6.3309
131
+ VER -6.33734
132
+ ▁JUST -6.34705
133
+ ▁PRO -6.38157
134
+ ION -6.38203
135
+ ▁PA -6.38311
136
+ ▁WHO -6.389
137
+ ▁SE -6.39145
138
+ EL -6.39577
139
+ IR -6.39902
140
+ ▁US -6.40366
141
+ ▁UP -6.40929
142
+ ▁YOUR -6.41111
143
+ CI -6.41571
144
+ RY -6.42878
145
+ ▁GO -6.43225
146
+ ▁SHE -6.44576
147
+ ▁LE -6.45467
148
+ ▁OUT -6.4618
149
+ ▁PO -6.46959
150
+ ▁HO -6.47283
151
+ ATE -6.48248
152
+ ▁BO -6.48598
153
+ ▁BY -6.48743
154
+ ▁FA -6.48779
155
+ ▁MI -6.48972
156
+ AS -6.49552
157
+ MP -6.50825
158
+ ▁HER -6.51444
159
+ VI -6.51584
160
+ ▁THINK -6.51821
161
+ ▁SOME -6.51966
162
+ ▁WHEN -6.53955
163
+ ▁AH -6.54789
164
+ ▁PEOPLE -6.56579
165
+ IG -6.56619
166
+ ▁WA -6.58446
167
+ ▁TE -6.58708
168
+ ▁LA -6.5924
169
+ ▁WERE -6.59558
170
+ ▁LI -6.59617
171
+ ▁WOULD -6.60333
172
+ ▁SEE -6.60421
173
+ ▁WHICH -6.60769
174
+ DE -6.61146
175
+ GE -6.61214
176
+ ▁K -6.61725
177
+ IGHT -6.63987
178
+ ▁HA -6.64097
179
+ ▁OUR -6.67441
180
+ UN -6.67953
181
+ ▁HOW -6.68709
182
+ ▁GET -6.6874
183
+ IS -6.69453
184
+ UT -6.70334
185
+ Z -6.70922
186
+ CO -6.71316
187
+ ET -6.71592
188
+ UL -6.71664
189
+ IES -6.72788
190
+ IVE -6.73038
191
+ AT -6.73107
192
+ ▁O -6.73302
193
+ ▁DON -6.73526
194
+ LU -6.73936
195
+ ▁TIME -6.74787
196
+ ▁WILL -6.75714
197
+ ▁MORE -6.75772
198
+ ▁SP -6.75795
199
+ ▁NOW -6.76281
200
+ RU -6.76441
201
+ ▁THEIR -6.76701
202
+ ▁UN -6.77337
203
+ ITY -6.78846
204
+ OL -6.78991
205
+ X -6.79216
206
+ TI -6.79565
207
+ US -6.79815
208
+ ▁VERY -6.81318
209
+ TION -6.81516
210
+ ▁FI -6.81739
211
+ ▁SAY -6.81764
212
+ ▁BECAUSE -6.83879
213
+ ▁EX -6.83974
214
+ ▁RO -6.84322
215
+ ERS -6.8685
216
+ IST -6.87091
217
+ ▁DA -6.87303
218
+ TING -6.87408
219
+ ▁EN -6.88514
220
+ OM -6.88529
221
+ ▁BA -6.8895
222
+ ▁BEEN -6.89199
223
+ ▁LO -6.89575
224
+ ▁UM -6.89988
225
+ AGE -6.9046
226
+ ABLE -6.90556
227
+ ▁WO -6.90762
228
+ ▁RA -6.90807
229
+ ▁OTHER -6.92762
230
+ ▁REALLY -6.92772
231
+ ENCE -6.93464
232
+ ▁GOING -6.93644
233
+ ▁HIM -6.93782
234
+ ▁HAS -6.9382
235
+ ▁THEM -6.93852
236
+ ▁DIS -6.94051
237
+ ▁WANT -6.94824
238
+ ID -6.95197
239
+ TA -6.95466
240
+ ▁LOOK -6.95859
241
+ KE -6.96683
242
+ ▁DID -6.96973
243
+ ▁SA -6.97504
244
+ ▁VI -6.97815
245
+ ▁SAID -6.98116
246
+ ▁RIGHT -6.98233
247
+ ▁THESE -6.98239
248
+ ▁WORK -6.98267
249
+ ▁COM -7.00591
250
+ ALLY -7.02101
251
+ FF -7.02191
252
+ QU -7.02982
253
+ AC -7.032
254
+ ▁DR -7.03706
255
+ ▁WAY -7.03766
256
+ ▁INTO -7.04973
257
+ MO -7.05136
258
+ TED -7.05518
259
+ EST -7.06036
260
+ ▁HERE -7.06279
261
+ OK -7.06659
262
+ ▁COULD -7.07484
263
+ ▁WELL -7.07681
264
+ MA -7.0798
265
+ ▁PRE -7.08375
266
+ ▁DI -7.08411
267
+ MAN -7.09183
268
+ ▁COMP -7.09251
269
+ ▁THEN -7.09311
270
+ IM -7.09454
271
+ ▁PER -7.09471
272
+ ▁NA -7.09814
273
+ ▁WHERE -7.0995
274
+ ▁TWO -7.10374
275
+ ▁WI -7.10433
276
+ ▁FE -7.12258
277
+ INE -7.13542
278
+ ▁ANY -7.13685
279
+ TURE -7.13787
280
+ ▁OVER -7.13806
281
+ BO -7.1436
282
+ ACH -7.15075
283
+ OW -7.16294
284
+ ▁MAKE -7.16967
285
+ ▁TRA -7.17924
286
+ HE -7.19414
287
+ UND -7.19438
288
+ ▁EVEN -7.1973
289
+ ANCE -7.19936
290
+ ▁YEAR -7.20074
291
+ HO -7.20126
292
+ AM -7.20803
293
+ ▁CHA -7.20944
294
+ ▁BACK -7.21247
295
+ VO -7.22034
296
+ ANT -7.24148
297
+ DI -7.24414
298
+ ▁ALSO -7.24441
299
+ ▁THOSE -7.2492
300
+ ▁MAN -7.25828
301
+ CTION -7.25994
302
+ ICAL -7.27095
303
+ ▁JO -7.27522
304
+ ▁OP -7.27657
305
+ ▁NEW -7.28026
306
+ ▁MU -7.28457
307
+ ▁HU -7.28719
308
+ ▁KIND -7.29056
309
+ ▁NE -7.29057
310
+ CA -7.29455
311
+ END -7.29991
312
+ TIC -7.30338
313
+ FUL -7.308
314
+ ▁YEAH -7.30971
315
+ SH -7.31052
316
+ ▁APP -7.31863
317
+ ▁THINGS -7.31873
318
+ SIDE -7.31876
319
+ ▁GOOD -7.32149
320
+ ONE -7.32385
321
+ ▁TAKE -7.33148
322
+ CU -7.33525
323
+ ▁EVERY -7.33606
324
+ ▁MEAN -7.33916
325
+ ▁FIRST -7.35633
326
+ OP -7.35807
327
+ ▁TH -7.35816
328
+ ▁MUCH -7.36035
329
+ ▁PART -7.36303
330
+ UGH -7.38075
331
+ ▁COME -7.39025
332
+ J -7.40206
333
+ ▁THAN -7.4027
334
+ ▁EXP -7.40635
335
+ ▁AGAIN -7.40638
336
+ ▁LITTLE -7.4123
337
+ MB -7.42061
338
+ ▁NEED -7.42982
339
+ ▁TALK -7.42993
340
+ IF -7.43499
341
+ FOR -7.43685
342
+ ▁SH -7.43707
343
+ ISH -7.43834
344
+ ▁STA -7.44406
345
+ ATED -7.44429
346
+ ▁GU -7.4457
347
+ ▁LET -7.44843
348
+ IA -7.45208
349
+ ▁MAR -7.46218
350
+ ▁DOWN -7.46252
351
+ ▁DAY -7.46636
352
+ ▁GA -7.46831
353
+ ▁SOMETHING -7.47373
354
+ ▁BU -7.47952
355
+ DUC -7.47968
356
+ HA -7.48577
357
+ ▁LOT -7.48964
358
+ ▁RU -7.49641
359
+ ▁THOUGH -7.50103
360
+ ▁GREAT -7.51571
361
+ AIN -7.53755
362
+ ▁THROUGH -7.53814
363
+ ▁THING -7.5397
364
+ OUS -7.54298
365
+ ▁PRI -7.54663
366
+ ▁GOT -7.54668
367
+ ▁SHOULD -7.54854
368
+ ▁AFTER -7.54887
369
+ ▁HEAR -7.54921
370
+ ▁TA -7.5583
371
+ ▁ONLY -7.56318
372
+ ▁CHI -7.56667
373
+ IOUS -7.58401
374
+ ▁SHA -7.58711
375
+ ▁MOST -7.5956
376
+ ▁ACTUALLY -7.59878
377
+ ▁START -7.60046
378
+ LIC -7.60951
379
+ ▁VA -7.61021
380
+ ▁RI -7.6118
381
+ DAY -7.61429
382
+ IAN -7.6194
383
+ ▁DOES -7.62247
384
+ ROW -7.62941
385
+ ▁GRA -7.63287
386
+ ITION -7.64608
387
+ ▁MANY -7.6498
388
+ ▁BEFORE -7.67839
389
+ ▁GIVE -7.67892
390
+ PORT -7.68357
391
+ QUI -7.70117
392
+ ▁LIFE -7.70155
393
+ ▁WORLD -7.70833
394
+ ▁PI -7.71249
395
+ ▁LONG -7.71586
396
+ ▁THREE -7.72143
397
+ IZE -7.72871
398
+ NESS -7.73046
399
+ ▁SHOW -7.73117
400
+ PH -7.74408
401
+ ▁WHY -7.75452
402
+ ▁QUESTION -7.7659
403
+ WARD -7.76633
404
+ ▁THANK -7.77763
405
+ ▁PH -7.78092
406
+ ▁DIFFERENT -7.79798
407
+ ▁OWN -7.8008
408
+ ▁FEEL -7.80762
409
+ ▁MIGHT -7.81212
410
+ ▁HAPPEN -7.82257
411
+ ▁MADE -7.82361
412
+ ▁BRO -7.83505
413
+ IBLE -7.83692
414
+ ▁HI -7.83977
415
+ ▁STATE -7.8432
416
+ ▁HAND -7.85373
417
+ ▁NEVER -7.8582
418
+ ▁PLACE -7.8697
419
+ ▁LOVE -7.87056
420
+ ▁DU -7.87177
421
+ ▁POINT -7.87926
422
+ ▁HELP -7.88273
423
+ ▁COUNT -7.88305
424
+ ▁STILL -7.88446
425
+ ▁MR -7.90123
426
+ ▁FIND -7.90288
427
+ ▁PERSON -7.92879
428
+ ▁CAME -7.93
429
+ ▁SAME -7.95337
430
+ ▁LAST -7.97701
431
+ ▁HIGH -7.97875
432
+ ▁OLD -8.00689
433
+ ▁UNDER -8.01208
434
+ ▁FOUR -8.01237
435
+ ▁AROUND -8.0163
436
+ ▁SORT -8.0255
437
+ ▁CHANGE -8.0293
438
+ ▁YES -8.03364
439
+ SHIP -8.04202
440
+ ▁ANOTHER -8.04909
441
+ ATIVE -8.07714
442
+ ▁FOUND -8.09227
443
+ ▁JA -8.09351
444
+ ▁ALWAYS -8.10023
445
+ ▁NEXT -8.10913
446
+ ▁TURN -8.11079
447
+ ▁JU -8.12961
448
+ ▁SIX -8.13018
449
+ ▁FACT -8.13809
450
+ ▁INTEREST -8.14199
451
+ ▁WORD -8.15103
452
+ ▁THOUSAND -8.16173
453
+ ▁HUNDRED -8.19128
454
+ ▁NUMBER -8.19271
455
+ ▁IDEA -8.2011
456
+ ▁PLAN -8.21386
457
+ ▁COURSE -8.21699
458
+ ▁SCHOOL -8.22226
459
+ ▁HOUSE -8.22885
460
+ ▁TWENTY -8.23354
461
+ ▁JE -8.23872
462
+ ▁PLAY -8.2479
463
+ ▁AWAY -8.25259
464
+ ▁LEARN -8.25403
465
+ ▁HARD -8.25523
466
+ ▁WEEK -8.25715
467
+ ▁BETTER -8.26711
468
+ ▁WHILE -8.27626
469
+ ▁FRIEND -8.28156
470
+ ▁OKAY -8.28222
471
+ ▁NINE -8.28927
472
+ ▁UNDERSTAND -8.30685
473
+ ▁KEEP -8.30703
474
+ ▁GONNA -8.30908
475
+ ▁SYSTEM -8.33115
476
+ ▁AMERICA -8.33344
477
+ ▁POWER -8.34085
478
+ ▁IMPORTANT -8.34508
479
+ ▁WITHOUT -8.35783
480
+ ▁MAYBE -8.36369
481
+ ▁SEVEN -8.36468
482
+ ▁BETWEEN -8.36808
483
+ ▁BUILD -8.38309
484
+ ▁CERTAIN -8.38571
485
+ ▁PROBLEM -8.40003
486
+ ▁MONEY -8.40454
487
+ ▁BELIEVE -8.42698
488
+ ▁SECOND -8.43244
489
+ ▁REASON -8.45837
490
+ ▁TOGETHER -8.49857
491
+ ▁PUBLIC -8.50598
492
+ ▁ANYTHING -8.50789
493
+ ▁SPEAK -8.51154
494
+ ▁BUSINESS -8.5218
495
+ ▁EVERYTHING -8.52868
496
+ ▁CLOSE -8.52921
497
+ ▁QUITE -8.5317
498
+ ▁ANSWER -8.5436
499
+ ▁ENOUGH -8.54754
500
+ Q -9.75572