rohitsroch commited on
Commit
d344ffa
1 Parent(s): 25e9c75

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +21 -2
README.md CHANGED
@@ -11,12 +11,15 @@ datasets:
11
 
12
  ## Paper
13
 
14
- ## [SEAD: SIMPLE ENSEMBLE AND KNOWLEDGE DISTILLATION FRAMEWORK FOR NATURAL LANGUAGE UNDERSTANDING](https://www.course5i.com/ai-labs/)
15
  Aurthors: *Moyan Mei*, *Rohit Sroch*
16
 
17
  ## Abstract
18
 
19
- With the widespread use of pre-trained language models (PLM), there has been increased research on how to make them applicable, especially in limited resource or low latency high throughput scenarios. One of the dominant approaches is knowledge distillation (KD), where a smaller model is trained by receiving guidance from a large PLM. While there are many successful designs for learning knowledge from teachers, it remains unclear how students can learn better. Inspired by real university teaching processes, in this work we further explore knowledge distillation and propose a very simple yet effective framework, SEAD, to further improve task-specific generalization by utilizing multiple teachers. Our experiments show that SEAD leads to better performance compared to other popular KD methods [[1](https://arxiv.org/abs/1910.01108)] [[2](https://arxiv.org/abs/1909.10351)] [[3](https://arxiv.org/abs/2002.10957)] and achieves comparable or superior performance to its teacher model such as BERT [[4](https://arxiv.org/abs/1810.04805)] on total 13 tasks for the GLUE [[5](https://arxiv.org/abs/1804.07461)] and SuperGLUE [[6](https://arxiv.org/abs/1905.00537)] benchmarks.
 
 
 
20
 
21
  ## SEAD-L-6_H-384_A-12-sst2
22
 
@@ -57,3 +60,19 @@ $ hyperparameters = torch.load(os.path.join('training_args.bin'))
57
  - Flax >=0.3.5
58
  - Datasets >=1.10.2
59
  - Tokenizers >=0.11.6
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
11
 
12
  ## Paper
13
 
14
+ ## [SEAD: SIMPLE ENSEMBLE AND KNOWLEDGE DISTILLATION FRAMEWORK FOR NATURAL LANGUAGE UNDERSTANDING](www.adasci.org/journals/lattice-35309407/?volumes=true&open=621a3b18edc4364e8a96cb63)
15
  Aurthors: *Moyan Mei*, *Rohit Sroch*
16
 
17
  ## Abstract
18
 
19
+ With the widespread use of pre-trained language models (PLM), there has been increased research on how to make them applicable, especially in limited-resource or low latency high throughput scenarios. One of the dominant approaches is knowledge distillation (KD), where a smaller model is trained by receiving guidance from a large PLM. While there are many successful designs for learning knowledge from teachers, it remains unclear how students can learn better. Inspired by real university teaching processes, in this work we further explore knowledge distillation and propose a very simple yet effective framework, SEAD, to further improve task-specific generalization by utilizing multiple teachers. Our experiments show that SEAD leads to better performance compared to other popular KD methods [[1](https://arxiv.org/abs/1910.01108)] [[2](https://arxiv.org/abs/1909.10351)] [[3](https://arxiv.org/abs/2002.10957)] and achieves comparable or superior performance to its teacher model such as BERT [[4](https://arxiv.org/abs/1810.04805)] on total 13 tasks for the GLUE [[5](https://arxiv.org/abs/1804.07461)] and SuperGLUE [[6](https://arxiv.org/abs/1905.00537)] benchmarks.
20
+
21
+ *Moyan Mei and Rohit Sroch. 2022. [SEAD: Simple ensemble and knowledge distillation framework for natural language understanding](www.adasci.org/journals/lattice-35309407/?volumes=true&open=621a3b18edc4364e8a96cb63).
22
+ Lattice, THE MACHINE LEARNING JOURNAL by Association of Data Scientists, 3(1).*
23
 
24
  ## SEAD-L-6_H-384_A-12-sst2
25
 
 
60
  - Flax >=0.3.5
61
  - Datasets >=1.10.2
62
  - Tokenizers >=0.11.6
63
+
64
+ If you use these models, please cite the following paper:
65
+
66
+ ```
67
+ @article{article,
68
+ author={Mei, Moyan and Sroch, Rohit},
69
+ title={SEAD: Simple Ensemble and Knowledge Distillation Framework for Natural Language Understanding},
70
+ volume={3},
71
+ number={1},
72
+ journal={Lattice, The Machine Learning Journal by Association of Data Scientists},
73
+ day={26},
74
+ year={2022},
75
+ month={Feb},
76
+ url = {www.adasci.org/journals/lattice-35309407/?volumes=true&open=621a3b18edc4364e8a96cb63}
77
+ }
78
+ ```