patrickvonplaten commited on
Commit
094079e
1 Parent(s): 6e58c63

Upload README.md

Browse files
Files changed (1) hide show
  1. README.md +109 -4
README.md CHANGED
@@ -1,8 +1,113 @@
1
  ---
 
 
 
 
2
  tags:
3
- - t5-new-success
 
 
 
4
  ---
5
 
6
- # Test
7
- Hf T5: -121.47478103637695
8
- MTF T5: -121.474853515625
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ language:
3
+ - en
4
+ datasets:
5
+ - c4
6
  tags:
7
+ - deep-narrow
8
+ inference: false
9
+
10
+ license: apache-2.0
11
  ---
12
 
13
+ # T5-Efficient-MINI (Deep-Narrow version)
14
+
15
+ T5-Efficient-MINI is a variation of [Google's original T5](https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html) following the [T5 model architecture](https://huggingface.co/docs/transformers/model_doc/t5).
16
+ It is a *pretrained-only* checkpoint and was released with the
17
+ paper **[Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers](https://arxiv.org/abs/2109.10686)**
18
+ by *Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, Donald Metzler*.
19
+
20
+ In a nutshell, the paper indicates that a **Deep-Narrow** model architecture is favorable for **downstream** performance compared to other model architectures
21
+ of similar parameter count.
22
+
23
+ To quote the paper:
24
+
25
+ > We generally recommend a DeepNarrow strategy where the model’s depth is preferentially increased
26
+ > before considering any other forms of uniform scaling across other dimensions. This is largely due to
27
+ > how much depth influences the Pareto-frontier as shown in earlier sections of the paper. Specifically, a
28
+ > tall small (deep and narrow) model is generally more efficient compared to the base model. Likewise,
29
+ > a tall base model might also generally more efficient compared to a large model. We generally find
30
+ > that, regardless of size, even if absolute performance might increase as we continue to stack layers,
31
+ > the relative gain of Pareto-efficiency diminishes as we increase the layers, converging at 32 to 36
32
+ > layers. Finally, we note that our notion of efficiency here relates to any one compute dimension, i.e.,
33
+ > params, FLOPs or throughput (speed). We report all three key efficiency metrics (number of params,
34
+ > FLOPS and speed) and leave this decision to the practitioner to decide which compute dimension to
35
+ > consider.
36
+
37
+ To be more precise, *model depth* is defined as the number of transformer blocks that are stacked sequentially.
38
+ A sequence of word embeddings is therefore processed sequentially by each transformer block.
39
+
40
+ ## Details model architecture
41
+
42
+ This model checkpoint - **t5-efficient-mini** - is of model type **Mini** with no variations.
43
+ It has **31.23** million parameters and thus requires *ca.* **124.92 MB** of memory in full precision (*fp32*)
44
+ or **62.46 MB** of memory in half precision (*fp16* or *bf16*).
45
+
46
+ A summary of the *original* T5 model architectures can be seen here:
47
+
48
+ | Model | nl (el/dl) | ff | dm | kv | nh | #Params|
49
+ | ----| ---- | ---- | ---- | ---- | ---- | ----|
50
+ | Tiny | 4/4 | 1024 | 256 | 32 | 4 | 16M|
51
+ | Mini | 4/4 | 1536 | 384 | 32 | 8 | 31M|
52
+ | Small | 6/6 | 2048 | 512 | 32 | 8 | 60M|
53
+ | Base | 12/12 | 3072 | 768 | 64 | 12 | 220M|
54
+ | Large | 24/24 | 4096 | 1024 | 64 | 16 | 738M|
55
+ | Xl | 24/24 | 16384 | 1024 | 128 | 32 | 3B|
56
+ | XXl | 24/24 | 65536 | 1024 | 128 | 128 | 11B|
57
+
58
+ whereas the following abbreviations are used:
59
+
60
+ | Abbreviation | Definition |
61
+ | ----| ---- |
62
+ | nl | Number of transformer blocks (depth) |
63
+ | dm | Dimension of embedding vector (output vector of transformers block) |
64
+ | kv | Dimension of key/value projection matrix |
65
+ | nh | Number of attention heads |
66
+ | ff | Dimension of intermediate vector within transformer block (size of feed-forward projection matrix) |
67
+ | el | Number of transformer blocks in the encoder (encoder depth) |
68
+ | dl | Number of transformer blocks in the decoder (decoder depth) |
69
+ | sh | Signifies that attention heads are shared |
70
+ | skv | Signifies that key-values projection matrices are tied |
71
+
72
+ If a model checkpoint has no specific, *el* or *dl* than both the number of encoder- and decoder layers correspond to *nl*.
73
+
74
+ ## Pre-Training
75
+
76
+ The checkpoint was pretrained on the [Colossal, Cleaned version of Common Crawl (C4)](https://huggingface.co/datasets/c4) for 524288 steps using
77
+ the span-based masked language modeling (MLM) objective.
78
+
79
+ ## Fine-Tuning
80
+
81
+ **Note**: This model is a **pretrained** checkpoint and has to be fine-tuned for practical usage.
82
+ The checkpoint was pretrained in English and is therefore only useful for English NLP tasks.
83
+ You can follow on of the following examples on how to fine-tune the model:
84
+
85
+ *PyTorch*:
86
+
87
+ - [Summarization](https://github.com/huggingface/transformers/tree/master/examples/pytorch/summarization)
88
+ - [Question Answering](https://github.com/huggingface/transformers/blob/master/examples/pytorch/question-answering/run_seq2seq_qa.py)
89
+ - [Text Classification](https://github.com/huggingface/transformers/tree/master/examples/pytorch/text-classification) - *Note*: You will have to slightly adapt the training example here to make it work with an encoder-decoder model.
90
+
91
+ *Tensorflow*:
92
+
93
+ - [Summarization](https://github.com/huggingface/transformers/tree/master/examples/tensorflow/summarization)
94
+ - [Text Classification](https://github.com/huggingface/transformers/tree/master/examples/tensorflow/text-classification) - *Note*: You will have to slightly adapt the training example here to make it work with an encoder-decoder model.
95
+
96
+ *JAX/Flax*:
97
+
98
+ - [Summarization](https://github.com/huggingface/transformers/tree/master/examples/flax/summarization)
99
+ - [Text Classification](https://github.com/huggingface/transformers/tree/master/examples/flax/text-classification) - *Note*: You will have to slightly adapt the training example here to make it work with an encoder-decoder model.
100
+
101
+ ## Downstream Performance
102
+
103
+ TODO: Add table if available
104
+
105
+ ## Computational Complexity
106
+
107
+ TODO: Add table if available
108
+
109
+ ## More information
110
+
111
+ We strongly recommend the reader to go carefully through the original paper **[Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers](https://arxiv.org/abs/2109.10686)** to get a more nuanced understanding of this model checkpoint.
112
+ As explained in the following [issue](https://github.com/google-research/google-research/issues/986#issuecomment-1035051145), checkpoints including the *sh* or *skv*
113
+ model architecture variations have *not* been ported to Transformers as they are probably of limited practical usage and are lacking a more detailed description. Those checkpoints are kept [here](https://huggingface.co/NewT5SharedHeadsSharedKeyValues) as they might be ported potentially in the future.