Hugging Face's logo

wuxianchao
/
lazylora-7bhf

5 papers
Model card Files Community
xianchaowu commited on
Commit
d88199e
•
1 Parent(s): fbd1288

use checkpoint-600 for llama2-7b with open assistant lazy lora

Browse files
Files changed (2) hide show
  1. README.md +132 -61
  2. adapter_model.bin +1 -1
README.md CHANGED
@@ -8,7 +8,15 @@ license: llama2
8
 
9
  0. using the updated [Meta's LLaMA-2 models](https://huggingface.co/meta-llama/Llama-2-7b-hf).
10
  1. support [4-bit qlora](https://arxiv.org/abs/2305.14314), extreme GPU memory and inference time saving;
11
- 2. comparable MMLU evaluation dataset results, llama2-7b's 45.3% to our 44.36% (-0.94%).
 
 
 
 
 
 
 
 
12
 
13
  ### Introduction
14
  Determine the rank of LoRA layers by the singular values of pretrained weight matrices.
@@ -84,67 +92,130 @@ model.print_trainable_parameters()
84
 
85
  ## MMLU result:
86
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
87
  ```json
88
- {"mmlu_loss": 1.8361594152170253,
89
- "mmlu_eval_accuracy_us_foreign_policy": 0.6363636363636364,
90
- "mmlu_eval_accuracy_world_religions": 0.7368421052631579,
91
- "mmlu_eval_accuracy_high_school_us_history": 0.6363636363636364,
92
- "mmlu_eval_accuracy_high_school_psychology": 0.6166666666666667,
93
- "mmlu_eval_accuracy_public_relations": 0.3333333333333333,
94
- "mmlu_eval_accuracy_high_school_european_history": 0.6666666666666666,
95
- "mmlu_eval_accuracy_econometrics": 0.16666666666666666,
96
- "mmlu_eval_accuracy_high_school_microeconomics": 0.34615384615384615,
97
- "mmlu_eval_accuracy_machine_learning": 0.18181818181818182,
98
- "mmlu_eval_accuracy_high_school_mathematics": 0.3448275862068966,
99
- "mmlu_eval_accuracy_high_school_computer_science": 0.5555555555555556,
100
- "mmlu_eval_accuracy_professional_accounting": 0.3548387096774194,
101
- "mmlu_eval_accuracy_high_school_world_history": 0.5,
102
- "mmlu_eval_accuracy_marketing": 0.72,
103
- "mmlu_eval_accuracy_sociology": 0.7272727272727273,
104
- "mmlu_eval_accuracy_nutrition": 0.5454545454545454,
105
- "mmlu_eval_accuracy_high_school_chemistry": 0.4090909090909091,
106
- "mmlu_eval_accuracy_logical_fallacies": 0.5555555555555556,
107
- "mmlu_eval_accuracy_college_mathematics": 0.18181818181818182,
108
- "mmlu_eval_accuracy_computer_security": 0.2727272727272727,
109
- "mmlu_eval_accuracy_miscellaneous": 0.6046511627906976,
110
- "mmlu_eval_accuracy_high_school_statistics": 0.2608695652173913,
111
- "mmlu_eval_accuracy_philosophy": 0.4117647058823529,
112
- "mmlu_eval_accuracy_global_facts": 0.4,
113
- "mmlu_eval_accuracy_management": 0.2727272727272727,
114
- "mmlu_eval_accuracy_human_aging": 0.6956521739130435,
115
- "mmlu_eval_accuracy_moral_scenarios": 0.25,
116
- "mmlu_eval_accuracy_human_sexuality": 0.5,
117
- "mmlu_eval_accuracy_abstract_algebra": 0.36363636363636365,
118
- "mmlu_eval_accuracy_high_school_macroeconomics": 0.3488372093023256,
119
- "mmlu_eval_accuracy_electrical_engineering": 0.375,
120
- "mmlu_eval_accuracy_professional_medicine": 0.45161290322580644,
121
- "mmlu_eval_accuracy_high_school_government_and_politics": 0.6666666666666666,
122
- "mmlu_eval_accuracy_high_school_biology": 0.3125,
123
- "mmlu_eval_accuracy_astronomy": 0.4375,
124
- "mmlu_eval_accuracy_security_studies": 0.4074074074074074,
125
- "mmlu_eval_accuracy_prehistory": 0.42857142857142855,
126
- "mmlu_eval_accuracy_conceptual_physics": 0.3076923076923077,
127
- "mmlu_eval_accuracy_college_medicine": 0.36363636363636365,
128
- "mmlu_eval_accuracy_moral_disputes": 0.39473684210526316,
129
- "mmlu_eval_accuracy_anatomy": 0.5,
130
- "mmlu_eval_accuracy_clinical_knowledge": 0.41379310344827586,
131
- "mmlu_eval_accuracy_college_computer_science": 0.5454545454545454,
132
- "mmlu_eval_accuracy_high_school_geography": 0.5909090909090909,
133
- "mmlu_eval_accuracy_college_chemistry": 0.125,
134
- "mmlu_eval_accuracy_professional_psychology": 0.36231884057971014,
135
- "mmlu_eval_accuracy_virology": 0.4444444444444444,
136
- "mmlu_eval_accuracy_international_law": 0.8461538461538461,
137
- "mmlu_eval_accuracy_medical_genetics": 0.8181818181818182,
138
- "mmlu_eval_accuracy_formal_logic": 0.14285714285714285,
139
- "mmlu_eval_accuracy_professional_law": 0.34705882352941175,
140
- "mmlu_eval_accuracy_college_biology": 0.25,
141
- "mmlu_eval_accuracy_jurisprudence": 0.45454545454545453,
142
- "mmlu_eval_accuracy_business_ethics": 0.5454545454545454,
143
- "mmlu_eval_accuracy_college_physics": 0.5454545454545454,
144
- "mmlu_eval_accuracy_high_school_physics": 0.29411764705882354,
145
- "mmlu_eval_accuracy_elementary_mathematics": 0.3170731707317073,
146
- "mmlu_eval_accuracy": 0.4435841258637352,
147
- "epoch": 1.36}
148
  ```
149
 
150
  ## License and intended use
 
8
 
9
  0. using the updated [Meta's LLaMA-2 models](https://huggingface.co/meta-llama/Llama-2-7b-hf).
10
  1. support [4-bit qlora](https://arxiv.org/abs/2305.14314), extreme GPU memory and inference time saving;
11
+ 2. comparable MMLU evaluation dataset results:
12
+
13
+ | | eval | test | comp-eval | comp-test |
14
+ |---------------|--------|--------|-----------|-----------|
15
+ |llama2-7b | 46.68% | 46.82% | | |
16
+ |ckpt-200 | 44.28% | 46.03% | -2.40% | -0.79% |
17
+ |ckpt-600 | 45.26% | 45.61% | -1.42% | -1.21% |
18
+
19
+ llama2-7b: "4e4d531bcab430a66c4d562b7e89e21c0fa235ea"
20
 
21
  ### Introduction
22
  Determine the rank of LoRA layers by the singular values of pretrained weight matrices.
 
92
 
93
  ## MMLU result:
94
 
95
+ ### MMLU eval result:
96
+ ```json
97
+ {"mmlu_loss": 1.9065961667247102,
98
+ "mmlu_eval_accuracy_professional_medicine": 0.3870967741935484,
99
+ "mmlu_eval_accuracy_college_physics": 0.45454545454545453,
100
+ "mmlu_eval_accuracy_conceptual_physics": 0.34615384615384615,
101
+ "mmlu_eval_accuracy_econometrics": 0.3333333333333333,
102
+ "mmlu_eval_accuracy_high_school_chemistry": 0.45454545454545453,
103
+ "mmlu_eval_accuracy_nutrition": 0.5151515151515151,
104
+ "mmlu_eval_accuracy_high_school_computer_science": 0.5555555555555556,
105
+ "mmlu_eval_accuracy_security_studies": 0.4444444444444444,
106
+ "mmlu_eval_accuracy_world_religions": 0.6842105263157895,
107
+ "mmlu_eval_accuracy_anatomy": 0.5,
108
+ "mmlu_eval_accuracy_prehistory": 0.42857142857142855,
109
+ "mmlu_eval_accuracy_high_school_government_and_politics": 0.6666666666666666,
110
+ "mmlu_eval_accuracy_professional_accounting": 0.3225806451612903,
111
+ "mmlu_eval_accuracy_philosophy": 0.4411764705882353,
112
+ "mmlu_eval_accuracy_astronomy": 0.3125,
113
+ "mmlu_eval_accuracy_medical_genetics": 0.8181818181818182,
114
+ "mmlu_eval_accuracy_jurisprudence": 0.5454545454545454,
115
+ "mmlu_eval_accuracy_professional_law": 0.38235294117647056,
116
+ "mmlu_eval_accuracy_college_chemistry": 0.125,
117
+ "mmlu_eval_accuracy_moral_disputes": 0.4473684210526316,
118
+ "mmlu_eval_accuracy_abstract_algebra": 0.36363636363636365,
119
+ "mmlu_eval_accuracy_computer_security": 0.5454545454545454,
120
+ "mmlu_eval_accuracy_business_ethics": 0.5454545454545454,
121
+ "mmlu_eval_accuracy_virology": 0.5,
122
+ "mmlu_eval_accuracy_electrical_engineering": 0.375,
123
+ "mmlu_eval_accuracy_high_school_biology": 0.34375,
124
+ "mmlu_eval_accuracy_public_relations": 0.3333333333333333,
125
+ "mmlu_eval_accuracy_high_school_physics": 0.35294117647058826,
126
+ "mmlu_eval_accuracy_high_school_psychology": 0.65,
127
+ "mmlu_eval_accuracy_college_computer_science": 0.5454545454545454,
128
+ "mmlu_eval_accuracy_high_school_european_history": 0.7222222222222222,
129
+ "mmlu_eval_accuracy_international_law": 0.8461538461538461,
130
+ "mmlu_eval_accuracy_high_school_microeconomics": 0.2692307692307692,
131
+ "mmlu_eval_accuracy_college_biology": 0.25,
132
+ "mmlu_eval_accuracy_formal_logic": 0.14285714285714285,
133
+ "mmlu_eval_accuracy_machine_learning": 0.18181818181818182,
134
+ "mmlu_eval_accuracy_human_aging": 0.6956521739130435,
135
+ "mmlu_eval_accuracy_logical_fallacies": 0.5555555555555556,
136
+ "mmlu_eval_accuracy_clinical_knowledge": 0.41379310344827586,
137
+ "mmlu_eval_accuracy_high_school_macroeconomics": 0.3488372093023256,
138
+ "mmlu_eval_accuracy_miscellaneous": 0.5930232558139535,
139
+ "mmlu_eval_accuracy_sociology": 0.7272727272727273,
140
+ "mmlu_eval_accuracy_high_school_us_history": 0.6363636363636364,
141
+ "mmlu_eval_accuracy_college_medicine": 0.4090909090909091,
142
+ "mmlu_eval_accuracy_high_school_world_history": 0.5,
143
+ "mmlu_eval_accuracy_marketing": 0.8,
144
+ "mmlu_eval_accuracy_human_sexuality": 0.4166666666666667,
145
+ "mmlu_eval_accuracy_professional_psychology": 0.36231884057971014,
146
+ "mmlu_eval_accuracy_moral_scenarios": 0.24,
147
+ "mmlu_eval_accuracy_college_mathematics": 0.18181818181818182,
148
+ "mmlu_eval_accuracy_us_foreign_policy": 0.6363636363636364,
149
+ "mmlu_eval_accuracy_high_school_geography": 0.6818181818181818,
150
+ "mmlu_eval_accuracy_high_school_statistics": 0.34782608695652173,
151
+ "mmlu_eval_accuracy_high_school_mathematics": 0.2413793103448276,
152
+ "mmlu_eval_accuracy_elementary_mathematics": 0.3170731707317073,
153
+ "mmlu_eval_accuracy_management": 0.36363636363636365,
154
+ "mmlu_eval_accuracy_global_facts": 0.2,
155
+ "mmlu_eval_accuracy": 0.4526436056641111}
156
+ ```
157
+
158
+ ### MMLU test result:
159
  ```json
160
+ {"mmlu_loss": 1.925738222594615,
161
+ "mmlu_test_accuracy_business_ethics": 0.53,
162
+ "mmlu_test_accuracy_medical_genetics": 0.53,
163
+ "mmlu_test_accuracy_international_law": 0.628099173553719,
164
+ "mmlu_test_accuracy_professional_law": 0.3363754889178618,
165
+ "mmlu_test_accuracy_econometrics": 0.32456140350877194,
166
+ "mmlu_test_accuracy_high_school_biology": 0.4806451612903226,
167
+ "mmlu_test_accuracy_computer_security": 0.57,
168
+ "mmlu_test_accuracy_global_facts": 0.34,
169
+ "mmlu_test_accuracy_clinical_knowledge": 0.46037735849056605,
170
+ "mmlu_test_accuracy_miscellaneous": 0.6347381864623244,
171
+ "mmlu_test_accuracy_high_school_microeconomics": 0.39915966386554624,
172
+ "mmlu_test_accuracy_public_relations": 0.5636363636363636,
173
+ "mmlu_test_accuracy_high_school_computer_science": 0.45,
174
+ "mmlu_test_accuracy_human_sexuality": 0.5572519083969466,
175
+ "mmlu_test_accuracy_virology": 0.43373493975903615,
176
+ "mmlu_test_accuracy_human_aging": 0.5695067264573991,
177
+ "mmlu_test_accuracy_high_school_world_history": 0.6371308016877637,
178
+ "mmlu_test_accuracy_college_medicine": 0.3699421965317919,
179
+ "mmlu_test_accuracy_marketing": 0.6923076923076923,
180
+ "mmlu_test_accuracy_world_religions": 0.6783625730994152,
181
+ "mmlu_test_accuracy_college_physics": 0.23529411764705882,
182
+ "mmlu_test_accuracy_high_school_chemistry": 0.33004926108374383,
183
+ "mmlu_test_accuracy_elementary_mathematics": 0.2751322751322751,
184
+ "mmlu_test_accuracy_high_school_psychology": 0.6018348623853211,
185
+ "mmlu_test_accuracy_sociology": 0.5920398009950248,
186
+ "mmlu_test_accuracy_astronomy": 0.4342105263157895,
187
+ "mmlu_test_accuracy_high_school_mathematics": 0.27037037037037037,
188
+ "mmlu_test_accuracy_high_school_us_history": 0.5343137254901961,
189
+ "mmlu_test_accuracy_logical_fallacies": 0.49693251533742333,
190
+ "mmlu_test_accuracy_high_school_statistics": 0.19907407407407407,
191
+ "mmlu_test_accuracy_management": 0.5825242718446602,
192
+ "mmlu_test_accuracy_moral_disputes": 0.5057803468208093,
193
+ "mmlu_test_accuracy_formal_logic": 0.24603174603174602,
194
+ "mmlu_test_accuracy_college_chemistry": 0.25,
195
+ "mmlu_test_accuracy_college_mathematics": 0.3,
196
+ "mmlu_test_accuracy_high_school_geography": 0.5050505050505051,
197
+ "mmlu_test_accuracy_machine_learning": 0.35714285714285715,
198
+ "mmlu_test_accuracy_philosophy": 0.5787781350482315,
199
+ "mmlu_test_accuracy_college_computer_science": 0.32,
200
+ "mmlu_test_accuracy_security_studies": 0.46938775510204084,
201
+ "mmlu_test_accuracy_abstract_algebra": 0.27,
202
+ "mmlu_test_accuracy_professional_psychology": 0.4526143790849673,
203
+ "mmlu_test_accuracy_college_biology": 0.4444444444444444,
204
+ "mmlu_test_accuracy_us_foreign_policy": 0.68,
205
+ "mmlu_test_accuracy_professional_medicine": 0.4522058823529412,
206
+ "mmlu_test_accuracy_prehistory": 0.48148148148148145,
207
+ "mmlu_test_accuracy_anatomy": 0.45925925925925926,
208
+ "mmlu_test_accuracy_moral_scenarios": 0.2346368715083799,
209
+ "mmlu_test_accuracy_nutrition": 0.4738562091503268,
210
+ "mmlu_test_accuracy_high_school_macroeconomics": 0.4461538461538462,
211
+ "mmlu_test_accuracy_high_school_european_history": 0.6181818181818182,
212
+ "mmlu_test_accuracy_jurisprudence": 0.5370370370370371,
213
+ "mmlu_test_accuracy_professional_accounting": 0.35815602836879434,
214
+ "mmlu_test_accuracy_high_school_government_and_politics": 0.6321243523316062,
215
+ "mmlu_test_accuracy_high_school_physics": 0.32450331125827814,
216
+ "mmlu_test_accuracy_electrical_engineering": 0.47586206896551725,
217
+ "mmlu_test_accuracy_conceptual_physics": 0.3872340425531915,
218
+ "mmlu_test_accuracy": 0.4560969792275357}
 
219
  ```
220
 
221
  ## License and intended use
adapter_model.bin CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:3aa846c06af3188a3d7b7c3e0a32ad4c1ecb48b1d02353610425c192c3ae4182
3
  size 320063949
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:af8bf0371e973d003ca99c45cd773c08ad27da0551f89548ba8462a2d3de9269
3
  size 320063949