Update README.md
Browse files
README.md
CHANGED
@@ -19,6 +19,47 @@ This is a layer-pruned pre-trained language model sliced with [mergekit](https:/
|
|
19 |
|
20 |
![plot](mistral-7b-v0_3-prune6.png)
|
21 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
22 |
## Merge Details
|
23 |
### Merge Method
|
24 |
|
|
|
19 |
|
20 |
![plot](mistral-7b-v0_3-prune6.png)
|
21 |
|
22 |
+
|
23 |
+
## Quick eval
|
24 |
+
|
25 |
+
Quick eval for: pszemraj/Mistral-7B-v0.3-prune6
|
26 |
+
|
27 |
+
|
28 |
+
hf (pretrained=pszemraj/Mistral-7B-v0.3-prune6,trust_remote_code=True,dtype=bfloat16), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 2
|
29 |
+
|
30 |
+
| Tasks |Version|Filter|n-shot| Metric | Value | |Stderr|
|
31 |
+
|--------------|------:|------|-----:|----------|------:|---|-----:|
|
32 |
+
|arc_easy | 1|none | 0|acc | 0.6393|± |0.0099|
|
33 |
+
| | |none | 0|acc_norm | 0.6309|± |0.0099|
|
34 |
+
|boolq | 2|none | 0|acc | 0.7599|± |0.0075|
|
35 |
+
|lambada_openai| 1|none | 0|perplexity|10.1184|± |0.2771|
|
36 |
+
| | |none | 0|acc | 0.5507|± |0.0069|
|
37 |
+
|openbookqa | 1|none | 0|acc | 0.2200|± |0.0185|
|
38 |
+
| | |none | 0|acc_norm | 0.3580|± |0.0215|
|
39 |
+
|piqa | 1|none | 0|acc | 0.7203|± |0.0105|
|
40 |
+
| | |none | 0|acc_norm | 0.7350|± |0.0103|
|
41 |
+
|winogrande | 1|none | 0|acc | 0.6906|± |0.0130|
|
42 |
+
|
43 |
+
|
44 |
+
### original
|
45 |
+
|
46 |
+
bootstrapping for stddev: perplexity
|
47 |
+
hf (pretrained=mistralai/Mistral-7B-v0.3,trust_remote_code=True,dtype=bfloat16), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 2
|
48 |
+
| Tasks |Version|Filter|n-shot| Metric |Value | |Stderr|
|
49 |
+
|--------------|------:|------|-----:|----------|-----:|---|-----:|
|
50 |
+
|arc_easy | 1|none | 0|acc |0.7959|± |0.0083|
|
51 |
+
| | |none | 0|acc_norm |0.7832|± |0.0085|
|
52 |
+
|boolq | 2|none | 0|acc |0.8202|± |0.0067|
|
53 |
+
|lambada_openai| 1|none | 0|perplexity|3.2578|± |0.0601|
|
54 |
+
| | |none | 0|acc |0.7518|± |0.0060|
|
55 |
+
|openbookqa | 1|none | 0|acc |0.3340|± |0.0211|
|
56 |
+
| | |none | 0|acc_norm |0.4420|± |0.0222|
|
57 |
+
|piqa | 1|none | 0|acc |0.8009|± |0.0093|
|
58 |
+
| | |none | 0|acc_norm |0.8215|± |0.0089|
|
59 |
+
|winogrande | 1|none | 0|acc |0.7380|± |0.0124|
|
60 |
+
|
61 |
+
|
62 |
+
|
63 |
## Merge Details
|
64 |
### Merge Method
|
65 |
|