Eyalyoli commited on
Commit
0a60aec
1 Parent(s): 2535101

use official model instead

Browse files
2_Dense/pytorch_model.bin CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:c5b0a3fed5fc14c404cca5da12bbfdf1f19456e8beeab233dcb17c03e40a6ce9
3
  size 3146603
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0fea898f34b36bf88914400ddd80005cfac4463c76fff37cabef719b3b58a4ad
3
  size 3146603
README.md CHANGED
@@ -1,47 +1,2610 @@
1
  ---
2
  pipeline_tag: sentence-similarity
3
- language: en
4
- license: apache-2.0
5
  tags:
 
 
 
 
 
 
 
 
 
 
 
6
  - sentence-transformers
7
  - feature-extraction
8
  - sentence-similarity
9
  - transformers
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
10
  ---
11
 
12
- # hku-nlp/instructor-xl
13
- This is a general embedding model: It maps **any** piece of text (e.g., a title, a sentence, a document, etc.) to a fixed-length vector in test time **without further training**. With instructions, the embeddings are **domain-specific** (e.g., specialized for science, finance, etc.) and **task-aware** (e.g., customized for classification, information retrieval, etc.)
 
14
 
15
- The model is easy to use with `sentence-transformer` library.
 
 
 
 
 
 
16
 
17
  ## Installation
18
  ```bash
19
- git clone https://github.com/HKUNLP/instructor-embedding
20
- cd sentence-transformers
21
- pip install -e .
22
  ```
23
 
24
  ## Compute your customized embeddings
25
  Then you can use the model like this to calculate domain-specific and task-aware embeddings:
26
  ```python
27
- from sentence_transformers import SentenceTransformer
 
28
  sentence = "3D ActionSLAM: wearable person tracking in multi-floor environments"
29
- instruction = "Represent the Science title; Input:"
30
- model = SentenceTransformer('hku-nlp/instructor-xl')
31
- embeddings = model.encode([[instruction,sentence,0]])
32
  print(embeddings)
33
  ```
34
 
 
 
 
 
 
 
 
 
 
 
 
35
  ## Calculate Sentence similarities
36
  You can further use the model to compute similarities between two groups of sentences, with **customized embeddings**.
37
  ```python
38
  from sklearn.metrics.pairwise import cosine_similarity
39
- sentences_a = [['Represent the Science sentence; Input: ','Parton energy loss in QCD matter',0],
40
- ['Represent the Financial statement; Input: ','The Federal Reserve on Wednesday raised its benchmark interest rate.',0]
41
- sentences_b = [['Represent the Science sentence; Input: ','The Chiral Phase Transition in Dissipative Dynamics', 0],
42
- ['Represent the Financial statement; Input: ','The funds rose less than 0.5 per cent on Friday',0]
43
  embeddings_a = model.encode(sentences_a)
44
  embeddings_b = model.encode(sentences_b)
45
  similarities = cosine_similarity(embeddings_a,embeddings_b)
46
  print(similarities)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
47
  ```
 
1
  ---
2
  pipeline_tag: sentence-similarity
 
 
3
  tags:
4
+ - text-embedding
5
+ - embeddings
6
+ - information-retrieval
7
+ - beir
8
+ - text-classification
9
+ - language-model
10
+ - text-clustering
11
+ - text-semantic-similarity
12
+ - text-evaluation
13
+ - prompt-retrieval
14
+ - text-reranking
15
  - sentence-transformers
16
  - feature-extraction
17
  - sentence-similarity
18
  - transformers
19
+ - t5
20
+ - English
21
+ - Sentence Similarity
22
+ - natural_questions
23
+ - ms_marco
24
+ - fever
25
+ - hotpot_qa
26
+ - mteb
27
+ language: en
28
+ inference: false
29
+ license: apache-2.0
30
+ model-index:
31
+ - name: final_xl_results
32
+ results:
33
+ - task:
34
+ type: Classification
35
+ dataset:
36
+ type: mteb/amazon_counterfactual
37
+ name: MTEB AmazonCounterfactualClassification (en)
38
+ config: en
39
+ split: test
40
+ revision: e8379541af4e31359cca9fbcf4b00f2671dba205
41
+ metrics:
42
+ - type: accuracy
43
+ value: 85.08955223880596
44
+ - type: ap
45
+ value: 52.66066378722476
46
+ - type: f1
47
+ value: 79.63340218960269
48
+ - task:
49
+ type: Classification
50
+ dataset:
51
+ type: mteb/amazon_polarity
52
+ name: MTEB AmazonPolarityClassification
53
+ config: default
54
+ split: test
55
+ revision: e2d317d38cd51312af73b3d32a06d1a08b442046
56
+ metrics:
57
+ - type: accuracy
58
+ value: 86.542
59
+ - type: ap
60
+ value: 81.92695193008987
61
+ - type: f1
62
+ value: 86.51466132573681
63
+ - task:
64
+ type: Classification
65
+ dataset:
66
+ type: mteb/amazon_reviews_multi
67
+ name: MTEB AmazonReviewsClassification (en)
68
+ config: en
69
+ split: test
70
+ revision: 1399c76144fd37290681b995c656ef9b2e06e26d
71
+ metrics:
72
+ - type: accuracy
73
+ value: 42.964
74
+ - type: f1
75
+ value: 41.43146249774862
76
+ - task:
77
+ type: Retrieval
78
+ dataset:
79
+ type: arguana
80
+ name: MTEB ArguAna
81
+ config: default
82
+ split: test
83
+ revision: None
84
+ metrics:
85
+ - type: map_at_1
86
+ value: 29.872
87
+ - type: map_at_10
88
+ value: 46.342
89
+ - type: map_at_100
90
+ value: 47.152
91
+ - type: map_at_1000
92
+ value: 47.154
93
+ - type: map_at_3
94
+ value: 41.216
95
+ - type: map_at_5
96
+ value: 44.035999999999994
97
+ - type: mrr_at_1
98
+ value: 30.939
99
+ - type: mrr_at_10
100
+ value: 46.756
101
+ - type: mrr_at_100
102
+ value: 47.573
103
+ - type: mrr_at_1000
104
+ value: 47.575
105
+ - type: mrr_at_3
106
+ value: 41.548
107
+ - type: mrr_at_5
108
+ value: 44.425
109
+ - type: ndcg_at_1
110
+ value: 29.872
111
+ - type: ndcg_at_10
112
+ value: 55.65
113
+ - type: ndcg_at_100
114
+ value: 58.88099999999999
115
+ - type: ndcg_at_1000
116
+ value: 58.951
117
+ - type: ndcg_at_3
118
+ value: 45.0
119
+ - type: ndcg_at_5
120
+ value: 50.09
121
+ - type: precision_at_1
122
+ value: 29.872
123
+ - type: precision_at_10
124
+ value: 8.549
125
+ - type: precision_at_100
126
+ value: 0.991
127
+ - type: precision_at_1000
128
+ value: 0.1
129
+ - type: precision_at_3
130
+ value: 18.658
131
+ - type: precision_at_5
132
+ value: 13.669999999999998
133
+ - type: recall_at_1
134
+ value: 29.872
135
+ - type: recall_at_10
136
+ value: 85.491
137
+ - type: recall_at_100
138
+ value: 99.075
139
+ - type: recall_at_1000
140
+ value: 99.644
141
+ - type: recall_at_3
142
+ value: 55.974000000000004
143
+ - type: recall_at_5
144
+ value: 68.35
145
+ - task:
146
+ type: Clustering
147
+ dataset:
148
+ type: mteb/arxiv-clustering-p2p
149
+ name: MTEB ArxivClusteringP2P
150
+ config: default
151
+ split: test
152
+ revision: a122ad7f3f0291bf49cc6f4d32aa80929df69d5d
153
+ metrics:
154
+ - type: v_measure
155
+ value: 42.452729850641276
156
+ - task:
157
+ type: Clustering
158
+ dataset:
159
+ type: mteb/arxiv-clustering-s2s
160
+ name: MTEB ArxivClusteringS2S
161
+ config: default
162
+ split: test
163
+ revision: f910caf1a6075f7329cdf8c1a6135696f37dbd53
164
+ metrics:
165
+ - type: v_measure
166
+ value: 32.21141846480423
167
+ - task:
168
+ type: Reranking
169
+ dataset:
170
+ type: mteb/askubuntudupquestions-reranking
171
+ name: MTEB AskUbuntuDupQuestions
172
+ config: default
173
+ split: test
174
+ revision: 2000358ca161889fa9c082cb41daa8dcfb161a54
175
+ metrics:
176
+ - type: map
177
+ value: 65.34710928952622
178
+ - type: mrr
179
+ value: 77.61124301983028
180
+ - task:
181
+ type: STS
182
+ dataset:
183
+ type: mteb/biosses-sts
184
+ name: MTEB BIOSSES
185
+ config: default
186
+ split: test
187
+ revision: d3fb88f8f02e40887cd149695127462bbcf29b4a
188
+ metrics:
189
+ - type: cos_sim_spearman
190
+ value: 84.15312230525639
191
+ - task:
192
+ type: Classification
193
+ dataset:
194
+ type: mteb/banking77
195
+ name: MTEB Banking77Classification
196
+ config: default
197
+ split: test
198
+ revision: 0fd18e25b25c072e09e0d92ab615fda904d66300
199
+ metrics:
200
+ - type: accuracy
201
+ value: 82.66233766233766
202
+ - type: f1
203
+ value: 82.04175284777669
204
+ - task:
205
+ type: Clustering
206
+ dataset:
207
+ type: mteb/biorxiv-clustering-p2p
208
+ name: MTEB BiorxivClusteringP2P
209
+ config: default
210
+ split: test
211
+ revision: 65b79d1d13f80053f67aca9498d9402c2d9f1f40
212
+ metrics:
213
+ - type: v_measure
214
+ value: 37.36697339826455
215
+ - task:
216
+ type: Clustering
217
+ dataset:
218
+ type: mteb/biorxiv-clustering-s2s
219
+ name: MTEB BiorxivClusteringS2S
220
+ config: default
221
+ split: test
222
+ revision: 258694dd0231531bc1fd9de6ceb52a0853c6d908
223
+ metrics:
224
+ - type: v_measure
225
+ value: 30.551241447593092
226
+ - task:
227
+ type: Retrieval
228
+ dataset:
229
+ type: BeIR/cqadupstack
230
+ name: MTEB CQADupstackAndroidRetrieval
231
+ config: default
232
+ split: test
233
+ revision: None
234
+ metrics:
235
+ - type: map_at_1
236
+ value: 36.797000000000004
237
+ - type: map_at_10
238
+ value: 48.46
239
+ - type: map_at_100
240
+ value: 49.968
241
+ - type: map_at_1000
242
+ value: 50.080000000000005
243
+ - type: map_at_3
244
+ value: 44.71
245
+ - type: map_at_5
246
+ value: 46.592
247
+ - type: mrr_at_1
248
+ value: 45.494
249
+ - type: mrr_at_10
250
+ value: 54.747
251
+ - type: mrr_at_100
252
+ value: 55.43599999999999
253
+ - type: mrr_at_1000
254
+ value: 55.464999999999996
255
+ - type: mrr_at_3
256
+ value: 52.361000000000004
257
+ - type: mrr_at_5
258
+ value: 53.727000000000004
259
+ - type: ndcg_at_1
260
+ value: 45.494
261
+ - type: ndcg_at_10
262
+ value: 54.989
263
+ - type: ndcg_at_100
264
+ value: 60.096000000000004
265
+ - type: ndcg_at_1000
266
+ value: 61.58
267
+ - type: ndcg_at_3
268
+ value: 49.977
269
+ - type: ndcg_at_5
270
+ value: 51.964999999999996
271
+ - type: precision_at_1
272
+ value: 45.494
273
+ - type: precision_at_10
274
+ value: 10.558
275
+ - type: precision_at_100
276
+ value: 1.6049999999999998
277
+ - type: precision_at_1000
278
+ value: 0.203
279
+ - type: precision_at_3
280
+ value: 23.796
281
+ - type: precision_at_5
282
+ value: 16.881
283
+ - type: recall_at_1
284
+ value: 36.797000000000004
285
+ - type: recall_at_10
286
+ value: 66.83
287
+ - type: recall_at_100
288
+ value: 88.34100000000001
289
+ - type: recall_at_1000
290
+ value: 97.202
291
+ - type: recall_at_3
292
+ value: 51.961999999999996
293
+ - type: recall_at_5
294
+ value: 57.940000000000005
295
+ - task:
296
+ type: Retrieval
297
+ dataset:
298
+ type: BeIR/cqadupstack
299
+ name: MTEB CQADupstackEnglishRetrieval
300
+ config: default
301
+ split: test
302
+ revision: None
303
+ metrics:
304
+ - type: map_at_1
305
+ value: 32.597
306
+ - type: map_at_10
307
+ value: 43.424
308
+ - type: map_at_100
309
+ value: 44.78
310
+ - type: map_at_1000
311
+ value: 44.913
312
+ - type: map_at_3
313
+ value: 40.315
314
+ - type: map_at_5
315
+ value: 41.987
316
+ - type: mrr_at_1
317
+ value: 40.382
318
+ - type: mrr_at_10
319
+ value: 49.219
320
+ - type: mrr_at_100
321
+ value: 49.895
322
+ - type: mrr_at_1000
323
+ value: 49.936
324
+ - type: mrr_at_3
325
+ value: 46.996
326
+ - type: mrr_at_5
327
+ value: 48.231
328
+ - type: ndcg_at_1
329
+ value: 40.382
330
+ - type: ndcg_at_10
331
+ value: 49.318
332
+ - type: ndcg_at_100
333
+ value: 53.839999999999996
334
+ - type: ndcg_at_1000
335
+ value: 55.82899999999999
336
+ - type: ndcg_at_3
337
+ value: 44.914
338
+ - type: ndcg_at_5
339
+ value: 46.798
340
+ - type: precision_at_1
341
+ value: 40.382
342
+ - type: precision_at_10
343
+ value: 9.274000000000001
344
+ - type: precision_at_100
345
+ value: 1.497
346
+ - type: precision_at_1000
347
+ value: 0.198
348
+ - type: precision_at_3
349
+ value: 21.592
350
+ - type: precision_at_5
351
+ value: 15.159
352
+ - type: recall_at_1
353
+ value: 32.597
354
+ - type: recall_at_10
355
+ value: 59.882000000000005
356
+ - type: recall_at_100
357
+ value: 78.446
358
+ - type: recall_at_1000
359
+ value: 90.88000000000001
360
+ - type: recall_at_3
361
+ value: 46.9
362
+ - type: recall_at_5
363
+ value: 52.222
364
+ - task:
365
+ type: Retrieval
366
+ dataset:
367
+ type: BeIR/cqadupstack
368
+ name: MTEB CQADupstackGamingRetrieval
369
+ config: default
370
+ split: test
371
+ revision: None
372
+ metrics:
373
+ - type: map_at_1
374
+ value: 43.8
375
+ - type: map_at_10
376
+ value: 57.293000000000006
377
+ - type: map_at_100
378
+ value: 58.321
379
+ - type: map_at_1000
380
+ value: 58.361
381
+ - type: map_at_3
382
+ value: 53.839999999999996
383
+ - type: map_at_5
384
+ value: 55.838
385
+ - type: mrr_at_1
386
+ value: 49.592000000000006
387
+ - type: mrr_at_10
388
+ value: 60.643
389
+ - type: mrr_at_100
390
+ value: 61.23499999999999
391
+ - type: mrr_at_1000
392
+ value: 61.251999999999995
393
+ - type: mrr_at_3
394
+ value: 58.265
395
+ - type: mrr_at_5
396
+ value: 59.717
397
+ - type: ndcg_at_1
398
+ value: 49.592000000000006
399
+ - type: ndcg_at_10
400
+ value: 63.364
401
+ - type: ndcg_at_100
402
+ value: 67.167
403
+ - type: ndcg_at_1000
404
+ value: 67.867
405
+ - type: ndcg_at_3
406
+ value: 57.912
407
+ - type: ndcg_at_5
408
+ value: 60.697
409
+ - type: precision_at_1
410
+ value: 49.592000000000006
411
+ - type: precision_at_10
412
+ value: 10.088
413
+ - type: precision_at_100
414
+ value: 1.2930000000000001
415
+ - type: precision_at_1000
416
+ value: 0.13899999999999998
417
+ - type: precision_at_3
418
+ value: 25.789
419
+ - type: precision_at_5
420
+ value: 17.541999999999998
421
+ - type: recall_at_1
422
+ value: 43.8
423
+ - type: recall_at_10
424
+ value: 77.635
425
+ - type: recall_at_100
426
+ value: 93.748
427
+ - type: recall_at_1000
428
+ value: 98.468
429
+ - type: recall_at_3
430
+ value: 63.223
431
+ - type: recall_at_5
432
+ value: 70.122
433
+ - task:
434
+ type: Retrieval
435
+ dataset:
436
+ type: BeIR/cqadupstack
437
+ name: MTEB CQADupstackGisRetrieval
438
+ config: default
439
+ split: test
440
+ revision: None
441
+ metrics:
442
+ - type: map_at_1
443
+ value: 27.721
444
+ - type: map_at_10
445
+ value: 35.626999999999995
446
+ - type: map_at_100
447
+ value: 36.719
448
+ - type: map_at_1000
449
+ value: 36.8
450
+ - type: map_at_3
451
+ value: 32.781
452
+ - type: map_at_5
453
+ value: 34.333999999999996
454
+ - type: mrr_at_1
455
+ value: 29.604999999999997
456
+ - type: mrr_at_10
457
+ value: 37.564
458
+ - type: mrr_at_100
459
+ value: 38.505
460
+ - type: mrr_at_1000
461
+ value: 38.565
462
+ - type: mrr_at_3
463
+ value: 34.727000000000004
464
+ - type: mrr_at_5
465
+ value: 36.207
466
+ - type: ndcg_at_1
467
+ value: 29.604999999999997
468
+ - type: ndcg_at_10
469
+ value: 40.575
470
+ - type: ndcg_at_100
471
+ value: 45.613
472
+ - type: ndcg_at_1000
473
+ value: 47.676
474
+ - type: ndcg_at_3
475
+ value: 34.811
476
+ - type: ndcg_at_5
477
+ value: 37.491
478
+ - type: precision_at_1
479
+ value: 29.604999999999997
480
+ - type: precision_at_10
481
+ value: 6.1690000000000005
482
+ - type: precision_at_100
483
+ value: 0.906
484
+ - type: precision_at_1000
485
+ value: 0.11199999999999999
486
+ - type: precision_at_3
487
+ value: 14.237
488
+ - type: precision_at_5
489
+ value: 10.056
490
+ - type: recall_at_1
491
+ value: 27.721
492
+ - type: recall_at_10
493
+ value: 54.041
494
+ - type: recall_at_100
495
+ value: 76.62299999999999
496
+ - type: recall_at_1000
497
+ value: 92.134
498
+ - type: recall_at_3
499
+ value: 38.582
500
+ - type: recall_at_5
501
+ value: 44.989000000000004
502
+ - task:
503
+ type: Retrieval
504
+ dataset:
505
+ type: BeIR/cqadupstack
506
+ name: MTEB CQADupstackMathematicaRetrieval
507
+ config: default
508
+ split: test
509
+ revision: None
510
+ metrics:
511
+ - type: map_at_1
512
+ value: 16.553
513
+ - type: map_at_10
514
+ value: 25.384
515
+ - type: map_at_100
516
+ value: 26.655
517
+ - type: map_at_1000
518
+ value: 26.778000000000002
519
+ - type: map_at_3
520
+ value: 22.733
521
+ - type: map_at_5
522
+ value: 24.119
523
+ - type: mrr_at_1
524
+ value: 20.149
525
+ - type: mrr_at_10
526
+ value: 29.705
527
+ - type: mrr_at_100
528
+ value: 30.672
529
+ - type: mrr_at_1000
530
+ value: 30.737
531
+ - type: mrr_at_3
532
+ value: 27.032
533
+ - type: mrr_at_5
534
+ value: 28.369
535
+ - type: ndcg_at_1
536
+ value: 20.149
537
+ - type: ndcg_at_10
538
+ value: 30.843999999999998
539
+ - type: ndcg_at_100
540
+ value: 36.716
541
+ - type: ndcg_at_1000
542
+ value: 39.495000000000005
543
+ - type: ndcg_at_3
544
+ value: 25.918999999999997
545
+ - type: ndcg_at_5
546
+ value: 27.992
547
+ - type: precision_at_1
548
+ value: 20.149
549
+ - type: precision_at_10
550
+ value: 5.858
551
+ - type: precision_at_100
552
+ value: 1.009
553
+ - type: precision_at_1000
554
+ value: 0.13799999999999998
555
+ - type: precision_at_3
556
+ value: 12.645000000000001
557
+ - type: precision_at_5
558
+ value: 9.179
559
+ - type: recall_at_1
560
+ value: 16.553
561
+ - type: recall_at_10
562
+ value: 43.136
563
+ - type: recall_at_100
564
+ value: 68.562
565
+ - type: recall_at_1000
566
+ value: 88.208
567
+ - type: recall_at_3
568
+ value: 29.493000000000002
569
+ - type: recall_at_5
570
+ value: 34.751
571
+ - task:
572
+ type: Retrieval
573
+ dataset:
574
+ type: BeIR/cqadupstack
575
+ name: MTEB CQADupstackPhysicsRetrieval
576
+ config: default
577
+ split: test
578
+ revision: None
579
+ metrics:
580
+ - type: map_at_1
581
+ value: 28.000999999999998
582
+ - type: map_at_10
583
+ value: 39.004
584
+ - type: map_at_100
585
+ value: 40.461999999999996
586
+ - type: map_at_1000
587
+ value: 40.566
588
+ - type: map_at_3
589
+ value: 35.805
590
+ - type: map_at_5
591
+ value: 37.672
592
+ - type: mrr_at_1
593
+ value: 33.782000000000004
594
+ - type: mrr_at_10
595
+ value: 44.702
596
+ - type: mrr_at_100
597
+ value: 45.528
598
+ - type: mrr_at_1000
599
+ value: 45.576
600
+ - type: mrr_at_3
601
+ value: 42.14
602
+ - type: mrr_at_5
603
+ value: 43.651
604
+ - type: ndcg_at_1
605
+ value: 33.782000000000004
606
+ - type: ndcg_at_10
607
+ value: 45.275999999999996
608
+ - type: ndcg_at_100
609
+ value: 50.888
610
+ - type: ndcg_at_1000
611
+ value: 52.879
612
+ - type: ndcg_at_3
613
+ value: 40.191
614
+ - type: ndcg_at_5
615
+ value: 42.731
616
+ - type: precision_at_1
617
+ value: 33.782000000000004
618
+ - type: precision_at_10
619
+ value: 8.200000000000001
620
+ - type: precision_at_100
621
+ value: 1.287
622
+ - type: precision_at_1000
623
+ value: 0.16199999999999998
624
+ - type: precision_at_3
625
+ value: 19.185
626
+ - type: precision_at_5
627
+ value: 13.667000000000002
628
+ - type: recall_at_1
629
+ value: 28.000999999999998
630
+ - type: recall_at_10
631
+ value: 58.131
632
+ - type: recall_at_100
633
+ value: 80.869
634
+ - type: recall_at_1000
635
+ value: 93.931
636
+ - type: recall_at_3
637
+ value: 44.161
638
+ - type: recall_at_5
639
+ value: 50.592000000000006
640
+ - task:
641
+ type: Retrieval
642
+ dataset:
643
+ type: BeIR/cqadupstack
644
+ name: MTEB CQADupstackProgrammersRetrieval
645
+ config: default
646
+ split: test
647
+ revision: None
648
+ metrics:
649
+ - type: map_at_1
650
+ value: 28.047
651
+ - type: map_at_10
652
+ value: 38.596000000000004
653
+ - type: map_at_100
654
+ value: 40.116
655
+ - type: map_at_1000
656
+ value: 40.232
657
+ - type: map_at_3
658
+ value: 35.205
659
+ - type: map_at_5
660
+ value: 37.076
661
+ - type: mrr_at_1
662
+ value: 34.932
663
+ - type: mrr_at_10
664
+ value: 44.496
665
+ - type: mrr_at_100
666
+ value: 45.47
667
+ - type: mrr_at_1000
668
+ value: 45.519999999999996
669
+ - type: mrr_at_3
670
+ value: 41.743
671
+ - type: mrr_at_5
672
+ value: 43.352000000000004
673
+ - type: ndcg_at_1
674
+ value: 34.932
675
+ - type: ndcg_at_10
676
+ value: 44.901
677
+ - type: ndcg_at_100
678
+ value: 50.788999999999994
679
+ - type: ndcg_at_1000
680
+ value: 52.867
681
+ - type: ndcg_at_3
682
+ value: 39.449
683
+ - type: ndcg_at_5
684
+ value: 41.929
685
+ - type: precision_at_1
686
+ value: 34.932
687
+ - type: precision_at_10
688
+ value: 8.311
689
+ - type: precision_at_100
690
+ value: 1.3050000000000002
691
+ - type: precision_at_1000
692
+ value: 0.166
693
+ - type: precision_at_3
694
+ value: 18.836
695
+ - type: precision_at_5
696
+ value: 13.447000000000001
697
+ - type: recall_at_1
698
+ value: 28.047
699
+ - type: recall_at_10
700
+ value: 57.717
701
+ - type: recall_at_100
702
+ value: 82.182
703
+ - type: recall_at_1000
704
+ value: 95.82000000000001
705
+ - type: recall_at_3
706
+ value: 42.448
707
+ - type: recall_at_5
708
+ value: 49.071
709
+ - task:
710
+ type: Retrieval
711
+ dataset:
712
+ type: BeIR/cqadupstack
713
+ name: MTEB CQADupstackRetrieval
714
+ config: default
715
+ split: test
716
+ revision: None
717
+ metrics:
718
+ - type: map_at_1
719
+ value: 27.861250000000005
720
+ - type: map_at_10
721
+ value: 37.529583333333335
722
+ - type: map_at_100
723
+ value: 38.7915
724
+ - type: map_at_1000
725
+ value: 38.90558333333335
726
+ - type: map_at_3
727
+ value: 34.57333333333333
728
+ - type: map_at_5
729
+ value: 36.187166666666656
730
+ - type: mrr_at_1
731
+ value: 32.88291666666666
732
+ - type: mrr_at_10
733
+ value: 41.79750000000001
734
+ - type: mrr_at_100
735
+ value: 42.63183333333333
736
+ - type: mrr_at_1000
737
+ value: 42.68483333333333
738
+ - type: mrr_at_3
739
+ value: 39.313750000000006
740
+ - type: mrr_at_5
741
+ value: 40.70483333333333
742
+ - type: ndcg_at_1
743
+ value: 32.88291666666666
744
+ - type: ndcg_at_10
745
+ value: 43.09408333333333
746
+ - type: ndcg_at_100
747
+ value: 48.22158333333333
748
+ - type: ndcg_at_1000
749
+ value: 50.358000000000004
750
+ - type: ndcg_at_3
751
+ value: 38.129583333333336
752
+ - type: ndcg_at_5
753
+ value: 40.39266666666666
754
+ - type: precision_at_1
755
+ value: 32.88291666666666
756
+ - type: precision_at_10
757
+ value: 7.5584999999999996
758
+ - type: precision_at_100
759
+ value: 1.1903333333333332
760
+ - type: precision_at_1000
761
+ value: 0.15658333333333332
762
+ - type: precision_at_3
763
+ value: 17.495916666666666
764
+ - type: precision_at_5
765
+ value: 12.373833333333332
766
+ - type: recall_at_1
767
+ value: 27.861250000000005
768
+ - type: recall_at_10
769
+ value: 55.215916666666665
770
+ - type: recall_at_100
771
+ value: 77.392
772
+ - type: recall_at_1000
773
+ value: 92.04908333333334
774
+ - type: recall_at_3
775
+ value: 41.37475
776
+ - type: recall_at_5
777
+ value: 47.22908333333333
778
+ - task:
779
+ type: Retrieval
780
+ dataset:
781
+ type: BeIR/cqadupstack
782
+ name: MTEB CQADupstackStatsRetrieval
783
+ config: default
784
+ split: test
785
+ revision: None
786
+ metrics:
787
+ - type: map_at_1
788
+ value: 25.064999999999998
789
+ - type: map_at_10
790
+ value: 31.635999999999996
791
+ - type: map_at_100
792
+ value: 32.596000000000004
793
+ - type: map_at_1000
794
+ value: 32.695
795
+ - type: map_at_3
796
+ value: 29.612
797
+ - type: map_at_5
798
+ value: 30.768
799
+ - type: mrr_at_1
800
+ value: 28.528
801
+ - type: mrr_at_10
802
+ value: 34.717
803
+ - type: mrr_at_100
804
+ value: 35.558
805
+ - type: mrr_at_1000
806
+ value: 35.626000000000005
807
+ - type: mrr_at_3
808
+ value: 32.745000000000005
809
+ - type: mrr_at_5
810
+ value: 33.819
811
+ - type: ndcg_at_1
812
+ value: 28.528
813
+ - type: ndcg_at_10
814
+ value: 35.647
815
+ - type: ndcg_at_100
816
+ value: 40.207
817
+ - type: ndcg_at_1000
818
+ value: 42.695
819
+ - type: ndcg_at_3
820
+ value: 31.878
821
+ - type: ndcg_at_5
822
+ value: 33.634
823
+ - type: precision_at_1
824
+ value: 28.528
825
+ - type: precision_at_10
826
+ value: 5.46
827
+ - type: precision_at_100
828
+ value: 0.84
829
+ - type: precision_at_1000
830
+ value: 0.11399999999999999
831
+ - type: precision_at_3
832
+ value: 13.547999999999998
833
+ - type: precision_at_5
834
+ value: 9.325
835
+ - type: recall_at_1
836
+ value: 25.064999999999998
837
+ - type: recall_at_10
838
+ value: 45.096000000000004
839
+ - type: recall_at_100
840
+ value: 65.658
841
+ - type: recall_at_1000
842
+ value: 84.128
843
+ - type: recall_at_3
844
+ value: 34.337
845
+ - type: recall_at_5
846
+ value: 38.849000000000004
847
+ - task:
848
+ type: Retrieval
849
+ dataset:
850
+ type: BeIR/cqadupstack
851
+ name: MTEB CQADupstackTexRetrieval
852
+ config: default
853
+ split: test
854
+ revision: None
855
+ metrics:
856
+ - type: map_at_1
857
+ value: 17.276
858
+ - type: map_at_10
859
+ value: 24.535
860
+ - type: map_at_100
861
+ value: 25.655
862
+ - type: map_at_1000
863
+ value: 25.782
864
+ - type: map_at_3
865
+ value: 22.228
866
+ - type: map_at_5
867
+ value: 23.612
868
+ - type: mrr_at_1
869
+ value: 21.266
870
+ - type: mrr_at_10
871
+ value: 28.474
872
+ - type: mrr_at_100
873
+ value: 29.398000000000003
874
+ - type: mrr_at_1000
875
+ value: 29.482000000000003
876
+ - type: mrr_at_3
877
+ value: 26.245
878
+ - type: mrr_at_5
879
+ value: 27.624
880
+ - type: ndcg_at_1
881
+ value: 21.266
882
+ - type: ndcg_at_10
883
+ value: 29.087000000000003
884
+ - type: ndcg_at_100
885
+ value: 34.374
886
+ - type: ndcg_at_1000
887
+ value: 37.433
888
+ - type: ndcg_at_3
889
+ value: 25.040000000000003
890
+ - type: ndcg_at_5
891
+ value: 27.116
892
+ - type: precision_at_1
893
+ value: 21.266
894
+ - type: precision_at_10
895
+ value: 5.258
896
+ - type: precision_at_100
897
+ value: 0.9299999999999999
898
+ - type: precision_at_1000
899
+ value: 0.13699999999999998
900
+ - type: precision_at_3
901
+ value: 11.849
902
+ - type: precision_at_5
903
+ value: 8.699
904
+ - type: recall_at_1
905
+ value: 17.276
906
+ - type: recall_at_10
907
+ value: 38.928000000000004
908
+ - type: recall_at_100
909
+ value: 62.529
910
+ - type: recall_at_1000
911
+ value: 84.44800000000001
912
+ - type: recall_at_3
913
+ value: 27.554000000000002
914
+ - type: recall_at_5
915
+ value: 32.915
916
+ - task:
917
+ type: Retrieval
918
+ dataset:
919
+ type: BeIR/cqadupstack
920
+ name: MTEB CQADupstackUnixRetrieval
921
+ config: default
922
+ split: test
923
+ revision: None
924
+ metrics:
925
+ - type: map_at_1
926
+ value: 27.297
927
+ - type: map_at_10
928
+ value: 36.957
929
+ - type: map_at_100
930
+ value: 38.252
931
+ - type: map_at_1000
932
+ value: 38.356
933
+ - type: map_at_3
934
+ value: 34.121
935
+ - type: map_at_5
936
+ value: 35.782000000000004
937
+ - type: mrr_at_1
938
+ value: 32.275999999999996
939
+ - type: mrr_at_10
940
+ value: 41.198
941
+ - type: mrr_at_100
942
+ value: 42.131
943
+ - type: mrr_at_1000
944
+ value: 42.186
945
+ - type: mrr_at_3
946
+ value: 38.557
947
+ - type: mrr_at_5
948
+ value: 40.12
949
+ - type: ndcg_at_1
950
+ value: 32.275999999999996
951
+ - type: ndcg_at_10
952
+ value: 42.516
953
+ - type: ndcg_at_100
954
+ value: 48.15
955
+ - type: ndcg_at_1000
956
+ value: 50.344
957
+ - type: ndcg_at_3
958
+ value: 37.423
959
+ - type: ndcg_at_5
960
+ value: 39.919
961
+ - type: precision_at_1
962
+ value: 32.275999999999996
963
+ - type: precision_at_10
964
+ value: 7.155
965
+ - type: precision_at_100
966
+ value: 1.123
967
+ - type: precision_at_1000
968
+ value: 0.14200000000000002
969
+ - type: precision_at_3
970
+ value: 17.163999999999998
971
+ - type: precision_at_5
972
+ value: 12.127
973
+ - type: recall_at_1
974
+ value: 27.297
975
+ - type: recall_at_10
976
+ value: 55.238
977
+ - type: recall_at_100
978
+ value: 79.2
979
+ - type: recall_at_1000
980
+ value: 94.258
981
+ - type: recall_at_3
982
+ value: 41.327000000000005
983
+ - type: recall_at_5
984
+ value: 47.588
985
+ - task:
986
+ type: Retrieval
987
+ dataset:
988
+ type: BeIR/cqadupstack
989
+ name: MTEB CQADupstackWebmastersRetrieval
990
+ config: default
991
+ split: test
992
+ revision: None
993
+ metrics:
994
+ - type: map_at_1
995
+ value: 29.142000000000003
996
+ - type: map_at_10
997
+ value: 38.769
998
+ - type: map_at_100
999
+ value: 40.292
1000
+ - type: map_at_1000
1001
+ value: 40.510000000000005
1002
+ - type: map_at_3
1003
+ value: 35.39
1004
+ - type: map_at_5
1005
+ value: 37.009
1006
+ - type: mrr_at_1
1007
+ value: 34.19
1008
+ - type: mrr_at_10
1009
+ value: 43.418
1010
+ - type: mrr_at_100
1011
+ value: 44.132
1012
+ - type: mrr_at_1000
1013
+ value: 44.175
1014
+ - type: mrr_at_3
1015
+ value: 40.547
1016
+ - type: mrr_at_5
1017
+ value: 42.088
1018
+ - type: ndcg_at_1
1019
+ value: 34.19
1020
+ - type: ndcg_at_10
1021
+ value: 45.14
1022
+ - type: ndcg_at_100
1023
+ value: 50.364
1024
+ - type: ndcg_at_1000
1025
+ value: 52.481
1026
+ - type: ndcg_at_3
1027
+ value: 39.466
1028
+ - type: ndcg_at_5
1029
+ value: 41.772
1030
+ - type: precision_at_1
1031
+ value: 34.19
1032
+ - type: precision_at_10
1033
+ value: 8.715
1034
+ - type: precision_at_100
1035
+ value: 1.6150000000000002
1036
+ - type: precision_at_1000
1037
+ value: 0.247
1038
+ - type: precision_at_3
1039
+ value: 18.248
1040
+ - type: precision_at_5
1041
+ value: 13.161999999999999
1042
+ - type: recall_at_1
1043
+ value: 29.142000000000003
1044
+ - type: recall_at_10
1045
+ value: 57.577999999999996
1046
+ - type: recall_at_100
1047
+ value: 81.428
1048
+ - type: recall_at_1000
1049
+ value: 94.017
1050
+ - type: recall_at_3
1051
+ value: 41.402
1052
+ - type: recall_at_5
1053
+ value: 47.695
1054
+ - task:
1055
+ type: Retrieval
1056
+ dataset:
1057
+ type: BeIR/cqadupstack
1058
+ name: MTEB CQADupstackWordpressRetrieval
1059
+ config: default
1060
+ split: test
1061
+ revision: None
1062
+ metrics:
1063
+ - type: map_at_1
1064
+ value: 22.039
1065
+ - type: map_at_10
1066
+ value: 30.669999999999998
1067
+ - type: map_at_100
1068
+ value: 31.682
1069
+ - type: map_at_1000
1070
+ value: 31.794
1071
+ - type: map_at_3
1072
+ value: 28.139999999999997
1073
+ - type: map_at_5
1074
+ value: 29.457
1075
+ - type: mrr_at_1
1076
+ value: 24.399
1077
+ - type: mrr_at_10
1078
+ value: 32.687
1079
+ - type: mrr_at_100
1080
+ value: 33.622
1081
+ - type: mrr_at_1000
1082
+ value: 33.698
1083
+ - type: mrr_at_3
1084
+ value: 30.407
1085
+ - type: mrr_at_5
1086
+ value: 31.552999999999997
1087
+ - type: ndcg_at_1
1088
+ value: 24.399
1089
+ - type: ndcg_at_10
1090
+ value: 35.472
1091
+ - type: ndcg_at_100
1092
+ value: 40.455000000000005
1093
+ - type: ndcg_at_1000
1094
+ value: 43.15
1095
+ - type: ndcg_at_3
1096
+ value: 30.575000000000003
1097
+ - type: ndcg_at_5
1098
+ value: 32.668
1099
+ - type: precision_at_1
1100
+ value: 24.399
1101
+ - type: precision_at_10
1102
+ value: 5.656
1103
+ - type: precision_at_100
1104
+ value: 0.874
1105
+ - type: precision_at_1000
1106
+ value: 0.121
1107
+ - type: precision_at_3
1108
+ value: 13.062000000000001
1109
+ - type: precision_at_5
1110
+ value: 9.242
1111
+ - type: recall_at_1
1112
+ value: 22.039
1113
+ - type: recall_at_10
1114
+ value: 48.379
1115
+ - type: recall_at_100
1116
+ value: 71.11800000000001
1117
+ - type: recall_at_1000
1118
+ value: 91.095
1119
+ - type: recall_at_3
1120
+ value: 35.108
1121
+ - type: recall_at_5
1122
+ value: 40.015
1123
+ - task:
1124
+ type: Retrieval
1125
+ dataset:
1126
+ type: climate-fever
1127
+ name: MTEB ClimateFEVER
1128
+ config: default
1129
+ split: test
1130
+ revision: None
1131
+ metrics:
1132
+ - type: map_at_1
1133
+ value: 10.144
1134
+ - type: map_at_10
1135
+ value: 18.238
1136
+ - type: map_at_100
1137
+ value: 20.143
1138
+ - type: map_at_1000
1139
+ value: 20.346
1140
+ - type: map_at_3
1141
+ value: 14.809
1142
+ - type: map_at_5
1143
+ value: 16.567999999999998
1144
+ - type: mrr_at_1
1145
+ value: 22.671
1146
+ - type: mrr_at_10
1147
+ value: 34.906
1148
+ - type: mrr_at_100
1149
+ value: 35.858000000000004
1150
+ - type: mrr_at_1000
1151
+ value: 35.898
1152
+ - type: mrr_at_3
1153
+ value: 31.238
1154
+ - type: mrr_at_5
1155
+ value: 33.342
1156
+ - type: ndcg_at_1
1157
+ value: 22.671
1158
+ - type: ndcg_at_10
1159
+ value: 26.540000000000003
1160
+ - type: ndcg_at_100
1161
+ value: 34.138000000000005
1162
+ - type: ndcg_at_1000
1163
+ value: 37.72
1164
+ - type: ndcg_at_3
1165
+ value: 20.766000000000002
1166
+ - type: ndcg_at_5
1167
+ value: 22.927
1168
+ - type: precision_at_1
1169
+ value: 22.671
1170
+ - type: precision_at_10
1171
+ value: 8.619
1172
+ - type: precision_at_100
1173
+ value: 1.678
1174
+ - type: precision_at_1000
1175
+ value: 0.23500000000000001
1176
+ - type: precision_at_3
1177
+ value: 15.592
1178
+ - type: precision_at_5
1179
+ value: 12.43
1180
+ - type: recall_at_1
1181
+ value: 10.144
1182
+ - type: recall_at_10
1183
+ value: 33.46
1184
+ - type: recall_at_100
1185
+ value: 59.758
1186
+ - type: recall_at_1000
1187
+ value: 79.704
1188
+ - type: recall_at_3
1189
+ value: 19.604
1190
+ - type: recall_at_5
1191
+ value: 25.367
1192
+ - task:
1193
+ type: Retrieval
1194
+ dataset:
1195
+ type: dbpedia-entity
1196
+ name: MTEB DBPedia
1197
+ config: default
1198
+ split: test
1199
+ revision: None
1200
+ metrics:
1201
+ - type: map_at_1
1202
+ value: 8.654
1203
+ - type: map_at_10
1204
+ value: 18.506
1205
+ - type: map_at_100
1206
+ value: 26.412999999999997
1207
+ - type: map_at_1000
1208
+ value: 28.13
1209
+ - type: map_at_3
1210
+ value: 13.379
1211
+ - type: map_at_5
1212
+ value: 15.529000000000002
1213
+ - type: mrr_at_1
1214
+ value: 66.0
1215
+ - type: mrr_at_10
1216
+ value: 74.13
1217
+ - type: mrr_at_100
1218
+ value: 74.48700000000001
1219
+ - type: mrr_at_1000
1220
+ value: 74.49799999999999
1221
+ - type: mrr_at_3
1222
+ value: 72.75
1223
+ - type: mrr_at_5
1224
+ value: 73.762
1225
+ - type: ndcg_at_1
1226
+ value: 54.50000000000001
1227
+ - type: ndcg_at_10
1228
+ value: 40.236
1229
+ - type: ndcg_at_100
1230
+ value: 44.690999999999995
1231
+ - type: ndcg_at_1000
1232
+ value: 52.195
1233
+ - type: ndcg_at_3
1234
+ value: 45.632
1235
+ - type: ndcg_at_5
1236
+ value: 42.952
1237
+ - type: precision_at_1
1238
+ value: 66.0
1239
+ - type: precision_at_10
1240
+ value: 31.724999999999998
1241
+ - type: precision_at_100
1242
+ value: 10.299999999999999
1243
+ - type: precision_at_1000
1244
+ value: 2.194
1245
+ - type: precision_at_3
1246
+ value: 48.75
1247
+ - type: precision_at_5
1248
+ value: 41.6
1249
+ - type: recall_at_1
1250
+ value: 8.654
1251
+ - type: recall_at_10
1252
+ value: 23.74
1253
+ - type: recall_at_100
1254
+ value: 50.346999999999994
1255
+ - type: recall_at_1000
1256
+ value: 74.376
1257
+ - type: recall_at_3
1258
+ value: 14.636
1259
+ - type: recall_at_5
1260
+ value: 18.009
1261
+ - task:
1262
+ type: Classification
1263
+ dataset:
1264
+ type: mteb/emotion
1265
+ name: MTEB EmotionClassification
1266
+ config: default
1267
+ split: test
1268
+ revision: 4f58c6b202a23cf9a4da393831edf4f9183cad37
1269
+ metrics:
1270
+ - type: accuracy
1271
+ value: 53.245
1272
+ - type: f1
1273
+ value: 48.74520523753552
1274
+ - task:
1275
+ type: Retrieval
1276
+ dataset:
1277
+ type: fever
1278
+ name: MTEB FEVER
1279
+ config: default
1280
+ split: test
1281
+ revision: None
1282
+ metrics:
1283
+ - type: map_at_1
1284
+ value: 51.729
1285
+ - type: map_at_10
1286
+ value: 63.904
1287
+ - type: map_at_100
1288
+ value: 64.363
1289
+ - type: map_at_1000
1290
+ value: 64.38199999999999
1291
+ - type: map_at_3
1292
+ value: 61.393
1293
+ - type: map_at_5
1294
+ value: 63.02100000000001
1295
+ - type: mrr_at_1
1296
+ value: 55.686
1297
+ - type: mrr_at_10
1298
+ value: 67.804
1299
+ - type: mrr_at_100
1300
+ value: 68.15299999999999
1301
+ - type: mrr_at_1000
1302
+ value: 68.161
1303
+ - type: mrr_at_3
1304
+ value: 65.494
1305
+ - type: mrr_at_5
1306
+ value: 67.01599999999999
1307
+ - type: ndcg_at_1
1308
+ value: 55.686
1309
+ - type: ndcg_at_10
1310
+ value: 70.025
1311
+ - type: ndcg_at_100
1312
+ value: 72.011
1313
+ - type: ndcg_at_1000
1314
+ value: 72.443
1315
+ - type: ndcg_at_3
1316
+ value: 65.32900000000001
1317
+ - type: ndcg_at_5
1318
+ value: 68.05600000000001
1319
+ - type: precision_at_1
1320
+ value: 55.686
1321
+ - type: precision_at_10
1322
+ value: 9.358
1323
+ - type: precision_at_100
1324
+ value: 1.05
1325
+ - type: precision_at_1000
1326
+ value: 0.11
1327
+ - type: precision_at_3
1328
+ value: 26.318
1329
+ - type: precision_at_5
1330
+ value: 17.321
1331
+ - type: recall_at_1
1332
+ value: 51.729
1333
+ - type: recall_at_10
1334
+ value: 85.04
1335
+ - type: recall_at_100
1336
+ value: 93.777
1337
+ - type: recall_at_1000
1338
+ value: 96.824
1339
+ - type: recall_at_3
1340
+ value: 72.521
1341
+ - type: recall_at_5
1342
+ value: 79.148
1343
+ - task:
1344
+ type: Retrieval
1345
+ dataset:
1346
+ type: fiqa
1347
+ name: MTEB FiQA2018
1348
+ config: default
1349
+ split: test
1350
+ revision: None
1351
+ metrics:
1352
+ - type: map_at_1
1353
+ value: 23.765
1354
+ - type: map_at_10
1355
+ value: 39.114
1356
+ - type: map_at_100
1357
+ value: 40.987
1358
+ - type: map_at_1000
1359
+ value: 41.155
1360
+ - type: map_at_3
1361
+ value: 34.028000000000006
1362
+ - type: map_at_5
1363
+ value: 36.925000000000004
1364
+ - type: mrr_at_1
1365
+ value: 46.451
1366
+ - type: mrr_at_10
1367
+ value: 54.711
1368
+ - type: mrr_at_100
1369
+ value: 55.509
1370
+ - type: mrr_at_1000
1371
+ value: 55.535000000000004
1372
+ - type: mrr_at_3
1373
+ value: 52.649
1374
+ - type: mrr_at_5
1375
+ value: 53.729000000000006
1376
+ - type: ndcg_at_1
1377
+ value: 46.451
1378
+ - type: ndcg_at_10
1379
+ value: 46.955999999999996
1380
+ - type: ndcg_at_100
1381
+ value: 53.686
1382
+ - type: ndcg_at_1000
1383
+ value: 56.230000000000004
1384
+ - type: ndcg_at_3
1385
+ value: 43.374
1386
+ - type: ndcg_at_5
1387
+ value: 44.372
1388
+ - type: precision_at_1
1389
+ value: 46.451
1390
+ - type: precision_at_10
1391
+ value: 13.256
1392
+ - type: precision_at_100
1393
+ value: 2.019
1394
+ - type: precision_at_1000
1395
+ value: 0.247
1396
+ - type: precision_at_3
1397
+ value: 29.115000000000002
1398
+ - type: precision_at_5
1399
+ value: 21.389
1400
+ - type: recall_at_1
1401
+ value: 23.765
1402
+ - type: recall_at_10
1403
+ value: 53.452999999999996
1404
+ - type: recall_at_100
1405
+ value: 78.828
1406
+ - type: recall_at_1000
1407
+ value: 93.938
1408
+ - type: recall_at_3
1409
+ value: 39.023
1410
+ - type: recall_at_5
1411
+ value: 45.18
1412
+ - task:
1413
+ type: Retrieval
1414
+ dataset:
1415
+ type: hotpotqa
1416
+ name: MTEB HotpotQA
1417
+ config: default
1418
+ split: test
1419
+ revision: None
1420
+ metrics:
1421
+ - type: map_at_1
1422
+ value: 31.918000000000003
1423
+ - type: map_at_10
1424
+ value: 46.741
1425
+ - type: map_at_100
1426
+ value: 47.762
1427
+ - type: map_at_1000
1428
+ value: 47.849000000000004
1429
+ - type: map_at_3
1430
+ value: 43.578
1431
+ - type: map_at_5
1432
+ value: 45.395
1433
+ - type: mrr_at_1
1434
+ value: 63.834999999999994
1435
+ - type: mrr_at_10
1436
+ value: 71.312
1437
+ - type: mrr_at_100
1438
+ value: 71.695
1439
+ - type: mrr_at_1000
1440
+ value: 71.714
1441
+ - type: mrr_at_3
1442
+ value: 69.82000000000001
1443
+ - type: mrr_at_5
1444
+ value: 70.726
1445
+ - type: ndcg_at_1
1446
+ value: 63.834999999999994
1447
+ - type: ndcg_at_10
1448
+ value: 55.879999999999995
1449
+ - type: ndcg_at_100
1450
+ value: 59.723000000000006
1451
+ - type: ndcg_at_1000
1452
+ value: 61.49400000000001
1453
+ - type: ndcg_at_3
1454
+ value: 50.964
1455
+ - type: ndcg_at_5
1456
+ value: 53.47
1457
+ - type: precision_at_1
1458
+ value: 63.834999999999994
1459
+ - type: precision_at_10
1460
+ value: 11.845
1461
+ - type: precision_at_100
1462
+ value: 1.4869999999999999
1463
+ - type: precision_at_1000
1464
+ value: 0.172
1465
+ - type: precision_at_3
1466
+ value: 32.158
1467
+ - type: precision_at_5
1468
+ value: 21.278
1469
+ - type: recall_at_1
1470
+ value: 31.918000000000003
1471
+ - type: recall_at_10
1472
+ value: 59.223000000000006
1473
+ - type: recall_at_100
1474
+ value: 74.328
1475
+ - type: recall_at_1000
1476
+ value: 86.05000000000001
1477
+ - type: recall_at_3
1478
+ value: 48.238
1479
+ - type: recall_at_5
1480
+ value: 53.193999999999996
1481
+ - task:
1482
+ type: Classification
1483
+ dataset:
1484
+ type: mteb/imdb
1485
+ name: MTEB ImdbClassification
1486
+ config: default
1487
+ split: test
1488
+ revision: 3d86128a09e091d6018b6d26cad27f2739fc2db7
1489
+ metrics:
1490
+ - type: accuracy
1491
+ value: 79.7896
1492
+ - type: ap
1493
+ value: 73.65166029460288
1494
+ - type: f1
1495
+ value: 79.71794693711813
1496
+ - task:
1497
+ type: Retrieval
1498
+ dataset:
1499
+ type: msmarco
1500
+ name: MTEB MSMARCO
1501
+ config: default
1502
+ split: dev
1503
+ revision: None
1504
+ metrics:
1505
+ - type: map_at_1
1506
+ value: 22.239
1507
+ - type: map_at_10
1508
+ value: 34.542
1509
+ - type: map_at_100
1510
+ value: 35.717999999999996
1511
+ - type: map_at_1000
1512
+ value: 35.764
1513
+ - type: map_at_3
1514
+ value: 30.432
1515
+ - type: map_at_5
1516
+ value: 32.81
1517
+ - type: mrr_at_1
1518
+ value: 22.908
1519
+ - type: mrr_at_10
1520
+ value: 35.127
1521
+ - type: mrr_at_100
1522
+ value: 36.238
1523
+ - type: mrr_at_1000
1524
+ value: 36.278
1525
+ - type: mrr_at_3
1526
+ value: 31.076999999999998
1527
+ - type: mrr_at_5
1528
+ value: 33.419
1529
+ - type: ndcg_at_1
1530
+ value: 22.908
1531
+ - type: ndcg_at_10
1532
+ value: 41.607
1533
+ - type: ndcg_at_100
1534
+ value: 47.28
1535
+ - type: ndcg_at_1000
1536
+ value: 48.414
1537
+ - type: ndcg_at_3
1538
+ value: 33.253
1539
+ - type: ndcg_at_5
1540
+ value: 37.486000000000004
1541
+ - type: precision_at_1
1542
+ value: 22.908
1543
+ - type: precision_at_10
1544
+ value: 6.645
1545
+ - type: precision_at_100
1546
+ value: 0.9490000000000001
1547
+ - type: precision_at_1000
1548
+ value: 0.105
1549
+ - type: precision_at_3
1550
+ value: 14.130999999999998
1551
+ - type: precision_at_5
1552
+ value: 10.616
1553
+ - type: recall_at_1
1554
+ value: 22.239
1555
+ - type: recall_at_10
1556
+ value: 63.42
1557
+ - type: recall_at_100
1558
+ value: 89.696
1559
+ - type: recall_at_1000
1560
+ value: 98.351
1561
+ - type: recall_at_3
1562
+ value: 40.77
1563
+ - type: recall_at_5
1564
+ value: 50.93
1565
+ - task:
1566
+ type: Classification
1567
+ dataset:
1568
+ type: mteb/mtop_domain
1569
+ name: MTEB MTOPDomainClassification (en)
1570
+ config: en
1571
+ split: test
1572
+ revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf
1573
+ metrics:
1574
+ - type: accuracy
1575
+ value: 95.06839945280439
1576
+ - type: f1
1577
+ value: 94.74276398224072
1578
+ - task:
1579
+ type: Classification
1580
+ dataset:
1581
+ type: mteb/mtop_intent
1582
+ name: MTEB MTOPIntentClassification (en)
1583
+ config: en
1584
+ split: test
1585
+ revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba
1586
+ metrics:
1587
+ - type: accuracy
1588
+ value: 72.25718194254446
1589
+ - type: f1
1590
+ value: 53.91164489161391
1591
+ - task:
1592
+ type: Classification
1593
+ dataset:
1594
+ type: mteb/amazon_massive_intent
1595
+ name: MTEB MassiveIntentClassification (en)
1596
+ config: en
1597
+ split: test
1598
+ revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
1599
+ metrics:
1600
+ - type: accuracy
1601
+ value: 71.47948890383323
1602
+ - type: f1
1603
+ value: 69.98520247230257
1604
+ - task:
1605
+ type: Classification
1606
+ dataset:
1607
+ type: mteb/amazon_massive_scenario
1608
+ name: MTEB MassiveScenarioClassification (en)
1609
+ config: en
1610
+ split: test
1611
+ revision: 7d571f92784cd94a019292a1f45445077d0ef634
1612
+ metrics:
1613
+ - type: accuracy
1614
+ value: 76.46603900470748
1615
+ - type: f1
1616
+ value: 76.44111526065399
1617
+ - task:
1618
+ type: Clustering
1619
+ dataset:
1620
+ type: mteb/medrxiv-clustering-p2p
1621
+ name: MTEB MedrxivClusteringP2P
1622
+ config: default
1623
+ split: test
1624
+ revision: e7a26af6f3ae46b30dde8737f02c07b1505bcc73
1625
+ metrics:
1626
+ - type: v_measure
1627
+ value: 33.19106070798198
1628
+ - task:
1629
+ type: Clustering
1630
+ dataset:
1631
+ type: mteb/medrxiv-clustering-s2s
1632
+ name: MTEB MedrxivClusteringS2S
1633
+ config: default
1634
+ split: test
1635
+ revision: 35191c8c0dca72d8ff3efcd72aa802307d469663
1636
+ metrics:
1637
+ - type: v_measure
1638
+ value: 30.78772205248094
1639
+ - task:
1640
+ type: Reranking
1641
+ dataset:
1642
+ type: mteb/mind_small
1643
+ name: MTEB MindSmallReranking
1644
+ config: default
1645
+ split: test
1646
+ revision: 3bdac13927fdc888b903db93b2ffdbd90b295a69
1647
+ metrics:
1648
+ - type: map
1649
+ value: 31.811231631488507
1650
+ - type: mrr
1651
+ value: 32.98200485378021
1652
+ - task:
1653
+ type: Retrieval
1654
+ dataset:
1655
+ type: nfcorpus
1656
+ name: MTEB NFCorpus
1657
+ config: default
1658
+ split: test
1659
+ revision: None
1660
+ metrics:
1661
+ - type: map_at_1
1662
+ value: 6.9
1663
+ - type: map_at_10
1664
+ value: 13.703000000000001
1665
+ - type: map_at_100
1666
+ value: 17.251
1667
+ - type: map_at_1000
1668
+ value: 18.795
1669
+ - type: map_at_3
1670
+ value: 10.366999999999999
1671
+ - type: map_at_5
1672
+ value: 11.675
1673
+ - type: mrr_at_1
1674
+ value: 47.059
1675
+ - type: mrr_at_10
1676
+ value: 55.816
1677
+ - type: mrr_at_100
1678
+ value: 56.434
1679
+ - type: mrr_at_1000
1680
+ value: 56.467
1681
+ - type: mrr_at_3
1682
+ value: 53.973000000000006
1683
+ - type: mrr_at_5
1684
+ value: 55.257999999999996
1685
+ - type: ndcg_at_1
1686
+ value: 44.737
1687
+ - type: ndcg_at_10
1688
+ value: 35.997
1689
+ - type: ndcg_at_100
1690
+ value: 33.487
1691
+ - type: ndcg_at_1000
1692
+ value: 41.897
1693
+ - type: ndcg_at_3
1694
+ value: 41.18
1695
+ - type: ndcg_at_5
1696
+ value: 38.721
1697
+ - type: precision_at_1
1698
+ value: 46.129999999999995
1699
+ - type: precision_at_10
1700
+ value: 26.533
1701
+ - type: precision_at_100
1702
+ value: 8.706
1703
+ - type: precision_at_1000
1704
+ value: 2.16
1705
+ - type: precision_at_3
1706
+ value: 38.493
1707
+ - type: precision_at_5
1708
+ value: 33.189
1709
+ - type: recall_at_1
1710
+ value: 6.9
1711
+ - type: recall_at_10
1712
+ value: 17.488999999999997
1713
+ - type: recall_at_100
1714
+ value: 34.583000000000006
1715
+ - type: recall_at_1000
1716
+ value: 64.942
1717
+ - type: recall_at_3
1718
+ value: 11.494
1719
+ - type: recall_at_5
1720
+ value: 13.496
1721
+ - task:
1722
+ type: Retrieval
1723
+ dataset:
1724
+ type: nq
1725
+ name: MTEB NQ
1726
+ config: default
1727
+ split: test
1728
+ revision: None
1729
+ metrics:
1730
+ - type: map_at_1
1731
+ value: 33.028999999999996
1732
+ - type: map_at_10
1733
+ value: 49.307
1734
+ - type: map_at_100
1735
+ value: 50.205
1736
+ - type: map_at_1000
1737
+ value: 50.23
1738
+ - type: map_at_3
1739
+ value: 44.782
1740
+ - type: map_at_5
1741
+ value: 47.599999999999994
1742
+ - type: mrr_at_1
1743
+ value: 37.108999999999995
1744
+ - type: mrr_at_10
1745
+ value: 51.742999999999995
1746
+ - type: mrr_at_100
1747
+ value: 52.405
1748
+ - type: mrr_at_1000
1749
+ value: 52.422000000000004
1750
+ - type: mrr_at_3
1751
+ value: 48.087999999999994
1752
+ - type: mrr_at_5
1753
+ value: 50.414
1754
+ - type: ndcg_at_1
1755
+ value: 37.08
1756
+ - type: ndcg_at_10
1757
+ value: 57.236
1758
+ - type: ndcg_at_100
1759
+ value: 60.931999999999995
1760
+ - type: ndcg_at_1000
1761
+ value: 61.522
1762
+ - type: ndcg_at_3
1763
+ value: 48.93
1764
+ - type: ndcg_at_5
1765
+ value: 53.561
1766
+ - type: precision_at_1
1767
+ value: 37.08
1768
+ - type: precision_at_10
1769
+ value: 9.386
1770
+ - type: precision_at_100
1771
+ value: 1.1480000000000001
1772
+ - type: precision_at_1000
1773
+ value: 0.12
1774
+ - type: precision_at_3
1775
+ value: 22.258
1776
+ - type: precision_at_5
1777
+ value: 16.025
1778
+ - type: recall_at_1
1779
+ value: 33.028999999999996
1780
+ - type: recall_at_10
1781
+ value: 78.805
1782
+ - type: recall_at_100
1783
+ value: 94.643
1784
+ - type: recall_at_1000
1785
+ value: 99.039
1786
+ - type: recall_at_3
1787
+ value: 57.602
1788
+ - type: recall_at_5
1789
+ value: 68.253
1790
+ - task:
1791
+ type: Retrieval
1792
+ dataset:
1793
+ type: quora
1794
+ name: MTEB QuoraRetrieval
1795
+ config: default
1796
+ split: test
1797
+ revision: None
1798
+ metrics:
1799
+ - type: map_at_1
1800
+ value: 71.122
1801
+ - type: map_at_10
1802
+ value: 85.237
1803
+ - type: map_at_100
1804
+ value: 85.872
1805
+ - type: map_at_1000
1806
+ value: 85.885
1807
+ - type: map_at_3
1808
+ value: 82.27499999999999
1809
+ - type: map_at_5
1810
+ value: 84.13199999999999
1811
+ - type: mrr_at_1
1812
+ value: 81.73
1813
+ - type: mrr_at_10
1814
+ value: 87.834
1815
+ - type: mrr_at_100
1816
+ value: 87.92
1817
+ - type: mrr_at_1000
1818
+ value: 87.921
1819
+ - type: mrr_at_3
1820
+ value: 86.878
1821
+ - type: mrr_at_5
1822
+ value: 87.512
1823
+ - type: ndcg_at_1
1824
+ value: 81.73
1825
+ - type: ndcg_at_10
1826
+ value: 88.85499999999999
1827
+ - type: ndcg_at_100
1828
+ value: 89.992
1829
+ - type: ndcg_at_1000
1830
+ value: 90.07
1831
+ - type: ndcg_at_3
1832
+ value: 85.997
1833
+ - type: ndcg_at_5
1834
+ value: 87.55199999999999
1835
+ - type: precision_at_1
1836
+ value: 81.73
1837
+ - type: precision_at_10
1838
+ value: 13.491
1839
+ - type: precision_at_100
1840
+ value: 1.536
1841
+ - type: precision_at_1000
1842
+ value: 0.157
1843
+ - type: precision_at_3
1844
+ value: 37.623
1845
+ - type: precision_at_5
1846
+ value: 24.742
1847
+ - type: recall_at_1
1848
+ value: 71.122
1849
+ - type: recall_at_10
1850
+ value: 95.935
1851
+ - type: recall_at_100
1852
+ value: 99.657
1853
+ - type: recall_at_1000
1854
+ value: 99.996
1855
+ - type: recall_at_3
1856
+ value: 87.80799999999999
1857
+ - type: recall_at_5
1858
+ value: 92.161
1859
+ - task:
1860
+ type: Clustering
1861
+ dataset:
1862
+ type: mteb/reddit-clustering
1863
+ name: MTEB RedditClustering
1864
+ config: default
1865
+ split: test
1866
+ revision: 24640382cdbf8abc73003fb0fa6d111a705499eb
1867
+ metrics:
1868
+ - type: v_measure
1869
+ value: 63.490029238193756
1870
+ - task:
1871
+ type: Clustering
1872
+ dataset:
1873
+ type: mteb/reddit-clustering-p2p
1874
+ name: MTEB RedditClusteringP2P
1875
+ config: default
1876
+ split: test
1877
+ revision: 282350215ef01743dc01b456c7f5241fa8937f16
1878
+ metrics:
1879
+ - type: v_measure
1880
+ value: 65.13153408508836
1881
+ - task:
1882
+ type: Retrieval
1883
+ dataset:
1884
+ type: scidocs
1885
+ name: MTEB SCIDOCS
1886
+ config: default
1887
+ split: test
1888
+ revision: None
1889
+ metrics:
1890
+ - type: map_at_1
1891
+ value: 4.202999999999999
1892
+ - type: map_at_10
1893
+ value: 10.174
1894
+ - type: map_at_100
1895
+ value: 12.138
1896
+ - type: map_at_1000
1897
+ value: 12.418
1898
+ - type: map_at_3
1899
+ value: 7.379
1900
+ - type: map_at_5
1901
+ value: 8.727
1902
+ - type: mrr_at_1
1903
+ value: 20.7
1904
+ - type: mrr_at_10
1905
+ value: 30.389
1906
+ - type: mrr_at_100
1907
+ value: 31.566
1908
+ - type: mrr_at_1000
1909
+ value: 31.637999999999998
1910
+ - type: mrr_at_3
1911
+ value: 27.133000000000003
1912
+ - type: mrr_at_5
1913
+ value: 29.078
1914
+ - type: ndcg_at_1
1915
+ value: 20.7
1916
+ - type: ndcg_at_10
1917
+ value: 17.355999999999998
1918
+ - type: ndcg_at_100
1919
+ value: 25.151
1920
+ - type: ndcg_at_1000
1921
+ value: 30.37
1922
+ - type: ndcg_at_3
1923
+ value: 16.528000000000002
1924
+ - type: ndcg_at_5
1925
+ value: 14.396999999999998
1926
+ - type: precision_at_1
1927
+ value: 20.7
1928
+ - type: precision_at_10
1929
+ value: 8.98
1930
+ - type: precision_at_100
1931
+ value: 2.015
1932
+ - type: precision_at_1000
1933
+ value: 0.327
1934
+ - type: precision_at_3
1935
+ value: 15.367
1936
+ - type: precision_at_5
1937
+ value: 12.559999999999999
1938
+ - type: recall_at_1
1939
+ value: 4.202999999999999
1940
+ - type: recall_at_10
1941
+ value: 18.197
1942
+ - type: recall_at_100
1943
+ value: 40.903
1944
+ - type: recall_at_1000
1945
+ value: 66.427
1946
+ - type: recall_at_3
1947
+ value: 9.362
1948
+ - type: recall_at_5
1949
+ value: 12.747
1950
+ - task:
1951
+ type: STS
1952
+ dataset:
1953
+ type: mteb/sickr-sts
1954
+ name: MTEB SICK-R
1955
+ config: default
1956
+ split: test
1957
+ revision: a6ea5a8cab320b040a23452cc28066d9beae2cee
1958
+ metrics:
1959
+ - type: cos_sim_spearman
1960
+ value: 81.69890989765257
1961
+ - task:
1962
+ type: STS
1963
+ dataset:
1964
+ type: mteb/sts12-sts
1965
+ name: MTEB STS12
1966
+ config: default
1967
+ split: test
1968
+ revision: a0d554a64d88156834ff5ae9920b964011b16384
1969
+ metrics:
1970
+ - type: cos_sim_spearman
1971
+ value: 75.31953790551489
1972
+ - task:
1973
+ type: STS
1974
+ dataset:
1975
+ type: mteb/sts13-sts
1976
+ name: MTEB STS13
1977
+ config: default
1978
+ split: test
1979
+ revision: 7e90230a92c190f1bf69ae9002b8cea547a64cca
1980
+ metrics:
1981
+ - type: cos_sim_spearman
1982
+ value: 87.44050861280759
1983
+ - task:
1984
+ type: STS
1985
+ dataset:
1986
+ type: mteb/sts14-sts
1987
+ name: MTEB STS14
1988
+ config: default
1989
+ split: test
1990
+ revision: 6031580fec1f6af667f0bd2da0a551cf4f0b2375
1991
+ metrics:
1992
+ - type: cos_sim_spearman
1993
+ value: 81.86922869270393
1994
+ - task:
1995
+ type: STS
1996
+ dataset:
1997
+ type: mteb/sts15-sts
1998
+ name: MTEB STS15
1999
+ config: default
2000
+ split: test
2001
+ revision: ae752c7c21bf194d8b67fd573edf7ae58183cbe3
2002
+ metrics:
2003
+ - type: cos_sim_spearman
2004
+ value: 88.9399170304284
2005
+ - task:
2006
+ type: STS
2007
+ dataset:
2008
+ type: mteb/sts16-sts
2009
+ name: MTEB STS16
2010
+ config: default
2011
+ split: test
2012
+ revision: 4d8694f8f0e0100860b497b999b3dbed754a0513
2013
+ metrics:
2014
+ - type: cos_sim_spearman
2015
+ value: 85.38015314088582
2016
+ - task:
2017
+ type: STS
2018
+ dataset:
2019
+ type: mteb/sts17-crosslingual-sts
2020
+ name: MTEB STS17 (en-en)
2021
+ config: en-en
2022
+ split: test
2023
+ revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d
2024
+ metrics:
2025
+ - type: cos_sim_spearman
2026
+ value: 90.53653527788835
2027
+ - task:
2028
+ type: STS
2029
+ dataset:
2030
+ type: mteb/sts22-crosslingual-sts
2031
+ name: MTEB STS22 (en)
2032
+ config: en
2033
+ split: test
2034
+ revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80
2035
+ metrics:
2036
+ - type: cos_sim_spearman
2037
+ value: 68.64526474250209
2038
+ - task:
2039
+ type: STS
2040
+ dataset:
2041
+ type: mteb/stsbenchmark-sts
2042
+ name: MTEB STSBenchmark
2043
+ config: default
2044
+ split: test
2045
+ revision: b0fddb56ed78048fa8b90373c8a3cfc37b684831
2046
+ metrics:
2047
+ - type: cos_sim_spearman
2048
+ value: 86.56156983963042
2049
+ - task:
2050
+ type: Reranking
2051
+ dataset:
2052
+ type: mteb/scidocs-reranking
2053
+ name: MTEB SciDocsRR
2054
+ config: default
2055
+ split: test
2056
+ revision: d3c5e1fc0b855ab6097bf1cda04dd73947d7caab
2057
+ metrics:
2058
+ - type: map
2059
+ value: 79.48610254648003
2060
+ - type: mrr
2061
+ value: 94.02481505422682
2062
+ - task:
2063
+ type: Retrieval
2064
+ dataset:
2065
+ type: scifact
2066
+ name: MTEB SciFact
2067
+ config: default
2068
+ split: test
2069
+ revision: None
2070
+ metrics:
2071
+ - type: map_at_1
2072
+ value: 48.983
2073
+ - type: map_at_10
2074
+ value: 59.077999999999996
2075
+ - type: map_at_100
2076
+ value: 59.536
2077
+ - type: map_at_1000
2078
+ value: 59.575
2079
+ - type: map_at_3
2080
+ value: 55.691
2081
+ - type: map_at_5
2082
+ value: 57.410000000000004
2083
+ - type: mrr_at_1
2084
+ value: 51.666999999999994
2085
+ - type: mrr_at_10
2086
+ value: 60.427
2087
+ - type: mrr_at_100
2088
+ value: 60.763
2089
+ - type: mrr_at_1000
2090
+ value: 60.79900000000001
2091
+ - type: mrr_at_3
2092
+ value: 57.556
2093
+ - type: mrr_at_5
2094
+ value: 59.089000000000006
2095
+ - type: ndcg_at_1
2096
+ value: 51.666999999999994
2097
+ - type: ndcg_at_10
2098
+ value: 64.559
2099
+ - type: ndcg_at_100
2100
+ value: 66.58
2101
+ - type: ndcg_at_1000
2102
+ value: 67.64
2103
+ - type: ndcg_at_3
2104
+ value: 58.287
2105
+ - type: ndcg_at_5
2106
+ value: 61.001000000000005
2107
+ - type: precision_at_1
2108
+ value: 51.666999999999994
2109
+ - type: precision_at_10
2110
+ value: 9.067
2111
+ - type: precision_at_100
2112
+ value: 1.0170000000000001
2113
+ - type: precision_at_1000
2114
+ value: 0.11100000000000002
2115
+ - type: precision_at_3
2116
+ value: 23.0
2117
+ - type: precision_at_5
2118
+ value: 15.6
2119
+ - type: recall_at_1
2120
+ value: 48.983
2121
+ - type: recall_at_10
2122
+ value: 80.289
2123
+ - type: recall_at_100
2124
+ value: 89.43299999999999
2125
+ - type: recall_at_1000
2126
+ value: 97.667
2127
+ - type: recall_at_3
2128
+ value: 62.978
2129
+ - type: recall_at_5
2130
+ value: 69.872
2131
+ - task:
2132
+ type: PairClassification
2133
+ dataset:
2134
+ type: mteb/sprintduplicatequestions-pairclassification
2135
+ name: MTEB SprintDuplicateQuestions
2136
+ config: default
2137
+ split: test
2138
+ revision: d66bd1f72af766a5cc4b0ca5e00c162f89e8cc46
2139
+ metrics:
2140
+ - type: cos_sim_accuracy
2141
+ value: 99.79009900990098
2142
+ - type: cos_sim_ap
2143
+ value: 94.94115052608419
2144
+ - type: cos_sim_f1
2145
+ value: 89.1260162601626
2146
+ - type: cos_sim_precision
2147
+ value: 90.599173553719
2148
+ - type: cos_sim_recall
2149
+ value: 87.7
2150
+ - type: dot_accuracy
2151
+ value: 99.79009900990098
2152
+ - type: dot_ap
2153
+ value: 94.94115052608419
2154
+ - type: dot_f1
2155
+ value: 89.1260162601626
2156
+ - type: dot_precision
2157
+ value: 90.599173553719
2158
+ - type: dot_recall
2159
+ value: 87.7
2160
+ - type: euclidean_accuracy
2161
+ value: 99.79009900990098
2162
+ - type: euclidean_ap
2163
+ value: 94.94115052608419
2164
+ - type: euclidean_f1
2165
+ value: 89.1260162601626
2166
+ - type: euclidean_precision
2167
+ value: 90.599173553719
2168
+ - type: euclidean_recall
2169
+ value: 87.7
2170
+ - type: manhattan_accuracy
2171
+ value: 99.7940594059406
2172
+ - type: manhattan_ap
2173
+ value: 94.95271414642431
2174
+ - type: manhattan_f1
2175
+ value: 89.24508790072387
2176
+ - type: manhattan_precision
2177
+ value: 92.3982869379015
2178
+ - type: manhattan_recall
2179
+ value: 86.3
2180
+ - type: max_accuracy
2181
+ value: 99.7940594059406
2182
+ - type: max_ap
2183
+ value: 94.95271414642431
2184
+ - type: max_f1
2185
+ value: 89.24508790072387
2186
+ - task:
2187
+ type: Clustering
2188
+ dataset:
2189
+ type: mteb/stackexchange-clustering
2190
+ name: MTEB StackExchangeClustering
2191
+ config: default
2192
+ split: test
2193
+ revision: 6cbc1f7b2bc0622f2e39d2c77fa502909748c259
2194
+ metrics:
2195
+ - type: v_measure
2196
+ value: 68.43866571935851
2197
+ - task:
2198
+ type: Clustering
2199
+ dataset:
2200
+ type: mteb/stackexchange-clustering-p2p
2201
+ name: MTEB StackExchangeClusteringP2P
2202
+ config: default
2203
+ split: test
2204
+ revision: 815ca46b2622cec33ccafc3735d572c266efdb44
2205
+ metrics:
2206
+ - type: v_measure
2207
+ value: 35.16579026551532
2208
+ - task:
2209
+ type: Reranking
2210
+ dataset:
2211
+ type: mteb/stackoverflowdupquestions-reranking
2212
+ name: MTEB StackOverflowDupQuestions
2213
+ config: default
2214
+ split: test
2215
+ revision: e185fbe320c72810689fc5848eb6114e1ef5ec69
2216
+ metrics:
2217
+ - type: map
2218
+ value: 52.518952473513934
2219
+ - type: mrr
2220
+ value: 53.292457134368895
2221
+ - task:
2222
+ type: Summarization
2223
+ dataset:
2224
+ type: mteb/summeval
2225
+ name: MTEB SummEval
2226
+ config: default
2227
+ split: test
2228
+ revision: cda12ad7615edc362dbf25a00fdd61d3b1eaf93c
2229
+ metrics:
2230
+ - type: cos_sim_pearson
2231
+ value: 31.12529588316604
2232
+ - type: cos_sim_spearman
2233
+ value: 32.31662126895294
2234
+ - type: dot_pearson
2235
+ value: 31.125303796647056
2236
+ - type: dot_spearman
2237
+ value: 32.31662126895294
2238
+ - task:
2239
+ type: Retrieval
2240
+ dataset:
2241
+ type: trec-covid
2242
+ name: MTEB TRECCOVID
2243
+ config: default
2244
+ split: test
2245
+ revision: None
2246
+ metrics:
2247
+ - type: map_at_1
2248
+ value: 0.219
2249
+ - type: map_at_10
2250
+ value: 1.7469999999999999
2251
+ - type: map_at_100
2252
+ value: 10.177999999999999
2253
+ - type: map_at_1000
2254
+ value: 26.108999999999998
2255
+ - type: map_at_3
2256
+ value: 0.64
2257
+ - type: map_at_5
2258
+ value: 0.968
2259
+ - type: mrr_at_1
2260
+ value: 82.0
2261
+ - type: mrr_at_10
2262
+ value: 89.067
2263
+ - type: mrr_at_100
2264
+ value: 89.067
2265
+ - type: mrr_at_1000
2266
+ value: 89.067
2267
+ - type: mrr_at_3
2268
+ value: 88.333
2269
+ - type: mrr_at_5
2270
+ value: 88.73299999999999
2271
+ - type: ndcg_at_1
2272
+ value: 78.0
2273
+ - type: ndcg_at_10
2274
+ value: 71.398
2275
+ - type: ndcg_at_100
2276
+ value: 55.574999999999996
2277
+ - type: ndcg_at_1000
2278
+ value: 51.771
2279
+ - type: ndcg_at_3
2280
+ value: 77.765
2281
+ - type: ndcg_at_5
2282
+ value: 73.614
2283
+ - type: precision_at_1
2284
+ value: 82.0
2285
+ - type: precision_at_10
2286
+ value: 75.4
2287
+ - type: precision_at_100
2288
+ value: 58.040000000000006
2289
+ - type: precision_at_1000
2290
+ value: 23.516000000000002
2291
+ - type: precision_at_3
2292
+ value: 84.0
2293
+ - type: precision_at_5
2294
+ value: 78.4
2295
+ - type: recall_at_1
2296
+ value: 0.219
2297
+ - type: recall_at_10
2298
+ value: 1.958
2299
+ - type: recall_at_100
2300
+ value: 13.797999999999998
2301
+ - type: recall_at_1000
2302
+ value: 49.881
2303
+ - type: recall_at_3
2304
+ value: 0.672
2305
+ - type: recall_at_5
2306
+ value: 1.0370000000000001
2307
+ - task:
2308
+ type: Retrieval
2309
+ dataset:
2310
+ type: webis-touche2020
2311
+ name: MTEB Touche2020
2312
+ config: default
2313
+ split: test
2314
+ revision: None
2315
+ metrics:
2316
+ - type: map_at_1
2317
+ value: 1.8610000000000002
2318
+ - type: map_at_10
2319
+ value: 8.705
2320
+ - type: map_at_100
2321
+ value: 15.164
2322
+ - type: map_at_1000
2323
+ value: 16.78
2324
+ - type: map_at_3
2325
+ value: 4.346
2326
+ - type: map_at_5
2327
+ value: 6.151
2328
+ - type: mrr_at_1
2329
+ value: 22.448999999999998
2330
+ - type: mrr_at_10
2331
+ value: 41.556
2332
+ - type: mrr_at_100
2333
+ value: 42.484
2334
+ - type: mrr_at_1000
2335
+ value: 42.494
2336
+ - type: mrr_at_3
2337
+ value: 37.755
2338
+ - type: mrr_at_5
2339
+ value: 40.102
2340
+ - type: ndcg_at_1
2341
+ value: 21.429000000000002
2342
+ - type: ndcg_at_10
2343
+ value: 23.439
2344
+ - type: ndcg_at_100
2345
+ value: 36.948
2346
+ - type: ndcg_at_1000
2347
+ value: 48.408
2348
+ - type: ndcg_at_3
2349
+ value: 22.261
2350
+ - type: ndcg_at_5
2351
+ value: 23.085
2352
+ - type: precision_at_1
2353
+ value: 22.448999999999998
2354
+ - type: precision_at_10
2355
+ value: 21.633
2356
+ - type: precision_at_100
2357
+ value: 8.02
2358
+ - type: precision_at_1000
2359
+ value: 1.5939999999999999
2360
+ - type: precision_at_3
2361
+ value: 23.810000000000002
2362
+ - type: precision_at_5
2363
+ value: 24.490000000000002
2364
+ - type: recall_at_1
2365
+ value: 1.8610000000000002
2366
+ - type: recall_at_10
2367
+ value: 15.876000000000001
2368
+ - type: recall_at_100
2369
+ value: 50.300999999999995
2370
+ - type: recall_at_1000
2371
+ value: 86.098
2372
+ - type: recall_at_3
2373
+ value: 5.892
2374
+ - type: recall_at_5
2375
+ value: 9.443
2376
+ - task:
2377
+ type: Classification
2378
+ dataset:
2379
+ type: mteb/toxic_conversations_50k
2380
+ name: MTEB ToxicConversationsClassification
2381
+ config: default
2382
+ split: test
2383
+ revision: d7c0de2777da35d6aae2200a62c6e0e5af397c4c
2384
+ metrics:
2385
+ - type: accuracy
2386
+ value: 70.3264
2387
+ - type: ap
2388
+ value: 13.249577616243794
2389
+ - type: f1
2390
+ value: 53.621518367695685
2391
+ - task:
2392
+ type: Classification
2393
+ dataset:
2394
+ type: mteb/tweet_sentiment_extraction
2395
+ name: MTEB TweetSentimentExtractionClassification
2396
+ config: default
2397
+ split: test
2398
+ revision: d604517c81ca91fe16a244d1248fc021f9ecee7a
2399
+ metrics:
2400
+ - type: accuracy
2401
+ value: 61.57611771363894
2402
+ - type: f1
2403
+ value: 61.79797478568639
2404
+ - task:
2405
+ type: Clustering
2406
+ dataset:
2407
+ type: mteb/twentynewsgroups-clustering
2408
+ name: MTEB TwentyNewsgroupsClustering
2409
+ config: default
2410
+ split: test
2411
+ revision: 6125ec4e24fa026cec8a478383ee943acfbd5449
2412
+ metrics:
2413
+ - type: v_measure
2414
+ value: 53.38315344479284
2415
+ - task:
2416
+ type: PairClassification
2417
+ dataset:
2418
+ type: mteb/twittersemeval2015-pairclassification
2419
+ name: MTEB TwitterSemEval2015
2420
+ config: default
2421
+ split: test
2422
+ revision: 70970daeab8776df92f5ea462b6173c0b46fd2d1
2423
+ metrics:
2424
+ - type: cos_sim_accuracy
2425
+ value: 87.55438993860642
2426
+ - type: cos_sim_ap
2427
+ value: 77.98702600017738
2428
+ - type: cos_sim_f1
2429
+ value: 71.94971653931476
2430
+ - type: cos_sim_precision
2431
+ value: 67.50693802035153
2432
+ - type: cos_sim_recall
2433
+ value: 77.01846965699208
2434
+ - type: dot_accuracy
2435
+ value: 87.55438993860642
2436
+ - type: dot_ap
2437
+ value: 77.98702925907986
2438
+ - type: dot_f1
2439
+ value: 71.94971653931476
2440
+ - type: dot_precision
2441
+ value: 67.50693802035153
2442
+ - type: dot_recall
2443
+ value: 77.01846965699208
2444
+ - type: euclidean_accuracy
2445
+ value: 87.55438993860642
2446
+ - type: euclidean_ap
2447
+ value: 77.98702951957925
2448
+ - type: euclidean_f1
2449
+ value: 71.94971653931476
2450
+ - type: euclidean_precision
2451
+ value: 67.50693802035153
2452
+ - type: euclidean_recall
2453
+ value: 77.01846965699208
2454
+ - type: manhattan_accuracy
2455
+ value: 87.54246885617214
2456
+ - type: manhattan_ap
2457
+ value: 77.95531413902947
2458
+ - type: manhattan_f1
2459
+ value: 71.93605683836589
2460
+ - type: manhattan_precision
2461
+ value: 69.28152492668622
2462
+ - type: manhattan_recall
2463
+ value: 74.80211081794195
2464
+ - type: max_accuracy
2465
+ value: 87.55438993860642
2466
+ - type: max_ap
2467
+ value: 77.98702951957925
2468
+ - type: max_f1
2469
+ value: 71.94971653931476
2470
+ - task:
2471
+ type: PairClassification
2472
+ dataset:
2473
+ type: mteb/twitterurlcorpus-pairclassification
2474
+ name: MTEB TwitterURLCorpus
2475
+ config: default
2476
+ split: test
2477
+ revision: 8b6510b0b1fa4e4c4f879467980e9be563ec1cdf
2478
+ metrics:
2479
+ - type: cos_sim_accuracy
2480
+ value: 89.47296930182016
2481
+ - type: cos_sim_ap
2482
+ value: 86.92853616302108
2483
+ - type: cos_sim_f1
2484
+ value: 79.35138351681047
2485
+ - type: cos_sim_precision
2486
+ value: 76.74820143884892
2487
+ - type: cos_sim_recall
2488
+ value: 82.13735756082538
2489
+ - type: dot_accuracy
2490
+ value: 89.47296930182016
2491
+ - type: dot_ap
2492
+ value: 86.92854339601595
2493
+ - type: dot_f1
2494
+ value: 79.35138351681047
2495
+ - type: dot_precision
2496
+ value: 76.74820143884892
2497
+ - type: dot_recall
2498
+ value: 82.13735756082538
2499
+ - type: euclidean_accuracy
2500
+ value: 89.47296930182016
2501
+ - type: euclidean_ap
2502
+ value: 86.92854191061649
2503
+ - type: euclidean_f1
2504
+ value: 79.35138351681047
2505
+ - type: euclidean_precision
2506
+ value: 76.74820143884892
2507
+ - type: euclidean_recall
2508
+ value: 82.13735756082538
2509
+ - type: manhattan_accuracy
2510
+ value: 89.47685023479644
2511
+ - type: manhattan_ap
2512
+ value: 86.90063722679578
2513
+ - type: manhattan_f1
2514
+ value: 79.30753865502702
2515
+ - type: manhattan_precision
2516
+ value: 76.32066068631639
2517
+ - type: manhattan_recall
2518
+ value: 82.53772713273791
2519
+ - type: max_accuracy
2520
+ value: 89.47685023479644
2521
+ - type: max_ap
2522
+ value: 86.92854339601595
2523
+ - type: max_f1
2524
+ value: 79.35138351681047
2525
  ---
2526
 
2527
+ # hkunlp/instructor-xl
2528
+ We introduce **Instructor**👨‍🏫, an instruction-finetuned text embedding model that can generate text embeddings tailored to any task (e.g., classification, retrieval, clustering, text evaluation, etc.) and domains (e.g., science, finance, etc.) ***by simply providing the task instruction, without any finetuning***. Instructor👨‍ achieves sota on 70 diverse embedding tasks!
2529
+ The model is easy to use with **our customized** `sentence-transformer` library. For more details, check out [our paper](https://arxiv.org/abs/2212.09741) and [project page](https://instructor-embedding.github.io/)!
2530
 
2531
+ **************************** **Updates** ****************************
2532
+
2533
+ * 01/21: We released a new [checkpoint](https://huggingface.co/hkunlp/instructor-xl) trained with hard negatives, which gives better performance.
2534
+ * 12/21: We released our [paper](https://arxiv.org/abs/2212.09741), [code](https://github.com/HKUNLP/instructor-embedding), [checkpoint](https://huggingface.co/hkunlp/instructor-xl) and [project page](https://instructor-embedding.github.io/)! Check them out!
2535
+
2536
+ ## Quick start
2537
+ <hr />
2538
 
2539
  ## Installation
2540
  ```bash
2541
+ pip install InstructorEmbedding
 
 
2542
  ```
2543
 
2544
  ## Compute your customized embeddings
2545
  Then you can use the model like this to calculate domain-specific and task-aware embeddings:
2546
  ```python
2547
+ from InstructorEmbedding import INSTRUCTOR
2548
+ model = INSTRUCTOR('hkunlp/instructor-xl')
2549
  sentence = "3D ActionSLAM: wearable person tracking in multi-floor environments"
2550
+ instruction = "Represent the Science title:"
2551
+ embeddings = model.encode([[instruction,sentence]])
 
2552
  print(embeddings)
2553
  ```
2554
 
2555
+ ## Use cases
2556
+ <hr />
2557
+
2558
+ ## Calculate embeddings for your customized texts
2559
+ If you want to calculate customized embeddings for specific sentences, you may follow the unified template to write instructions:
2560
+
2561
+ &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Represent the `domain` `text_type` for `task_objective`:
2562
+ * `domain` is optional, and it specifies the domain of the text, e.g., science, finance, medicine, etc.
2563
+ * `text_type` is required, and it specifies the encoding unit, e.g., sentence, document, paragraph, etc.
2564
+ * `task_objective` is optional, and it specifies the objective of embedding, e.g., retrieve a document, classify the sentence, etc.
2565
+
2566
  ## Calculate Sentence similarities
2567
  You can further use the model to compute similarities between two groups of sentences, with **customized embeddings**.
2568
  ```python
2569
  from sklearn.metrics.pairwise import cosine_similarity
2570
+ sentences_a = [['Represent the Science sentence: ','Parton energy loss in QCD matter'],
2571
+ ['Represent the Financial statement: ','The Federal Reserve on Wednesday raised its benchmark interest rate.']]
2572
+ sentences_b = [['Represent the Science sentence: ','The Chiral Phase Transition in Dissipative Dynamics'],
2573
+ ['Represent the Financial statement: ','The funds rose less than 0.5 per cent on Friday']]
2574
  embeddings_a = model.encode(sentences_a)
2575
  embeddings_b = model.encode(sentences_b)
2576
  similarities = cosine_similarity(embeddings_a,embeddings_b)
2577
  print(similarities)
2578
+ ```
2579
+
2580
+ ## Information Retrieval
2581
+ You can also use **customized embeddings** for information retrieval.
2582
+ ```python
2583
+ import numpy as np
2584
+ from sklearn.metrics.pairwise import cosine_similarity
2585
+ query = [['Represent the Wikipedia question for retrieving supporting documents: ','where is the food stored in a yam plant']]
2586
+ corpus = [['Represent the Wikipedia document for retrieval: ','Capitalism has been dominant in the Western world since the end of feudalism, but most feel[who?] that the term "mixed economies" more precisely describes most contemporary economies, due to their containing both private-owned and state-owned enterprises. In capitalism, prices determine the demand-supply scale. For example, higher demand for certain goods and services lead to higher prices and lower demand for certain goods lead to lower prices.'],
2587
+ ['Represent the Wikipedia document for retrieval: ',"The disparate impact theory is especially controversial under the Fair Housing Act because the Act regulates many activities relating to housing, insurance, and mortgage loans—and some scholars have argued that the theory's use under the Fair Housing Act, combined with extensions of the Community Reinvestment Act, contributed to rise of sub-prime lending and the crash of the U.S. housing market and ensuing global economic recession"],
2588
+ ['Represent the Wikipedia document for retrieval: ','Disparate impact in United States labor law refers to practices in employment, housing, and other areas that adversely affect one group of people of a protected characteristic more than another, even though rules applied by employers or landlords are formally neutral. Although the protected classes vary by statute, most federal civil rights laws protect based on race, color, religion, national origin, and sex as protected traits, and some laws include disability status and other traits as well.']]
2589
+ query_embeddings = model.encode(query)
2590
+ corpus_embeddings = model.encode(corpus)
2591
+ similarities = cosine_similarity(query_embeddings,corpus_embeddings)
2592
+ retrieved_doc_id = np.argmax(similarities)
2593
+ print(retrieved_doc_id)
2594
+ ```
2595
+
2596
+ ## Clustering
2597
+ Use **customized embeddings** for clustering texts in groups.
2598
+ ```python
2599
+ import sklearn.cluster
2600
+ sentences = [['Represent the Medicine sentence for clustering: ','Dynamical Scalar Degree of Freedom in Horava-Lifshitz Gravity'],
2601
+ ['Represent the Medicine sentence for clustering: ','Comparison of Atmospheric Neutrino Flux Calculations at Low Energies'],
2602
+ ['Represent the Medicine sentence for clustering: ','Fermion Bags in the Massive Gross-Neveu Model'],
2603
+ ['Represent the Medicine sentence for clustering: ',"QCD corrections to Associated t-tbar-H production at the Tevatron"],
2604
+ ['Represent the Medicine sentence for clustering: ','A New Analysis of the R Measurements: Resonance Parameters of the Higher, Vector States of Charmonium']]
2605
+ embeddings = model.encode(sentences)
2606
+ clustering_model = sklearn.cluster.MiniBatchKMeans(n_clusters=2)
2607
+ clustering_model.fit(embeddings)
2608
+ cluster_assignment = clustering_model.labels_
2609
+ print(cluster_assignment)
2610
  ```
config.json CHANGED
@@ -1,5 +1,5 @@
1
  {
2
- "_name_or_path": "/scratch/acd13578qu/huggingface_models/sentence-transformers_gtr-t5-xl/",
3
  "architectures": [
4
  "T5EncoderModel"
5
  ],
 
1
  {
2
+ "_name_or_path": "/home2/huggingface/outputs/xl_30000_fever/checkpoint-300/",
3
  "architectures": [
4
  "T5EncoderModel"
5
  ],
pytorch_model.bin CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:904227ee22ccbd02b0a60a5040ee49bde055892fd0f663ac43397f003dbab34b
3
  size 4963705019
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9b065dd74cb4423b155cada6ebf11f97cdcb67b2f10841a8a810d674b10bbf99
3
  size 4963705019
tokenizer_config.json CHANGED
@@ -104,7 +104,7 @@
104
  "eos_token": "</s>",
105
  "extra_ids": 100,
106
  "model_max_length": 512,
107
- "name_or_path": "sentence-transformers/gtr-t5-xl",
108
  "pad_token": "<pad>",
109
  "special_tokens_map_file": null,
110
  "tokenizer_class": "T5Tokenizer",
 
104
  "eos_token": "</s>",
105
  "extra_ids": 100,
106
  "model_max_length": 512,
107
+ "name_or_path": "/home2/huggingface/outputs/xl_30000_fever/checkpoint-300",
108
  "pad_token": "<pad>",
109
  "special_tokens_map_file": null,
110
  "tokenizer_class": "T5Tokenizer",