instruction-pretrain
commited on
Commit
•
9242acf
1
Parent(s):
fd7cbde
Update README.md
Browse files
README.md
CHANGED
@@ -45,7 +45,7 @@ We explore supervised multitask pre-training by proposing ***Instruction Pre-Tra
|
|
45 |
## Domain-Adaptive Continued Pre-Training
|
46 |
Following [AdaptLLM](https://huggingface.co/AdaptLLM/finance-chat), we augment the domain-specific raw corpora with instruction-response pairs generated by our [context-based instruction synthesizer](https://huggingface.co/instruction-pretrain/instruction-synthesizer).
|
47 |
|
48 |
-
### 1. chat with the finance-Llama3-8B model:
|
49 |
```python
|
50 |
from transformers import AutoModelForCausalLM, AutoTokenizer
|
51 |
|
@@ -71,7 +71,7 @@ pred = tokenizer.decode(outputs[answer_start:], skip_special_tokens=True)
|
|
71 |
print(pred)
|
72 |
```
|
73 |
|
74 |
-
### 2. evaluate any Huggingface LMs on domain-
|
75 |
You can use the following script to reproduce our results and evaluate any other Huggingface models on domain-specific tasks. Note that the script is NOT applicable to models that require specific prompt templates (e.g., Llama2-chat, Llama3-Instruct).
|
76 |
|
77 |
1). Set Up Dependencies
|
|
|
45 |
## Domain-Adaptive Continued Pre-Training
|
46 |
Following [AdaptLLM](https://huggingface.co/AdaptLLM/finance-chat), we augment the domain-specific raw corpora with instruction-response pairs generated by our [context-based instruction synthesizer](https://huggingface.co/instruction-pretrain/instruction-synthesizer).
|
47 |
|
48 |
+
### 1. To chat with the finance-Llama3-8B model:
|
49 |
```python
|
50 |
from transformers import AutoModelForCausalLM, AutoTokenizer
|
51 |
|
|
|
71 |
print(pred)
|
72 |
```
|
73 |
|
74 |
+
### 2. To evaluate any Huggingface LMs on domain-specific tasks (💡New!)
|
75 |
You can use the following script to reproduce our results and evaluate any other Huggingface models on domain-specific tasks. Note that the script is NOT applicable to models that require specific prompt templates (e.g., Llama2-chat, Llama3-Instruct).
|
76 |
|
77 |
1). Set Up Dependencies
|