--- license: llama3 language: - en tags: - finance datasets: - Open-Orca/OpenOrca - GAIR/lima - WizardLM/WizardLM_evol_instruct_V2_196k --- # Instruction Pre-Training: Language Models are Supervised Multitask Learners This repo contains the **finance model developed from Llama3-8B** in our paper [Instruction Pre-Training: Language Models are Supervised Multitask Learners](https://huggingface.co/papers/2406.14491). We explore supervised multitask pre-training by proposing ***Instruction Pre-Training***, a framework that scalably augments massive raw corpora with instruction-response pairs to pre-train language models. The instruction-response pairs are generated by an efficient instruction synthesizer built on open-source models. ***Instruction Pre-Training* outperforms *Vanilla Pre-training* in both general pre-training from scratch and domain-adaptive continual pre-training.** In pre-training from scratch, *Instruction Pre-Training* not only improves pre-trained base models but also benefits more from further instruction tuning. **In continual pre-training, *Instruction Pre-Training* enables Llama3-8B to be comparable to or even outperform Llama3-70B.**

**************************** **Updates** **************************** * 2024/7/31: Updated pre-training suggestions in the `Advanced Usage` section of [instruction-synthesizer](https://huggingface.co/instruction-pretrain/instruction-synthesizer) * 2024/7/15: We scaled up the pre-trained tokens from 100B to 250B, with the number of synthesized instruction-response pairs reaching 500M! Below, we show the performance trend on downstream tasks throughout the pre-training process:

* 2024/6/21: Released the [paper](https://huggingface.co/papers/2406.14491), [code](https://github.com/microsoft/LMOps), and [resources](https://huggingface.co/instruction-pretrain) ## Resources **🤗 We share our data and models with example usages, feel free to open any discussions at [this page](https://huggingface.co/papers/2406.14491)! 🤗** - Thanks to the demo [davanstrien/instruction-synthesizer](https://huggingface.co/spaces/davanstrien/instruction-synthesizer) for implementing our approach - Context-Based Instruction Synthesizer: [instruction-synthesizer](https://huggingface.co/instruction-pretrain/instruction-synthesizer) - Fine-Tuning Data for the Synthesizer: [ft-instruction-synthesizer-collection](https://huggingface.co/datasets/instruction-pretrain/ft-instruction-synthesizer-collection) - General Models Pre-Trained from Scratch (on 100B tokes): - [InstructLM-500M](https://huggingface.co/instruction-pretrain/InstructLM-500M) - [InstructLM-1.3B](https://huggingface.co/instruction-pretrain/InstructLM-1.3B) - Domain-Specific Models Pre-Trained from Llama3-8B: - [Finance-Llama3-8B](https://huggingface.co/instruction-pretrain/finance-Llama3-8B) - [Biomedicine-Llama3-8B](https://huggingface.co/instruction-pretrain/medicine-Llama3-8B) - General Instruction-Augmented Corpora: [general-instruction-augmented-corpora](https://huggingface.co/datasets/instruction-pretrain/general-instruction-augmented-corpora) - Domain-Specific Instruction-Augmented Corpora (no finance data to avoid ethical issues): [medicine-instruction-augmented-corpora](https://huggingface.co/datasets/instruction-pretrain/medicine-instruction-augmented-corpora) ## Domain-Adaptive Continued Pre-Training Following [AdaptLLM](https://huggingface.co/AdaptLLM/finance-chat), we augment the domain-specific raw corpora with instruction-response pairs generated by our [context-based instruction synthesizer](https://huggingface.co/instruction-pretrain/instruction-synthesizer). ### 1. chat with the finance-Llama3-8B model: ```python from transformers import AutoModelForCausalLM, AutoTokenizer model = AutoModelForCausalLM.from_pretrained("instruction-pretrain/finance-Llama3-8B") tokenizer = AutoTokenizer.from_pretrained("instruction-pretrain/finance-Llama3-8B") # Put your input here, NO prompt template is required user_input = '''Use this fact to answer the question: Title of each class Trading Symbol(s) Name of each exchange on which registered Common Stock, Par Value $.01 Per Share MMM New York Stock Exchange MMM Chicago Stock Exchange, Inc. 1.500% Notes due 2026 MMM26 New York Stock Exchange 1.750% Notes due 2030 MMM30 New York Stock Exchange 1.500% Notes due 2031 MMM31 New York Stock Exchange Which debt securities are registered to trade on a national securities exchange under 3M's name as of Q2 of 2023?''' inputs = tokenizer(user_input, return_tensors="pt", add_special_tokens=True).input_ids.to(model.device) outputs = model.generate(input_ids=inputs, max_new_tokens=400)[0] answer_start = int(inputs.shape[-1]) pred = tokenizer.decode(outputs[answer_start:], skip_special_tokens=True) print(pred) ``` ### 2. evaluate any Huggingface LMs on domain-dpecific tasks (💡New!) You can use the following scripts to reproduce our results and evaluate any other Huggingface models on domain-specific tasks. Note that these scripts are not applicable to models that require specific prompt templates (e.g., Llama2-chat, Llama3-Instruct). 1). Set Up Dependencies ```bash git clone https://github.com/microsoft/LMOps cd LMOps/adaptllm pip install -r requirements.txt ``` 2). Evaluate the Model ```bash # Select the domain from ['biomedicine', 'finance', 'law'] DOMAIN='finance' # Specify any Huggingface LM name (Not applicable to models requiring specific prompt templates) MODEL='instruction-pretrain/finance-Llama3-8B' # Model parallelization: # - Set MODEL_PARALLEL=False if the model fits on a single GPU. # We observe that LMs smaller than 10B always meet this requirement. # - Set MODEL_PARALLEL=True if the model is too large and encounters OOM on a single GPU. MODEL_PARALLEL=False # Choose the number of GPUs from [1, 2, 4, 8] N_GPU=1 # Whether to add a BOS token at the beginning of the prompt input: # - Set to False for AdaptLLM. # - Set to True for instruction-pretrain models. # If unsure, we recommend setting it to False, as this is suitable for most LMs. add_bos_token=True # Run the evaluation script bash scripts/inference.sh ${DOMAIN} ${MODEL} ${add_bos_token} ${MODEL_PARALLEL} ${N_GPU} ``` ## Citation If you find our work helpful, please cite us: Instruction Pre-Training ```bibtex @article{cheng2024instruction, title={Instruction Pre-Training: Language Models are Supervised Multitask Learners}, author={Cheng, Daixuan and Gu, Yuxian and Huang, Shaohan and Bi, Junyu and Huang, Minlie and Wei, Furu}, journal={arXiv preprint arXiv:2406.14491}, year={2024} } ``` [Adapt LLM to Domains](https://huggingface.co/papers/2309.09530) ```bibtex @inproceedings{ cheng2024adapting, title={Adapting Large Language Models via Reading Comprehension}, author={Daixuan Cheng and Shaohan Huang and Furu Wei}, booktitle={The Twelfth International Conference on Learning Representations}, year={2024}, url={https://openreview.net/forum?id=y886UXPEZ0} } ```