---
license: cc-by-nc-4.0
datasets:
- vicgalle/alpaca-gpt4
language:
- en
---

## Model Overview
Model license: cc-by-nc-4.0<br>
This model is trained based on [EleutherAI/pythia-1.4b-deduped](https://huggingface.co/EleutherAI/pythia-1.4b-deduped) model that is LoRA finetuned on [vicgalle/alpaca-gpt4](https://huggingface.co/datasets/vicgalle/alpaca-gpt4) dataset.<br>

## Prompt Template: `Alpaca`
```
<system_prompt>

### Instruction:
<user_message>

### Response:
<assistant_response>
```

## Intended Use
THIS IS A TEST MODEL, IT IS NOT INTENDED FOR REAL APPLICATIONS BY ANY MEANS. HOWEVER, A NEW MODEL IS COMING IN THE SAME TOPIC.<br>
This model series will be used for small but intense applications.

## Training Details
This model took `2:31:23` to train in QLoRA on a single  `T4`  GPU.<br>
 - *epochs*:  `1`
 - *train batch size*:  `12`
 - *eval batch size*:  `12`
 - *gradient accumulation steps*:  `1`
 - *maximum gradient normal*:  `0.3`
 - *learning rate*:  `2e-4`
 - *weight decay*:  `0.001`
 - *optimizer*:  `paged_adamw_32bit`
 - *learning rate schedule*:  `cosine`
 - *warmup ratio (linear)*:  `0.03`
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_AtAndDev__ShortKingv0.1)

| Metric                | Value                     |
|-----------------------|---------------------------|
| Avg.                  | 31.17   |
| ARC (25-shot)         | 34.22          |
| HellaSwag (10-shot)   | 54.59    |
| MMLU (5-shot)         | 25.78         |
| TruthfulQA (0-shot)   | 41.64   |
| Winogrande (5-shot)   | 56.04   |
| GSM8K (5-shot)        | 0.45        |
| DROP (3-shot)         | 5.47         |