---
library_name: peft
tags:
- generated_from_trainer
base_model: scb10x/typhoon-7b
model-index:
- name: work/out
  results: []
datasets:
- scb_mt_enth_2020
language:
- th
pipeline_tag: text-generation
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

[<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl)
<details><summary>See axolotl config</summary>

axolotl version: `0.4.0`
```yaml
base_model: ./models/scb10x_typhoon-7b
model_type: MistralForCausalLM
tokenizer_type: LlamaTokenizer
is_mistral_derived_model: true

load_in_8bit: false
load_in_4bit: true
strict: false


datasets:
  - path: ./work/scb-mt-en-th-2020/apdf.csv
    type:
      system_prompt: ""
      field_system: system
      field_instruction: en_text
      field_output: th_text
      format: "{instruction}<translate>"
  - path: ./work/scb-mt-en-th-2020/assorted_government.csv
    type:
      system_prompt: ""
      field_system: system
      field_instruction: en_text
      field_output: th_text
      format: "{instruction}<translate>"
  - path: ./work/scb-mt-en-th-2020/generated_reviews_crowd.csv
    type:
      system_prompt: ""
      field_system: system
      field_instruction: en_text
      field_output: th_text
      format: "{instruction}<translate>"
  - path: ./work/scb-mt-en-th-2020/generated_reviews_translator.csv
    type:
      system_prompt: ""
      field_system: system
      field_instruction: en_text
      field_output: th_text
      format: "{instruction}<translate>"
  - path: ./work/scb-mt-en-th-2020/generated_reviews_yn.csv
    type:
      system_prompt: ""
      field_system: system
      field_instruction: en_text
      field_output: th_text
      format: "{instruction}<translate>"
  - path: ./work/scb-mt-en-th-2020/mozilla_common_voice.csv
    type:
      system_prompt: ""
      field_system: system
      field_instruction: en_text
      field_output: th_text
      format: "{instruction}<translate>"
  - path: ./work/scb-mt-en-th-2020/msr_paraphrase.csv
    type:
      system_prompt: ""
      field_system: system
      field_instruction: en_text
      field_output: th_text
      format: "{instruction}<translate>"
  - path: ./work/scb-mt-en-th-2020/nus_sms.csv
    type:
      system_prompt: ""
      field_system: system
      field_instruction: en_text
      field_output: th_text
      format: "{instruction}<translate>"
  - path: ./work/scb-mt-en-th-2020/paracrawl.csv
    type:
      system_prompt: ""
      field_system: system
      field_instruction: en_text
      field_output: th_text
      format: "{instruction}<translate>"
  - path: ./work/scb-mt-en-th-2020/task_master_1.csv
    type:
      system_prompt: ""
      field_system: system
      field_instruction: en_text
      field_output: th_text
      format: "{instruction}<translate>"
  - path: ./work/scb-mt-en-th-2020/thai_websites.csv
    type:
      system_prompt: ""
      field_system: system
      field_instruction: en_text
      field_output: th_text
      format: "{instruction}<translate>"
  - path: ./work/scb-mt-en-th-2020/wikipedia.csv
    type:
      system_prompt: ""
      field_system: system
      field_instruction: en_text
      field_output: th_text
      format: "{instruction}<translate>"

dataset_prepared_path: ./work/last_run_prepared
val_set_size: 0.02
output_dir: ./work/out


adapter: qlora
lora_model_dir:

sequence_len: 4096
sample_packing: true
pad_to_sequence_len: true

gpu_memory_limit: 20

lora_r: 32
lora_alpha: 16
lora_dropout: 0.05
lora_target_linear: true
lora_fan_in_fan_out:


wandb_project: typhoon-7b
wandb_entity:
wandb_watch:
wandb_name:
wandb_log_model:

gradient_accumulation_steps: 8
micro_batch_size: 2
num_epochs: 1
optimizer: paged_adamw_8bit
lr_scheduler: cosine
learning_rate: 0.0004

train_on_inputs: false
group_by_length: false
bf16: true
fp16: false
tf32: false

gradient_checkpointing: true
resume_from_checkpoint: true
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true


warmup_ratio: 0.01
eval_steps: 10
eval_table_size:
eval_table_max_new_tokens: 128
save_steps: 10
save_total_limit: 10
debug:
deepspeed:
weight_decay: 0.0
fsdp:
fsdp_config:
```

</details><br>

# ping98k/typhoon-7b-en-to-th-lora

This model was qlora finetuned on the scb_mt_enth_2020 dataset.
It achieves the following results on the evaluation set:
- Loss: 0.8657

## Model description

### prompt
```
Why can camels survive for long without water?<translate>
```
### output
```
ทำไมอูฐสามารถอยู่รอดได้นานโดยไม่มีน้ำ
```
### known issue
model not train with end translate token correctly. some time model will output `<translate>` or `</translate>`
```
Why can camels survive for long without water?<translate>ทำไมอูฐสามารถอยู่รอดได้นานโดยไม่มีน้ำ<translate>

Why can camels survive for long without water?<translate>ทำไมอูฐสามารถอยู่รอดได้นานโดยไม่มีน้ำ</translate>
```

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0004
- train_batch_size: 2
- eval_batch_size: 2
- seed: 42
- gradient_accumulation_steps: 8
- total_train_batch_size: 16
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 90
- num_epochs: 1

### Training results

| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 2.8002        | 0.01  | 10   | 2.7164          |
| 2.1186        | 0.02  | 20   | 2.0709          |
| 1.717         | 0.03  | 30   | 1.6999          |
| 1.5327        | 0.04  | 40   | 1.5332          |
| 1.3684        | 0.04  | 50   | 1.4293          |
| 1.3992        | 0.05  | 60   | 1.3651          |
| 1.3031        | 0.06  | 70   | 1.3198          |
| 1.3067        | 0.07  | 80   | 1.2831          |
| 1.2685        | 0.08  | 90   | 1.2542          |
| 1.2469        | 0.09  | 100  | 1.2293          |
| 1.2067        | 0.1   | 110  | 1.2096          |
| 1.1458        | 0.11  | 120  | 1.1942          |
| 1.1679        | 0.11  | 130  | 1.1732          |
| 1.1914        | 0.12  | 140  | 1.1609          |
| 1.2329        | 0.13  | 150  | 1.1491          |
| 1.1151        | 0.14  | 160  | 1.1365          |
| 1.1138        | 0.15  | 170  | 1.1252          |
| 1.1607        | 0.16  | 180  | 1.1188          |
| 1.083         | 0.17  | 190  | 1.1095          |
| 1.1068        | 0.18  | 200  | 1.1016          |
| 1.1214        | 0.18  | 210  | 1.0921          |
| 1.061         | 0.19  | 220  | 1.0862          |
| 1.1072        | 0.2   | 230  | 1.0792          |
| 1.0275        | 0.21  | 240  | 1.0739          |
| 1.0735        | 0.22  | 250  | 1.0666          |
| 1.0549        | 0.23  | 260  | 1.0634          |
| 1.0336        | 0.24  | 270  | 1.0561          |
| 1.0784        | 0.25  | 280  | 1.0519          |
| 1.0313        | 0.26  | 290  | 1.0459          |
| 1.0459        | 0.26  | 300  | 1.0415          |
| 1.0824        | 0.27  | 310  | 1.0390          |
| 1.0543        | 0.28  | 320  | 1.0327          |
| 1.0732        | 0.29  | 330  | 1.0287          |
| 1.0071        | 0.3   | 340  | 1.0237          |
| 1.0336        | 0.31  | 350  | 1.0200          |
| 1.0694        | 0.32  | 360  | 1.0155          |
| 0.9799        | 0.33  | 370  | 1.0111          |
| 1.0025        | 0.33  | 380  | 1.0073          |
| 0.9805        | 0.34  | 390  | 1.0044          |
| 0.9398        | 0.35  | 400  | 1.0011          |
| 1.0133        | 0.36  | 410  | 0.9957          |
| 1.0465        | 0.37  | 420  | 0.9916          |
| 0.9711        | 0.38  | 430  | 0.9887          |
| 0.9786        | 0.39  | 440  | 0.9858          |
| 0.9687        | 0.4   | 450  | 0.9835          |
| 0.988         | 0.4   | 460  | 0.9810          |
| 1.021         | 0.41  | 470  | 0.9770          |
| 0.9754        | 0.42  | 480  | 0.9734          |
| 0.9677        | 0.43  | 490  | 0.9705          |
| 1.0114        | 0.44  | 500  | 0.9667          |
| 0.978         | 0.45  | 510  | 0.9643          |
| 0.9762        | 0.46  | 520  | 0.9611          |
| 0.9795        | 0.47  | 530  | 0.9597          |
| 0.9419        | 0.48  | 540  | 0.9558          |
| 0.9403        | 0.48  | 550  | 0.9519          |
| 0.9408        | 0.49  | 560  | 0.9495          |
| 0.9704        | 0.5   | 570  | 0.9460          |
| 0.9426        | 0.51  | 580  | 0.9447          |
| 0.9288        | 0.52  | 590  | 0.9406          |
| 0.9986        | 0.53  | 600  | 0.9394          |
| 0.9129        | 0.54  | 610  | 0.9374          |
| 0.9797        | 0.55  | 620  | 0.9349          |
| 0.9269        | 0.55  | 630  | 0.9317          |
| 0.9258        | 0.56  | 640  | 0.9296          |
| 0.9041        | 0.57  | 650  | 0.9268          |
| 0.9383        | 0.58  | 660  | 0.9240          |
| 0.9289        | 0.59  | 670  | 0.9220          |
| 0.8906        | 0.6   | 680  | 0.9201          |
| 0.9275        | 0.61  | 690  | 0.9171          |
| 0.99          | 0.62  | 700  | 0.9150          |
| 0.9063        | 0.62  | 710  | 0.9124          |
| 0.8757        | 0.63  | 720  | 0.9107          |
| 0.9276        | 0.64  | 730  | 0.9087          |
| 0.9315        | 0.65  | 740  | 0.9064          |
| 0.9442        | 0.66  | 750  | 0.9037          |
| 0.8848        | 0.67  | 760  | 0.9015          |
| 0.8901        | 0.68  | 770  | 0.8993          |
| 0.8714        | 0.69  | 780  | 0.8973          |
| 0.8641        | 0.7   | 790  | 0.8956          |
| 0.8915        | 0.7   | 800  | 0.8938          |
| 0.9069        | 0.71  | 810  | 0.8921          |
| 0.8798        | 0.72  | 820  | 0.8901          |
| 0.9195        | 0.73  | 830  | 0.8884          |
| 0.8936        | 0.74  | 840  | 0.8868          |
| 0.8284        | 0.75  | 850  | 0.8851          |
| 0.9469        | 0.76  | 860  | 0.8833          |
| 0.8854        | 0.77  | 870  | 0.8820          |
| 0.8865        | 0.77  | 880  | 0.8809          |
| 0.8982        | 0.78  | 890  | 0.8799          |
| 0.8683        | 0.79  | 900  | 0.8786          |
| 0.9326        | 0.8   | 910  | 0.8773          |
| 0.8937        | 0.81  | 920  | 0.8758          |
| 0.8995        | 0.82  | 930  | 0.8746          |
| 0.9263        | 0.83  | 940  | 0.8735          |
| 0.907         | 0.84  | 950  | 0.8725          |
| 0.8467        | 0.84  | 960  | 0.8715          |
| 0.9037        | 0.85  | 970  | 0.8708          |
| 0.833         | 0.86  | 980  | 0.8699          |
| 0.878         | 0.87  | 990  | 0.8693          |
| 0.8897        | 0.88  | 1000 | 0.8686          |
| 0.8931        | 0.89  | 1010 | 0.8681          |
| 0.8766        | 0.9   | 1020 | 0.8676          |
| 0.839         | 0.91  | 1030 | 0.8672          |
| 0.8973        | 0.92  | 1040 | 0.8669          |
| 0.8806        | 0.92  | 1050 | 0.8666          |
| 0.8683        | 0.93  | 1060 | 0.8664          |
| 0.8736        | 0.94  | 1070 | 0.8662          |
| 0.8495        | 0.95  | 1080 | 0.8660          |
| 0.8364        | 0.96  | 1090 | 0.8659          |
| 0.8934        | 0.97  | 1100 | 0.8658          |
| 0.8954        | 0.98  | 1110 | 0.8658          |
| 0.8783        | 0.99  | 1120 | 0.8657          |
| 0.8678        | 0.99  | 1130 | 0.8657          |


### Framework versions

- PEFT 0.7.1
- Transformers 4.37.0
- Pytorch 2.0.1+cu118
- Datasets 2.16.1
- Tokenizers 0.15.0