MammoLLM_bz32_acc4_lr1e4_large_epoch20

This model is a fine-tuned version of gpt2 on the None dataset. It achieves the following results on the evaluation set:

• Loss: 1.0667

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 64
• eval_batch_size: 64
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 256
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 1000
• num_epochs: 20

Training results

Training Loss	Epoch	Step	Validation Loss
3.5007	0.48	500	2.1723
1.9621	0.96	1000	1.7328
1.6708	1.44	1500	1.5700
1.5458	1.92	2000	1.4780
1.453	2.4	2500	1.4207
1.4098	2.87	3000	1.3808
1.3492	3.35	3500	1.3499
1.3205	3.83	4000	1.3220
1.2694	4.31	4500	1.2945
1.2459	4.79	5000	1.2724
1.2006	5.27	5500	1.2482
1.1749	5.75	6000	1.2302
1.139	6.23	6500	1.2146
1.105	6.71	7000	1.1944
1.0754	7.19	7500	1.1791
1.0407	7.66	8000	1.1618
1.0188	8.14	8500	1.1497
0.9787	8.62	9000	1.1374
0.9652	9.1	9500	1.1300
0.9177	9.58	10000	1.1139
0.9165	10.06	10500	1.1088
0.8636	10.54	11000	1.0979
0.8693	11.02	11500	1.0909
0.812	11.5	12000	1.0895
0.8243	11.98	12500	1.0779
0.7734	12.46	13000	1.0796
0.7792	12.93	13500	1.0717
0.74	13.41	14000	1.0763
0.7403	13.89	14500	1.0681
0.7088	14.37	15000	1.0699
0.708	14.85	15500	1.0650
0.6846	15.33	16000	1.0684
0.6811	15.81	16500	1.0652
0.6644	16.29	17000	1.0688
0.6582	16.77	17500	1.0665
0.6512	17.25	18000	1.0669
0.6433	17.72	18500	1.0663
0.6403	18.2	19000	1.0668
0.6347	18.68	19500	1.0666
0.6318	19.16	20000	1.0668
0.6304	19.64	20500	1.0667

Framework versions

• Transformers 4.31.0
• Pytorch 2.0.1+cu117
• Datasets 2.14.3
• Tokenizers 0.13.3