---
license: llama3
---

# Weights from the Llama-3-8B Self-Align Experiments

[WEIGHTS TO BE UPLOADED ONCE DONE]

## Training Config

The `config.yaml` should be used during `accelerate launch`, and `run.sh` was used to launch the training using the [StarCoder2 Self-Align training script](https://github.com/bigcode-project/starcoder2-self-align?tab=readme-ov-file#training-details).
Some tweaks were performed to get this working on 48GB vRAM:
- FSDP was used
- `per_device_batch_size` is `2`
- A learning rate of 3e-6 was used


## Environment:

- Trained with 2x4090 GPUs
- 128GB RAM