What is the context length in training?

by xuxiu - opened Jul 16

Jul 16

I am reproducing this model using the 0625 dataset and qwen2-7B, I want to know what is the context length you use in training. I saw that you concat the samples in one sequence. I am doing the same, but I don't know what exactly the length you use.

hyxmmm

Beijing Academy of Artificial Intelligence org Jul 17

Hi, we set it to 4096.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment