open-llm-leaderboard/open_llm_leaderboard · Running Evaluation Queue appears to be stuck

16 days ago

It seems a whole bunch of models were benchmarked but are currently stuck in the Running Evaluation Queue.

I'm very much looking forward to Pantheon-RP-1.6-12b-Nemo-KTO's position compared to its SFT counterpart, cause science! (And hoping it did something...)

alozowski

Open LLM Leaderboard org 16 days ago

Hi @Gryphe ,

Thanks for your patience!

Our research cluster is currently running at full capacity, which is why several models are still in the RUNNING queue. Just a quick note: the queue status updates when the Leaderboard restarts, so the best way to track your model's progress is through the Requests dataset. Here’s the request file for Gryphe/Pantheon-RP-1.6-12b-Nemo-KTO – link. According to the file, the evaluation is FINISHED, so usually your model should appear on the Leaderboard after the next restart.

I've checked the Leaderboard, you can find your model now:

alozowski changed discussion status to closed 16 days ago