Leaderboard
|
Agent
|
% Progress
|
BabyAI
|
Crafter
|
TextWorld
|
BabaIsAI
|
MiniHack
|
NetHack
|
Date
|
|---|---|---|---|---|---|---|---|---|
|
🥈 🕹️ ✔️ Claude-3.5-Sonnet |
30.0 ± 2.0 |
52.0 ± 7.1 |
32.7 ± 3.2 |
42.1 ± 5.4 |
37.5 ± 4.4 |
15.0 ± 5.6 |
0.6 ± 0.5 |
2024-11-11 |
|
🕹️ ✔️ Gemini-1.5-Flash |
10.6 ± 1.0 |
25.6 ± 3.9 |
20.0 ± 0.7 |
0.0 ± 0.0 |
12.8 ± 2.3 |
5.0 ± 3.5 |
0.0 ± 0.0 |
2024-11-11 |
|
🕹️ ✔️ Llama-3.1-8B-it |
14.1 ± 1.5 |
30.0 ± 6.5 |
25.5 ± 3.2 |
6.1 ± 2.4 |
18.3 ± 3.5 |
5.0 ± 3.4 |
0.0 ± 0.0 |
2024-11-11 |
|
🕹️ ✔️ Llama-3.2-3B-it |
8.5 ± 1.1 |
10.0 ± 4.2 |
17.3 ± 2.8 |
3.5 ± 1.1 |
17.5 ± 3.5 |
2.5 ± 2.5 |
0.0 ± 0.0 |
2024-11-11 |
|
🥇 🕹️ ✔️ GPT-4o-2024-05-13 |
32.3 ± 1.5 |
77.6 ± 3.7 |
33.1 ± 2.3 |
39.3 ± 5.2 |
33.7 ± 3.3 |
10.0 ± 4.7 |
0.4 ± 0.4 |
2024-11-11 |
|
🕹️ ✔️ Gemini-1.5-Pro |
21.0 ± 1.2 |
58.4 ± 4.4 |
30.2 ± 2.9 |
0.0 ± 0.0 |
32.0 ± 3.3 |
5.0 ± 3.5 |
0.4 ± 0.4 |
2024-11-11 |
|
🥉 🕹️ ✔️ Llama-3.1-70B-it |
27.9 ± 1.4 |
73.2 ± 4.0 |
31.2 ± 2.7 |
15.0 ± 4.6 |
40.0 ± 3.4 |
7.5 ± 4.2 |
0.3 ± 0.3 |
2024-11-17 |
|
🕹️ ✔️ GPT-4o-mini |
17.4 ± 1.4 |
50.4 ± 4.5 |
15.9 ± 2.0 |
12.2 ± 3.5 |
15.6 ± 2.5 |
10.0 ± 4.7 |
0.0 ± 0.0 |
2024-11-11 |
|
🕹️ ✔️ Llama-3.2-90B-it |
24.5 ± 1.2 |
55.2 ± 4.5 |
31.7 ± 1.4 |
11.2 ± 3.0 |
43.9 ± 3.5 |
5.0 ± 3.4 |
0.0 ± 0.0 |
2024-11-11 |
|
🕹️ ✔️ Llama-3.2-11B-it |
14.0 ± 1.1 |
32.8 ± 4.2 |
26.2 ± 3.3 |
6.7 ± 2.2 |
15.6 ± 2.5 |
2.5 ± 2.5 |
0.0 ± 0.0 |
2024-11-11 |
|
🕹️ ✔️ Llama-3.2-1B-it |
6.3 ± 1.0 |
6.0 ± 3.4 |
12.7 ± 1.9 |
3.3 ± 0.9 |
10.8 ± 2.8 |
5.0 ± 3.4 |
0.0 ± 0.0 |
2024-11-11 |
|
Agent
|
% Progress
|
BabyAI
|
Crafter
|
BabaIsAI
|
MiniHack
|
NetHack
|
Date
|
|---|---|---|---|---|---|---|---|
|
🥇 🕹️ ✔️ Claude-3.5-Sonnet |
29.1 ± 2.2 |
50.0 ± 7.1 |
37.3 ± 3.1 |
34.5 ± 4.4 |
22.5 ± 6.6 |
1.2 ± 0.4 |
2024-11-11 |
|
🕹️ ✔️ Gemini-1.5-Flash |
14.9 ± 1.4 |
43.2 ± 4.4 |
20.7 ± 4.4 |
8.3 ± 1.9 |
2.5 ± 2.5 |
0.0 ± 0.0 |
2024-11-08 |
|
🥉 🕹️ ✔️ GPT-4o-2024-05-13 |
22.6 ± 1.4 |
62.0 ± 4.3 |
26.8 ± 3.7 |
18.6 ± 2.7 |
5.0 ± 3.4 |
0.4 ± 0.4 |
2024-11-08 |
|
🥈 🕹️ ✔️ Gemini-1.5-Pro |
25.8 ± 1.4 |
58.4 ± 4.4 |
33.5 ± 2.1 |
31.4 ± 3.2 |
5.0 ± 3.4 |
0.5 ± 0.5 |
2024-11-08 |
|
🕹️ ✔️ Llama-3.2-90B-it |
13.4 ± 1.2 |
28.2 ± 4.0 |
14.5 ± 1.8 |
21.9 ± 2.9 |
2.5 ± 2.5 |
0.0 ± 0.0 |
2024-11-11 |
|
🕹️ ✔️ Llama-3.2-11B-it |
6.9 ± 0.8 |
10.4 ± 2.7 |
15.9 ± 1.2 |
5.8 ± 1.6 |
2.5 ± 2.5 |
0.0 ± 0.0 |
2024-11-11 |
|
🕹️ ✔️ GPT-4o-mini |
15.4 ± 1.3 |
38.0 ± 4.3 |
19.9 ± 3.1 |
16.4 ± 2.6 |
2.5 ± 2.5 |
0.0 ± 0.0 |
2024-11-08 |
- The % Progress metric refers to the average
completion
percentage of BALROG
environments of the model.
- ✔️ Checked indicates that we, the BALROG team,
received
access to the system and
were able to reproduce the patch generations.
- 🕹️ Open refers to submissions that have
open-source
code. This
does not
necessarily mean the underlying model is open-source.
- The leaderboard is updated once a week on Monday.
- If you would like to submit your model to the leaderboard, please check the submission page.
About
BALROG is a benchmark designed to evaluate the agentic capabilities of large language and vision-language models (LLMs and VLMs) on long-horizon tasks, testing their ability to plan, reason spatially, and explore in dynamic environments. Our benchmark reveals that while current models show some success on simpler tasks, they struggle with more complex, procedurally generated environments like NetHack, especially when vision-based decision-making is involved. We provide an open, fine-grained evaluation framework to drive progress in autonomous agent research. Read more about agent quest in our paper!
Citation
@inproceedings{
author2024BALROG,
title={BALROG: Benchmarking Agentic LLM/VLM Reasoning On Games},
author={Davide Paglieri*, Bartłomiej Cupiał*, Samuel Coward, Ulyana Piterbarg,
Maciej Wolczyk, Akbir Khan, Eduardo Pignatelli, Łukasz Kuciński, Lerrel Pinto,
Rob Fergus, Jakob Nicolaus Foerster, Jack Parker-Holder, Tim Rocktäschel},
booktitle={pre-print},
year={2024},
url={https://example.com/BALROG}
}
Usage: BALROG's website and leaderboard use the template made available by SWE-bench. If you would like to use this template for your own leaderboard, please visit their website and request permission
Correspondence to: d.paglieri@cs.ucl.ac.uk
