Back to AI News
arxiv

3D GameDev Arena benchmarks coding agents on long-horizon browser game development

Researchers introduce 3D GameDev Arena, a benchmark of 108 from-scratch 3D browser game-development tasks spanning 14 genres, where an automated plan-implement-test-repair pipeline builds each game from a text spec and UI/UX reference images. Runs average roughly 1,000 LLM calls, with the longest using 14,407 calls, 2.13B tokens, and 608 assets. Evaluation combines code-level functional acceptance, agent-based QA and playtesting, and blind pairwise human preference in a public arena. Preliminary findings indicate long-horizon 3D game development remains far from solved and that code-level correctness alone does not determine player preference.

AI Brief

It offers a concrete, public benchmark for evaluating coding agents on extreme long-horizon, multi-stage tasks with human preference signals, useful for tracking agent capability beyond repository-level issue fixing.

arXiv:2611.24073v1 Announce Type: new Abstract: Coding agents now resolve repository issues and generate compact playable games, but whether they can build a large 3D game from scratch remains open. Such games tightly couple mechanics, scenes, assets, and interaction logic, and their quality rests on player enjoyment, which existing benchmarks rarely assess. Thus, we introduce 3D GameDev Arena, a first extreme long-horizon benchmark of 108 from-scratch game-development tasks across 14 genres. Given only a text specification and UI/UX reference images, a fully automated plan--implement--test--repair pipeline builds each task into a self-contained browser game. Runs average $\sim$1,000 LLM calls ; the longest uses 14,407 calls, 2.13B tokens, and 608 assets. Evaluation spans (1) objective code-level functional acceptance against the game specification, (2) agent-based QA and playtesting of 3D presentation and playability, and (3) blind pairwise Bradley-Terry human preference in a public arena. The preliminary findings indicate that long-horizon 3D game development remains far from solved, and code-level correctness alone does not dictate player preference. The public arena is hosted online.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!