Assessing Frontier LLMs in Solving Game Development Problems: Preliminary Findings Across Three Game Engines - Vasconcelos, Thiago Guedes Cruz de; Castro Filho, Adams Amaral de; Ribeiro, Guadalupe Prado Saldanha; Rodrigues, Maria Andréia F.; Mendonça, Nabor C. (Anais do Workshop sobre Engenharia de Software para Desenvolvimento de Jogos (SE4Games); 2025: Anais do I Workshop sobre Engenharia de Software para Desenvolvimento de JogosDO - 10.5753/se4games.2025.14869, 2025)
This paper evaluates three frontier LLMs (ChatGPT-4o, o3, and Gemini 2.5 Pro) against expert-rated human answers for 30 technical questions collected from online Q&A forums about three popular game engines: Unreal, Unity, and Godot. Our results reveal significant performance variance, with o3 demonstrating superior capabilities over Gemini 2.5 Pro and ChatGPT-4o. A primary weakness identified across all models was response completeness, where AI-generated answers often lacked the comprehensive detail of the human baseline. These findings suggest that although LLMs are powerful assistants, they are not yet a substitute for human expertise in engine-based game development tasks.