Can LLMs Write Your Tests? Evaluating Automated ISO/IEC 25010-Based Test Generation for VR Metaverse Environments
Resumo
Virtual Reality (VR) metaverse environments impose demanding non-functional requirements (NFRs) on quality attributes. Manual design of NFR test cases is labour-intensive, expert-dependent, and difficult to keep aligned with evolving quality standards. We investigate whether four state-of-the-art Large Language Models (LLMs) can generate automated, ISO/IEC 25010-aligned NFR test cases for a VR metaverse, and whether human assessor ratings agree with an LLM-as-a-Judge evaluation of the same tests. We applied a five-stage Automated Test Case Generation (ATCG) workflow to five ISO/IEC 25010 quality attributes. Each generated test case was evaluated under two independent perspectives: (i) a two-assessor human panel rating; and (ii) an LLM-as-a-Judge protocol implemented as a custom skill applying the same rubric. Both perspectiveswere aggregated into weighted composite scorecards using domain-prioritised attribute weights. The two evaluation perspectives yield different rankings. Under human assessment, DeepSeek-V2 ranked first, followed by Copilot, ChatGPT, and Gemini. Under the LLM-as-a-Judge evaluation, Copilot ranked first, followed by ChatGPT, Gemini, and DeepSeek-V2 last. The rank inversion of DeepSeek-V2 traces to the judge penalising missing preconditions and measurement tool specifications that human assessors filled in with domain knowledge. LLMs are viable ATCG tools for VR metaverse NFRs when prompts are anchored to ISO/IEC 25010 sub-characteristics. Evaluation perspective is a first-class variable: human assessors favour conceptual breadth, while rubric-anchored LLM judges enforce specification completeness.
Referências
Natália André and Everton Alves. 2025. Generating Exploratory Test Suites with ChatGPT: Insights from an Empirical Study. In Anais do X Simpósio Brasileiro de Testes de Software Sistemático e Automatizado (Recife/PE). SBC, Porto Alegre, RS, Brasil, 65–74. DOI: 10.5753/sast.2025.14094
Nathalia Barbosa, Paulo Borba, and Léuson Silva. 2025. Detecção de Conflitos Semânticos com Testes Gerados por LLM. In Anais do X Simpósio Brasileiro de Testes de Software Sistemático e Automatizado (Recife/PE). SBC, Porto Alegre, RS, Brasil, 18–27. DOI: 10.5753/sast.2025.13826
Ana Cardoso, Cleicy Santos, Eliane Collins, Kelen Lima, Pablo Quiroga, and Marlon Griego. 2023. Evaluation of Automatic Test Case Generation for the Android Operating System Using Deep Reinforcement Learning. In Anais do XXII Simpósio Brasileiro de Qualidade de Software (SBQS). SBC, Porto Alegre, RS, Brasil, 228–235. [link]
Arda Celik and Qusay H. Mahmoud. 2025. A Review of Large Language Models for Automated Test Case Generation. Machine Learning and Knowledge Extraction 7, 3 (2025). DOI: 10.3390/make7030097
Sunghyup Chang and Jungwook Suh. 2025. The Impact of VR Exhibition Experiences on Presence, Interaction, Immersion, and Satisfaction: Focusing on the Experience Economy Theory (4Es). Systems 13, 1 (2025), 55. DOI: 10.3390/systems13010055, Article no. 55.
Nayse Fagundes and Leopoldo Teixeira. 2025. Evaluating LLMs for Multimodal GUI Test Generation in Android Applications. In Anais do X Simpósio Brasileiro de Testes de Software Sistemático e Automatizado (Recife/PE). SBC, Porto Alegre, RS, Brasil, 28–36. DOI: 10.5753/sast.2025.13852
Erica Ferreira, Beatriz Oliveira, Carina Silveira, France Arnaut, Crescencio Lima, Andrea Machado, José Amado, and Juliendrios Oliveira. 2025. A gamificação como estratégia de memorabilidade em prol da aprendizagem. In Anais do XXIV Simpósio Brasileiro de Jogos e Entretenimento Digital (Salvador/BA). SBC, Porto Alegre, RS, Brasil, 1637–1647. DOI: 10.5753/sbgames.2025.10179
Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Zhouchi Lin, Bowen Zhang, Lionel Ni, Wen Gao, Yuanzhuo Wang, and Jian Guo. 2026. A survey on LLM-as-a-judge. The Innovation 7, 6 (2026), 101253. DOI: 10.1016/j.xinn.2025.101253
Mahmudul Islam, Farhan Khan, Sabrina Alam, and Mahady Hasan. 2023. Artificial Intelligence in Software Testing: A Systematic Review. DOI: 10.1109/TENCON58879.2023.10322349
Shanshan Li, Liguo Wei, Yicong Liu, Cuiyun Gao, Shing-Chi Cheung, and Michael R. Lyu. 2023. Towards Modeling Software Quality of Virtual Reality Applications from Users’ Perspectives. arXiv:2308.06783 [cs.SE] [link]
Diogo Lima, Danyllo Albuquerque, Emanuel Dantas Filho, Mirko Perkusich, and Angelo Perkusich. 2023. Integrating Reinforcement Learning in Software Testing Automation: A Promising Approach. In Anais do III Workshop Brasileiro de Engenharia de Software Inteligente (Campo Grande/MS). SBC, Porto Alegre, RS, Brasil, 39–41. DOI: 10.5753/ise.2023.235976
Emmanuel Peters and George Kwamina Aggrey. 2020. An ISO 25010 Based Quality Model for ERP Systems. Advances in Science, Technology and Engineering Systems Journal 5, 2 (2020), 578–583.
Dhia Elhaq Rzig, Nafees Iqbal, Isabella Attisano, Xue Qin, and Foyzul Hassan. 2023. Virtual Reality (VR) Automated Testing in the Wild: A Case Study on Unity-Based VR Applications. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis (Seattle, WA, USA) (ISSTA 2023). Association for Computing Machinery, New York, NY, USA, 1269–1281. DOI: 10.1145/3597926.3598134
Murat Tasarsu, Ahmet Vedat Tokmak, and Cagatay Catal. 2026. Test Case Generation Using Large Language Models: A Systematic Literature Review. Cluster Computing 29, 4 (2026), 227. DOI: 10.1007/s10586-026-06021-z
Junjie Wang, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang, and Qing Wang. 2024. Software Testing With Large Language Models: Survey, Landscape, and Vision. IEEE Transactions on Software Engineering 50, 4 (2024), 911–936. DOI: 10.1109/TSE.2024.3368208
Zhiqiang Yuan, Mingwei Liu, Shiji Ding, Kaixin Wang, Yixuan Chen, Xin Peng, and Yiling Lou. 2024. Evaluating and Improving ChatGPT for Unit Test Generation. Proc. ACM Softw. Eng. 1, FSE, Article 76 (July 2024), 24 pages. DOI: 10.1145/3660783
Quanjun Zhang, Chunrong Fang, Yuxiang Ma, Weisong Sun, and Zhenyu Chen. 2023. A Survey of Learning-based Automated Program Repair. ACM Trans. Softw. Eng. Methodol. 33, 2, Article 55 (Dec. 2023), 69 pages. DOI: 10.1145/3631974
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-judge with MTbench and Chatbot Arena. In Proceedings of the 37th International Conference on Neural Information Processing Systems (New Orleans, LA, USA) (NIPS ’23). Curran Associates Inc., Red Hook, NY, USA, Article 2020, 29 pages.
