Automated testing framework to evaluate multi-agent chat assistants - Ramalho, Lucas; Sousa, Jose; Nascimento, Maria; Hanada, Raiza; Souza, Cristian; Collins, Eliane (Anais do Congresso Ibero-Americano em Engenharia de Software (CIbSE), 2026)
This applied R&D project proposes an automated framework for evaluating multi-agent conversational assistants equipped with retrieval-augmented generation (RAG) capabilities. The solution addresses the high cost and time demands of manual evaluation by introducing a synthetic persona dataset and an automated pipeline that executes large-scale tests on mobile devices. Tests with English and Portuguese personas revealed recurring weaknesses in multi-agent systems, particularly in Portuguese interactions, highlighting the importance of multilingual evaluation. The project is a collaboration between INDT and Motorola Mobility and aims to provide a systematic methodology and testing infrastructure for industry conversational systems.