Evaluating LLM-Based Chatbots through Touchpoint-Driven Process Models - Silva, Carlos H. Camillo da; Maita, Ana R. Cárdenas; Almeida Filho, Gregorio Assagra de; Silva, Caio V. Melo da; Dimant, Gabriel; Almeida, Renato A.; Costa, Guilherme M. Lopes; Araujo, Anna M. Cintra; Santana, Barbara Correia Dos Santos; Blay, Enio Alterman; Peres, Sarajane Marques (Anais do Workshop sobre Bots na Engenharia de Software (WBOTS); 2025: Anais do II Workshop sobre Bots na Engenharia de Software, 2025)
This article introduces a method for evaluating the quality of large language model (LLM)-based chatbots, focusing on the user journey. Each dialogue utterance, whether from the user or the bot, is mapped to a predefined set of touchpoints that represent key interaction goals. The resulting dialogue sequences are treated as business process activities, enabling process mining to discover and analyze the process model underlying the dialogue context. This method identifies inefficiencies such as bottlenecks, conversational loops, and shifting responsibility to human agents. It was tested on simulated dialogues generated by an LLM-based financial chatbot, with user personas and dialogue goals also synthesized by LLMs. Although based on synthetic data, the results demonstrate the potential of process mining to uncover structural strengths and weaknesses in LLM-based chatbots.