Pre-Deployment Evaluation of LLM-based Conversational Agents through a Digital Twin-Inspired Approach
Resumo
The dynamic and non-deterministic nature of large language models (LLMs) complicates conversational agent evaluation, particularly when assessing pre-deployment configurations under realistic conditions. To address this, a digital-twin-inspired architecture is proposed to monitor interactions and fork ongoing dialogues, enabling what-if analyses of alternative agent configurations using observed conversational contexts. To assess technical feasibility, we conducted an exploratory experiment using 3,996 simulated dialogues as a production proxy, generating 949 evaluation dialogues across four twin agents with varying parameters. The resulting interaction traces were analyzed with process mining techniques, showing the feasibility of the proposed architecture and its potential to support comparative evaluation of conversational agent modifications before deployment.
Palavras-chave:
LLM-based Conversational Agents, Digital Twins, LLM Testing, LLM Evaluation, Process Mining
Referências
Merav Allouch, Amos Azaria, and Rina Azoulay. 2021. Conversational Agents: Goals, Technologies, Vision and Challenges. Sensors 21, 24 (2021), 48 pages.
Amir Reza Asadi. 2026. Building Conversational Information Sources as HumanCentered Digital Persona Twins for Probation Officer Training. In Proc. Conf. Hum. Inf. Interact. Retr. (USA). ACM, USA, 394–398.
Tobias Brockhoff, Malte Heithoff, István Koren, Judith Michael, Jérôme Pfeiffer, Bernhard Rumpe, Merih Seran Uysal, Wil M. P. Van Der Aalst, and Andreas Wortmann. 2021. Process Prediction with Digital Twins. In ACM/IEEE Int. Conf. Model Driven Eng. Lang. Syst. Companion. IEEE, Japan, 182–187.
Fabio Caffaro and Giuseppe Rizzo. 2024. Knowledge-Enhanced Conversational Agents. J. Comput. Sci. Technol. 39, 3 (2024), 585–609.
Xin Chen, Yan Li, and Xiaoming Wang. 2025. Design Principles and Guidelines for LLM Observability: Insights from Developers. In Proc. Ext. Abstr. CHI Conf. Hum. Factors Comput. Syst. ACM, USA, 9 pages.
Manuela Dalibor, Malte Heithoff, et al. 2022. Generating customized low-code development platforms for digital twins. J. Comp. Lang. 70 (2022), 101117.
Manuela Dalibor, Nico Jansen, Bernhard Rumpe, et al. 2022. A Cross-Domain Systematic Mapping Study on Software Engineering for Digital Twins. J. Syst. Software 193 (2022), 111361.
Thomas G. Dietterich. 2000. Ensemble Methods in Machine Learning. In Multiple Classifier Systems. Lecture Notes in Computer Science, Vol. 1857. Springer, Germany, 1–15.
Fabrizio Fornari, Ivan Compagnucci, Massimo Callisto De Donato, et al. 2025. Digital Twins of Business Processes: A Research Manifesto. Internet of Things 30 (2025), 101477.
Zhiqi Guo, Fengxiao Tang, Linfeng Luo, Ming Zhao, and Nei Kato. 2025. A Survey on Applications of Large Language Model-Driven Digital Twins for Intelligent Network Optimization. IEEE Commun. Surv. Tutorials 28 (2025), 3388–3411.
Gyunam Park. 2021. Realizing A Digital Twin of An Organization Using Action-oriented Process Mining. In Int. Conf. Process Min. IEEE, Eindhoven, Netherlands, 104–111.
J. Han, M. Kamber, and J. Pei. 2011. Data Mining: Concepts and Techniques (3rd ed.). Morgan Kaufmann Publishers Inc., USA.
Yizheng Huang and Jimmy X. Huang. 2026. A Survey on Retrieval-Augmented Text Generation for Large Language Models. ACM Comput. Surv. 58, 12 (2026), 38 pages.
Yijun Huang, Jihan Zhang, Xi Chen, Alan H. F. Lam, and BenM. Chen. 2024. From Simulation to Prediction: Enhancing Digital Twins with Advanced Generative AI Technologies. In IEEE Int. Conf. Control Autom. IEEE, Iceland, 490–495.
Daniel Jurafsky and James H. Martin. 2026. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition with Language Models (3rd ed.). Stanford, USA.
J. Richard Landis and Gary G. Koch. 1977. An Application of Hierarchical Kappa-Type Statistics in the Assessment of Majority Agreement among Multiple Observers. Biometrics 33, 2 (1977), 363–374. DOI: 10.2307/2529786
LangChain. 2026. LangChain Documentation: Retrieval. [link]
Jung-Hyun Lee, Eunhee Choi, Sergio L. Angulo, Robert A. McDougal, and William W. Lytton. 2025. Neurological history both twinned and queried by generative artificial intelligence. Frontiers in Medicine 11 (2025), 9 pages.
Pengze Li, Yutong Hu, Jianfu Li, Garit Gemeinhardt, et al. 2026. Developing provider digital twins for personalized provider-patient communication via a RAG-based conversational framework. Computational and Structural Biotechnology Journal 32 (2026), 24–33.
Kalle Lyytinen, Barbara Weber, Markus C. Becker, and Brian T. Pentland. 2023. Digital twins of organization: implications for organization design. J. Organ. Des. 13, 3 (2023), 77–93.
Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, et al. 2025. Large Language Models: A Survey. arXiv:2402.06196 [cs.CR]
Mehreen Naeem, Andrew Melnik, and Michael Beetz. 2025. Grounding Language Models with Semantic Digital Twins for Robotic Planning. arXiv:2506.16493 [cs].
Carlos Silva, Ana Maita, Gregorio Almeida Filho, Caio Silva, Gabriel Dimant, Renato Almeida, Guilherme Costa, Anna Araujo, Barbara Santana, Enio Blay, and Sarajane Peres. 2025. Evaluating LLM-Based Chatbots through Touchpoint-Driven Process Models. In Anais do II Workshop sobre Bots na Engenharia de Software. SBC, Recife, PE, Brazil, 1–10.
Twinkll Sisodia. 2026. AI Observability for Large Language Model Systems: A Multi-Layer Analysis of Monitoring Approaches from Confidence Calibration to Infrastructure Tracing. arXiv:2604.26152 [cs.SE]
W. M. P. van der, Aalst. 2016. Process Mining: Data Science in Action (2 ed.). Springer, Berlin, Germany.
Francesco Vitale, Simone Guarino, Francesco Flammini, Luca Faramondi, Nicola Mazzocca, and Roberto Setola. 2025. Process Mining for Digital Twin Development of Industrial Cyber-Physical Systems. IEEE Trans. Ind. Inf. 21, 1 (2025), 866–875.
Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. 2025. 𝜏-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. In Int. Conf. Learn. Represent. Curran Associates, Singapore, 53 pages.
Asaf Yehudai, Lilach Eden, Alan Li, Guy Uziel, Yilun Zhao, Roy Bar-Haim, Arman Cohan, and Michal Shmueli-Scheuer. 2026. Survey on Evaluation of LLM-based Agents. arXiv:2503.16416 [cs.AI].
Amir Reza Asadi. 2026. Building Conversational Information Sources as HumanCentered Digital Persona Twins for Probation Officer Training. In Proc. Conf. Hum. Inf. Interact. Retr. (USA). ACM, USA, 394–398.
Tobias Brockhoff, Malte Heithoff, István Koren, Judith Michael, Jérôme Pfeiffer, Bernhard Rumpe, Merih Seran Uysal, Wil M. P. Van Der Aalst, and Andreas Wortmann. 2021. Process Prediction with Digital Twins. In ACM/IEEE Int. Conf. Model Driven Eng. Lang. Syst. Companion. IEEE, Japan, 182–187.
Fabio Caffaro and Giuseppe Rizzo. 2024. Knowledge-Enhanced Conversational Agents. J. Comput. Sci. Technol. 39, 3 (2024), 585–609.
Xin Chen, Yan Li, and Xiaoming Wang. 2025. Design Principles and Guidelines for LLM Observability: Insights from Developers. In Proc. Ext. Abstr. CHI Conf. Hum. Factors Comput. Syst. ACM, USA, 9 pages.
Manuela Dalibor, Malte Heithoff, et al. 2022. Generating customized low-code development platforms for digital twins. J. Comp. Lang. 70 (2022), 101117.
Manuela Dalibor, Nico Jansen, Bernhard Rumpe, et al. 2022. A Cross-Domain Systematic Mapping Study on Software Engineering for Digital Twins. J. Syst. Software 193 (2022), 111361.
Thomas G. Dietterich. 2000. Ensemble Methods in Machine Learning. In Multiple Classifier Systems. Lecture Notes in Computer Science, Vol. 1857. Springer, Germany, 1–15.
Fabrizio Fornari, Ivan Compagnucci, Massimo Callisto De Donato, et al. 2025. Digital Twins of Business Processes: A Research Manifesto. Internet of Things 30 (2025), 101477.
Zhiqi Guo, Fengxiao Tang, Linfeng Luo, Ming Zhao, and Nei Kato. 2025. A Survey on Applications of Large Language Model-Driven Digital Twins for Intelligent Network Optimization. IEEE Commun. Surv. Tutorials 28 (2025), 3388–3411.
Gyunam Park. 2021. Realizing A Digital Twin of An Organization Using Action-oriented Process Mining. In Int. Conf. Process Min. IEEE, Eindhoven, Netherlands, 104–111.
J. Han, M. Kamber, and J. Pei. 2011. Data Mining: Concepts and Techniques (3rd ed.). Morgan Kaufmann Publishers Inc., USA.
Yizheng Huang and Jimmy X. Huang. 2026. A Survey on Retrieval-Augmented Text Generation for Large Language Models. ACM Comput. Surv. 58, 12 (2026), 38 pages.
Yijun Huang, Jihan Zhang, Xi Chen, Alan H. F. Lam, and BenM. Chen. 2024. From Simulation to Prediction: Enhancing Digital Twins with Advanced Generative AI Technologies. In IEEE Int. Conf. Control Autom. IEEE, Iceland, 490–495.
Daniel Jurafsky and James H. Martin. 2026. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition with Language Models (3rd ed.). Stanford, USA.
J. Richard Landis and Gary G. Koch. 1977. An Application of Hierarchical Kappa-Type Statistics in the Assessment of Majority Agreement among Multiple Observers. Biometrics 33, 2 (1977), 363–374. DOI: 10.2307/2529786
LangChain. 2026. LangChain Documentation: Retrieval. [link]
Jung-Hyun Lee, Eunhee Choi, Sergio L. Angulo, Robert A. McDougal, and William W. Lytton. 2025. Neurological history both twinned and queried by generative artificial intelligence. Frontiers in Medicine 11 (2025), 9 pages.
Pengze Li, Yutong Hu, Jianfu Li, Garit Gemeinhardt, et al. 2026. Developing provider digital twins for personalized provider-patient communication via a RAG-based conversational framework. Computational and Structural Biotechnology Journal 32 (2026), 24–33.
Kalle Lyytinen, Barbara Weber, Markus C. Becker, and Brian T. Pentland. 2023. Digital twins of organization: implications for organization design. J. Organ. Des. 13, 3 (2023), 77–93.
Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, et al. 2025. Large Language Models: A Survey. arXiv:2402.06196 [cs.CR]
Mehreen Naeem, Andrew Melnik, and Michael Beetz. 2025. Grounding Language Models with Semantic Digital Twins for Robotic Planning. arXiv:2506.16493 [cs].
Carlos Silva, Ana Maita, Gregorio Almeida Filho, Caio Silva, Gabriel Dimant, Renato Almeida, Guilherme Costa, Anna Araujo, Barbara Santana, Enio Blay, and Sarajane Peres. 2025. Evaluating LLM-Based Chatbots through Touchpoint-Driven Process Models. In Anais do II Workshop sobre Bots na Engenharia de Software. SBC, Recife, PE, Brazil, 1–10.
Twinkll Sisodia. 2026. AI Observability for Large Language Model Systems: A Multi-Layer Analysis of Monitoring Approaches from Confidence Calibration to Infrastructure Tracing. arXiv:2604.26152 [cs.SE]
W. M. P. van der, Aalst. 2016. Process Mining: Data Science in Action (2 ed.). Springer, Berlin, Germany.
Francesco Vitale, Simone Guarino, Francesco Flammini, Luca Faramondi, Nicola Mazzocca, and Roberto Setola. 2025. Process Mining for Digital Twin Development of Industrial Cyber-Physical Systems. IEEE Trans. Ind. Inf. 21, 1 (2025), 866–875.
Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. 2025. 𝜏-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. In Int. Conf. Learn. Represent. Curran Associates, Singapore, 53 pages.
Asaf Yehudai, Lilach Eden, Alan Li, Guy Uziel, Yilun Zhao, Roy Bar-Haim, Arman Cohan, and Michal Shmueli-Scheuer. 2026. Survey on Evaluation of LLM-based Agents. arXiv:2503.16416 [cs.AI].
Publicado
11/09/2026
Como Citar
BERNARDO, Caio Vinícius Cunha; NEUBAUER, Thais Rodrigues; ALMEIDA FILHO, Gregório A. de; BLAY, Enio Alterman; PERES, Sarajane Marques.
Pre-Deployment Evaluation of LLM-based Conversational Agents through a Digital Twin-Inspired Approach. In: WORKSHOP SOBRE BOTS NA ENGENHARIA DE SOFTWARE (WBOTS), 3. , 2026, Cuiabá/MT.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2026
.
p. 1-7.
DOI: https://doi.org/10.5753/wbots.2026.30606.
