Evaluating LLM Models for Model-to-Text Transformation: Singletons, Ensembles, and Statistical Analysis

• Pedro Henrique Bernardes Caetano Milken UFU
• Marcelo de Almeida Maia UFU
• Leandro Nogueira Couto UFU
• Nicole Maia Argondizzi UFU
• Marcelo Keese Albertini UFU

DOI: https://doi.org/10.5753/eniac.2026.30800

Resumo

Business Process Model and Notation (BPMN) diagrams are essential for illustrating organizational procedures but can be inaccessible to non-technical stakeholders. Model-to-text (M2T) transformation helps bridge this gap. Research identifies GPT-4 as a leading model for M2T, but highlights hallucination, which introduces inaccuracies as a main issue. This study explores the potential reduction of hallucinations by employing contemporary LLMs in an iterative self-refinement cycle, testing eight models under various feedback and temperature conditions. Key performance indicators reveal that while self-refinement reduces task omission, it increases misrepresentation and spurious generation. The results indicate that top-performing configurations excel in one-shot generation, and modeling-register leakage is minimal due to effective jargon avoidance. However, automated feedback may inadvertently amplify hallucinations, challenging the assumptions of self-refinement strategies in M2T for requirements engineering. The study’s reproducibility package is publicly available, detailing the methodology and results.

Referências

Bellan, P., van der Aa, H., Dragoni, M., Ghidini, C., and Ponzetto, S. P. (2023). PET: An annotated dataset for process extraction from natural language text tasks. In Cabanillas, C., Garmann-Johnsen, N. F., and Koschmider, A., editors, Business Process Management Workshops, volume 460 of Lecture Notes in Business Information Processing, pages 315–321. Springer, Cham.

Caetano Milken, P. H. B. (2026). Replication package for GPT vs. Claude for model-to-text transformation: Singletons, ensembles, and statistical significance. DOI: 10.5281/zenodo.19979264.

Demšar, J. (2006). Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 7:1–30.

IrisAgent (2024). So, which LLM models are the best for building an AI chatbot? [link]. Accessed: 2 May 2026.

Jiang, Q., Gao, Z., and Karniadakis, G. E. (2025). Deepseek vs. chatgpt vs. claude: A comparative study for scientific computing and scientific machine learning tasks. Theoretical and Applied Mechanics Letters. Article 100583.

Klievtsova, N., Mangler, J., Kampik, T., and Rinderle-Ma, S. (2024). Utilizing process models in the requirements engineering process through model2text transformation. In Liebel, G., Hadar, I., and Spoletini, P., editors, Proceedings of the 32nd IEEE International Requirements Engineering Conference (RE 2024), pages 205–217. IEEE Computer Society.

Silva, M. O., Oliveira, G. P., Costa, L. G. L., and Pappa, G. L. (2025). GovBERT-BR: A BERT-based language model for brazilian portuguese governmental data. In Paes, A. and Verri, F. A. N., editors, Intelligent Systems, pages 19–32. Springer Nature Switzerland, Cham.

Wimmer, M. and Burgueño, L. (2013). Testing M2T/T2M transformations. In Moreira, A., Schätz, B., Gray, J., Vallecillo, A., and Clarke, P., editors, Model-Driven Engineering Languages and Systems, volume 8107 of Lecture Notes in Computer Science, pages 203–219. Springer, Berlin, Heidelberg.