Integrating Large Language Models into a Business-Critical Application in a Regulated Environment: An Experience Report
Resumo
Organizations are increasingly exploring Large Language Models (LLMs) to augment business-critical applications with document analysis, information extraction, and decision support. This paper reports an ongoing one-year pilot project in a governmentowned financial institution to integrate LLM-based capabilities into a client application portal. The solution supports analysts in checking whether information manually provided by clients is consistent with corporate legal documents submitted during the application process. We summarize the project context, the two-phase integration strategy, the prototype evaluation approach, preliminary outcomes, threats to validity, and reusable design guidelines. The report also distinguishes the project-management aspect of the integration strategy from the technical prompt-construction mechanism based on traceable page retrieval.
Palavras-chave:
Intelligent Software Engineering, Large Language Models, Generative AI, Business-Critical Systems, Regulated Environments
Referências
Syed Arham Akheel. 2025. Guardrails for large language models: A review of techniques and challenges. Journal of Artificial Intelligence, Machine Learning and Data Science 3, 1 (2025), 2504–2512.
Farid Ariai, Joel Mackenzie, and Gianluca Demartini. 2025. Natural language processing for the legal domain: A survey of tasks, datasets, models, and challenges. Comput. Surveys 58, 6 (2025), 1–37.
Fred J Damerau. 1964. A technique for computer detection and correction of spelling errors. Commun. ACM 7, 3 (1964), 171–176.
Maximillien De Bayser, Vinícius Segura, Leonardo G Azevedo, Leonardo P Tizzei, Raphael Thiago, Elton Soares, and Renato Cerqueira. 2022. DevOps and microservices in scientific system development: Experience on a multi-year industry research project. In Proceedings of the 37th ACM/SIGAPP symposium on applied computing. 1452–1455.
Jez Humble and David Farley. 2010. Continuous delivery: reliable software releases through build, test, and deployment automation. Addison-Wesley Boston.
Alfirna Rizqi Lahitani, Adhistya Erna Permanasari, and Noor Akhmad Setiawan. 2016. Cosine similarity to determine similarity measure: Study case in online essay assessment. In 2016 4th International conference on cyber and IT service management. IEEE, 1–6.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems 33 (2020), 9459–9474.
Yupei Liu, Yuqi Jia, Runpeng Geng, Jinyuan Jia, and Neil Zhenqiang Gong. 2024. Formalizing and benchmarking prompt injection attacks and defenses. In 33rd USENIX Security Symposium (USENIX Security 24). 1831–1847.
Linke Ouyang, Yuan Qu, Hongbin Zhou, Jiawei Zhu, Rui Zhang, Qunshu Lin, Bin Wang, Zhiyuan Zhao, Man Jiang, Xiaomeng Zhao, et al. 2025. Omnidocbench: Benchmarking diverse pdf document parsing with comprehensive annotations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 24838–24848.
Ken Schwaber and Jeff Sutherland. 2020. The scrum guide. Scrum Alliance 21, 19 (2020), 1.
Leonardo P Tizzei, Leonardo Azevedo, Elton Soares, Raphael Thiago, and Rodrigo Costa. 2020. On themaintenance of a scientific application based onmicroservices: an experience report. In 2020 IEEE International Conference onWeb Services (ICWS). IEEE, 102–109.
Li Yujian and Liu Bo. 2007. A normalized Levenshtein distance metric. IEEE transactions on pattern analysis and machine intelligence 29, 6 (2007), 1091–1095.
Farid Ariai, Joel Mackenzie, and Gianluca Demartini. 2025. Natural language processing for the legal domain: A survey of tasks, datasets, models, and challenges. Comput. Surveys 58, 6 (2025), 1–37.
Fred J Damerau. 1964. A technique for computer detection and correction of spelling errors. Commun. ACM 7, 3 (1964), 171–176.
Maximillien De Bayser, Vinícius Segura, Leonardo G Azevedo, Leonardo P Tizzei, Raphael Thiago, Elton Soares, and Renato Cerqueira. 2022. DevOps and microservices in scientific system development: Experience on a multi-year industry research project. In Proceedings of the 37th ACM/SIGAPP symposium on applied computing. 1452–1455.
Jez Humble and David Farley. 2010. Continuous delivery: reliable software releases through build, test, and deployment automation. Addison-Wesley Boston.
Alfirna Rizqi Lahitani, Adhistya Erna Permanasari, and Noor Akhmad Setiawan. 2016. Cosine similarity to determine similarity measure: Study case in online essay assessment. In 2016 4th International conference on cyber and IT service management. IEEE, 1–6.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems 33 (2020), 9459–9474.
Yupei Liu, Yuqi Jia, Runpeng Geng, Jinyuan Jia, and Neil Zhenqiang Gong. 2024. Formalizing and benchmarking prompt injection attacks and defenses. In 33rd USENIX Security Symposium (USENIX Security 24). 1831–1847.
Linke Ouyang, Yuan Qu, Hongbin Zhou, Jiawei Zhu, Rui Zhang, Qunshu Lin, Bin Wang, Zhiyuan Zhao, Man Jiang, Xiaomeng Zhao, et al. 2025. Omnidocbench: Benchmarking diverse pdf document parsing with comprehensive annotations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 24838–24848.
Ken Schwaber and Jeff Sutherland. 2020. The scrum guide. Scrum Alliance 21, 19 (2020), 1.
Leonardo P Tizzei, Leonardo Azevedo, Elton Soares, Raphael Thiago, and Rodrigo Costa. 2020. On themaintenance of a scientific application based onmicroservices: an experience report. In 2020 IEEE International Conference onWeb Services (ICWS). IEEE, 102–109.
Li Yujian and Liu Bo. 2007. A normalized Levenshtein distance metric. IEEE transactions on pattern analysis and machine intelligence 29, 6 (2007), 1091–1095.
Publicado
11/09/2026
Como Citar
SOARES, Elton F. de S.; CORNELIO, Marcio F.; SENA, Pedro H. de A.; N., Caio H. D. Milão do; O., Otavio A. Gomes de; A. JR., Gladstone M.; LOPES, João A. dos S..
Integrating Large Language Models into a Business-Critical Application in a Regulated Environment: An Experience Report. In: WORKSHOP BRASILEIRO DE ENGENHARIA DE SOFTWARE INTELIGENTE (ISE), 5. , 2026, São Paulo/SP.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2026
.
p. 133-136.
DOI: https://doi.org/10.5753/ise.2026.32946.
