Exploring Conversational Agents for Developer Guidance on Software Dependability Issues

• João Correia PUC-Rio
• Alessandro Garcia PUC-Rio
• Rafael de Mello UFRJ

DOI: https://doi.org/10.5753/cbsoft_estendido.2026.29839

Resumo

Software dependability is a critical non-functional requirement that encompasses a system’s ability to remain available, reliable, maintainable, robust, and resilient in the presence of faults. Dependability requirements are essential in almost every software project. Exception handling plays a central role in operationalizing these attributes and serves as the means to detect, report and handle failures; yet it remains under-supported by tools and often under-emphasized in developer workflows. In this context, this work empirically investigates recurring exception handling issues in software projects. Then, it further investigates how conversational agents, particularly those powered by Large Language Models (LLMs), can support dependable software development. With this aim, we evaluated the effectiveness of LLM-based assistants in real-world software engineering contexts: (i) answering technical questions of developers, and (ii) automatically detecting dependability-related issues and generating candidate fixes for developer review. To this end, four complementary studies were conducted in the context of open-source and closed-source systems. DevMentorAI, a retrieval-augmented conversational agent, was evaluated through a case study involving Mozilla developers in the PDF.js project. Dev-MentorAI demonstrated superior or similar performance compared to humans in many scenarios. Then, another comparative study in Mozilla Firefox showed that retrieval-augmented models produce more helpful and comprehensive responses than generic LLMs and human experts. Finally, we designed and evaluated a multi-agent architecture that extends conversational assistance by proactively identifying and repairing dependability issues. The approach was evaluated through case studies in three industrial software projects from different domains, with assessments performed by the projects’ own developers. Our findings revealed that LLM-driven agents can generate effective, explainable, and reviewable code changes that align with developers’ expectations. When grounded in project-specific knowledge, they can effectively support the identification and resolution of software dependability issues, advancing both research and practice in AI-assisted software engineering. These studies are reported in premier vehicles in software engineering and specific themes. The agent systems were used by industry partners.

Palavras-chave: Software Dependability, Developer Assistance, Large Language Models, Conversational Agents

Referências

Algirdas Avizienis, Jean-Claude Laprie, Brian Randell, and Carl Landwehr. 2004. Basic concepts and taxonomy of dependable and secure computing. IEEE Transactions on Dependable and Secure Computing 1, 1 (2004), 11–33. DOI: 10.1109/TDSC.2004.2

Martin Balfroid, Benoît Vanderose, and Xavier Devroey. 2024. Towards LLMGenerated Code Tours for Onboarding. In Workshop on NL-based Software Engineering (NLBSE’24).

Eiji Adachi Barbosa, Alessandro F. Garcia, and Simone Diniz Junqueira Barbosa. 2014. Categorizing Faults in Exception Handling: A Study of Open Source Projects. In 2014 Brazilian Symposium on Software Engineering. IEEE Computer Society, 11–20. DOI: 10.1109/SBES.2014.19

Joshua Bloch. 2018. Effective Java (3rd ed.). Addison-Wesley.

Nélio Cacho, Eiji Adachi Barbosa, Juliana Araujo, Frederico Pranto, Alessandro Garcia, Thiago Cesar, Eliezio Soares, Arthur Cassio, Thomas Filipe, and Israel Garcia. 2014. How Does Exception Handling Behavior Evolve? An Exploratory Study in Java and C# Applications. In IEEE International Conference on Software Maintenance and Evolution. 31–40. DOI: 10.1109/ICSME.2014.25

Nélio Cacho, Thiago César, Thomas Filipe, Eliezio Soares, Arthur Cassio, Rafael Souza, Israel Garcia, Eiji Adachi Barbosa, and Alessandro Garcia. 2014. Trading robustness for maintainability: an empirical study of evolving C# programs. In 36th International Conference on Software Engineering. ACM, 584–595. DOI: 10.1145/2568225.2568308

Cognita. [n. d.]. Cognita. [link]. (Accessed on 06/05/2026).

João Correia, Daniel Coutinho, Marco Castelluccio, Caio Barbosa, Rafael de Mello, Anita Sarma, Alessandro Garcia, Marco Gerosa, and Igor Steinmacher. 2026. A Comparison of Conversational Models and Humans in Answering Technical Questions: the Firefox Case. In Proceedings of the 48th IEEE/ACM International Conference on Software Engineering (ICSE). IEEE/ACM.

João Correia, Daniel Coutinho, Marco Castelluccio, Caio Barbosa, Rafael de Mello, Anita Sarma, Alessandro Garcia, Marco Gerosa, and Igor Steinmacher. 2026. Replication Package: A Comparison of Conversational Models and Humans in Answering Technical Questions: the Firefox Case. [link]. Accessed: 2026-06-05.

João Correia, Daniel Coutinho, Alessandro Garcia, Rafael de Mello, Caio Barbosa, Anderson Oliveira, Wesley K. G. Assunção, Juliana Alves Pereira, Igor Steinmacher, Marco Gerosa, Jairo Souza, and Johny Arriel. 2024. On the Investigation of Exception Pull Request Characteristics: Exploring the Apache Ecosystem. In 2024 IEEE International Conference on Source Code Analysis and Manipulation (SCAM), in conjunction with International Conference on Software Maitenance and Evolution (ICSME 2024). 143–153. DOI: 10.1109/SCAM63643.2024.00023

João Correia, Daniel Coutinho, Alessandro Garcia, Rafael de Mello, Caio Barbosa, Anderson Oliveira, Wesley K. G. Assunção, Juliana Alves Pereira, Igor Steinmacher, Marco Gerosa, Jairo Souza, and Johny Arriel. 2024. Replication Package: On the Investigation of Exception Pull Request Characteristics: Exploring the Apache Ecosystem. (2024). [link] Accessed on: 2026-06-05.

João Correia, Morgan C. Nicholson, Daniel Coutinho, Caio Barbosa, Marco Castelluccio, Marco Gerosa, Alessandro Garcia, and Igor Steinmacher. 2024. Replication package: Unveiling the Potential of a Conversational Agent in Developer Support: Insights from Mozilla’s PDF.js Project. [link]. Accessed: 2026-06-05.

João Correia, Morgan C. Nicholson, Daniel Coutinho, Caio Barbosa, Marco Castelluccio, Marco Gerosa, Alessandro Garcia, and Igor Steinmacher. 2024. Unveiling the Potential of a Conversational Agent in Developer Support: Insights from Mozilla’s PDF.js Project. In Proceedings of the 1st ACM International Conference on AI-Powered Software, Foundations of Software Engineering (FSE) Conference (Porto de Galinhas, Brazil) (AIware 2024). Association for Computing Machinery, New York, NY, USA, 10–18. DOI: 10.1145/3664646.3664758

João Lucas Correia. 2025. Exploring Conversational Agents for Developer Guidance on Software Dependability Issues. PhD Thesis. Pontifical Catholic University of Rio de Janeiro, Informatics Department, Rio de Janeiro, Brazil. [link]

João Lucas Correia, Juliana Alves Pereira, Rafael Maiani de Mello, Alessandro Garcia, Baldoino Fonseca, Márcio Ribeiro, Rohit Gheyi, Marcos Kalinowski, Renato Cerqueira, and Willy Tiengo. 2020. Brazilian Data Scientists: Revealing their Challenges and Practices on Machine Learning Model Development. In 19th Brazilian Symposium on Software Quality, SBQS 2020, São Luís, Brazil, December, 2020, Davi Viana and Marcelo Schots (Eds.). ACM, 10. DOI: 10.1145/3439961.3439971

João Lucas Marques Correia, Daniel Coutinho, Alessandro Garcia, Rafael Maiani de Mello, Wesley K. G. Assunção, Caio Barbosa, Nélio Cacho, and Thaís Vasconcelos Batista. 2026. A Conversational Multiagent Architecture for Resolving Issues on Multi-Attribute Dependability. submitted to a journal (2026), 1–20.

João Lucas Marques Correia, Daniel Coutinho, Alessandro Garcia, Rafael Maiani de Mello, Wesley K. G. Assunção, Caio Barbosa, Nélio Cacho, and Thaís Vasconcelos Batista. 2026. Developers’ Perceptions of Automated Detection and Resolution of Software Dependability Issues. In Proceedings of the 40th Brazilian Symposium on Software Engineering (SBES) (São Paulo, Brazil).

João Lucas Marques Correia, Daniel Coutinho, Alessandro Garcia, Rafael Maiani de Mello, Wesley K. G. Assunção, Caio Barbosa, Nélio Cacho, and Thaís Vasconcelos Batista. 2026. Research Artifacts for “Developers’ Perceptions of Automated Detection and Resolution of Software Dependability Issues”. [link]. Research artifact and replication package. Accessed: 2026-06-05.

Laura Dabbish, Colleen Stuart, Jason Tsay, and Jim Herbsleb. 2012. Social coding in GitHub: transparency and collaboration in an open software repository. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work (Seattle, Washington, USA) (CSCW ’12). Association for Computing Machinery, New York, NY, USA, 1277–1286. DOI: 10.1145/2145204.2145396

Felipe Ebert, Fernando Castor, and Alexander Serebrenik. 2015. An exploratory study on exception handling bugs in Java programs. Journal of Systems and Software 106 (2015), 82–101. DOI: 10.1016/j.jss.2015.04.066

Felipe Ebert, Fernando Castor, and Alexander Serebrenik. 2020. A Reflection on “An Exploratory Study on Exception Handling Bugs in Java Programs”. In 2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER). 552–556. DOI: 10.1109/SANER48275.2020.9054791

Mats Finsås and Joachim Maksim. 2024. Optimizing RAG Systems for Technical Support with LLM-based Relevance Feedback and Multi-Agent Patterns. Master’s thesis. NTNU.

Felipe Fronchetti, David C. Shepherd, Igor Wiese, Christoph Treude, Marco Aurélio Gerosa, and Igor Steinmacher. 2023. Do CONTRIBUTING Files Provide Information about OSS Newcomers’ Onboarding Barriers?. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (San Francisco, CA, USA) (ESEC/FSE 2023). Association for Computing Machinery, New York, NY, USA, 16–28. DOI: 10.1145/3611643.3616288

Md Saleh Ibtasham, Sarmad Bashir, Muhammad Abbas, Zulqarnain Haider, Mehrdad Saadatmand, and Antonio Cicchetti. 2025. ReqRAG: Enhancing Software Release Management through Retrieval-Augmented LLMs: An Industrial Study. In Requirements Engineering: Foundation for Software Quality: 31st International Working Conference, REFSQ 2025, Barcelona, Spain, April 7–10, 2025, Proceedings (Barcelona, Spain). Springer-Verlag, Berlin, Heidelberg, 277–292. DOI: 10.1007/978-3-031-88531-0_20

Carlos E. Jimenez, John Yang, AlexanderWettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? arXiv:2310.06770 [cs.CL] [link]

J.C. C. Laprie, A. Avizienis, and H. Kopetz. 1992. Dependability: Basic Concepts and Terminology. Springer-Verlag, Berlin, Heidelberg.

Xiangxin Meng, Zexiong Ma, Pengfei Gao, and Chao Peng. 2025. An Empirical Study on LLM-based Agents for Automated Bug Fixing. arXiv:2411.10213 [cs.SE] [link]

Arghavan Moradi Dakhel, Vahid Majdinasab, Amin Nikanjam, Foutse Khomh, Michel C. Desmarais, and Zhen Ming (Jack) Jiang. 2023. GitHub Copilot AI pair programmer: Asset or Liability? Journal of Systems and Software 203 (2023), 111734. DOI: 10.1016/j.jss.2023.111734

Suman Nakshatri, Maithri Hegde, and Sahithi Thandra. 2016. Analysis of exception handling patterns in Java projects: an empirical study. In 13th International Conference on Mining Software Repositories. ACM, 500–503. DOI: 10.1145/2901739.2903499

Anderson Oliveira, João Correia, Leonardo Sousa, Wesley K. G. Assunção, Daniel Coutinho, Alessandro Garcia, Willian Oizumi, Caio Barbosa, Anderson Uchôa, and Juliana Alves Pereira. 2023. Don’t Forget the Exception! : Considering Robustness Changes to Identify Design Problems. In 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR). 417–429. DOI: 10.1109/MSR59073.2023.00064

Anderson Oliveira, João Correia, Wesley K. G. Assunção, Juliana Alves Pereira, Rafael de Mello, Daniel Coutinho, Caio Barbosa, Paulo Libório, and Alessandro Garcia. 2024. Understanding Developers’ Discussions and Perceptions on Nonfunctional Requirements: The Case of the Spring Ecosystem. Proceedings of the ACM on Software Engineering Journal - Foundations on Software Engineering Conference (FSE 2024) 1, FSE, Article 24 (July 2024), 22 pages. DOI: 10.1145/3643750

Parametrix Solutions, Inc. 2025. CrowdStrike’s Impact on the Fortune 500. [link]. Accessed: 2025-09-19.

Rohith Pudari and Neil A Ernst. 2023. From copilot to pilot: Towards AI supported software development. arXiv preprint arXiv:2303.04142 (2023).

Daniel Ramos, Claudia Mamede, Kush Jain, Paulo Canelas, Catarina Gamboa, and Claire Le Goues. 2025. Are Large Language Models Memorizing Bug Benchmarks? arXiv:2411.13323 [cs.SE] [link]

June Sallou, Thomas Durieux, and Annibale Panichella. 2023. Breaking the silence: the threats of using llms in software engineering. arXiv preprint arXiv:2312.08055 (2023). CBSoft 2026, September 8-11, 2026, São Paulo, SP, Brazil Correia et al.

Puntitra Sawadpong, Edward B. Allen, and Byron J. Williams. 2012. Exception Handling Defects: An Empirical Study. In 14th International Symposium on High-Assurance Systems Engineering. 90–97. DOI: 10.1109/HASE.2012.24

Hina Shah, Carsten Görg, and Mary Jean Harrold. 2008. Why do developers neglect exception handling?. In 4th InternationalWorkshop on Exception Handling. ACM, 62–68. DOI: 10.1145/1454268.1454277

Hina Shah, Carsten Gorg, and Mary Jean Harrold. 2010. Understanding Exception Handling: Viewpoints of Novices and Experts. IEEE Transactions on Software Engineering 36, 2 (2010), 150–161. DOI: 10.1109/TSE.2010.7

Dominik Sobania, Martin Briesch, Carol Hanna, and Justyna Petke. 2023. An Analysis of the Automatic Bug Fixing Performance of ChatGPT. (May 2023), 23–30. DOI: 10.1109/APR59189.2023.00012

Giovanni Viviani, Michalis Famelis, Xin Xia, Calahan Janik-Jones, and Gail C. Murphy. 2021. Locating Latent Design Information in Developer Discussions: A Study on Pull Requests. IEEE Transactions on Software Engineering 47, 7 (2021), 1402–1413. DOI: 10.1109/TSE.2019.2924006

Giovanni Viviani, Calahan Janik-Jones, Michalis Famelis, Xin Xia, and Gail C. Murphy. 2018. What design topics do developers discuss?. In 26th Conference on Program Comprehension. ACM, 328–331. DOI: 10.1145/3196321.3196357

Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. 2025. Demystifying LLM-Based Software Engineering Agents. Proc. ACM Softw. Eng. 2, FSE, Article FSE037 (June 2025), 24 pages. DOI: 10.1145/3715754

Zhentao Xu, Mark Jerome Cruz, Matthew Guevara, Tie Wang, Manasi Deshpande, Xiaofeng Wang, and Zheng Li. 2024. Retrieval-Augmented Generation with Knowledge Graphs for Customer Service Question Answering. Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (2024). [link]

Xuanming Zhang, Yuxuan Chen, Yuan Yuan, and Minlie Huang. 2025. Towards Exception Safety Code Generation with Intermediate Representation Agents Framework. arXiv:2410.06949 [cs.SE] [link]

Hao Zhong and Hong Mei. 2018. Mining repair model for exception-related bug. Journal of Systems and Software 141 (2018), 16–31. DOI: 10.1016/j.jss.2018.03.046