Automatically Generating Security Guidelines from Pull Request Discussions: The Case of Django Framework
Resumo
Security in open-source software relies on documentation, yet a major gap exists regarding project-specific instructions for secure usage. We introduce a multi-stage framework that automatically generates project-specific User Guidelines from historical Pull Request (PR) code reviews. Adapting PRIMES (Prompt Refinement and Insights for Mining Empirical Software repositories) [5] — a methodology for using LLMs (Large Language Models) in repository mining — our approach uses iterative prompt engineering and operational constraints to classify unstructured PR threads into OWASP Top 10 2025 categories [11]. Using a pilot validation set, the classification pipeline achieved an accuracy of 0.98, which we then applied to classify the full corpus of 18,524 Django PRs. Large-scale analysis identified 1,917 security discussions, predominantly involving the Mishandling of Exceptional Conditions (A10). Finally, a qualitative peer review by senior developers confirmed that the framework reliably transforms historical developer dialogs into production-ready security policies.
Palavras-chave:
Large Language Models, Software Security, Repository Mining, Pull Requests, OWASP Top 10, Security Documentation
Referências
Anthropic. 2026. Claude.ai. [link] AI assistant platform. Accessed: 2026-08-08.
Jessy Ayala and Joshua Garcia. 2023. An Empirical Study on Workflows and Security Policies in Popular GitHub Repositories. In 2023 IEEE/ACM 1st International Workshop on Software Vulnerability (SVM). IEEE, 6–9. DOI: 10.1109/svm59160.2023.00006
Noah Bühlmann and Mohammad Ghafari. 2022. How do developers deal with security issue reports on GitHub?. In Proceedings of the 37th ACM/SIGAPP Symposium on Applied Computing (Virtual Event) (SAC ’22). Association for Computing Machinery, New York, NY, USA, 1580–1589. DOI: 10.1145/3477314.3507123
Morakot Choetkiertikul, Sushawapak Kancharoendee, Chanikarn Jongyingyos, Thanat Phichitphanphong, Chaiyong Ragkhitwetsagul, Brittany Reid, Raula Kula, and Thanwadee Sunetnanta. 2026. Security by documentation? characterizing GitHub SECURITY.md policy and their adoption in Python libraries. Empirical Software Engineering 31 (02 2026). DOI: 10.1007/s10664-025-10794-z
Vincenzo De Martino, Joel Castano, Fabio Palomba, Xavier Franch, and Silverio Martinez-Fernandez. 2025. A Framework for Using LLMs for Repository Mining Studies in Empirical Software Engineering . In 2025 IEEE/ACM International Workshop on Methodological Issues with Empirical Studies in Software Engineering (WSESE). IEEE Computer Society, 6–11.
Django Software Foundation. 2026. Django: A high-level Python web framework. [link] GitHub repository. Accessed: 2026-06-17.
GitHub, Inc. 2026. GitHub REST API documentation. [link] Official documentation. Accessed: 2026-06-17.
Google AI for Developers. 2026. Prompting strategies: Example template combining best practices. [link] Gemini API Documentation. Accessed: 2026-06-17.
Google Cloud. 2026. Google Gen AI SDK. [link] Google Cloud Documentation. Accessed: 2026-08-08.
Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. Large Language Models for Software Engineering: A Systematic Literature Review. arXiv:2308.10620 [cs.SE] [link]
OWASP Foundation. 2025. OWASP Top 10:2025. [link] Accessed: 2026-06-17.
Antonio Della Porta, Vincenzo De Martino, Gilberto Recupito, Carmine Iemmino, Gemma Catolino, Dario Di Nucci, and Fabio Palomba. 2024. Using Large Language Models to Support Software Engineering Documentation in Waterfall Life Cycles: Are We There Yet?. In Proceedings of the Ital-IA Intelligenza Artificiale - Thematic Workshops co-located with the 4th CINI National Lab AIIS Conference on Artificial Intelligence (Ital-IA 2024), Naples, Italy, May 29-30, 2024 (CEUR Workshop Proceedings, Vol. 3762), Sergio Di Martino, Carlo Sansone, Elio Masciari, Silvia Rossi, and Michela Gravina (Eds.). CEUR-WS.org, 42–47. [link]
Patrick Serrano, Luwen Huangfu, Chunhua Liao, Dongqing Lin, Kai Williams, Jaden Perleoni, and Thomas Brettin. 2025. Large Language Model (LLM)-Driven Document Clustering: Improving Real-Time Security Intelligence Extraction and Threat Analysis. In 2025 IEEE International Conference on Intelligence and Security Informatics (ISI). 7–14. DOI: 10.1109/ISI65680.2025.11201090
Luciana Lourdes Silva, Janio Rosa da Silva, Joao Eduardo Montandon, Marcus Andrade, and Marco Tulio Valente. 2024. Detecting Code Smells using ChatGPT: Initial Insights. In Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (Barcelona, Spain) (ESEM ’24). Association for Computing Machinery, New York, NY, USA, 400–406. DOI: 10.1145/3674805.3690742
Rosalia Tufano, Antonio Mastropaolo, Federica Pepe, Ozren Dabić, Massimiliano Di Penta, and Gabriele Bavota. 2024. Unveiling ChatGPT’s Usage in Open Source Projects: A Mining-based Study. arXiv:2402.16480 [cs.SE] [link]
Nusrat Zahan, Shohanuzzaman Shohan, Dan Harris, and Laurie Williams. 2023. Do Software Security Practices Yield Fewer Vulnerabilities?. In 2023 IEEE/ACM 45th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). 292–303. DOI: 10.1109/ICSE-SEIP58684.2023.00032
Jessy Ayala and Joshua Garcia. 2023. An Empirical Study on Workflows and Security Policies in Popular GitHub Repositories. In 2023 IEEE/ACM 1st International Workshop on Software Vulnerability (SVM). IEEE, 6–9. DOI: 10.1109/svm59160.2023.00006
Noah Bühlmann and Mohammad Ghafari. 2022. How do developers deal with security issue reports on GitHub?. In Proceedings of the 37th ACM/SIGAPP Symposium on Applied Computing (Virtual Event) (SAC ’22). Association for Computing Machinery, New York, NY, USA, 1580–1589. DOI: 10.1145/3477314.3507123
Morakot Choetkiertikul, Sushawapak Kancharoendee, Chanikarn Jongyingyos, Thanat Phichitphanphong, Chaiyong Ragkhitwetsagul, Brittany Reid, Raula Kula, and Thanwadee Sunetnanta. 2026. Security by documentation? characterizing GitHub SECURITY.md policy and their adoption in Python libraries. Empirical Software Engineering 31 (02 2026). DOI: 10.1007/s10664-025-10794-z
Vincenzo De Martino, Joel Castano, Fabio Palomba, Xavier Franch, and Silverio Martinez-Fernandez. 2025. A Framework for Using LLMs for Repository Mining Studies in Empirical Software Engineering . In 2025 IEEE/ACM International Workshop on Methodological Issues with Empirical Studies in Software Engineering (WSESE). IEEE Computer Society, 6–11.
Django Software Foundation. 2026. Django: A high-level Python web framework. [link] GitHub repository. Accessed: 2026-06-17.
GitHub, Inc. 2026. GitHub REST API documentation. [link] Official documentation. Accessed: 2026-06-17.
Google AI for Developers. 2026. Prompting strategies: Example template combining best practices. [link] Gemini API Documentation. Accessed: 2026-06-17.
Google Cloud. 2026. Google Gen AI SDK. [link] Google Cloud Documentation. Accessed: 2026-08-08.
Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. Large Language Models for Software Engineering: A Systematic Literature Review. arXiv:2308.10620 [cs.SE] [link]
OWASP Foundation. 2025. OWASP Top 10:2025. [link] Accessed: 2026-06-17.
Antonio Della Porta, Vincenzo De Martino, Gilberto Recupito, Carmine Iemmino, Gemma Catolino, Dario Di Nucci, and Fabio Palomba. 2024. Using Large Language Models to Support Software Engineering Documentation in Waterfall Life Cycles: Are We There Yet?. In Proceedings of the Ital-IA Intelligenza Artificiale - Thematic Workshops co-located with the 4th CINI National Lab AIIS Conference on Artificial Intelligence (Ital-IA 2024), Naples, Italy, May 29-30, 2024 (CEUR Workshop Proceedings, Vol. 3762), Sergio Di Martino, Carlo Sansone, Elio Masciari, Silvia Rossi, and Michela Gravina (Eds.). CEUR-WS.org, 42–47. [link]
Patrick Serrano, Luwen Huangfu, Chunhua Liao, Dongqing Lin, Kai Williams, Jaden Perleoni, and Thomas Brettin. 2025. Large Language Model (LLM)-Driven Document Clustering: Improving Real-Time Security Intelligence Extraction and Threat Analysis. In 2025 IEEE International Conference on Intelligence and Security Informatics (ISI). 7–14. DOI: 10.1109/ISI65680.2025.11201090
Luciana Lourdes Silva, Janio Rosa da Silva, Joao Eduardo Montandon, Marcus Andrade, and Marco Tulio Valente. 2024. Detecting Code Smells using ChatGPT: Initial Insights. In Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (Barcelona, Spain) (ESEM ’24). Association for Computing Machinery, New York, NY, USA, 400–406. DOI: 10.1145/3674805.3690742
Rosalia Tufano, Antonio Mastropaolo, Federica Pepe, Ozren Dabić, Massimiliano Di Penta, and Gabriele Bavota. 2024. Unveiling ChatGPT’s Usage in Open Source Projects: A Mining-based Study. arXiv:2402.16480 [cs.SE] [link]
Nusrat Zahan, Shohanuzzaman Shohan, Dan Harris, and Laurie Williams. 2023. Do Software Security Practices Yield Fewer Vulnerabilities?. In 2023 IEEE/ACM 45th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). 292–303. DOI: 10.1109/ICSE-SEIP58684.2023.00032
Publicado
11/09/2026
Como Citar
SILVA, Ester; VALENTE, Marco Tulio; BORGES, Hudson.
Automatically Generating Security Guidelines from Pull Request Discussions: The Case of Django Framework. In: WORKSHOP DE VISUALIZAÇÃO, EVOLUÇÃO E MANUTENÇÃO DE SOFTWARE (VEM), 14. , 2026, São Paulo/SP.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2026
.
p. 123-129.
DOI: https://doi.org/10.5753/vem.2026.32980.
