Literature-Grounded Generation of README Files with Large Language Models: A Dual Human–LLM Evaluation

Allan Assis de Andrade; Talita Vieira Ribeiro

doi:10.5753/ise.2026.32979

Allan Assis de Andrade UERJ
Talita Vieira Ribeiro UERJ

DOI: https://doi.org/10.5753/ise.2026.32979

Resumo

README files are the primary entry point to a software project, yet they are frequently incomplete or outdated. Large Language Models (LLMs) can automate their creation, but existing generators either leave the document structure to the model or fix it in templates with no documented basis in studies of README content. This paper presents README-Gen, a generation approach whose README structure is derived from studies of README content and from the Theory of Robust API Knowledge (ATRAK). We apply it, backed by GPT-4.1-mini, to twelve GitHub repositories across three languages, and assess the output with a dual protocol pairing a human assessment with an automated evaluator (Claude Fable 5) driven by a rule-based per-section rubric; the two agree substantially (Krippendorff’s 𝛼 = 0.90). Structured prompting reliably enforces the expected structure (98% completeness) and full ATRAK adherence, and README-Gen outscores README-AI, under the same model and rubric, on the user-oriented dimension that rubric measures (78% vs. 48% correctness). Correctness, however, is stratified by repository popularity (99% vs. 28%, human evaluation), which we report as an exploratory association with how much information about a project exists outside its repository.

Palavras-chave: Software documentation, API knowledge, README generation, large language models, LLM-as-a-judge

Referências

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, et al. 2020. Language models are few-shot learners. In Proceedings of the 34th International Conference on Neural Information Processing Systems (Vancouver, BC, Canada) (NIPS ’20). Curran Associates Inc., Red Hook, NY, USA, Article 159, 25 pages.

Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramèr, and Chiyuan Zhang. 2023. Quantifying Memorization Across Neural Language Models. In The Eleventh International Conference on Learning Representations (ICLR). [link]

Xing Cui, Jingzheng Wu, Zhiyuan Li, Tianyue Luo, and Xiang Ling. 2025. RM-Genie: An LLM-Based Agent Framework for Open Source Software README Generation. In 2025 IEEE International Conference on Software Maintenance and Evolution (ICSME). 505–516.

Thu T. H. Doan, Phuong T. Nguyen, Juri Di Rocco, and Davide Di Ruscio. 2023. Too long; didn’t read: Automatic summarization of GitHub README.MD with Transformers. In Proceedings of the 27th International Conference on Evaluation and Assessment in Software Engineering (Oulu, Finland) (EASE ’23). Association for Computing Machinery, New York, NY, USA, 267–272. DOI: 10.1145/3593434.3593448

Karl Fogel. 2017. Producing Open Source Software: How to Run a Successful Free Software Project (second ed.). O’Reilly Media, Sebastopol, CA, USA. [link].

Andrew Forward and Timothy C. Lethbridge. 2002. The relevance of software documentation, tools and technologies: a survey. In Proceedings of the 2002 ACM Symposium on Document Engineering (DocEng ’02). Association for Computing Machinery, New York, NY, USA, 26–33. DOI: 10.1145/585058.585065

GitHub. 2017. Open Source Survey. [link]. Accessed: 2026-07-10.

Shohei Ikeda, Akinori Ihara, Raula Gaikovina Kula, and Kenichi Matsumoto. 2019. An Empirical Study of README contents for JavaScript Packages. IEICE Transactions on Information and Systems E102.D, 2 (Feb. 2019), 280–288. DOI: 10.1587/transinf.2018EDP7071

Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of Hallucination in Natural Language Generation. Comput. Surveys 55, 12, Article 248 (2023), 38 pages. DOI: 10.1145/3571730

Naoki Kobayakawa and Kenichi Yoshida. 2017. How GitHub Contributing.md Contributes to Contributors. In 2017 IEEE 41st Annual Computer Software and Applications Conference (COMPSAC). IEEE, 694–696. DOI: 10.1109/compsac.2017.139

Yuta Koreeda, Terufumi Morishita, Osamu Imaichi, and Yasuhiro Sogawa. 2023. LARCH: Large Language Model-based Automatic Readme Creation with Heuristics. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management (CIKM ’23). ACM, New York, NY, USA, 5066–5070. DOI: 10.1145/3583780.3614744

Yuyang Liu, Ehsan Noei, and Kelly Lyons. 2022. How ReadMe files are structured in open source Java projects. Information and Software Technology 148 (2022), 106924. DOI: 10.1016/j.infsof.2022.106924

Gede Artha Prana, Christoph Treude, Ferdian Thung, Thushari Atapattu, and David Lo. 2019. Categorizing the Content of GitHub README Files. Empirical Software Engineering 24, 3 (June 2019), 1296–1327. DOI: 10.1007/s10664-018-9660-3

Eli Salamie. 2023. README-AI: A README File Generator, Powered by AI. [link]. Accessed: 2026-06-18.

Stack Overflow. 2024. 2024 Developer Survey: Technology. [link]. Accessed: 2026-07-10.

Kyle Thayer, Sarah E. Chasins, and Amy J. Ko. 2021. A Theory of Robust API Knowledge. ACM Trans. Comput. Educ. 21, 1, Article 8 (Jan. 2021), 32 pages. DOI: 10.1145/3444945

Jianyu Wu, Hao He, Wenxin Xiao, Kai Gao, and Minghui Zhou. 2022. Demystifying Software Release Note Issues on GitHub. In Proceedings of the 30th IEEE/ACM International Conference on Program Comprehension (ICPC ’22). Association for Computing Machinery, New York, NY, USA, 602–613. DOI: 10.1145/3524610.3527919

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, et al. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track. [link]