Same Volatility, Different Domains: A Reproduction of CodeLL Across ML and Software Engineering Projects
Resumo
Code change datasets rarely capture the long-term temporal dimension of repositories, which limits their use in lifelong learning scenarios for code language models. The CodeLL dataset fills this gap by mapping changes in files, methods, and API calls across the entire release history of 71 Python machine learning (ML) projects. The released dataset, however, ships only descriptive analyses without aggregated churn metrics, covers a single application domain, and its target repositories have continued to evolve since the original mining in 2023/24. This paper addresses these gaps through a reproduction and extension of the CodeLL study: we re-extracted six projects from Software Heritage, three ML projects and three software engineering (SE) tool projects, totaling 615 releases, and re-executed the original mining pipeline and statistics notebook on the newly mined data. The pipeline ran end to end on all six projects, and for Gensim—the only project allowing a direct comparison against the published figures—the descriptive patterns were reproduced, with coefficient deviations compatible with repository growth between the two mining dates. Our extension adds two churn metrics for API calls and methods, plus two derived analyses (a volatility ranking and a per-release API change rate). The cross-domain comparison showed that this first operationalization does not distinguish the six sampled projects: its value ranges are nearly identical in ML and SE. Codebase size and maturity are a candidate explanation, but the available size proxy confirms it only in part, so it remains a hypothesis. We further document qualitative evidence of real, datable API evolution (Python 2→3 migration) in both domains, and report the practical difficulties of the reproduction, notably the rate limiting of Software Heritage.
Referências
Victor R. Basili, Gianluigi Caldiera, and H. Dieter Rombach. 1994. Goal Question Metric Paradigm. In Encyclopedia of Software Engineering, John J. Marciniak (Ed.). Vol. 1. John Wiley & Sons, New York, NY, 528–532.
Roberto Di Cosmo and Stefano Zacchiroli. 2017. Software Heritage: Why and How to Preserve Software Source Code. In Proceedings of the 14th International Conference on Digital Preservation (iPRES). Kyoto, Japan.
Danny Dig and Ralph Johnson. 2006. How do APIs evolve? A story of refactoring. Journal of Software Maintenance and Evolution: Research and Practice 18, 2 (2006), 83–107. DOI: 10.1002/smr.328
Malinda Dilhara, Ameya Ketkar, and Danny Dig. 2021. Understanding Software-2.0: A Study of Machine Learning Library Usage and Evolution. ACM Transactions on Software Engineering and Methodology 30, 4 (2021), 55:1–55:42. DOI: 10.1145/3453478
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020. The Pile: An 800GB Dataset of Diverse Text for Language Modeling. arXiv preprint arXiv:2101.00027 (2020).
ShuzhengGao, Hongyu Zhang, CuiyunGao, and ChaozhengWang. 2023. Keeping Pace with Ever-Increasing Data: Towards Continual Learning of Code Intelligence Models. In Proceedings of the 45th International Conference on Software Engineering (ICSE ’23). 30–42. DOI: 10.1109/ICSE48619.2023.00015
Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019. CodeSearchNet Challenge: Evaluating the State of Semantic Code Search. arXiv preprint arXiv:1909.09436 (2019).
Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, Leandro von Werra, and Harm de Vries. 2022. The Stack: 3 TB of Permissively Licensed Source Code. arXiv preprint arXiv:2211.15533 (2022).
Romain Lefeuvre, Jessie Galasso, Benoit Combemale, Houari Sahraoui, and Stefano Zacchiroli. 2023. Fingerprinting and Building Large Reproducible Datasets. In Proceedings of the 2023 ACM Conference on Reproducibility and Replicability (ACM REP ’23). ACM, New York, NY, USA, 27–36. DOI: 10.1145/3589806.3600043
Meir M. Lehman. 1980. Programs, Life Cycles, and Laws of Software Evolution. Proc. IEEE 68, 9 (1980), 1060–1076. DOI: 10.1109/PROC.1980.11805
Bo Lin, Shangwen Wang, Zhongxin Liu, Yepang Liu, Xin Xia, and Xiaoguang Mao. 2023. CCT5: A Code-Change-Oriented Pre-Trained Model. arXiv preprint arXiv:2305.10785 (2023).
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. 2021. CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation. arXiv preprint arXiv:2102.04664 (2021).
Tyler McDonnell, Baishakhi Ray, and Miryung Kim. 2013. An Empirical Study of API Stability and Adoption in the Android Ecosystem. In Proceedings of the 29th IEEE International Conference on Software Maintenance (ICSM). IEEE, 70–79. DOI: 10.1109/ICSM.2013.18
Martin Monperrus, Matias Martinez, He Ye, Fernanda Madeiral, Thomas Durieux, and Zhongxing Yu. 2021. Megadiff: A Dataset of 600k Java Source Code Changes Categorized by Diff Size. arXiv preprint arXiv:2108.04631 (2021).
Meiyappan Nagappan, Thomas Zimmermann, and Christian Bird. 2013. Diversity in Software Engineering Research. In Proceedings of the 9th Joint Meeting on Foundations of Software Engineering (ESEC/FSE). ACM, New York, NY, USA, 466–476. DOI: 10.1145/2491411.2491415
Nachiappan Nagappan and Thomas Ball. 2005. Use of Relative Code Churn Measures to Predict SystemDefect Density. In Proceedings of the 27th International Conference on Software Engineering (ICSE). ACM, New York, NY, USA, 284–292. DOI: 10.1145/1062455.1062514
Martin Weyssow, Claudio Di Sipio, Davide Di Ruscio, and Houari Sahraoui. 2024. CodeLL: A Lifelong Learning Dataset to Support the Co-Evolution of Data and Language Models of Code. In Proceedings of the 21st International Conference on Mining Software Repositories (MSR ’24). ACM, Lisbon, Portugal, 637–641. DOI: 10.1145/3643991.3644864
Martin Weyssow, Claudio Di Sipio, Davide Di Ruscio, and Houari Sahraoui. 2024. CodeLL-Dataset: source code and artifact repository. [link]. Accessed: July 2026.
Martin Weyssow, Xin Zhou, Kisub Kim, David Lo, and Houari Sahraoui. 2023. On the Usage of Continual Learning for Out-of-Distribution Generalization in Pre-trained Language Models of Code. arXiv preprint arXiv:2305.04106 (2023). DOI: 10.48550/arXiv.2305.04106
