What Developers Ask to ChatGPT in GitHub Pull Requests? an Exploratory Study

• Julyanara R. Silva IFTM
• Carlos Eduardo C. Dantas IFTM
• Marcelo A. Maia UFU

DOI: https://doi.org/10.5753/vem.2024.3913

Resumo

The emergence of Large Language Models (LLMs), such as ChatGPT, has introduced a new set of tools to support software developers in solving programming tasks. However, our understanding of the interactions (i.e., prompts) between developers and ChatGPT that result in contributions to the codebase remains limited. To explore this limitation, we conducted a manual evaluation of 155 valid ChatGPT share links extracted from 139 merged Pull Requests (PRs), revealing the interactions between developers and reviewers with ChatGPT that led to merges into the main codebase. Our results produced a catalog of 14 types of ChatGPT requests categorized into four main groups. We found a significant number of requests involving code review and the implementation of code snippets based on specific tasks. Developers also sought to clarify doubts by requesting technical explanations or by asking for text refinements for their web pages. Furthermore, we verified that prompts involving code generation generally required more interactions to produce the desired answer compared to prompts requesting text review or technical information.

Referências

Chouchen, M., Bessghaier, N., Begoug, M., Ouni, A., AlOmar, E. A., and Mkaouer, M. W. (2024). How do software developers use chatgpt? an exploratory study on github pull requests.

Dabic, O., Aghajani, E., and Bavota, G. (2021). Sampling projects in github for msr studies. 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR), pages 560–564.

Dantas, C. C., Rocha, A. M., and Maia, M. A. (2023a). How do developers improve code readability? an empirical study of pull requests. In 2023 IEEE International Conference on Software Maintenance and Evolution (ICSME), pages 110–122, Los Alamitos, CA, USA. IEEE Computer Society.

Dantas, C. E., Rocha, A. M., and Maia, M. A. (2023b). Assessing the readability of chatgpt code snippet recommendations: A comparative study. In Proceedings of the XXXVII Brazilian Symposium on Software Engineering, SBES ’23, page 283–292, New York, NY, USA. Association for Computing Machinery.

Ebert, C. and Louridas, P. (2023). Generative AI for software practitioners. IEEE Software, 40:30–38.

Ernst, N. A. and Bavota, G. (2022). AI-driven development is here: Should you worry? IEEE Software, 39(2):106–110.

Hao, H., Hasan, K. A., Qin, H., Macedo, M., Tian, Y., Ding, S. H. H., and Hassan, A. E. (2024). An empirical study on developers shared conversations with chatgpt in github pull requests and issues.

Li Tsz On, Wenxi Zong, Y. W. H. T. Y. W. S.-C. C. J. K. (2023). Nuances are the key: Unlocking chatgpt to find failure-inducing tests with differential prompting. In International Conference on Automated Software Engineering.

Mondal, S., Bappon, S. D., and Roy, C. (2024). Enhancing user interaction in ChatGPT: Characterizing and consolidating multiple prompts for issue resolution. In Proceedings of the International Conference on Mining Software Repositories (MSR 2024).

Pantiuchina, J., Zampetti, F., Scalabrino, S., Piantadosi, V., Oliveto, R., Bavota, G., and Penta, M. D. (2020). Why developers refactor source code: A mining-based study. 29(4).

Qi Guo, Junming Cao, X. X. S. L. X. L. B. C.-X. P. (2024). Exploring the potential of chatgpt in automated code refinement: An empirical study. In International Conference on Software Engineering.

Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. (2018). Language models are unsupervised multitask learners.

Silva, D., Tsantalis, N., and Valente, M. T. (2016). Why we refactor? confessions of github contributors. CoRR, abs/1607.02459.

Silva, J. R., Dantas, C. E. C., and Maia, M. A. (2024). What Developers Ask to ChatGPT in GitHub Pull Requests? an Exploratory Study. DOI: 10.5281/zenodo.13346101.

Spencer, D. and Garrett, J. (2009). Card Sorting: Designing Usable Categories. Rosenfeld Media.

Tufano, R., Mastropaolo, A., Pepe, F., Dabić, O., Penta, M. D., and Bavota, G. (2024). Unveiling ChatGPT’s usage in open source projects: A mining-based study.

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need.

Xiao, T., Treude, C., Hata, H., and Matsumoto, K. (2024). Devgpt: Studying developer-chatgpt conversations. In Proceedings of the International Conference on Mining Software Repositories (MSR 2024).