Study case: Use of ChatGPT for programming problem-solving

• Débora Souza UFCG
• Rohit Gheyi UFCG

DOI: https://doi.org/10.5753/cbsoft_estendido.2023.235666

Abstract

The generation of programs from natural language aims to transform sentences into programming code. ChatGPT, based on the GPT-3 model, is a chatbot capable of generating text and code like a human. To assess its reliability in the programming context, 100 programming problems were randomly selected, and distributed among Easy, Intermediate, and Hard levels of complexity from popular platforms for the model to solve. Over 3 attempts, ChatGPT correctly solved 71 problems, with 50 from the LeetCode platform and 21 from the BeeCrowd platform. The remaining 29 unsolved problems were mostly distributed among Intermediate and Hard levels of complexity. In this context, the study demonstrates that the use of ChatGPT can be beneficial for programmers as the model can provide examples of well-crafted code and offer solutions to their challenges. However, it also highlights that ChatGPT does not achieve 100% accuracy; therefore, programmers should not consider it a complete replacement.

Keywords: Software metrics, software quality, software testing and verification

References

ChatGPT. Disponível em: [link]. Acesso em: Setembro 2023.

Getting started with GitHub Copilot. Disponível em: [link]. Acesso em: Setembro 2023.

ChatGPT. Disponível em: [link]. Acesso em: Setembro 2023.

Recruitment team unwittingly recommends ChatGPT for job interview. Disponível em: [link]. Acesso em: Setembro 2023.

Plataforma LeetCode. Disponível em: [link]. Acesso em: Setembro 2023.

Plataforma BeeCrowd. Disponível em: [link]. Acesso em: Setembro 2023.

ELO Rating System. Disponível em: [link]. Acesso em: Setembro 2023.

FAQS Judge - BeeCrowd. Disponível em: [link]. Acesso em: Setembro 2023.

FAQS Answer - BeeCrowd. Disponível em: [link]. Acesso em: Setembro 2023.

Concorrente ou aliado? Especialista defende que ChatGPT pode alavancar a carreira de programador. Disponível em: [link]. Acesso em: Setembro 2023.

GPT-4. Disponível em: [link]. Acesso em: Setembro 2023.

Rohit Gheyi, Paulo Borba. Refactoring Alloy Specifications. In Cavalcanti, A., Machado, P., eds.: Electronic Notes in Theoretical Computer Science, Proceedings of the Brazilian Workshop on Formal Methods. Volume 95. Elsevier (2004) 227–243.

Categorias dos problemas da plataforma BeeCrowd. Disponível em: [link]. Acesso em: Setembro 2023.

Débora Souza, Rohit Gheyi. Questões submetidas ao ChatGPT. Disponível em: [link]. Acesso em: Setembro 2023.

F. Medeiros et al.. An empirical study on configuration-related issues: investigating undeclared and unused identifiers. GPCE 2015, pages 35–44, 2015.

F. Medeiros et al.. Investigating misunderstanding code patterns in C open-source software projects. Empirical Software Engineering, 24:1693–1726, 2019.

Samia Kabir, David N. Udo-Imeh, Bonan Kou, Tianyi Zhang. Who Answers It Better? An In-Depth Analysis of ChatGPT and Stack Overflow Answers to Software Engineering Questions. Disponível em: [link]. Acesso em: Setembro 2023.

Raphaël Khoury, Anderson Avila, Jacob Brunelle, Baba Camara. How Secure is Code Generated by ChatGPT?. Disponível em: [link]. Acesso em: Setembro 2023.