Uma análise do uso de ferramentas de geração de código por alunos de computação
Resumo
O GitHub Copilot é um assistente de código que utiliza inteligência artificial para auxiliar desenvolvedores em suas tarefas de codificação. Por ser uma ferramenta relativamente nova, muitos pesquisadores tem dirigido esforços para avaliar a sua eficiência. No intuito de realizar uma avaliação dessa ferramenta dentro do ambiente acadêmico, foi proposto um experimento, no qual alguns alunos de graduação em computação resolveram problemas simples de programação com e sem o auxilio dessa ferramenta, para assim, avaliar o impacto de ferramentas desse tipo no processo de ensino/aprendizado. Os resultados mostram que os alunos que utilizaram o GitHub Copilot resolveram mais problemas corretamente e em menos tempo. A partir da análise estatística realizada concluiu-se que os tempos médios dos alunos que utilizaram o GitHub Copilot e os que não utilizaram são estatisticamente significativos.
Palavras-chave:
GitHub Copilot, IA Generativa, Large Language Model, GPT-3
Referências
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. arXiv:2005.14165 [cs.CL]
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021. Evaluating Large Language Models Trained on Code. arXiv:2107.03374 [link]
Arghavan Moradi Dakhel, Vahid Majdinasab, Amin Nikanjam, Foutse Khomh, Michel C. Desmarais, Zhen Ming, and Jiang. 2023. GitHub Copilot AI pair programmer: Asset or Liability? arXiv:2206.15331 [cs.SE]
W.W. Daniel. 2000. Applied Nonparametric Statistics. Duxbury. [link].
Pedro Domingos. 2012. A Few Useful Things to Know about Machine Learning. Commun. ACM 55, 10, 78–87
Neil A. Ernst and Gabriele Bavota. 2022. AI-Driven Development Is Here: Should You Worry? IEEE Software 39, 2, 106–110.
GitHub. 2023. Your AI pair programmer. [link]. Acessado: 20 de agos. de 2023.
Shudong Huang, Zhao Kang, Zenglin Xu, and Quanhui Liu. 2021. Robust deep k-means: An effective and simple method for data clustering. Pattern Recognition 117, 107996.
Mlađan Jovanović and Mark Campbell. 2022. Generative Artificial Intelligence: Trends and Prospects. Computer 55, 10, 107–112.
M.H. Kutner. 2005. Applied Linear Statistical Models. McGraw-Hill Irwin
Daniel Leiker, Ashley Ricker Gyllen, Ismail Eldesouky, and Mutlu Cukurova. 2023. Generative AI for Learning: Investigating the Potential of Learning Videos with Synthetic Virtual Instructors. In Artificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners, Doctoral Consortium and Blue Sky, Ning Wang, Genaro RebolledoMendez, Vania Dimitrova, Noboru Matsuda, and Olga C. Santos (Eds.). Springer Nature Switzerland, Cham, 523–529.
Weng Marc Lim, Asanka Gunasekara, Jessica Leigh Pallant, Jason Ian Pallant, and Ekaterina Pechenkina. 2023. Generative AI and the future of education: Ragnarök or reformation? A paradoxical perspective from management educators. The International Journal of Management Education 21, 2, 100790.
T.J. McCabe. 1976. A Complexity Measure. IEEE Transactions on Software Engineering SE-2, 4, 308–320.
Tom Michael Mitchell. 1997. Machine Learning. WCB/McGraw-Hill, Boston, MA. 414 pages.
JunSeong Moon, RaeEun Yang, SoMin Cha, and Seong Baeg Kim. 2023. chatGPT vs Mentor : Programming Language Learning Assistance System for Beginners. In 2023 IEEE 8th International Conference On Software Engineering and Computer Systems (ICSECS). 2023 IEEE 8th International Conference On Software Engineering and Computer Systems (ICSECS), Penang, Malaysia, 106–110.
Arghavan Moradi Dakhel, Vahid Majdinasab, Amin Nikanjam, Foutse Khomh, Michel C. Desmarais, and Zhen Ming (Jack) Jiang. 2023. GitHub Copilot AI pair programmer: Asset or Liability? Journal of Systems and Software 203, 111734.
Ekaterina A. Moroz, Vladimir O. Grizkevich, and Igor M. Novozhilov. 2022. The Potential of Artificial Intelligence as a Method of Software Developer’s Productivity Improvement. In 2022 Conference of Russian Young Researchers in Electrical and Electronic Engineering (ElConRus). 2022 Conference of Russian Young Researchers in Electrical and Electronic Engineering (ElConRus), Saint Petersburg, Russian Federation, 386–390.
Nhan Nguyen and Sarah Nadi. 2022. An Empirical Evaluation of GitHub Copilot’s Code Suggestions. In Proceedings of the 19th International Conference on Mining Software Repositories (Pittsburgh, Pennsylvania) (MSR ’22). Association for Computing Machinery, New York, NY, USA, 1–5.
Alberto S. Nuñez-Varela, Héctor G. Pérez-Gonzalez, Francisco E. Martínez-Perez, and Carlos Soubervielle-Montalvo. 2017. Source code metrics: A systematic mapping study. Journal of Systems and Software 128, 164–197.
NVIDIA. 2023. Large Language Models Explained. [link]. Acessado: 20 de agos. de 2023.
James Prather, Brent N. Reeves, Paul Denny, Brett A. Becker, Juho Leinonen, Andrew Luxton-Reilly, Garrett Powell, James Finnie-Ansley, and Eddie Antonio Santos. 2023.
Ben Puryear and Gina Sprint. 2022. Github Copilot in the Classroom: Learning to Code with AI Assistance. J. Comput. Sci. Coll. 38, 1, 37–47.
Damien Raftery. 2023. Will ChatGPT pass the online quizzes? Adapting an assessment strategy in the age of generative AI. Irish Journal of Technology Enhanced Learning 7.
Linda Rosenberg, Ted Hammer, and Jack Shaw. 1998. Software metrics and reliability. In 9th international symposium on software reliability engineering.
S.J. Russell, P. Norvig, and E. Davis. 2010. Artificial Intelligence: A Modern Approach. Prentice Hal.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2023. Attention Is All You Need. arXiv:1706.03762 [cs.CL]
Michel Wermelinger. 2023. Using GitHub Copilot to Solve Simple Programming Problems. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 (Toronto ON, Canada) (SIGCSE 2023). Association for Computing Machinery, New York, NY, USA, 172–178
Han Xue and Yanmin Niu. 2023. Exercise Generation and Student Cognitive Ability Research Based on ChatGPT and Rasch Model. IEEE Access 11, 1–1.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021. Evaluating Large Language Models Trained on Code. arXiv:2107.03374 [link]
Arghavan Moradi Dakhel, Vahid Majdinasab, Amin Nikanjam, Foutse Khomh, Michel C. Desmarais, Zhen Ming, and Jiang. 2023. GitHub Copilot AI pair programmer: Asset or Liability? arXiv:2206.15331 [cs.SE]
W.W. Daniel. 2000. Applied Nonparametric Statistics. Duxbury. [link].
Pedro Domingos. 2012. A Few Useful Things to Know about Machine Learning. Commun. ACM 55, 10, 78–87
Neil A. Ernst and Gabriele Bavota. 2022. AI-Driven Development Is Here: Should You Worry? IEEE Software 39, 2, 106–110.
GitHub. 2023. Your AI pair programmer. [link]. Acessado: 20 de agos. de 2023.
Shudong Huang, Zhao Kang, Zenglin Xu, and Quanhui Liu. 2021. Robust deep k-means: An effective and simple method for data clustering. Pattern Recognition 117, 107996.
Mlađan Jovanović and Mark Campbell. 2022. Generative Artificial Intelligence: Trends and Prospects. Computer 55, 10, 107–112.
M.H. Kutner. 2005. Applied Linear Statistical Models. McGraw-Hill Irwin
Daniel Leiker, Ashley Ricker Gyllen, Ismail Eldesouky, and Mutlu Cukurova. 2023. Generative AI for Learning: Investigating the Potential of Learning Videos with Synthetic Virtual Instructors. In Artificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners, Doctoral Consortium and Blue Sky, Ning Wang, Genaro RebolledoMendez, Vania Dimitrova, Noboru Matsuda, and Olga C. Santos (Eds.). Springer Nature Switzerland, Cham, 523–529.
Weng Marc Lim, Asanka Gunasekara, Jessica Leigh Pallant, Jason Ian Pallant, and Ekaterina Pechenkina. 2023. Generative AI and the future of education: Ragnarök or reformation? A paradoxical perspective from management educators. The International Journal of Management Education 21, 2, 100790.
T.J. McCabe. 1976. A Complexity Measure. IEEE Transactions on Software Engineering SE-2, 4, 308–320.
Tom Michael Mitchell. 1997. Machine Learning. WCB/McGraw-Hill, Boston, MA. 414 pages.
JunSeong Moon, RaeEun Yang, SoMin Cha, and Seong Baeg Kim. 2023. chatGPT vs Mentor : Programming Language Learning Assistance System for Beginners. In 2023 IEEE 8th International Conference On Software Engineering and Computer Systems (ICSECS). 2023 IEEE 8th International Conference On Software Engineering and Computer Systems (ICSECS), Penang, Malaysia, 106–110.
Arghavan Moradi Dakhel, Vahid Majdinasab, Amin Nikanjam, Foutse Khomh, Michel C. Desmarais, and Zhen Ming (Jack) Jiang. 2023. GitHub Copilot AI pair programmer: Asset or Liability? Journal of Systems and Software 203, 111734.
Ekaterina A. Moroz, Vladimir O. Grizkevich, and Igor M. Novozhilov. 2022. The Potential of Artificial Intelligence as a Method of Software Developer’s Productivity Improvement. In 2022 Conference of Russian Young Researchers in Electrical and Electronic Engineering (ElConRus). 2022 Conference of Russian Young Researchers in Electrical and Electronic Engineering (ElConRus), Saint Petersburg, Russian Federation, 386–390.
Nhan Nguyen and Sarah Nadi. 2022. An Empirical Evaluation of GitHub Copilot’s Code Suggestions. In Proceedings of the 19th International Conference on Mining Software Repositories (Pittsburgh, Pennsylvania) (MSR ’22). Association for Computing Machinery, New York, NY, USA, 1–5.
Alberto S. Nuñez-Varela, Héctor G. Pérez-Gonzalez, Francisco E. Martínez-Perez, and Carlos Soubervielle-Montalvo. 2017. Source code metrics: A systematic mapping study. Journal of Systems and Software 128, 164–197.
NVIDIA. 2023. Large Language Models Explained. [link]. Acessado: 20 de agos. de 2023.
James Prather, Brent N. Reeves, Paul Denny, Brett A. Becker, Juho Leinonen, Andrew Luxton-Reilly, Garrett Powell, James Finnie-Ansley, and Eddie Antonio Santos. 2023.
Ben Puryear and Gina Sprint. 2022. Github Copilot in the Classroom: Learning to Code with AI Assistance. J. Comput. Sci. Coll. 38, 1, 37–47.
Damien Raftery. 2023. Will ChatGPT pass the online quizzes? Adapting an assessment strategy in the age of generative AI. Irish Journal of Technology Enhanced Learning 7.
Linda Rosenberg, Ted Hammer, and Jack Shaw. 1998. Software metrics and reliability. In 9th international symposium on software reliability engineering.
S.J. Russell, P. Norvig, and E. Davis. 2010. Artificial Intelligence: A Modern Approach. Prentice Hal.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2023. Attention Is All You Need. arXiv:1706.03762 [cs.CL]
Michel Wermelinger. 2023. Using GitHub Copilot to Solve Simple Programming Problems. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 (Toronto ON, Canada) (SIGCSE 2023). Association for Computing Machinery, New York, NY, USA, 172–178
Han Xue and Yanmin Niu. 2023. Exercise Generation and Student Cognitive Ability Research Based on ChatGPT and Rasch Model. IEEE Access 11, 1–1.
Publicado
22/04/2024
Como Citar
LIRA, Werney Ayala Luz; SANTOS NETO, Pedro de A. dos; OSORIO, Luiz Fernando Mendes.
Uma análise do uso de ferramentas de geração de código por alunos de computação. In: SIMPÓSIO BRASILEIRO DE EDUCAÇÃO EM COMPUTAÇÃO (EDUCOMP), 4. , 2024, Evento Online.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2024
.
p. 63-71.
DOI: https://doi.org/10.5753/educomp.2024.237427.
