The Role of LLM-Based Tools in Shaping Data Science Student Behavior

• Felipe de S. Siqueira UFCG
• Claudio E. C. Campelo UFCG

DOI: https://doi.org/10.5753/sbie.2025.12715

Resumo

This study evaluates the impact of tools based on Large Language Models (LLMs) in data science education, focusing on how reliance on these tools affects students’ analytical and technical skills. The research examines two main aspects: students’ perceptions of the tools’ impact on their analytical abilities and their behavior during data science activities. The study involved 27 students who participated in a practical exercise using a chatbot and completed questionnaires to assess their perceptions of such tools. The results suggest that students with higher levels of reliance tend to have a more positive perception of the tools, although this perception does not necessarily correlate with better performance in analytical and visualization tasks. Moreover, students who actively explore alternative approaches and modify the generated code perform better in data clustering tasks. These findings suggest that LLM-based tools can support data science education, provided that students are encouraged to develop critical thinking and go beyond the solutions automatically generated by the models.

Referências

Baker, J. and Smith, E. (2023). The impact of large language models on academic integrity and learning outcomes. Journal of Educational Technology, 58:45–60.

Budhiraja, R., Joshi, I., Challa, J. S., Akolekar, H. D., and Kumar, D. (2024). “it’s not like jarvis, but it’s pretty close!”-examining chatgpt’s usage among undergraduate students in computer science. In Proceedings of the 26th Australasian Computing Education Conference, pages 124–133.

Creswell, J. and Creswell, J. (2017). Research Design: Qualitative, Quantitative, and Mixed Methods Approaches. SAGE Publications.

Gil, A. C. (2008). Métodos e técnicas de pesquisa social. 6. ed. Ediitora Atlas SA.

Kirova, V. D., Ku, C. S., Laracy, J. R., and Marlowe, T. J. (2024). Software engineering education must adapt and evolve for an llm environment. In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1, pages 666–672.

Liu, W. and Gonzalez, M. (2024). Ai chatbots in higher education: Effects on student engagement and performance. In Proceedings of the International Conference on Educational Data Mining.

Lopes, C., da Costa, A. B., and Zoppo, B. M. (2024). A influência das metodologias ativas de aprendizagem na promoção da autonomia e inovação em pesquisadores em formação: um estudo transdisciplinar. volume 21, pages e3108–e3108.

Malinka, K., Peresíni, M., Firc, A., Hujnák, O., and Janus, F. (2023). On the educational impact of chatgpt: Is artificial intelligence ready to obtain a university degree? In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1, pages 47–53.

Shen, Y., Ai, X., Soosai Raj, A. G., Leo John, R. J., and Syamkumar, M. (2024). Implications of chatgpt for data science education. In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1, pages 1230–1236.

Sigolo, B. d. O. O. and Casarin, H. d. C. S. (2024). Contribuições da teoria da carga cognitiva para compreensão da sobrecarga informacional: uma revis˜ao de literatura. volume 22, page e024027. SciELO Brasil.

Tu, X., Zou, J., Su, W. J., and Zhang, L. (2023). What should data science education do with large language models?

Zheng, Y. (2023). Chatgpt for teaching and learning: an experience from data science education. In Proceedings of the 24th Annual Conference on Information Technology Education, pages 66–72.