Hello, Freire! How Can You Help Me? A Smart Chatbot to Enhance Access to Academic Opportunities and Institutional Matters
Resumo
Context: Universities offer various resources and services that students often fail to utilize due to difficulties in accessing relevant information. Chatbots can improve access and interaction between students and institutions. Problem: Students struggle to find information about academic opportunities and institutional matters due to fragmented data, limiting engagement. Solution: This paper introduces Freire Assistant (Freire), a chatbot system powered by a Large Language Model (LLM) and built using Retrieval Augmented Generation (RAG) pipelines. Freire aims to streamline access to academic information through a fast, intuitive communication channel. IS theory: Thiswork was developed under the aegis of Soft Systems Theory, as it addresses the complexity of accessing academic information as a sociotechnical problem. Method: To evaluate Freire’s effectiveness in providing accurate and relevant information about academic opportunities and institutional matters, we conducted a case study at a Brazilian university. We quantitatively evaluated the contextual quality of Freire’s responses using the DeepEval framework. Additionally, we qualitatively evaluated Freire’s usability with 45 students through the Chatbot Usability Questionnaire (CUQ) which measures aspects related to Chatbot Personality, Onboarding, User Experience, and Error Handling. Summary of Results: The findings demonstrated Freire’s efficiency in generating contextually relevant and accurate responses. Furthermore, usability scores were generally high due to their simple interface and conversation-driven functionality. Contributions and Impact in the IS area: This study highlights how smart chatbots can enhance user experience and access to information in universities, providing a replicable model for other institutions facing similar challenges.
Palavras-chave:
chatbot, education context, innovation, students, llm
Referências
Suha Khalil Assayed, Khaled Shaalan, and Manar Alkhatib. 2022. A chatbot intent classifier for supporting high school students. EAI Endorsed Transactions on Scalable Information Systems 10, 3 (2022).
John Brooke et al. 1996. SUS-A quick and dirty usability scale. Usability evaluation in industry 189, 194 (1996), 4–7.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, et al. 2020. Language models are few-shot learners. In 34th International Conference on Neural Information Processing Systems (Vancouver, BC, Canada) (NIPS ’20). Curran Associates Inc., Red Hook, NY, USA, Article 159, 25 pages.
Victor R Basili-Gianluigi Caldiera and H Dieter Rombach. 1994. Goal question metric paradigm. Encyclopedia of software engineering 1, 528-532 (1994), 6.
Yogi Wisesa Chandra and Suyanto Suyanto. 2019. Indonesian chatbot of university admission using a question answering system based on sequence-to-sequence model. Procedia Computer Science 157 (2019), 367–374.
Lijia Chen, Pingping Chen, and Zhijian Lin. 2020. Artificial intelligence in education: A review. Ieee Access 8 (2020), 75264–75278.
Yu-Hung Chien and Chun-Kai Yao. 2020. Development of an ai userbot for engineering design education using an intent and flow combined framework. Applied Sciences 10, 22 (2020), 7970.
Juliet Corbin and Anselm Strauss. 2014. Basics of qualitative research: Techniques and procedures for developing grounded theory. Sage publications.
Mozilla Corporation. 2024. Web Speech API. [link]. (Accessed on 17/10/2024).
Samuel Holmes, Anne Moorhead, Raymond Bond, Huiru Zheng, Vivien Coates, and Michael McTear. 2019. Usability testing of a healthcare chatbot: Can we use conventional methods to assess conversational user interfaces?. In 31st European Conference on Cognitive Ergonomics. 207–214.
Confident AI Inc. 2024. Deepeval documentation. [link]. (Accessed on 17/10/2024).
Jina. 2024. Jina framework. [link]. (Accessed on 17/10/2024).
langchain. 2024. Langchain framework. [link]. (Accessed on 17/10/2024).
Langchain. 2024. RecursiveCharacterTextSplitter langchain documentation. [link]
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems 33 (2020), 9459–9474.
R. Likert. 1932. A Technique for the Measurement of Attitudes. Number Nº 136-165 in A Technique for the Measurement of Attitudes. Archives of Psychology.
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out. 74–81.
Rongxin Liu, Carter Zenke, Charlie Liu, Andrew Holmes, Patrick Thornton, and David J Malan. 2024. Teaching CS50 with AI: leveraging generative artificial intelligence in computer science education. In 55th ACM Technical Symposium on Computer Science Education V. 1. 750–756.
Bei Luo, Raymond YK Lau, Chunping Li, and Yain-Whar Si. 2022. A critical reviewof state-of-the-art chatbot designs and applications. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 12, 1 (2022), e1434.
Yu A Malkov and Dmitry A Yashunin. 2018. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. IEEE transactions on pattern analysis and machine intelligence 42, 4 (2018), 824–836.
C Martinez-Araneda,MGutiérrez, D Maldonado, P Gómez, A Segura, and C Vidal-Castro. 2024. Designing a Chatbot to Support Problem-Solving in a Programming Course. In INTED2024 Proceedings. IATED, 966–975.
Ábalos N. Martín J., Muñoz-Romero C. 2024. chatbottest - Improve your chatbot’s design. [link]. (Accessed on 17/10/2024).
Arthur Willame Mesquita, Carlos Freire, Antonio Gomes, Bruna Amazonas, and Anderson Uchôa. 2025. Replication package for the paper: "Hello, Freire! How Can You Help Me? A Smart Chatbot to Enhance Access to Academic Opportunities and Institutional Matters". DOI: 10.5281/zenodo.15014430
Subash Neupane, Elias Hossain, Jason Keith, Himanshu Tripathi, Farbod Ghiasi, Noorbakhsh Amiri Golilarz, Amin Amirlatifi, Sudip Mittal, and Shahram Rahimi. 2024. From Questions to Insightful Answers: Building an Informed Chatbot for University Resources. arXiv preprint arXiv:2405.08120 (2024).
Pedro Filipe Oliveira and Paulo Matos. 2023. Introducing a chatbot to the web portal of a higher education institution to enhance student interaction. Engineering Proceedings 56, 1 (2023), 128.
OpenAI. 2023. GPT-4 Technical Report. [link]
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In 40th annual meeting of the Association for Computational Linguistics. 311–318.
Anupam Purwar et al. 2024. Evaluating the Efficacy of Open-Source LLMs in Enterprise-Specific RAG Systems: A Comparative Study of Performance and Scalability. arXiv preprint arXiv:2406.11424 (2024).
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research 21, 140 (2020), 1–67.
J Jinu Sophia and T Prem Jacob. 2021. Edubot-a chatbot for education in covid-19 pandemic and vqabot comparison. In 2nd International Conference on Electronics and Sustainable Communication Systems (ICESC). IEEE, 1707–1714.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023).
Yujia Wang, Shuqi Liu, and Linqi Song. 2022. Designing an educational chatbot with joint intent classification and slot filling. In International Conference on Teaching, Assessment and Learning for Engineering (TALE). IEEE, 381–388.
Claes Wohlin, Per Runeson, Martin Höst, Magnus C Ohlsson, Björn Regnell, and Anders Wesslén. 2012. Experimentation in software engineering. Springer Science & Business Media.
Jia-Yu Yao, Kun-Peng Ning, Zhen-Hui Liu, Mu-Nan Ning, Yu-Yang Liu, and Li Yuan. 2023. Llm lies: Hallucinations are not bugs, but features as adversarial examples. arXiv preprint arXiv:2310.01469 (2023).
John Brooke et al. 1996. SUS-A quick and dirty usability scale. Usability evaluation in industry 189, 194 (1996), 4–7.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, et al. 2020. Language models are few-shot learners. In 34th International Conference on Neural Information Processing Systems (Vancouver, BC, Canada) (NIPS ’20). Curran Associates Inc., Red Hook, NY, USA, Article 159, 25 pages.
Victor R Basili-Gianluigi Caldiera and H Dieter Rombach. 1994. Goal question metric paradigm. Encyclopedia of software engineering 1, 528-532 (1994), 6.
Yogi Wisesa Chandra and Suyanto Suyanto. 2019. Indonesian chatbot of university admission using a question answering system based on sequence-to-sequence model. Procedia Computer Science 157 (2019), 367–374.
Lijia Chen, Pingping Chen, and Zhijian Lin. 2020. Artificial intelligence in education: A review. Ieee Access 8 (2020), 75264–75278.
Yu-Hung Chien and Chun-Kai Yao. 2020. Development of an ai userbot for engineering design education using an intent and flow combined framework. Applied Sciences 10, 22 (2020), 7970.
Juliet Corbin and Anselm Strauss. 2014. Basics of qualitative research: Techniques and procedures for developing grounded theory. Sage publications.
Mozilla Corporation. 2024. Web Speech API. [link]. (Accessed on 17/10/2024).
Samuel Holmes, Anne Moorhead, Raymond Bond, Huiru Zheng, Vivien Coates, and Michael McTear. 2019. Usability testing of a healthcare chatbot: Can we use conventional methods to assess conversational user interfaces?. In 31st European Conference on Cognitive Ergonomics. 207–214.
Confident AI Inc. 2024. Deepeval documentation. [link]. (Accessed on 17/10/2024).
Jina. 2024. Jina framework. [link]. (Accessed on 17/10/2024).
langchain. 2024. Langchain framework. [link]. (Accessed on 17/10/2024).
Langchain. 2024. RecursiveCharacterTextSplitter langchain documentation. [link]
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems 33 (2020), 9459–9474.
R. Likert. 1932. A Technique for the Measurement of Attitudes. Number Nº 136-165 in A Technique for the Measurement of Attitudes. Archives of Psychology.
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out. 74–81.
Rongxin Liu, Carter Zenke, Charlie Liu, Andrew Holmes, Patrick Thornton, and David J Malan. 2024. Teaching CS50 with AI: leveraging generative artificial intelligence in computer science education. In 55th ACM Technical Symposium on Computer Science Education V. 1. 750–756.
Bei Luo, Raymond YK Lau, Chunping Li, and Yain-Whar Si. 2022. A critical reviewof state-of-the-art chatbot designs and applications. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 12, 1 (2022), e1434.
Yu A Malkov and Dmitry A Yashunin. 2018. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. IEEE transactions on pattern analysis and machine intelligence 42, 4 (2018), 824–836.
C Martinez-Araneda,MGutiérrez, D Maldonado, P Gómez, A Segura, and C Vidal-Castro. 2024. Designing a Chatbot to Support Problem-Solving in a Programming Course. In INTED2024 Proceedings. IATED, 966–975.
Ábalos N. Martín J., Muñoz-Romero C. 2024. chatbottest - Improve your chatbot’s design. [link]. (Accessed on 17/10/2024).
Arthur Willame Mesquita, Carlos Freire, Antonio Gomes, Bruna Amazonas, and Anderson Uchôa. 2025. Replication package for the paper: "Hello, Freire! How Can You Help Me? A Smart Chatbot to Enhance Access to Academic Opportunities and Institutional Matters". DOI: 10.5281/zenodo.15014430
Subash Neupane, Elias Hossain, Jason Keith, Himanshu Tripathi, Farbod Ghiasi, Noorbakhsh Amiri Golilarz, Amin Amirlatifi, Sudip Mittal, and Shahram Rahimi. 2024. From Questions to Insightful Answers: Building an Informed Chatbot for University Resources. arXiv preprint arXiv:2405.08120 (2024).
Pedro Filipe Oliveira and Paulo Matos. 2023. Introducing a chatbot to the web portal of a higher education institution to enhance student interaction. Engineering Proceedings 56, 1 (2023), 128.
OpenAI. 2023. GPT-4 Technical Report. [link]
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In 40th annual meeting of the Association for Computational Linguistics. 311–318.
Anupam Purwar et al. 2024. Evaluating the Efficacy of Open-Source LLMs in Enterprise-Specific RAG Systems: A Comparative Study of Performance and Scalability. arXiv preprint arXiv:2406.11424 (2024).
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research 21, 140 (2020), 1–67.
J Jinu Sophia and T Prem Jacob. 2021. Edubot-a chatbot for education in covid-19 pandemic and vqabot comparison. In 2nd International Conference on Electronics and Sustainable Communication Systems (ICESC). IEEE, 1707–1714.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023).
Yujia Wang, Shuqi Liu, and Linqi Song. 2022. Designing an educational chatbot with joint intent classification and slot filling. In International Conference on Teaching, Assessment and Learning for Engineering (TALE). IEEE, 381–388.
Claes Wohlin, Per Runeson, Martin Höst, Magnus C Ohlsson, Björn Regnell, and Anders Wesslén. 2012. Experimentation in software engineering. Springer Science & Business Media.
Jia-Yu Yao, Kun-Peng Ning, Zhen-Hui Liu, Mu-Nan Ning, Yu-Yang Liu, and Li Yuan. 2023. Llm lies: Hallucinations are not bugs, but features as adversarial examples. arXiv preprint arXiv:2310.01469 (2023).
Publicado
19/05/2025
Como Citar
MESQUITA, Arthur Willame; FREIRE, Carlos; GOMES, Antônio; AMAZONAS, Bruna; UCHÔA, Anderson.
Hello, Freire! How Can You Help Me? A Smart Chatbot to Enhance Access to Academic Opportunities and Institutional Matters. In: SIMPÓSIO BRASILEIRO DE SISTEMAS DE INFORMAÇÃO (SBSI), 21. , 2025, Recife/PE.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2025
.
p. 469-478.
DOI: https://doi.org/10.5753/sbsi.2025.246546.
