Collaborative Feedback Generation in Intelligent Tutoring Systems through Collective Intelligence

• Thyago Tenório Universidade Federal do Alagoas (UFAL)
• Seiji Isotani University of Pennsylvania https://orcid.org/0000-0003-1574-0784

DOI: https://doi.org/10.5753/sbie.2026.28181

Resumo

Inner loops play a critical role in intelligent tutoring systems (ITS) by providing the step-by-step feedback necessary to guide students through tasks and enhance their learning experience. However, delivering this detailed feedback requires highly granular domain knowledge, placing an substantial burden on domain experts. While automatic approaches are typically limited to specific contexts, both manual and automatic methods often ignore the potential of student interactions within the system. To address this challenge, it was proposed a novel, hybrid approach to designing the knowledge model with inner loops by leveraging students' collective intelligence through data derived from their interactions. The approach was evaluated within an ITS with 147 students, and the expert evaluations showed high accuracy, precision, and F-scores. Consequently, our approach rapidly generates highly accurate ITS inner loops while requiring significantly less expert involvement than traditional methods. To the best of our knowledge, this is the first domain-independent approach that integrates collective intelligence into authoring of ITS inner loops.

Palavras-chave: Intelligent Tutoring Systems, Collective Intelligence, Inner Loop

Referências

Aleven, V., McLaren, B. M., Sewall, J., and Koedinger, K. R. (2006). The cognitive tutor authoring tools (ctat): preliminary evaluation of efficiency gains. In International Conference on Intelligent Tutoring Systems, pages 61–70. Springer.

Aleven, V., Mclaren, B. M., Sewall, J., and Koedinger, K. R. (2009). A new paradigm for intelligent tutoring systems: Example-tracing tutors. International Journal of Artificial Intelligence in Education, 19(2):105–154.

Chicco, D. and Jurman, G. (2020). The advantages of the matthews correlation coefficient (mcc) over f1 score and accuracy in binary classification evaluation. BMC Genomics, 21(1).

Cross, A., Bayyapunedi, M., Ravindran, D., Cutrell, E., and Thies, W. (2014). Vidwiki: Enabling the crowd to improve the legibility of online educational videos. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing, page 1167–1175, New York, NY, USA.

Heffernan, N. T. and Heffernan, C. L. (2014). The assistments ecosystem: Building a platform that brings scientists and teachers together for minimally invasive research on human learning and teaching. International Journal of Artificial Intelligence in Education, 24(4):470–497.

Heffernan, N. T., Ostrow, K. S., Kelly, K., Selent, D., Van Inwegen, E. G., Xiong, X., and Williams, J. J. (2016). The future of adaptive learning: Does the crowd hold the key? International Journal of Artificial Intelligence in Education, 26(2):615–644.

Leimeister, J. M. (2010). Collective intelligence. Business & Information Systems Engineering, 2(4):245–248.

MacLellan, C. J., Koedinger, K. R., and Matsuda, N. (2014). Authoring tutors with simstudent: An evaluation of efficiency and model quality. In International conference on intelligent tutoring systems, pages 551–560. Springer.

Matthews, B. (1975). Comparison of the predicted and observed secondary structure of t4 phage lysozyme. Biochimica et Biophysica Acta (BBA) - Protein Structure, 405(2):442–451.

Meza, J., Jimenez, A., Mendoza, K., and Vaca-Cárdenas, L. (2018). Collective intelligence education, enhancing the collaborative learning. In International Conference on eDemocracy & eGovernment (ICEDEG), pages 24–30. IEEE.

Mitrovic, A. (2003). An intelligent sql tutor on the web. International Journal of Artificial Intelligence in Education, 13(2-4):173–197.

Nkambou, R., Mizoguchi, R., and Bourdeau, J. (2010). Advances in intelligent tutoring systems, volume 308. Springer Science & Business Media.

Pane, J. F., Griffin, B. A., McCaffrey, D. F., and Karam, R. (2014). Effectiveness of cognitive tutor algebra i. Educational Evaluation and Policy Analysis, 36(2):127–144.

Polson, M. C. and Richardson, J. J. (2013). Foundations of intelligent tutoring systems. Psychology Press.

Psotka, J., Massey, L. D., and Mutter, S. A. (1988). Intelligent tutoring systems: Lessons learned. Psychology Press.

Razzaq, R., Ostrow, K. S., and Heffernan, N. T. (2020). Effect of immediate feedback on math achievement at the high school level. In Artificial Intelligence in Education, pages 263–267.

Schulze, K. G., Shelby, R. N., Treacy, D. J., Wintersgill, M. C., Vanlehn, K., and Gertner, A. (2000). Andes: An intelligent tutor for classical physics. Journal of Electronic Publishing, 6(1).

Seffrin, H., Rubi, G., and Jaques, P. (2012). O modelo cognitivo do sistema tutor inteligente pat2math. In Simpósio Brasileiro de Informática na Educação-SBIE.

Sun, C., Li, S.-w., and Lin, L. (2016). Thread structure prediction for mooc discussion forum. In Social Computing, pages 92–101, Singapore. Springer Singapore.

Tacoma, S., Drijvers, P., and Jeuring, J. (2021). Combined inner and outer loop feedback in an intelligent tutoring system for statistics in higher education. Journal of Computer Assisted Learning, 37(2):319–332.

Tenório, T., Isotani, S., and Bittencourt, I. I. (2022). Authoring inner loops of intelligent tutoring systems using collective intelligence. In Artificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners' and Doctoral Consortium: 23rd International Conference, AIED 2022, Durham, UK, July 27–31, 2022, Proceedings, Part II, page 400–404, Berlin, Heidelberg. Springer-Verlag.

Tenório, T., Isotani, S., Bittencourt, I. I., and Lu, Y. (2021). The state-of-the-art on collective intelligence in online educational technologies. IEEE Transactions on Learning Technologies, 14(2):257–271.

Tenório, T., Isotani, S., and Bittencourt, I. I. (2020). Inteligência coletiva como ferramenta de apoio na construção de loops internos em sistemas tutores inteligentes. In Anais do XXXI Simpósio Brasileiro de Informática na Educação, pages 1233–1242, Porto Alegre, RS, Brasil. SBC.

Tenório, T., Rodrigues, L., Isotani, S., and Bittencourt, I. (2023). Empowering instructors with collective intelligence: Learning problem-solving paths to facilitate feedback generation. In Anais do XXXIV Simpósio Brasileiro de Informática na Educação, pages 1249–1259, Porto Alegre, RS, Brasil. SBC.

Vaish, R., Goyal, S., Saberi, A., and Goel, S. (2018). Creating crowdsourced research talks at scale. In Proceedings of the 2018 World Wide Web Conference, page 1–11, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee.

Vanlehn, K. (2006). The behavior of tutoring systems. International journal of artificial intelligence in education, 16(3):227–265.

VanLehn, K. (2011). The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educational Psychologist, 46(4):197–221.

Weitekamp, D., Harpstead, E., and Koedinger, K. R. (2020). An interaction design for machine teaching to develop ai tutors. In Proceedings of the 2020 CHI conference on human factors in computing systems, pages 1–11.

Zaidan, O. F. and Callison-Burch, C. (2011). Crowdsourcing translation: Professional quality from non-professionals. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, volume 1, pages 1220–1229. Association for Computational Linguistics.