A Framework for Search as Learning Experiments: Design, Implementation, and Usability Insights
Resumo
Search as Learning (SAL) explores how users engage with search systems to acquire knowledge and develop understanding. Despite advances in SAL, the lack of general-purpose tools hinders reproducibility and standardization in experimental studies. This paper presents a framework to support researchers in designing SAL experiments, encompassing task creation, data collection, and learning assessment. To evaluate the proposal, we conducted a usability study with 12 participants, which yielded a score of 83.07, indicating excellent usability. Feedback of the participants also provided suggestions for improvement, guiding future development. This work contributes to strengthening methodological practices and fostering reproducibility in SAL research.Referências
Azzopardi, L. (2021). Cognitive biases in search: a review and reflection of cognitive biases in information retrieval. In CHIIR 2021 Proceedings of the 2021 Conference on Human Information Interaction and Retrieval, CHIIR 2021 - Proceedings of the 2021 Conference on Human Information Interaction and Retrieval, page 27–37.
Bangor, A., Kortum, P., and Miller, J. (2009). Determining what individual sus scores mean: adding an adjective rating scale. J. Usability Studies, 4(3):114–123.
Brooke, J. (1996). Sus: A quick and dirty usability scale. In Jordan, P. W., Thomas, B., Weerdmeester, B. A., and McClelland, I. L., editors, Usability Evaluation in Industry, pages 189–194. Taylor & Francis, London. Capítulo de livro.
Costa, L., Barbosa, S., and Cunha, J. (2025). Mind the gap: The missing features of the tools to support user studies in software engineering. Journal of Computer Languages, 84:101345.
Fox, M. C., Ericsson, K. A., and Best, R. (2011). Do procedures for verbal reporting of thinking have to be reactive? a meta-analysis and recommendations for best reporting methods. Psychological bulletin, 137(2):316.
Ghosh, S., Rath, M., and Shah, C. (2018). Searching as learning: Exploring search behavior and learning outcomes in learning-related tasks. In Proceedings of the 2018 Conference on Human Information Interaction & Retrieval, CHIIR ’18, page 22–31, New York, NY, USA. Association for Computing Machinery.
Gimenez, P., Machado, M., Pinelli, C., and Siqueira, S. (2020). Investigating the learning perspective of searching as learning, a review of the state of the art. In Anais do XXXI Simpósio Brasileiro de Informática na Educação, pages 302–311, Porto Alegre, RS, Brasil. SBC.
Hoppe, A., Yu, R., Liu, J., and Bhattacharya, N. (2025). Iwilds’25: The 5th international workshop on investigating learning during web search. In Proceedings of the Eighteenth ACM International Conference on Web Search and Data Mining, WSDM ’25, page 1116–1117, New York, NY, USA. Association for Computing Machinery.
Lewis, J. R. and Sauro, J. (2018). Item benchmarks for the system usability scale. J. Usability Studies, 13(3):158–167.
Machado, M., Assis, E. C., Souza, J. F., and Siqueira, S. W. M. (2024a). A framework to support experimentation in the context of cognitive biases in search as a learning process. In Proceedings of the 20th Brazilian Symposium on Information Systems, SBSI ’24, New York, NY, USA. Association for Computing Machinery.
Machado, M., Gimenez, P., and Siqueira, S. (2020). Raising the dimensions and variables for searching as a learning process: A systematic mapping of the literature. In Anais do XXXI Simpósio Brasileiro de Informática na Educação, pages 1393–1402, Porto Alegre, RS, Brasil. SBC.
Machado, M., Souza, J., and Siqueira, S. (2024b). Identifying confirmation bias in a search as learning task: A study on the use of artificial intelligence in education. In Anais do XXXV Simpósio Brasileiro de Informática na Educação, pages 1208–1221, Porto Alegre, RS, Brasil. SBC.
Marchionini, G. (2006). Exploratory search: from finding to understanding. Commun. ACM, 49(4):41–46.
Pimentel, M., Filippo, D., and Dos Santos, T. M. (2020). Design science research: pesquisa científica atrelada ao design de artefatos. RE@ D-Revista de Educação a Distância e eLearning, 3(1):37–61.
Renaud, G. and Azzopardi, L. (2012). Scamp: A tool for conducting interactive information retrieval experiments. IIiX 2012 - Proceedings 4th Information Interaction in Context Symposium: Behaviors, Interactions, Interfaces, Systems.
Rieh, S. Y., Collins-Thompson, K., Hansen, P., and Lee, H.-J. (2016). Towards searching as a learning process: A review of current perspectives and future directions. Journal of Information Science, 42(1):19–34.
Sünkler, S., Yagci, N., Sygulla, D., Mach, S., Schultheiß, S., and Lewandowski, D. (2023). Result assessment tool: A software toolkit for conducting studies based on search results. Proceedings of the Association for Information Science and Technology, 60:1143–1145.
Tibau, M., Siqueira, S. W. M., and Nunes, B. P. (2021). Think-aloud exploratory search: Understanding search behaviors and knowledge flows. In Visvizi, A., Lytras, M. D., and Aljohani, N. R., editors, Research and Innovation Forum 2020, pages 303–315, Cham. Springer International Publishing.
Tibau, M., Siqueira, S. W. M., and Nunes, B. P. (2022). The impact of non-verbalization in think-aloud: Understanding knowledge gain indicators considering think-aloud web searches. In Proceedings of the 33rd ACM Conference on Hypertext and Social Media, HT ’22, page 107–120, New York, NY, USA. Association for Computing Machinery.
Urgo, K. and Arguello, J. (2022). Understanding the “pathway” towards a searcher’s learning objective. ACM Trans. Inf. Syst., 40(4).
Vakkari, P. (2016). Searching as learning: A systematization based on literature. Journal of Information Science, 42(1):7–18.
von Hoyer, J., Hoppe, A., Kammerer, Y., Otto, C., Pardi, G., Rokicki, M., Yu, R., Dietze, S., Ewerth, R., and Holtz, P. (2022). The search as learning spaceship: Toward a comprehensive model of psychological and technological facets of search as learning. Frontiers in Psychology, Volume 13 - 2022.
Yang, Y., Urgo, K., Arguello, J., and Capra, R. (2025). Search+chat: Integrating search and genai to support users with learning-oriented search tasks. In Proceedings of the 2025 ACM SIGIR Conference on Human Information Interaction and Retrieval, CHIIR ’25, page 57–70, New York, NY, USA. Association for Computing Machinery.
Bangor, A., Kortum, P., and Miller, J. (2009). Determining what individual sus scores mean: adding an adjective rating scale. J. Usability Studies, 4(3):114–123.
Brooke, J. (1996). Sus: A quick and dirty usability scale. In Jordan, P. W., Thomas, B., Weerdmeester, B. A., and McClelland, I. L., editors, Usability Evaluation in Industry, pages 189–194. Taylor & Francis, London. Capítulo de livro.
Costa, L., Barbosa, S., and Cunha, J. (2025). Mind the gap: The missing features of the tools to support user studies in software engineering. Journal of Computer Languages, 84:101345.
Fox, M. C., Ericsson, K. A., and Best, R. (2011). Do procedures for verbal reporting of thinking have to be reactive? a meta-analysis and recommendations for best reporting methods. Psychological bulletin, 137(2):316.
Ghosh, S., Rath, M., and Shah, C. (2018). Searching as learning: Exploring search behavior and learning outcomes in learning-related tasks. In Proceedings of the 2018 Conference on Human Information Interaction & Retrieval, CHIIR ’18, page 22–31, New York, NY, USA. Association for Computing Machinery.
Gimenez, P., Machado, M., Pinelli, C., and Siqueira, S. (2020). Investigating the learning perspective of searching as learning, a review of the state of the art. In Anais do XXXI Simpósio Brasileiro de Informática na Educação, pages 302–311, Porto Alegre, RS, Brasil. SBC.
Hoppe, A., Yu, R., Liu, J., and Bhattacharya, N. (2025). Iwilds’25: The 5th international workshop on investigating learning during web search. In Proceedings of the Eighteenth ACM International Conference on Web Search and Data Mining, WSDM ’25, page 1116–1117, New York, NY, USA. Association for Computing Machinery.
Lewis, J. R. and Sauro, J. (2018). Item benchmarks for the system usability scale. J. Usability Studies, 13(3):158–167.
Machado, M., Assis, E. C., Souza, J. F., and Siqueira, S. W. M. (2024a). A framework to support experimentation in the context of cognitive biases in search as a learning process. In Proceedings of the 20th Brazilian Symposium on Information Systems, SBSI ’24, New York, NY, USA. Association for Computing Machinery.
Machado, M., Gimenez, P., and Siqueira, S. (2020). Raising the dimensions and variables for searching as a learning process: A systematic mapping of the literature. In Anais do XXXI Simpósio Brasileiro de Informática na Educação, pages 1393–1402, Porto Alegre, RS, Brasil. SBC.
Machado, M., Souza, J., and Siqueira, S. (2024b). Identifying confirmation bias in a search as learning task: A study on the use of artificial intelligence in education. In Anais do XXXV Simpósio Brasileiro de Informática na Educação, pages 1208–1221, Porto Alegre, RS, Brasil. SBC.
Marchionini, G. (2006). Exploratory search: from finding to understanding. Commun. ACM, 49(4):41–46.
Pimentel, M., Filippo, D., and Dos Santos, T. M. (2020). Design science research: pesquisa científica atrelada ao design de artefatos. RE@ D-Revista de Educação a Distância e eLearning, 3(1):37–61.
Renaud, G. and Azzopardi, L. (2012). Scamp: A tool for conducting interactive information retrieval experiments. IIiX 2012 - Proceedings 4th Information Interaction in Context Symposium: Behaviors, Interactions, Interfaces, Systems.
Rieh, S. Y., Collins-Thompson, K., Hansen, P., and Lee, H.-J. (2016). Towards searching as a learning process: A review of current perspectives and future directions. Journal of Information Science, 42(1):19–34.
Sünkler, S., Yagci, N., Sygulla, D., Mach, S., Schultheiß, S., and Lewandowski, D. (2023). Result assessment tool: A software toolkit for conducting studies based on search results. Proceedings of the Association for Information Science and Technology, 60:1143–1145.
Tibau, M., Siqueira, S. W. M., and Nunes, B. P. (2021). Think-aloud exploratory search: Understanding search behaviors and knowledge flows. In Visvizi, A., Lytras, M. D., and Aljohani, N. R., editors, Research and Innovation Forum 2020, pages 303–315, Cham. Springer International Publishing.
Tibau, M., Siqueira, S. W. M., and Nunes, B. P. (2022). The impact of non-verbalization in think-aloud: Understanding knowledge gain indicators considering think-aloud web searches. In Proceedings of the 33rd ACM Conference on Hypertext and Social Media, HT ’22, page 107–120, New York, NY, USA. Association for Computing Machinery.
Urgo, K. and Arguello, J. (2022). Understanding the “pathway” towards a searcher’s learning objective. ACM Trans. Inf. Syst., 40(4).
Vakkari, P. (2016). Searching as learning: A systematization based on literature. Journal of Information Science, 42(1):7–18.
von Hoyer, J., Hoppe, A., Kammerer, Y., Otto, C., Pardi, G., Rokicki, M., Yu, R., Dietze, S., Ewerth, R., and Holtz, P. (2022). The search as learning spaceship: Toward a comprehensive model of psychological and technological facets of search as learning. Frontiers in Psychology, Volume 13 - 2022.
Yang, Y., Urgo, K., Arguello, J., and Capra, R. (2025). Search+chat: Integrating search and genai to support users with learning-oriented search tasks. In Proceedings of the 2025 ACM SIGIR Conference on Human Information Interaction and Retrieval, CHIIR ’25, page 57–70, New York, NY, USA. Association for Computing Machinery.
Publicado
24/11/2025
Como Citar
SILVA, Joel H. N. de O.; NETO, Alfredo; ROSADO, Breno; MACHADO, Marcelo; SOUZA, Jairo F. de; SIQUEIRA, Sean W. M..
A Framework for Search as Learning Experiments: Design, Implementation, and Usability Insights. In: SIMPÓSIO BRASILEIRO DE INFORMÁTICA NA EDUCAÇÃO (SBIE), 36. , 2025, Curitiba/PR.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2025
.
p. 304-315.
DOI: https://doi.org/10.5753/sbie.2025.12265.
