Evaluating Knowledge Gain in Search Environments: An Exploratory Study of Learning Measurement
Resumo
Research Context: Searching as Learning (SaL) frames web search as a process where users construct and refine knowledge. However, measuring knowledge gain in natural search environments remains a methodological challenge. Scientific and/or Practical Problem: Traditional behavioral proxies (e.g., dwell time, clicks) scale well but fail to capture conceptual change, while pre/post-tests provide richer insights but are intrusive. This gap limits the development of search systems that can evaluate and promote learning. Proposed Solution and/or Analysis: This study advances a computational measure based on entropy reduction and semantic similarity, and novelly operationalizes it through a browser plug-in that enables real-time measurement in natural search environments, extending prior formalizations and prototype-based validations of the DKG metric. Related IS Theory: The study draws on Shannon’s Information Theory and Information Processing Theory in IS to conceptualize knowledge gain as uncertainty reduction supported by socio-technical processes. Research Method: An experiment combined three structured search tasks, pre/post-tests, and Concurrent Think-Aloud protocols. Quantitative measures (Transfer of Learning scores, also known as ToL, and values from the proposed metric) were triangulated with qualitative coding using OISS and ESKiP frameworks. Summary of Results: Statistical analysis showed a moderate positive correlation between ToL and the proposed metric (r = 0.62, p < 0.01). Bland–Altman analysis revealed systematic differences in scale, with ToL showing higher values, yet relative patterns were consistent. Transcripts emphasized how strategies such as query specialization, evaluation of sources, and persistence in reformulation aligned with higher values. Contributions and Impact to IS area: The study contributes a validated computational metric and artifacts for measuring knowledge gain in real search environments. It reinforces the sociotechnical view of IS by linking human strategies, processes, and technological advantages, and points to adaptive search systems that could measure and promote learning.Referências
Borsci, S., Macredie, R. D., Barnett, J., Martin, J., Kuljis, J., and Young, T. (2013). Reviewing and extending the five-user assumption: A grounded procedure for interaction evaluation. ACM Trans. Comput.-Hum. Interact., 20(5).
Boscarioli, C., Araujo, R. M., and Maciel, R. S. P. (2017). I GranDSI-BR: Grand Research Challenges in Information Systems in Brazil 2016–2026. Brazilian Computer Society (SBC), Special Committee on Information Systems (CE-SI).
Boucher, S. (2018). Stances and epistemology: Values, pragmatics, and rationality. Metaphilosophy, 49(4):521–547.
Câmara, A. (2024). Designing Search-as-Learning Systems. Ph.d. thesis, Delft University of Technology. DOI: 10.4233/uuid:0fe3a6bb-1bc1-40e2-86b0-ec3d3aef9c77, ISBN: 978-94-6384-569-4.
Câmara, A., Roy, N., Maxwell, D., and Hauff, C. (2021). Searching to learn with instructional scaffolding. In Proceedings of the 2021 Conference on Human Information Interaction and Retrieval, CHIIR ’21, page 209–218, New York, NY, USA. Association for Computing Machinery.
Chi, Y., Han, S., He, D., and Meng, R. (2016). Exploring knowledge learning in collaborative information seeking process. In SAL@SIGIR.
Daft, R. L. and Lengel, R. H. (1986). Organizational information requirements, media richness and structural design. Management Science, 32(5):554–571.
Dupret, G. E. and Piwowarski, B. (2008). A user browsing model to predict search engine click data from past observations. In Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’08, page 331–338, New York, NY, USA. Association for Computing Machinery.
El Zein, D. and da Costa Pereira, C. (2022). User’s knowledge and information needs in information retrieval evaluation. In Proceedings of the 30th ACM Conference on User Modeling, Adaptation and Personalization, UMAP ’22, page 170–178, New York, NY, USA. Association for Computing Machinery.
El Zein, D. and Da Costa Pereira, C. (2023). The evolution of user knowledge during search-as-learning sessions: A benchmark and baseline. In Proceedings of the 2023 Conference on Human Information Interaction and Retrieval, CHIIR ’23, page 454–458, New York, NY, USA. Association for Computing Machinery.
Galbraith, J. R. (1973). Designing Complex Organizations. Addison-Wesley, Reading, MA.
Gritz, W., Hoppe, A., and Ewerth, R. (2021). On the impact of features and classifiers for measuring knowledge gain during web search-a case study.
Gritz, W., Hoppe, A., and Ewerth, R. (2024). On the influence of reading sequences on knowledge gain during web search. In Goharian, N., Tonellotto, N., He, Y., Lipani, A., McDonald, G., Macdonald, C., and Ounis, I., editors, Advances in Information Retrieval, pages 364–373, Cham. Springer Nature Switzerland.
Hart, S. L., Steinheider, B., and Hoffmeister, V. E. (2019). Team-based learning and training transfer: a case study of training for the implementation of enterprise resources planning software. International Journal of Training and Development, 23(2):135–152.
Haußmann, C., Dwivedi, Y. K., Venkitachalam, K., and Williams, M. D. (2012). A summary and review of galbraith’s organizational information processing theory. Information Systems Theory, pages 71–93.
Jaynes, E. T. (1985). Entropy and Search Theory, pages 443–454. Springer Netherlands, Dordrecht.
Kelley, T., Capobianco, B., and Kaluf, K. (2015). Concurrent think-aloud protocols to assess elementary design students. International Journal of Technology & Design Education, 25(4):521 – 540.
Kivinen, O. and Ristela, P. (2003). From constructivism to a pragmatist conception of learning. Oxford review of education, 29(3):363–375.
Lazar, J., Feng, J. H., and Hochheiser, H. (2017). Chapter 10 - usability testing. In Lazar, J., Feng, J. H., and Hochheiser, H., editors, Research Methods in Human Computer Interaction (Second Edition), pages 263–298. Morgan Kaufmann, Boston, second edition edition.
Liu, C., Song, X., and Hansen, P. (2023). Characterising users’ task completion process in learning-related tasks:a search pace model. Journal of Information Science, 49(6):1462–1480.
Liu, Y., Qin, C., Ma, X., Chen, J., He, H., and Mao, J. (2025). Characterising exploratory search tasks: Evidence from different fields. Journal of Information Science, 0(0):01655515251330611.
Lu, J. 2025). A systematic literature review of usability definitions and psychometric properties of instruments in the field of learning design and technology. Journal of Research on Technology in Education, 0(0):1–22.
Marchionini, G. (2006). Exploratory search: From finding to understanding. Commun. ACM, 49(4):41–46.
Miles, M. B., Huberman, A. M., and Saldaña, J. (2013). Qualitative Data Analysis: A Methods Sourcebook. SAGE Publications, Thousand Oaks, CA, 3 edition.
Milne, P. (2012). Probability as a Measure of Information Added. Journal of Logic, Language and Information, 21(2):163–188.
Nielsen, J. and Landauer, T. K. (1993). A mathematical model of the finding of usability problems. In Proceedings of the INTERACT ’93 and CHI ’93 Conference on Human Factors in Computing Systems, CHI ’93, page 206–213, New York, NY, USA. Association for Computing Machinery.
Otto, C., Rokicki, M., Pardi, G., Gritz, W., Hienert, D., Yu, R., von Hoyer, J., Hoppe, A., Dietze, S., Holtz, P., Kammerer, Y., and Ewerth, R. (2022). SaL-Lightning Dataset: Search and Eye Gaze Behavior, Resource Interactions and Knowledge Gain during Web Search. In ACM SIGIR Conference on Human Information Interaction and Retrieval, pages 347–352, Regensburg Germany. ACM.
Otto, C., Yu, R., Pardi, G., von Hoyer, J., Rokicki, M., Hoppe, A., Holtz, P., Kammerer, Y., Dietze, S., and Ewerth, R. (2021). Predicting knowledge gain during web search based on multimedia resource consumption. In Roll, I., McNamara, D., Sosnovsky, S., Luckin, R., and Dimitrova, V., editors, Artificial Intelligence in Education, pages 318–330, Cham. Springer International Publishing.
Plano Clark, V. L. (2017). Mixed methods research. The Journal of Positive Psychology, 12(3):305–306.
Prieto-Guerrero, A. and Espinosa-Paredes, G. (2019). 7 - nonlinear signal processing methods: Dr estimation and nonlinear stability indicators. In Prieto-Guerrero, A. and Espinosa-Paredes, G., editors, Linear and Non-Linear Stability Analysis in Boiling Water Reactors, Woodhead Publishing Series in Energy, pages 315 – 398. Woodhead Publishing.
Reisoğlu, I., Çebi, A., and Bahçekapılı, T. (2019). Online information searching behaviours: examining the impact of task complexity, information searching experience, and cognitive style. Interactive Learning Environments, 0(0):1–18.
Roy, N., Moraes, F., and Hauff, C. (2020). Exploring users’ learning gains within search sessions. In Proceedings of the 2020 Conference on Human Information Interaction and Retrieval, CHIIR ’20, page 432–436, New York, NY, USA. Association for Computing Machinery.
Saldaña, J. (2021). The Coding Manual for Qualitative Researchers. SAGE Publications, London, 4 edition.
Schmuckler, M. A. (2001). What is ecological validity? a dimensional analysis. Infancy, 2(4):419–436.
Shannon, C. E. (1948). A mathematical theory of communication. The Bell system technical journal, 27(3):379–423.
Smith, G. T. (2005). On construct validity: issues of method and measurement. Psychological assessment, 17(4):396.
Strauss, M. E. and Smith, G. T. (2009). Construct validity: Advances in theory and methodology. Annual review of clinical psychology, 5:1–25.
Tibau, M. (2024). Quantifying Knowledge Gain in Online Searches: The DKG Metric. Tese (doutorado em informática), Universidade Federal do Estado do Rio de Janeiro (UNIRIO), Rio de Janeiro. Programa de Pós-Graduação em Informática.
Tibau, M., Siqueira, S. W. M., and Nunes, B. P. (2022). The impact of non-verbalization in think-aloud: Understanding knowledge gain indicators considering think-aloud web searches. In Proceedings of the 33rd ACM Conference on Hypertext and Social Media, HT ’22, page 107–120, New York, NY, USA. Association for Computing Machinery.
Tibau, M., Siqueira, S. W. M., and Nunes, B. P. (2023). Accounting for the knowledge gained during a web search: An empirical study on learning transfer indicators. Library & Information Science Research, 45(1):101222.
Tibau, M., Siqueira, S. W. M., Nunes, B. P., Nurmikko-Fuller, T., and Manrique, R. F. (2019). Using query reformulation to compare learning behaviors in web search engines. In 2019 IEEE 19th International Conference on Advanced Learning Technologies (ICALT), volume 2161-377X, pages 219–223.
Tomczyk, P., Brüggemann, P., Mergner, N., and Petrescu, M. (2024). Are ai tools better than traditional tools in literature searching? evidence from e-commerce research. Journal of Librarianship and Information Science, 0(0):09610006241295802.
Urgo, K. and Arguello, J. (2022). Learning assessments in search-as-learning: A survey of prior work and opportunities for future research. Information Processing & Management, 59(2):102821.
Vakkari, P. (2016). Searching as learning: A systematization based on literature. In Journal of Information Science, 42(1), pages 7–18.
Virzi, R. A. (1992). Refining the test phase of usability evaluation: How many subjects is enough? Hum. Factors, 34(4):457–468.
Xu, L., Zhou, X., and Gadiraju, U. (2020). How does team composition affect knowledge gain of users in collaborative web search? In Proceedings of the 31st ACM Conference on Hypertext and Social Media, HT ’20, page 91–100, New York, NY, USA. Association for Computing Machinery.
Yu, R., Tang, R., Rokicki, M., Gadiraju, U., and Dietze, S. (2021). Topic-independent modeling of user knowledge in informational search sessions. Information Retrieval Journal, 24(3):240–268.
Boscarioli, C., Araujo, R. M., and Maciel, R. S. P. (2017). I GranDSI-BR: Grand Research Challenges in Information Systems in Brazil 2016–2026. Brazilian Computer Society (SBC), Special Committee on Information Systems (CE-SI).
Boucher, S. (2018). Stances and epistemology: Values, pragmatics, and rationality. Metaphilosophy, 49(4):521–547.
Câmara, A. (2024). Designing Search-as-Learning Systems. Ph.d. thesis, Delft University of Technology. DOI: 10.4233/uuid:0fe3a6bb-1bc1-40e2-86b0-ec3d3aef9c77, ISBN: 978-94-6384-569-4.
Câmara, A., Roy, N., Maxwell, D., and Hauff, C. (2021). Searching to learn with instructional scaffolding. In Proceedings of the 2021 Conference on Human Information Interaction and Retrieval, CHIIR ’21, page 209–218, New York, NY, USA. Association for Computing Machinery.
Chi, Y., Han, S., He, D., and Meng, R. (2016). Exploring knowledge learning in collaborative information seeking process. In SAL@SIGIR.
Daft, R. L. and Lengel, R. H. (1986). Organizational information requirements, media richness and structural design. Management Science, 32(5):554–571.
Dupret, G. E. and Piwowarski, B. (2008). A user browsing model to predict search engine click data from past observations. In Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’08, page 331–338, New York, NY, USA. Association for Computing Machinery.
El Zein, D. and da Costa Pereira, C. (2022). User’s knowledge and information needs in information retrieval evaluation. In Proceedings of the 30th ACM Conference on User Modeling, Adaptation and Personalization, UMAP ’22, page 170–178, New York, NY, USA. Association for Computing Machinery.
El Zein, D. and Da Costa Pereira, C. (2023). The evolution of user knowledge during search-as-learning sessions: A benchmark and baseline. In Proceedings of the 2023 Conference on Human Information Interaction and Retrieval, CHIIR ’23, page 454–458, New York, NY, USA. Association for Computing Machinery.
Galbraith, J. R. (1973). Designing Complex Organizations. Addison-Wesley, Reading, MA.
Gritz, W., Hoppe, A., and Ewerth, R. (2021). On the impact of features and classifiers for measuring knowledge gain during web search-a case study.
Gritz, W., Hoppe, A., and Ewerth, R. (2024). On the influence of reading sequences on knowledge gain during web search. In Goharian, N., Tonellotto, N., He, Y., Lipani, A., McDonald, G., Macdonald, C., and Ounis, I., editors, Advances in Information Retrieval, pages 364–373, Cham. Springer Nature Switzerland.
Hart, S. L., Steinheider, B., and Hoffmeister, V. E. (2019). Team-based learning and training transfer: a case study of training for the implementation of enterprise resources planning software. International Journal of Training and Development, 23(2):135–152.
Haußmann, C., Dwivedi, Y. K., Venkitachalam, K., and Williams, M. D. (2012). A summary and review of galbraith’s organizational information processing theory. Information Systems Theory, pages 71–93.
Jaynes, E. T. (1985). Entropy and Search Theory, pages 443–454. Springer Netherlands, Dordrecht.
Kelley, T., Capobianco, B., and Kaluf, K. (2015). Concurrent think-aloud protocols to assess elementary design students. International Journal of Technology & Design Education, 25(4):521 – 540.
Kivinen, O. and Ristela, P. (2003). From constructivism to a pragmatist conception of learning. Oxford review of education, 29(3):363–375.
Lazar, J., Feng, J. H., and Hochheiser, H. (2017). Chapter 10 - usability testing. In Lazar, J., Feng, J. H., and Hochheiser, H., editors, Research Methods in Human Computer Interaction (Second Edition), pages 263–298. Morgan Kaufmann, Boston, second edition edition.
Liu, C., Song, X., and Hansen, P. (2023). Characterising users’ task completion process in learning-related tasks:a search pace model. Journal of Information Science, 49(6):1462–1480.
Liu, Y., Qin, C., Ma, X., Chen, J., He, H., and Mao, J. (2025). Characterising exploratory search tasks: Evidence from different fields. Journal of Information Science, 0(0):01655515251330611.
Lu, J. 2025). A systematic literature review of usability definitions and psychometric properties of instruments in the field of learning design and technology. Journal of Research on Technology in Education, 0(0):1–22.
Marchionini, G. (2006). Exploratory search: From finding to understanding. Commun. ACM, 49(4):41–46.
Miles, M. B., Huberman, A. M., and Saldaña, J. (2013). Qualitative Data Analysis: A Methods Sourcebook. SAGE Publications, Thousand Oaks, CA, 3 edition.
Milne, P. (2012). Probability as a Measure of Information Added. Journal of Logic, Language and Information, 21(2):163–188.
Nielsen, J. and Landauer, T. K. (1993). A mathematical model of the finding of usability problems. In Proceedings of the INTERACT ’93 and CHI ’93 Conference on Human Factors in Computing Systems, CHI ’93, page 206–213, New York, NY, USA. Association for Computing Machinery.
Otto, C., Rokicki, M., Pardi, G., Gritz, W., Hienert, D., Yu, R., von Hoyer, J., Hoppe, A., Dietze, S., Holtz, P., Kammerer, Y., and Ewerth, R. (2022). SaL-Lightning Dataset: Search and Eye Gaze Behavior, Resource Interactions and Knowledge Gain during Web Search. In ACM SIGIR Conference on Human Information Interaction and Retrieval, pages 347–352, Regensburg Germany. ACM.
Otto, C., Yu, R., Pardi, G., von Hoyer, J., Rokicki, M., Hoppe, A., Holtz, P., Kammerer, Y., Dietze, S., and Ewerth, R. (2021). Predicting knowledge gain during web search based on multimedia resource consumption. In Roll, I., McNamara, D., Sosnovsky, S., Luckin, R., and Dimitrova, V., editors, Artificial Intelligence in Education, pages 318–330, Cham. Springer International Publishing.
Plano Clark, V. L. (2017). Mixed methods research. The Journal of Positive Psychology, 12(3):305–306.
Prieto-Guerrero, A. and Espinosa-Paredes, G. (2019). 7 - nonlinear signal processing methods: Dr estimation and nonlinear stability indicators. In Prieto-Guerrero, A. and Espinosa-Paredes, G., editors, Linear and Non-Linear Stability Analysis in Boiling Water Reactors, Woodhead Publishing Series in Energy, pages 315 – 398. Woodhead Publishing.
Reisoğlu, I., Çebi, A., and Bahçekapılı, T. (2019). Online information searching behaviours: examining the impact of task complexity, information searching experience, and cognitive style. Interactive Learning Environments, 0(0):1–18.
Roy, N., Moraes, F., and Hauff, C. (2020). Exploring users’ learning gains within search sessions. In Proceedings of the 2020 Conference on Human Information Interaction and Retrieval, CHIIR ’20, page 432–436, New York, NY, USA. Association for Computing Machinery.
Saldaña, J. (2021). The Coding Manual for Qualitative Researchers. SAGE Publications, London, 4 edition.
Schmuckler, M. A. (2001). What is ecological validity? a dimensional analysis. Infancy, 2(4):419–436.
Shannon, C. E. (1948). A mathematical theory of communication. The Bell system technical journal, 27(3):379–423.
Smith, G. T. (2005). On construct validity: issues of method and measurement. Psychological assessment, 17(4):396.
Strauss, M. E. and Smith, G. T. (2009). Construct validity: Advances in theory and methodology. Annual review of clinical psychology, 5:1–25.
Tibau, M. (2024). Quantifying Knowledge Gain in Online Searches: The DKG Metric. Tese (doutorado em informática), Universidade Federal do Estado do Rio de Janeiro (UNIRIO), Rio de Janeiro. Programa de Pós-Graduação em Informática.
Tibau, M., Siqueira, S. W. M., and Nunes, B. P. (2022). The impact of non-verbalization in think-aloud: Understanding knowledge gain indicators considering think-aloud web searches. In Proceedings of the 33rd ACM Conference on Hypertext and Social Media, HT ’22, page 107–120, New York, NY, USA. Association for Computing Machinery.
Tibau, M., Siqueira, S. W. M., and Nunes, B. P. (2023). Accounting for the knowledge gained during a web search: An empirical study on learning transfer indicators. Library & Information Science Research, 45(1):101222.
Tibau, M., Siqueira, S. W. M., Nunes, B. P., Nurmikko-Fuller, T., and Manrique, R. F. (2019). Using query reformulation to compare learning behaviors in web search engines. In 2019 IEEE 19th International Conference on Advanced Learning Technologies (ICALT), volume 2161-377X, pages 219–223.
Tomczyk, P., Brüggemann, P., Mergner, N., and Petrescu, M. (2024). Are ai tools better than traditional tools in literature searching? evidence from e-commerce research. Journal of Librarianship and Information Science, 0(0):09610006241295802.
Urgo, K. and Arguello, J. (2022). Learning assessments in search-as-learning: A survey of prior work and opportunities for future research. Information Processing & Management, 59(2):102821.
Vakkari, P. (2016). Searching as learning: A systematization based on literature. In Journal of Information Science, 42(1), pages 7–18.
Virzi, R. A. (1992). Refining the test phase of usability evaluation: How many subjects is enough? Hum. Factors, 34(4):457–468.
Xu, L., Zhou, X., and Gadiraju, U. (2020). How does team composition affect knowledge gain of users in collaborative web search? In Proceedings of the 31st ACM Conference on Hypertext and Social Media, HT ’20, page 91–100, New York, NY, USA. Association for Computing Machinery.
Yu, R., Tang, R., Rokicki, M., Gadiraju, U., and Dietze, S. (2021). Topic-independent modeling of user knowledge in informational search sessions. Information Retrieval Journal, 24(3):240–268.
Publicado
25/05/2026
Como Citar
TIBAU, Marcelo; SILVA, Rafael Tavares da; SIQUEIRA, Sean Wolfgand Matsui; NUNES, Bernardo Pereira.
Evaluating Knowledge Gain in Search Environments: An Exploratory Study of Learning Measurement. In: SIMPÓSIO BRASILEIRO DE SISTEMAS DE INFORMAÇÃO (SBSI), 22. , 2026, Vitória/ES.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2026
.
p. 478-496.
DOI: https://doi.org/10.5753/sbsi.2026.248557.
