How Query Violations Relate to Relevance in Product Search
Resumo
The concept of relevance has underpinned the research in Information Retrieval since its early days. Specifically, in eCommerce search, where queries typically involve attributes that express constraints, relevance is affected by how well those constraints are satisfied. However, the relationship between relevance and constraint violations was overlooked in existing research. In this work, we analyze 5,000 queries and 50,000 query-product pairs to investigate whether query-intent violation constitutes a ranking signal related to, but distinct from, graded relevance. Our results show that violation is related to, but not reducible to, graded relevance. We further show that simple violation-aware features substantially reduce query violations and remain competitive with, or even slightly improve upon, strong baselines on retrieval quality metrics.
Palavras-chave:
eCommerce search, query constraint violation
Referências
Bencke, L., Paula, F. S., dos Santos, B. G., and Moreira, V. P. (2024). Can we trust llms as relevance judges? In Simpósio Brasileiro de Banco de Dados (SBBD), pages 600–612. SBC.
Chen, Y., Liu, S., Liu, Z., Sun, W., Baltrunas, L., and Schroeder, B. (2022). Wands: Dataset for product search relevance assessment. In European Conference on IR Research, page 128–141.
Cleverdon, C. (1960). The aslib cranfield research project on the comparative efficiency of indexing systems. Aslib Proceedings, 12(12):421–431.
Di Nunzio, G. M., Ferro, N., Jones, G. J. F., and Peters, C. (2006). Clef 2005: Ad hoc track overview. In Accessing Multilingual Information Repositories, pages 11–36.
Faggioli, G., Dietz, L., Clarke, C. L. A., Demartini, G., Hagen, M., Hauff, C., Kando, N., Kanoulas, E., Potthast, M., Stein, B., and Wachsmuth, H. (2023). Perspectives on large language models for relevance judgment. In Conference on Theory of Information Retrieval, ICTIR ’23, page 39–50.
Fang, C., Li, X., Fan, Z., Xu, J., Nag, K., Korpeoglu, E., Kumar, S., and Achan, K. (2024). Llm-ensemble: Optimal large language model ensemble method for e-commerce product attribute value extraction. In Conference on Research and Development in Information Retrieval, SIGIR ’24, page 2910–2914.
Järvelin, K. and Kekäläinen, J. (2002). Cumulated gain-based evaluation of ir techniques. ACM Transactions on Information Systems (TOIS), 20(4):422–446.
Kando, N. (1999). Overview of ir tasks at the first ntcir workshop. In NTCIR Workshop on Research in Japanese Text Retrieval and Term Recognition, Tokyo, Japan, 1999.
Loughnane, R., Liu, J., Chen, Z., Wang, Z., Giroux, J., Du, T., Schroeder, B., and Sun, W. (2024). Explicit attribute extraction in e-commerce search. In Workshop on e-Commerce and NLP @ LREC-COLING 2024, pages 125–135.
Luo, C., Goutam, R., Zhang, H., Zhang, C., Song, Y., and Yin, B. (2023). Implicit query parsing at amazon product search. In Conference on Research and Development in Information Retrieval, SIGIR ’23, page 3380–3384.
Luo, C., Tang, X., Lu, H., Xie, Y., Liu, H., Dai, Z., Cui, L., Joshi, A., Nag, S., Li, Y., Li, Z., Goutam, R., Tang, J., Zhang, H., and He, Q. (2024). Exploring query understanding for amazon product search.
Peikos, G. and Pasi, G. (2024). A systematic review of multidimensional relevance estimation in information retrieval. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 14(5):e1541.
Reddy, C. K., Màrquez, L., Valero, F., Rao, N., Zaragoza, H., Bandyopadhyay, S., Biswas, A., Xing, A., and Subbian, K. (2022). Shopping queries dataset: A large-scale esci benchmark for improving product search.
Sachdev, J., D Rosario, S., Phatak, A., Wen, H., Kirti, S., and Tripathy, C. (2025). Automated query-product relevance labeling using large language models for e-commerce search. In International Conference on Natural Language Processing and Information Retrieval, NLPIR ’24, page 32–40.
Schamber, L., Eisenberg, M. B., and Nilan, M. S. (1990). A re-examination of relevance: toward a dynamic, situational definition. Information processing & management, 26(6):755–776.
Sondhi, P., Sharma, M., Kolari, P., and Zhai, C. (2018). A taxonomy of queries for e-commerce search. In Conference on Research & Development in Information Retrieval, SIGIR ’18, page 1245–1248.
Soviero, B., Kuhn, D., Salle, A., and Moreira, V. P. (2024). Chatgpt goes shopping: Llms can predict relevance in ecommerce search. In Advances in Information Retrieval, pages 3–11.
Su, N., He, J., Liu, Y., Zhang, M., and Ma, S. (2018). User intent, behaviour, and perceived satisfaction in product search. In ACM International Conference on Web Search and Data Mining, WSDM ’18, page 547–555.
Thomas, P., Spielman, S., Craswell, N., and Mitra, B. (2024). Large language models can accurately predict searcher preferences. In Conference on Research and Development in Information Retrieval, SIGIR ’24, page 1930–1940.
Tsagkias, M., King, T. H., Kallumadi, S., Murdock, V., and De Rijke, M. (2021). Challenges and research opportunities in ecommerce search and recommendations. In ACM Sigir Forum, volume 54, pages 1–23.
Voorhees, E. (2007). Overview of trec 2006.
Wu, C.-Y., Ahmed, A., Kumar, G. R., and Datta, R. (2017). Predicting latent structured intents from shopping queries. In International Conference on World Wide Web, WWW ’17, page 1133–1141.
Yang, H., Gupta, P., Fernández Galán, R., Bu, D., and Jia, D. (2021). Seasonal relevance in e-commerce search. In ACM International Conference on Information & Knowledge Management, CIKM ’21, page 4293–4301.
Zhang, D., Li, Z., Cao, T., Luo, C., Wu, T., Lu, H., Song, Y., Yin, B., Zhao, T., and Yang, Q. (2021). Queaco: Borrowing treasures from weakly-labeled behavior data for query attribute value extraction. In ACM International Conference on Information & Knowledge Management, CIKM ’21, page 4362–4372.
Zhu, Y., Vedula, N., and Malmasi, S. (2025). Hint-augmented re-ranking: Efficient product search using LLM-based query decomposition. In Joint Conference on Natural Language Processing and Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, pages 200–216.
Chen, Y., Liu, S., Liu, Z., Sun, W., Baltrunas, L., and Schroeder, B. (2022). Wands: Dataset for product search relevance assessment. In European Conference on IR Research, page 128–141.
Cleverdon, C. (1960). The aslib cranfield research project on the comparative efficiency of indexing systems. Aslib Proceedings, 12(12):421–431.
Di Nunzio, G. M., Ferro, N., Jones, G. J. F., and Peters, C. (2006). Clef 2005: Ad hoc track overview. In Accessing Multilingual Information Repositories, pages 11–36.
Faggioli, G., Dietz, L., Clarke, C. L. A., Demartini, G., Hagen, M., Hauff, C., Kando, N., Kanoulas, E., Potthast, M., Stein, B., and Wachsmuth, H. (2023). Perspectives on large language models for relevance judgment. In Conference on Theory of Information Retrieval, ICTIR ’23, page 39–50.
Fang, C., Li, X., Fan, Z., Xu, J., Nag, K., Korpeoglu, E., Kumar, S., and Achan, K. (2024). Llm-ensemble: Optimal large language model ensemble method for e-commerce product attribute value extraction. In Conference on Research and Development in Information Retrieval, SIGIR ’24, page 2910–2914.
Järvelin, K. and Kekäläinen, J. (2002). Cumulated gain-based evaluation of ir techniques. ACM Transactions on Information Systems (TOIS), 20(4):422–446.
Kando, N. (1999). Overview of ir tasks at the first ntcir workshop. In NTCIR Workshop on Research in Japanese Text Retrieval and Term Recognition, Tokyo, Japan, 1999.
Loughnane, R., Liu, J., Chen, Z., Wang, Z., Giroux, J., Du, T., Schroeder, B., and Sun, W. (2024). Explicit attribute extraction in e-commerce search. In Workshop on e-Commerce and NLP @ LREC-COLING 2024, pages 125–135.
Luo, C., Goutam, R., Zhang, H., Zhang, C., Song, Y., and Yin, B. (2023). Implicit query parsing at amazon product search. In Conference on Research and Development in Information Retrieval, SIGIR ’23, page 3380–3384.
Luo, C., Tang, X., Lu, H., Xie, Y., Liu, H., Dai, Z., Cui, L., Joshi, A., Nag, S., Li, Y., Li, Z., Goutam, R., Tang, J., Zhang, H., and He, Q. (2024). Exploring query understanding for amazon product search.
Peikos, G. and Pasi, G. (2024). A systematic review of multidimensional relevance estimation in information retrieval. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 14(5):e1541.
Reddy, C. K., Màrquez, L., Valero, F., Rao, N., Zaragoza, H., Bandyopadhyay, S., Biswas, A., Xing, A., and Subbian, K. (2022). Shopping queries dataset: A large-scale esci benchmark for improving product search.
Sachdev, J., D Rosario, S., Phatak, A., Wen, H., Kirti, S., and Tripathy, C. (2025). Automated query-product relevance labeling using large language models for e-commerce search. In International Conference on Natural Language Processing and Information Retrieval, NLPIR ’24, page 32–40.
Schamber, L., Eisenberg, M. B., and Nilan, M. S. (1990). A re-examination of relevance: toward a dynamic, situational definition. Information processing & management, 26(6):755–776.
Sondhi, P., Sharma, M., Kolari, P., and Zhai, C. (2018). A taxonomy of queries for e-commerce search. In Conference on Research & Development in Information Retrieval, SIGIR ’18, page 1245–1248.
Soviero, B., Kuhn, D., Salle, A., and Moreira, V. P. (2024). Chatgpt goes shopping: Llms can predict relevance in ecommerce search. In Advances in Information Retrieval, pages 3–11.
Su, N., He, J., Liu, Y., Zhang, M., and Ma, S. (2018). User intent, behaviour, and perceived satisfaction in product search. In ACM International Conference on Web Search and Data Mining, WSDM ’18, page 547–555.
Thomas, P., Spielman, S., Craswell, N., and Mitra, B. (2024). Large language models can accurately predict searcher preferences. In Conference on Research and Development in Information Retrieval, SIGIR ’24, page 1930–1940.
Tsagkias, M., King, T. H., Kallumadi, S., Murdock, V., and De Rijke, M. (2021). Challenges and research opportunities in ecommerce search and recommendations. In ACM Sigir Forum, volume 54, pages 1–23.
Voorhees, E. (2007). Overview of trec 2006.
Wu, C.-Y., Ahmed, A., Kumar, G. R., and Datta, R. (2017). Predicting latent structured intents from shopping queries. In International Conference on World Wide Web, WWW ’17, page 1133–1141.
Yang, H., Gupta, P., Fernández Galán, R., Bu, D., and Jia, D. (2021). Seasonal relevance in e-commerce search. In ACM International Conference on Information & Knowledge Management, CIKM ’21, page 4293–4301.
Zhang, D., Li, Z., Cao, T., Luo, C., Wu, T., Lu, H., Song, Y., Yin, B., Zhao, T., and Yang, Q. (2021). Queaco: Borrowing treasures from weakly-labeled behavior data for query attribute value extraction. In ACM International Conference on Information & Knowledge Management, CIKM ’21, page 4362–4372.
Zhu, Y., Vedula, N., and Malmasi, S. (2025). Hint-augmented re-ranking: Efficient product search using LLM-based query decomposition. In Joint Conference on Natural Language Processing and Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, pages 200–216.
Publicado
08/09/2026
Como Citar
S. F. PAULA, Felipe; KUHN, Daniel Matheus; P. MOREIRA, Viviane.
How Query Violations Relate to Relevance in Product Search. In: SIMPÓSIO BRASILEIRO DE BANCO DE DADOS (SBBD), 41. , 2026, São Carlos/SP.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2026
.
p. 631-644.
ISSN 2763-8979.
DOI: https://doi.org/10.5753/sbbd.2026.249280.
