Improving Search Quality with Automatic Ranking Evaluation and Tuning

• Larícia Cavalcante Federal University of Pernambuco / VTEX
• Ullayne Lima Federal University of Pernambuco / VTEX
• Luciano Barbosa Federal University of Pernambuco
• Ana Luiza Gomes VTEX
• Éden Santana Federal University of Pernambuco
• Thiago Martins Federal University of Pernambuco

DOI: https://doi.org/10.5753/sbbd.2020.13634

Abstract

Search is a common feature available in document-based applications. It allows users to find information of interest easier. Two essential aspects for building an effective search is to evaluate the ranking quality and be able to efficiently tune it based on this evaluation. In this paper, we present our Automatic Ranking Tuning and Evaluation System (ARTES) that measures the ranking performance based on users’ clicks on search resulting pages and automatically tunes the search ranking function by applying a Bayesian Optimization algorithm. Our system is integrated with Elasticsearch, a widely-used search engine, which provides the search functionality. The whole solution is currently used by our customer support platform to help users effectively find relevant information, as our experimental evaluation confirms.

Keywords: ranking evaluation, rank tuning, Bayesiam optimization

References

Akiba, T., Sano, S., Yanase, T., Ohta, T., and Koyama, M. (2019). Optuna: A next- generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD, pages 2623–2631.

Baeza-Yates, R., Ribeiro-Neto, B., et al. (1999). Modern information retrieval, volume 463. ACM press New York.

Bergstra, J.S., Bardenet, R., Bengio, Y., and Kégl,B.(2011). Algorithms for hyper-parameter optimization. In NeurIPS, pages 2546–2554.

Shahriari, B., Swersky, K., Wang, Z., Adams, R. P., and De Freitas, N. (2015). Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE, 104(1):148–175.

Sprent, P. and Smeeton, N. C. (2016). Applied nonparametric statistical methods. CRC press.