APS-RAG-Benchmark: Um Recurso Inicial para Avaliação de RAG na Atenção Primária à Saúde
Resumo
Este trabalho apresenta o APS-RAG-Benchmark, um benchmark preliminar para avaliação de sistemas Retrieval-Augmented Generation (RAG) no domínio do e-SUS Atenção Primária à Saúde (APS). O benchmark reúne um dataset validado por especialistas, composto por perguntas reais de tickets de suporte, fontes documentais relevantes e respostas geradas por modelos de linguagem e revisadas por humanos. Além de disponibilizar um recurso para um domínio ainda pouco explorado em português, o trabalho propõe um protocolo experimental que avalia separadamente as etapas de recuperação e geração. Os resultados mostram que a combinação de estratégias léxicas (BM25) e semânticas melhora significativamente a recuperação, elevando o MRR de 0,298 para 0,596. Além disso, observou-se alinhamento frequente entre as fontes recuperadas e as consideradas corretas pelos especialistas, com até 69% das respostas avaliadas como adequadas.
Palavras-chave:
Geração Aumentada por Recuperação, Atenção Primária à Saúde, Grandes Modelos de Linguagem
Referências
Andrade, C. M., Magalhães, G., Asevedo, R., Paes, J., Oliveira, I., Pagano, A., Reis, Z., & Gonçalves, M. A. (2025). Estudo do impacto de dados sintéticos e paráfrases na mitigação do desbalanceamento em tarefas de classificação de textos em português com baixa amostragem. In Anais do XL Simpósio Brasileiro de Bancos de Dados.
Fernandes, D., de Moura, E. S., Ribeiro-Neto, B., da Silva, A. S., & Gonçalves, M. A. (2007). Computing block importance for searching on web sites. In Proceedings of the Sixteenth ACM Conference on Conference on Information and Knowledge Management.
França, C., Rabbi, G., Salles, T., Cunha, W., Rocha, L., & Gonçalves, M. A. (2025). Ranking-based fusion algorithms for extreme multi-label text classification (xmtc).
Gao, Y. et al. (2023). Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997.
Gwet, K. L. (2008). Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology, 61(1):29–48.
Krippendorff, K. (2011). Computing krippendorff’s alpha-reliability. Technical report, Annenberg School for Communication, University of Pennsylvania.
Lebeis, G., Reis, Z., Oliveira, I., & Pereira, F. (2025). Engagement strategies for digital literacy to support the brazilian e-sus primary care system. In MEDINFO 2025.
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t., Rocktäschel, T., Riedel, S., & Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive nlp tasks. In NeurIPS.
Malkov, Y. A. & Yashunin, D. A. (2019). Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. IEEE Transactions on Pattern Analysis and Machine Intelligence.
Montague, M. H. & Aslam, J. A. (2002). Evaluating score normalization methods in data fusion. In Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval.
Ram, O. et al. (2023). In-context retrieval-augmented language models. Transactions of the Association for Computational Linguistics.
Robertson, S. & Zaragoza, H. (2009). The probabilistic relevance framework: BM25 and beyond. Foundations and Trends® in Information Retrieval.
Thakur, N., Reimers, N., Rücklé, A., Srivastava, A., & Gupta, A. (2021). BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Advances in Neural Information Processing Systems (NeurIPS).
Wongpakaran, N., Wongpakaran, T., Wedding, D., & Gwet, K. L. (2013). A comparison of cohen’s kappa and gwet’s ac1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples. BMC Medical Research Methodology.
Zec, S., Soriani, N., Comoretto, R. I., & Baldi, I. (2017). The paradox of cohen’s kappa. Open Nursing Journal, 11:211–218.
Zha, D., Bhat, Z. P., Lai, K.-H., Yang, F., Jiang, Z., Zhong, S., & Hu, X. (2025). Data-centric artificial intelligence: A survey. ACM Comput. Surv.
Fernandes, D., de Moura, E. S., Ribeiro-Neto, B., da Silva, A. S., & Gonçalves, M. A. (2007). Computing block importance for searching on web sites. In Proceedings of the Sixteenth ACM Conference on Conference on Information and Knowledge Management.
França, C., Rabbi, G., Salles, T., Cunha, W., Rocha, L., & Gonçalves, M. A. (2025). Ranking-based fusion algorithms for extreme multi-label text classification (xmtc).
Gao, Y. et al. (2023). Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997.
Gwet, K. L. (2008). Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology, 61(1):29–48.
Krippendorff, K. (2011). Computing krippendorff’s alpha-reliability. Technical report, Annenberg School for Communication, University of Pennsylvania.
Lebeis, G., Reis, Z., Oliveira, I., & Pereira, F. (2025). Engagement strategies for digital literacy to support the brazilian e-sus primary care system. In MEDINFO 2025.
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t., Rocktäschel, T., Riedel, S., & Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive nlp tasks. In NeurIPS.
Malkov, Y. A. & Yashunin, D. A. (2019). Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. IEEE Transactions on Pattern Analysis and Machine Intelligence.
Montague, M. H. & Aslam, J. A. (2002). Evaluating score normalization methods in data fusion. In Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval.
Ram, O. et al. (2023). In-context retrieval-augmented language models. Transactions of the Association for Computational Linguistics.
Robertson, S. & Zaragoza, H. (2009). The probabilistic relevance framework: BM25 and beyond. Foundations and Trends® in Information Retrieval.
Thakur, N., Reimers, N., Rücklé, A., Srivastava, A., & Gupta, A. (2021). BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Advances in Neural Information Processing Systems (NeurIPS).
Wongpakaran, N., Wongpakaran, T., Wedding, D., & Gwet, K. L. (2013). A comparison of cohen’s kappa and gwet’s ac1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples. BMC Medical Research Methodology.
Zec, S., Soriani, N., Comoretto, R. I., & Baldi, I. (2017). The paradox of cohen’s kappa. Open Nursing Journal, 11:211–218.
Zha, D., Bhat, Z. P., Lai, K.-H., Yang, F., Jiang, Z., Zhong, S., & Hu, X. (2025). Data-centric artificial intelligence: A survey. ACM Comput. Surv.
Publicado
08/09/2026
Como Citar
ANDRADE, Claudio M. V. de; MAGALHÃES, Gestefane Rabbi; ASEVEDO, Raiane; PAES, Julia; OLIVEIRA, Isaias José Ramos; PAGANO, Adriana; REIS, Zilma; GONÇALVES, Marcos André.
APS-RAG-Benchmark: Um Recurso Inicial para Avaliação de RAG na Atenção Primária à Saúde. In: SIMPÓSIO BRASILEIRO DE BANCO DE DADOS (SBBD), 41. , 2026, São Carlos/SP.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2026
.
p. 1029-1035.
ISSN 2763-8979.
DOI: https://doi.org/10.5753/sbbd.2026.249660.
