Evaluating Generative AI Tools on the Brazilian Federal Revenue’s Q&A Dataset

• Erick de Brito UFCA
• Matheus Teotonio UFCA
• Roberto Lotufo UNICAMP
• Jayr Pereira UFCA / UNICAMP

DOI: https://doi.org/10.5753/stil.2025.37872

Abstract

Artificial Intelligence language models revolutionized the pursuit of digital information, therefore being broadly utilized due to their ease of use and range of topics. However, it is impratical to guarantee that these models maintain an adequate knowledge on unsatisfactorily documented topics, even when such topics are relevant or useful, such as the Brazilian individual income tax return (DIRPF). This study evaluates the coherence of responses provided by these generative tools by submitting to them a set of frequently asked questions on the subject. The results indicate that commercial LLMs are a convenient means of obtaining support related to the DIRPF.

References

Cabello, O. G. and Nakao, S. H. (2021). Complexidade, conformidade e arrecadação tributária. Economia e Sociedade, 30(3):1033–1050.

Coneglian, C. S., Torino, E., Segundo, J. E. S., and Vidotti, S. A. B. G. (2024). Inteligência artificial generativa e recuperação da informação: Tendências e oportunidades de pesquisa. In XXIII ENCONTRO NACIONAL DE PESQUISA E PÓS-GRADUAÇÃO EM CIÊNCIA DA INFORMAÇÃO.

Es, S., James, J., Espinosa Anke, L., and Schockaert, S. (2024). RAGAs: Automated evaluation of retrieval augmented generation. In Aletras, N. and De Clercq, O., editors, Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 150–158, St. Julians, Malta. Association for Computational Linguistics.

Júnior, J. D., Faria, A., de Oliveira, E. S., de Brito, E., Teotonio, M., Assumpção, A., Carmo, D., Lotufo, R., and Pereira, J. (2025). Br-taxqa-r: A dataset for question answering with references for brazilian personal income tax law, including case law. arXiv preprint arXiv:2505.15916.

Lin, C.-Y. (2004). ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.

Magalhães, B. and Matos, F. (2025). Falsas verdades. o impacto das alucinações de ia nos processos judiciais administrativos. Revista Eletrônica de Direito Processual, 26(2).

Maia, A. S. (2017). Declaração de imposto de renda de pessoas físicas: principais dificuldades dos contribuintes.

Maleki, N., Padmanabhan, B., and Dutta, K. (2024). Ai hallucinations: a misnomer worth clarifying. In 2024 IEEE conference on artificial intelligence (CAI), pages 133–138. IEEE.

Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2002). Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.

Trench, E. E. (2024). Raça, gênero e outros atributos do contribuinte e probabilidade de cair na “malha fina” da receita federal.