Análise de Falhas na Detecção de Alucinação em Textos Jurídico-Policiais em Português
Resumo
Este artigo documenta as falhas dos mecanismos de detecção de alucinação quando aplicados a textos jurídico-policiais brasileiros gerados por modelos de linguagem. Três pipelines de avaliação foram testadas, utilizando o modelo Lynx como auditor com votação por Self-Consistency (k=3) em infraestrutura AWS SageMaker. Os resultados revelam alto recall, porém precisão baixa em todas as configurações, com falsos positivos causados por dois padrões: rigidez referencial e opacidade normativa. Os achados indicam que as ferramentas atuais de verificação de factualidade não distinguem adequadamente inferências institucionais de conteúdo fabricado neste domínio especializado.
Referências
BANG, Yejin; JI, Ziwei; SCHELTEN, Alan; HARTSHORN, Anthony; FOWLER, Tara; ZHANG, Cheng; CANCEDDA, Nicola; FUNG, Pascale. HalluLens: LLM hallucination benchmark. In: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Vienna: ACL, 2025. p. 24128–24156. DOI: 10.18653/v1/2025.acl-long.1176
BENDER, E. M.; GEBRU, T.; MCMILLAN-MAJOR, A.; SHMITCHELL, S. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?. In: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (FAccT '21). Association for Computing Machinery, New York, NY, USA, p. 610–623, 2021.
CARD, D.; HENDERSON, P.; KHANDELWAL, U.; MOORE, R.; SMITH, N. A. With Little Power Comes Great Responsibility: Efficient and Effective Evaluation in NLP. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), p. 9263–9274, 2020.
CHEN, Wilbur Xinyuan; SRINIVASAN, Suraj; ZAKERINIA, Saleh. Displacement or complementarity? The labor market impact of generative AI. Harvard Business School Working Paper, n. 25-039, 2024.
COSTA, Maria Izabel Plath da. Terminologia jurídico-policial: seleção e validação de termos em textos-base e mapa-domínio. Tradterm, São Paulo, Brasil, v. 24, p. 301–324, 2015. DOI: 10.11606/issn.2317-9511.tradterm.2014.96574.
DAHL, M.; MAGESH, V.; SUZGUN, M.; HO, D. E. Large legal fictions: profiling legal hallucinations in large language models. Journal of Legal Analysis, v. 16, n. 1, p. 64–93, 2024. DOI: 10.1093/jla/laae003.
GOYAL, Shubh; HIRA, Medha; MISHRA, Shubham; GOYAL, Sukriti; GOEL, Arnav; DADU, Niharika; DB, Kirushikesh; MEHTA, Sameep; MADAAN, Nishtha. LLMGuard: guarding against unsafe LLM behavior. Proceedings of the AAAI Conference on Artificial Intelligence, v. 38, n. 21, p. 23790–23792, 2024. DOI: 10.1609/aaai.v38i21.30566
JI, Ziwei; LEE, Nayeon; FRIESKE, Rita; et al. Survey of Hallucination in Natural Language Generation. ACM Computing Surveys, v. 55, n. 12, 2022. DOI: 10.1145/3571730.
LEBRET, Rémi; GRANGIER, David; AULI, Michael. Neural text generation from structured data with application to the biography domain. In: SU, Jian; DUH, Kevin; CARRERAS, Xavier (ed.). Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Austin, Texas: Association for Computational Linguistics, 2016. p. 1203–1213. DOI: 10.18653/v1/D16-1128.
LIU, Yang; ITER, Dan; XU, Yichong; WANG, Shuohang; XU, Ruochen; and ZHU, Chenguang. G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Singapore: ACL, 2023. p. 2511-2522. DOI: 10.18653/v1/2023.emnlp-main.153.
LIU, Xuannan; YANG, Xiao; ZEKUN, Li; LI, Peipei; HE, Ran. AgentHallu: Benchmarking Automated Hallucination Attribution of LLM-based Agents. arXiv preprint arXiv:2601.06818, 2026.
MANAKUL, Potsawee; LIUSIE, Adian; GALES, Mark. SelfCheckGPT: zero-resource black-box hallucination detection for generative large language models. In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Singapore: ACL, 2023. p. 9004–9017. DOI: 10.18653/v1/2023.emnlp-main.557
RAVI, S. S. et al. Lynx: An Open Source Hallucination Evaluation Model. arXiv preprint arXiv:2407.08488, 2024.
TAMBER, Manveer Singh; BAO, Forrest Sheng; XU, Chenyu; LUO, Ge; KAZI, Suleman; BAE, Minseok; LI, Miaoran; MENDELEVITCH, Ofer; QU, Renyi; LIN, Jimmy. Benchmarking LLM faithfulness in RAG with evolving leaderboards. In: Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track. Suzhou: ACL, 2025. p. 799–811. DOI: 10.18653/v1/2025.emnlp-industry.54.
YANG, An et al. Qwen3 Technical Report. arXiv preprint arXiv:2505.09388, 2025.
BENDER, E. M.; GEBRU, T.; MCMILLAN-MAJOR, A.; SHMITCHELL, S. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?. In: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (FAccT '21). Association for Computing Machinery, New York, NY, USA, p. 610–623, 2021.
CARD, D.; HENDERSON, P.; KHANDELWAL, U.; MOORE, R.; SMITH, N. A. With Little Power Comes Great Responsibility: Efficient and Effective Evaluation in NLP. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), p. 9263–9274, 2020.
CHEN, Wilbur Xinyuan; SRINIVASAN, Suraj; ZAKERINIA, Saleh. Displacement or complementarity? The labor market impact of generative AI. Harvard Business School Working Paper, n. 25-039, 2024.
COSTA, Maria Izabel Plath da. Terminologia jurídico-policial: seleção e validação de termos em textos-base e mapa-domínio. Tradterm, São Paulo, Brasil, v. 24, p. 301–324, 2015. DOI: 10.11606/issn.2317-9511.tradterm.2014.96574.
DAHL, M.; MAGESH, V.; SUZGUN, M.; HO, D. E. Large legal fictions: profiling legal hallucinations in large language models. Journal of Legal Analysis, v. 16, n. 1, p. 64–93, 2024. DOI: 10.1093/jla/laae003.
GOYAL, Shubh; HIRA, Medha; MISHRA, Shubham; GOYAL, Sukriti; GOEL, Arnav; DADU, Niharika; DB, Kirushikesh; MEHTA, Sameep; MADAAN, Nishtha. LLMGuard: guarding against unsafe LLM behavior. Proceedings of the AAAI Conference on Artificial Intelligence, v. 38, n. 21, p. 23790–23792, 2024. DOI: 10.1609/aaai.v38i21.30566
JI, Ziwei; LEE, Nayeon; FRIESKE, Rita; et al. Survey of Hallucination in Natural Language Generation. ACM Computing Surveys, v. 55, n. 12, 2022. DOI: 10.1145/3571730.
LEBRET, Rémi; GRANGIER, David; AULI, Michael. Neural text generation from structured data with application to the biography domain. In: SU, Jian; DUH, Kevin; CARRERAS, Xavier (ed.). Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Austin, Texas: Association for Computational Linguistics, 2016. p. 1203–1213. DOI: 10.18653/v1/D16-1128.
LIU, Yang; ITER, Dan; XU, Yichong; WANG, Shuohang; XU, Ruochen; and ZHU, Chenguang. G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Singapore: ACL, 2023. p. 2511-2522. DOI: 10.18653/v1/2023.emnlp-main.153.
LIU, Xuannan; YANG, Xiao; ZEKUN, Li; LI, Peipei; HE, Ran. AgentHallu: Benchmarking Automated Hallucination Attribution of LLM-based Agents. arXiv preprint arXiv:2601.06818, 2026.
MANAKUL, Potsawee; LIUSIE, Adian; GALES, Mark. SelfCheckGPT: zero-resource black-box hallucination detection for generative large language models. In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Singapore: ACL, 2023. p. 9004–9017. DOI: 10.18653/v1/2023.emnlp-main.557
RAVI, S. S. et al. Lynx: An Open Source Hallucination Evaluation Model. arXiv preprint arXiv:2407.08488, 2024.
TAMBER, Manveer Singh; BAO, Forrest Sheng; XU, Chenyu; LUO, Ge; KAZI, Suleman; BAE, Minseok; LI, Miaoran; MENDELEVITCH, Ofer; QU, Renyi; LIN, Jimmy. Benchmarking LLM faithfulness in RAG with evolving leaderboards. In: Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track. Suzhou: ACL, 2025. p. 799–811. DOI: 10.18653/v1/2025.emnlp-industry.54.
YANG, An et al. Qwen3 Technical Report. arXiv preprint arXiv:2505.09388, 2025.
Publicado
19/07/2026
Como Citar
BARCELAR, Ricardo Rodrigues; VENTURA, Thiago Meirelles.
Análise de Falhas na Detecção de Alucinação em Textos Jurídico-Policiais em Português. In: SIMPÓSIO DE INFRAESTRUTURA DIGITAL/NUVEM PARA PESQUISA (PESQUISA@NUVEM), 1. , 2026, Gramado/RS.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2026
.
p. 11-19.
DOI: https://doi.org/10.5753/pesquisanuvem.2026.23945.
