JACV: A Benchmark for Legal-Argumentative Coherence Verification in Brazilian Portuguese

• Willgnner F. Santos UFG / TJGO
• João P. C. Presa UFG
• Sávio S. T. de Oliveira UFG
• Antônio P. de Castro Júnior TJGO
• Fernando R. Trindade TJGO
• Marcella S. R. Martins UTFPR
• Paulo V. dos Santos UFG
• Ronaldo M. da Costa UFG
• Nádia F. F. da Silva UFCAT

DOI: https://doi.org/10.5753/eniac.2026.30903

Resumo

We present JACV, a benchmark for legal-argumentative coherence in Brazilian Portuguese procedural documents, built from 120 valid anchors drawn from 1,000 authentic anonymized documents. JACV comprises two tasks: three-class classification (JACV-CLS) and adversarial relief selection (JACV-ADV). Across statistical baselines, encoders, fine-tuned models, LLMs, and Legal Chain-of-Thought prompting, the best models reach 99.58% on JACV-ADV and 76.94% on JACV-CLS; a TF-IDF baseline achieves 98.33% on JACV-ADV, exposing residual lexical separability, while Legal-CoT yields parsing error rates of 47–58%, revealing protocol-level limitations. Code and benchmark instances are available in a public repository.

Referências

Anthropic (2026a). Claude Opus 4.7. Access em: 24 abr. 2026.

Anthropic (2026b). System Card: Claude Sonnet 4.6. Access em: 24 abr. 2026.

Brasil (2018). Lei no 13.709, de 14 de agosto de 2018. lei geral de proteção de dados pessoais (lgpd). [link]. Access em 22 abr. 2026.

Brito, M., Pinheiro, V., Furtado, V., Monteiro Neto, J., Bomfim, F., Costa, A. d., and Silveira, R. (2023). CDJUR-BR - uma coleção dourada do judiciário brasileiro com entidades nomeadas refinadas. In Proceedings of the 14th Brazilian Symposium in Information and Human Language Technology, pages 176–195, Belo Horizonte, Brasil. Association for Computational Linguistics.

Camilo-Junior, C. G., Oliveira, S. S. T., Pereira, L. A., Amadeus, M., Scotti, R., Fazzioni, D., Novais, A. M. A., and Jordão, S. A. A. (2025). GAIA: An open language model for brazilian portuguese. [link].

Chu, Y. Y., Huang, S.-c., and Shao, H.-L. (2025). Unpacking legal reasoning in LLMs: Chain-of-thought as a key to human-machine alignment in essay-based NLU tasks. In Proceedings of the 5th Workshop on Natural Logic Meets Machine Learning (NALOMA), pages 1–7, Bochum, Alemanha. Association for Computational Linguistics.

Conselho Nacional de Justiça (2025a). Justiça em números: Judiciário reduziu acervo e alcançou produtividade histórica em 2024. [link]. Access em 22 abr. 2026.

Conselho Nacional de Justiça (2025b). Resolução no 615, de 11 de março de 2025. [link]. Access em 22 abr. 2026.

Dahl, M., Magesh, V., Suzgun, M., and Ho, D. E. (2024). Large legal fictions: Profiling legal hallucinations in large language models. arXiv preprint arXiv:2401.01301.

Devlin, J., Chang, M., Lee, K., and Toutanova, K. (2018). BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805.

Fei, Z., Shen, X., Zhu, D., Zhou, F., Han, Z., Huang, A., Zhang, S., Chen, K., Yin, Z., Shen, Z., Ge, J., and Ng, V. (2024). Lawbench: Benchmarking legal knowledge of large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Miami, Florida, USA. Association for Computational Linguistics.

Garcia, E. A. S., Silva, N. F. F., Siqueira, F., Albuquerque, H. O., Gomes, J. R. S., Souza, E., and Lima, E. A. (2024). RoBERTaLexPT: A legal RoBERTa model pretrained with deduplication for Portuguese. In Gamallo, P., Claro, D., Teixeira, A., Real, L., Garcia, M., Oliveira, H. G., and Amaro, R., editors, Proceedings of the 16th International Conference on Computational Processing of Portuguese, pages 374–383, Santiago de Compostela, Galicia/Spain. Association for Computational Lingustics.

Goebel, R., Kano, Y., Kim, M.-Y., Rabelo, J., Satoh, K., and Yoshioka, M. (2024). Overview of benchmark datasets and methods for the legal information extraction/entailment competition (COLIEE) 2024. In JSAI International Symposium on Artificial Intelligence, pages 109–124. Springer.

Guha, N., Nyarko, J., Ho, D. E., Ré, C., Chilton, A., Narayana, A., Chohlas-Wood, A., Peters, A., Waldon, B., Rockmore, D. N., et al. (2023). Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models. In Advances in Neural Information Processing Systems, volume 36.

Laitz, T., Almeida, T. S., Abonizio, H., Junior, R. M., Bonás, G. K., Piau, M., Larcher, C., Pires, R., and Nogueira, R. (2026). Sabi\’a-4 technical report. arXiv preprint arXiv:2603.10213.

Luz de Araujo, P. H., de Campos, T. E., Ataides Braz, F., and Correia da Silva, N. (2020). VICTOR: a dataset for brazilian legal documents classification. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 1449–1458, Marseille, France. European Language Resources Association.

McCoy, R. T., Pavlick, E., and Linzen, T. (2019). Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428–3448. Association for Computational Linguistics.

Mistral AI (2025). Mistral Large 3. Access em: 24 abr. 2026.

Polo, F. M., Mendonça, G. C. F., Parreira, K. C. J., Gianvechio, L. d. G., Cordeiro, P. S., Ferreira, J. B., Lima, L. M. P. d., Maia, A. C. d. A., and Vicente, R. (2021). Legalnlp: Natural language processing methods for the brazilian legal language. In Anais do Encontro Nacional de Inteligência Artificial e Computacional, Porto Alegre, Brasil. SBC.

Servantez, S., Barrow, J., Hammond, K., and Jain, R. (2024). Chain of logic: Rule-based reasoning with large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 2721–2733, Bangkok, Tailândia. Association for Computational Linguistics.

Silveira, R., Ponte, C., Almeida, V., Pinheiro, V., and Furtado, V. (2023). Legalbert-pt: A pretrained language model for the brazilian portuguese legal domain. In Brazilian Conference on Intelligent Systems, pages 268–282. Springer.

Souza, F., Nogueira, R., and Lotufo, R. (2020). Bertimbau: Pretrained bert models for brazilian portuguese. In Intelligent Systems, volume 12319 of Lecture Notes in Computer Science, pages 403–417. Springer.

T.y.s.s., S., Haddad, R., and Grabmair, M. (2024). ECtHR-PCR: A dataset for precedent understanding and prior case retrieval in the european court of human rights. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 5473–5483, Torino, Italia. ELRA and ICCL.

Viegas, C. F. O., Costa, B. C., and Ishii, R. P. (2023). JurisBERT: A new approach that converts a classification corpus into an STS one. In Computational Science and Its Applications – ICCSA 2023, pages 349–365. Springer Nature Switzerland.

Yu, F., Quartey, L., and Schilder, F. (2023). Exploring the effectiveness of prompt engineering for legal reasoning tasks. In Findings of the Association for Computational Linguistics: ACL 2023, pages 13582–13596, Toronto, Canada. Association for Computational Linguistics.