Hierarchical Approach with BERT for Classification of Long Legal Documents: An Analysis of Feature Aggregation Techniques

• Jasson Carvalho da Silva UFPI
• Ricardo Andrade Lira Rabelo UFPI
• Weslley Emmanuel Martins Lima UFPI
• Vitor Augusto Correa Cortez Almeida UFPI

DOI: https://doi.org/10.5753/ercemapi.2024.243441

Abstract

The study aims to evaluate and implement feature aggregation techniques in hierarchical models to deal with long texts in the legal domain. Transform-based models, although effective in many Natural Language Processing problems, face difficulties due to their quadratic complexity. due to quadratic complexity, especially with texts that exceed 512 tokens. To overcome these challenges, the study proposes investigating methods such as hierarchical models, text division strategies, local and sparse attention, with the aim of improving the processing and analysis of these texts, providing valuable insights for understanding and effective use in legal contexts.

References

Chalkidis, I., Androutsopoulos, I., and Aletras, N. (2019). Neural legal judgment prediction in English. In Korhonen, A., Traum, D., and Màrquez, L., editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4317–4323, Florence, Italy. Association for Computational Linguistics.

Chalkidis, I., Fergadiotis, M., Tsarapatsanis, D., Aletras, N., Androutsopoulos, I., and Malakasiotis, P. (2021). Paragraph-level rationale extraction through regularization: A case study on european court of human rights cases. arXiv preprint arXiv:2103.13084.

Chalkidis, I., Jana, A., Hartung, D., Bommarito, M., Androutsopoulos, I., Katz, D., and Aletras, N. (2022). LexGLUE: A benchmark dataset for legal language understanding in English. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4310–4330, Dublin, Ireland. Association for Computational Linguistics.

Chen, B., Sun, R., Dai, Y., Zheng, H.-T., and Zhang, R. (2022). Lordbert: Embedding long text by segment ordering with bert. In 2022 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE.

Conselho Nacional de Justiça (2023). Justiça em números 2023.

Dai, X., Chalkidis, I., Darkner, S., and Elliott, D. (2022). Revisiting transformer-based models for long document classification. arXiv preprint arXiv:2204.06683.

Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv pre-print arXiv:1810.04805.

Jawahar, G., Sagot, B., and Seddah, D. (2019). What does bert learn about the structure of language? In ACL 2019-57th Annual Meeting of the Association for Computational Linguistics.

Spaeth, H. J., Epstein, L., Segal, J. A., Martin, A. D., Ruger, T. J., and Benesh, S. C. (2020). Supreme court database, version 2020 release 01. [link]. Accessed: 2024-06-02.

Tuteja, M. and Juclà, D. G. (2023). Long text classification using transformers with paragraph selection strategies. In Proceedings of the Natural Legal Language Processing Workshop 2023, pages 17–24.

Wagh, V., Khandve, S., Joshi, I., Wani, A., Kale, G., and Joshi, R. (2021). Comparative study of long document classification. In TENCON 2021-2021 IEEE Region 10 Conference (TENCON), pages 732–737. IEEE.