LexDiverse: A Diversity-Aware Data Selection Strategy for Legal NLP
Resumo
Large Language Models (LLMs) have shown strong potential for legal Natural Language Processing (NLP), but their successful adaptation to the legal domain remains highly dependent on instruction-tuning data. Recent studies suggest that carefully selected subsets of training data can rival or even outperform full-dataset fine-tuning. However, little attention has been given to the legal domain, where documents are substantially longer and linguistically distinct from those typically used to evaluate existing approaches. To address this gap, we propose LexDiverse, a self-supervised, diversity-aware data selection strategy based on contrastive learning and clustering. The method learns document representations for long legal texts and clusters the resulting embedding space to promote semantically diverse sample selection without relying on external teacher models. We evaluate LexDiverse using LawInstruct for instruction tuning and LegalBench for downstream evaluation. Experimental results show that LexDiverse is the only evaluated strategy to improve aggregated performance on both legal-jargon and plain-English legal tasks. An ablation study further demonstrates the importance of long-context modeling and hierarchical representation learning, suggesting that the relative benefits of diversityand quality-based selection depend on the target legal task.Referências
Chalkidis, I., Fergadiotis, M., Malakasiotis, P., Aletras, N., and Androutsopoulos, I. (2020). Legal-bert: The muppets straight out of law school.
Chalkidis, I., Garneau, N., Goanta, C., Katz, D. M., and Søgaard, A. (2023). LeXFiles and LegalLAMA: Facilitating English Multinational Legal Language Model Development. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, Toronto, Canada. Association for Computational Linguistics.
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. (2020). A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, ICML’20. JMLR.org.
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., Webson, A., Gu, S. S., Dai, Z., Suzgun, M., Chen, X., Chowdhery, A., Castro-Ros, A., Pellat, M., Robinson, K., Valter, D., Narang, S., Mishra, G., Yu, A., Zhao, V., Huang, Y., Dai, A., Yu, H., Petrov, S., Chi, E. H., Dean, J., Devlin, J., Roberts, A., Zhou, D., Le, Q. V., and Wei, J. (2022). Scaling instruction-finetuned language models.
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). Bert: Pre-training of deep bidirectional transformers for language understanding.
Fama, I. C., Bueno, B. D., Alcoforado, A. T. d. S. G., Ferraz, T. P., Moya, A., and Reali Costa, A. H. (2024). No argument left behind: overlapping chunks for faster processing of arbitrarily long legal texts. In Simposio Brasileiro de Tecnologia da Informação e da Linguagem Humana. SBC.
Gao, T., Yao, X., and Chen, D. (2021). SimCSE: Simple contrastive learning of sentence embeddings. In Moens, M.-F., Huang, X., Specia, L., and Yih, S. W.-t., editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894–6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Guha, N., Nyarko, J., Ho, D. E., Ré, C., Chilton, A., Narayana, A., Chohlas-Wood, A., Peters, A., Waldon, B., Rockmore, D. N., Zambrano, D., Talisman, D., Hoque, E., Surani, F., Fagan, F., Sarfaty, G., Dickinson, G. M., Porat, H., Hegland, J., Wu, J., Nudell, J., Niklaus, J., Nay, J., Choi, J. H., Tobia, K., Hagan, M., Ma, M., Livermore, M., Rasumov-Rahe, N., Holzenberger, N., Kolt, N., Henderson, P., Rehaag, S., Goel, S., Gao, S., Williams, S., Gandhi, S., Zur, T., Iyer, V., and Li, Z. (2023). Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models.
He, Y., Wang, Z., Shen, Z., Sun, G., Dai, Y., Wu, Y., Wang, H., and Li, A. (2024). Shed: Shapley-based automated dataset refinement for instruction fine-tuning.
Katz, D. M., Bommarito, M. J., Gao, S., and Arredondo, P. (2024). Gpt-4 passes the bar exam. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 382(2270):20230254.
Li, M., Zhang, Y., Li, Z., Chen, J., Chen, L., Cheng, N., Wang, J., Zhou, T., and Xiao, J. (2024). From quantity to quality: Boosting llm performance with self-guided data selection for instruction tuning.
Liu, P., Li, Q., Tu, Q., Zhu, S., and Bi, S. (2026). Lead: legal efficiency and diversity in fine-tuning data selection through dual-metric optimization and syntactic clustering. Artificial Intelligence and Law, pages 1–23.
Liu, W., Zeng, W., He, K., Jiang, Y., and He, J. (2024). What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning.
Liu, Z., Ke, R., Liu, Y., Jiang, F., and Li, H. (2025). Take the essence and discard the dross: A rethinking on data selection for fine-tuning large language models. In Chiruzzo, L., Ritter, A., and Wang, L., editors, Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6595–6611, Albuquerque, New Mexico. Association for Computational Linguistics.
Martínez, E. (2025). Re-evaluating gpt-4’s bar exam performance. Artificial Intelligence and Law, 33(3):581–604.
Niklaus, J., Zheng, L., McCarthy, A. D., Hahn, C., Rosen, B. M., Henderson, P., Ho, D. E., Honke, G., Liang, P., and Manning, C. (2025). Lawinstruct: A resource for studying language model adaptation to the legal domain.
Ortakci, Y. and Borhan, B. (2025). Optimizing sbert for long text clustering: two novel approaches with empirical insights. The Journal of Supercomputing, 81:1–41.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. (2019). Language models are unsupervised multitask learners.
Reimers, N. and Gurevych, I. (2019). Sentence-bert: Sentence embeddings using siamese bert-networks.
Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. (2023). Stanford alpaca: An instruction-following llama model. [link].
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S., Lachaux, M.-A., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subramanian, R., Tan, X. E., Tang, B., Taylor, R., Williams, A., Kuan, J. X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T. (2023). Llama 2: Open foundation and fine-tuned chat models.
Xia, M., Malladi, S., Gururangan, S., Arora, S., and Chen, D. (2024). Less: Selecting influential data for targeted instruction tuning.
Zhang, J., Zhang, C.-X., Liu, Y., Jin, Y.-X., Yang, X.-W., Zheng, B., Liu, Y., and Guo, L.-Z. (2025a). D3: Diversity, difficulty, and dependability-aware data selection for sample-efficient llm instruction tuning.
Zhang, S., Dong, L., Li, X., Zhang, S., Sun, X., Wang, S., Li, J., Hu, R., Zhang, T., Wu, F., and Wang, G. (2025b). Instruction tuning for large language models: A survey.
Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., Yu, L., Zhang, S., Ghosh, G., Lewis, M., Zettlemoyer, L., and Levy, O. (2023). Lima: less is more for alignment. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA. Curran Associates Inc.
Chalkidis, I., Garneau, N., Goanta, C., Katz, D. M., and Søgaard, A. (2023). LeXFiles and LegalLAMA: Facilitating English Multinational Legal Language Model Development. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, Toronto, Canada. Association for Computational Linguistics.
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. (2020). A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, ICML’20. JMLR.org.
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., Webson, A., Gu, S. S., Dai, Z., Suzgun, M., Chen, X., Chowdhery, A., Castro-Ros, A., Pellat, M., Robinson, K., Valter, D., Narang, S., Mishra, G., Yu, A., Zhao, V., Huang, Y., Dai, A., Yu, H., Petrov, S., Chi, E. H., Dean, J., Devlin, J., Roberts, A., Zhou, D., Le, Q. V., and Wei, J. (2022). Scaling instruction-finetuned language models.
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). Bert: Pre-training of deep bidirectional transformers for language understanding.
Fama, I. C., Bueno, B. D., Alcoforado, A. T. d. S. G., Ferraz, T. P., Moya, A., and Reali Costa, A. H. (2024). No argument left behind: overlapping chunks for faster processing of arbitrarily long legal texts. In Simposio Brasileiro de Tecnologia da Informação e da Linguagem Humana. SBC.
Gao, T., Yao, X., and Chen, D. (2021). SimCSE: Simple contrastive learning of sentence embeddings. In Moens, M.-F., Huang, X., Specia, L., and Yih, S. W.-t., editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894–6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Guha, N., Nyarko, J., Ho, D. E., Ré, C., Chilton, A., Narayana, A., Chohlas-Wood, A., Peters, A., Waldon, B., Rockmore, D. N., Zambrano, D., Talisman, D., Hoque, E., Surani, F., Fagan, F., Sarfaty, G., Dickinson, G. M., Porat, H., Hegland, J., Wu, J., Nudell, J., Niklaus, J., Nay, J., Choi, J. H., Tobia, K., Hagan, M., Ma, M., Livermore, M., Rasumov-Rahe, N., Holzenberger, N., Kolt, N., Henderson, P., Rehaag, S., Goel, S., Gao, S., Williams, S., Gandhi, S., Zur, T., Iyer, V., and Li, Z. (2023). Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models.
He, Y., Wang, Z., Shen, Z., Sun, G., Dai, Y., Wu, Y., Wang, H., and Li, A. (2024). Shed: Shapley-based automated dataset refinement for instruction fine-tuning.
Katz, D. M., Bommarito, M. J., Gao, S., and Arredondo, P. (2024). Gpt-4 passes the bar exam. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 382(2270):20230254.
Li, M., Zhang, Y., Li, Z., Chen, J., Chen, L., Cheng, N., Wang, J., Zhou, T., and Xiao, J. (2024). From quantity to quality: Boosting llm performance with self-guided data selection for instruction tuning.
Liu, P., Li, Q., Tu, Q., Zhu, S., and Bi, S. (2026). Lead: legal efficiency and diversity in fine-tuning data selection through dual-metric optimization and syntactic clustering. Artificial Intelligence and Law, pages 1–23.
Liu, W., Zeng, W., He, K., Jiang, Y., and He, J. (2024). What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning.
Liu, Z., Ke, R., Liu, Y., Jiang, F., and Li, H. (2025). Take the essence and discard the dross: A rethinking on data selection for fine-tuning large language models. In Chiruzzo, L., Ritter, A., and Wang, L., editors, Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6595–6611, Albuquerque, New Mexico. Association for Computational Linguistics.
Martínez, E. (2025). Re-evaluating gpt-4’s bar exam performance. Artificial Intelligence and Law, 33(3):581–604.
Niklaus, J., Zheng, L., McCarthy, A. D., Hahn, C., Rosen, B. M., Henderson, P., Ho, D. E., Honke, G., Liang, P., and Manning, C. (2025). Lawinstruct: A resource for studying language model adaptation to the legal domain.
Ortakci, Y. and Borhan, B. (2025). Optimizing sbert for long text clustering: two novel approaches with empirical insights. The Journal of Supercomputing, 81:1–41.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. (2019). Language models are unsupervised multitask learners.
Reimers, N. and Gurevych, I. (2019). Sentence-bert: Sentence embeddings using siamese bert-networks.
Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. (2023). Stanford alpaca: An instruction-following llama model. [link].
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S., Lachaux, M.-A., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subramanian, R., Tan, X. E., Tang, B., Taylor, R., Williams, A., Kuan, J. X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T. (2023). Llama 2: Open foundation and fine-tuned chat models.
Xia, M., Malladi, S., Gururangan, S., Arora, S., and Chen, D. (2024). Less: Selecting influential data for targeted instruction tuning.
Zhang, J., Zhang, C.-X., Liu, Y., Jin, Y.-X., Yang, X.-W., Zheng, B., Liu, Y., and Guo, L.-Z. (2025a). D3: Diversity, difficulty, and dependability-aware data selection for sample-efficient llm instruction tuning.
Zhang, S., Dong, L., Li, X., Zhang, S., Sun, X., Wang, S., Li, J., Hu, R., Zhang, T., Wu, F., and Wang, G. (2025b). Instruction tuning for large language models: A survey.
Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., Yu, L., Zhang, S., Ghosh, G., Lewis, M., Zettlemoyer, L., and Levy, O. (2023). Lima: less is more for alignment. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA. Curran Associates Inc.
Publicado
19/10/2026
Como Citar
FAMA, Israel; ALCOFORADO, Alexandre; REALI, Anna Helena.
LexDiverse: A Diversity-Aware Data Selection Strategy for Legal NLP. In: ENCONTRO NACIONAL DE INTELIGÊNCIA ARTIFICIAL E COMPUTACIONAL (ENIAC), 23. , 2026, Cuiabá/MT.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2026
.
p. 313-324.
ISSN 2763-9061.
DOI: https://doi.org/10.5753/eniac.2026.29933.
