Open-Weight Language Models for Long-Context Academic Summarization

• João M. Souto UFC
• Vinicius S. Pereira IFCE
• Paulo Vinícius Gomes IFCE
• Valberto R. Feitosa IFCE
• Carlos Caminha UFC

DOI: https://doi.org/10.5753/eniac.2026.30709

Resumo

This work evaluates open-weight language models for the automatic generation of abstracts from full-length doctoral theses. We build a benchmark comprising 30 theses from the University of Oxford, removing the original abstracts from the input and using them only as references. The approach combines local inference with 19 open-weight models and Chain of Density-based prompting, comparing the results against Gemini 2.5 Pro and the Lead-N extractive baseline. The evaluation uses a custom Abstract Quality metric based on LLM-as-a-Judge. The results indicate better performance for larger models, with gpt-oss-120b standing out, while an ablation study showed modest gains from Chain of Density.

Referências

Adams, G., Fabbri, A., Ladhak, F., Lehman, E., and Elhadad, N. (2023). From sparse to dense: Gpt-4 summarization with chain of density prompting. arXiv preprint arXiv:2309.04269.

Bomfim, F., Neto, J. M., Filho, G. B., Furtado, V., and Pinheiro, V. (2024). Sara - a generative ai for legal process summarization based on chain of density prompt engineering. In Anais da XXXIV Brazilian Conference on Intelligent Systems, pages 370–383, Porto Alegre, RS, Brasil. SBC.

Esashika, R., Figueiredo, C. M. S., and de Melo, T. (2025). Structuring information from initial petitions using llms: A study in brazilian courts of justice. Anais do XXII Encontro Nacional de Inteligência Artificial e Computacional (ENIAC).

Fei, W., Niu, X., Zhou, P., Hou, L., Bai, B., Deng, L., and Han, W. (2024). Extending context window of large language models via semantic compression. Findings of the Association for Computational Linguistics: ACL 2024.

Garcez, L. R. and Cozman, F. G. (2025). Reasoning with llms aided by knowledge bases and by context. Anais do XXII Encontro Nacional de Inteligência Artificial e Computacional (ENIAC).

Järvinen, E. (2023). Long-input summarization using large language models. Master’s Thesis, Aalto University.

Jin, Y., Wang, Z., Lin, J., et al. (2024). A comprehensive evaluation of quantization strategies for large language models. arXiv preprint arXiv:2402.16775.

Justino, A. F., Jacob Junior, A. F. L., Marcacini, R. M., and Lobato, F. M. F. (2025). Evaluating bert models for semantic retrieval in long portuguese legal documents. Anais do XXII Encontro Nacional de Inteligência Artificial e Computacional (ENIAC).

Koh, H. Y., Ju, J., Liu, M., and Pan, S. (2022). An empirical survey on long document summarization: Datasets, models, and metrics. ACM Computing Surveys, 55(8):1–35.

Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P. (2023). Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.03172.

Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. (2023). Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36:46534–46594.

Maynez, J., Narayan, S., Bohnet, B., and McDonald, R. (2020). On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906–1919. Association for Computational Linguistics.

Narayan, S., Zhao, Y., Maynez, J., Simões, G., Nikolaev, V., and McDonald, R. (2021). Planning with learned entity prompts for abstractive summarization. Transactions of the Association for Computational Linguistics, 9:1475–1492.

Oukelmoun, A., Semmar, N., de Chalendar, G., Cormi, C., Oukelmoun, M., Vibert, E., and Allard, M.-A. (2025). Detecting omissions in LLM-generated medical summaries. In Potdar, S., Rojas-Barahona, L., and Montella, S., editors, Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 325–337, Suzhou (China). Association for Computational Linguistics.

Oxford (2025). Oxford university research archive: Diverse domain dataset for doctoral research. Acessado em: 12 mai. 2026.

Saito, K., Wataoka, K., Wachi, A., and Akimoto, Y. (2023). Verbosity bias in preference labeling by large language models.

Santos, W. F., Galvão Filho, A. R., de Oliveira, S. S. T., and Presa, J. P. C. (2025). Comparing prompt-based llms, fine-tuning, and classical models for legal text classification in portuguese. Anais do XXII Encontro Nacional de Inteligência Artificial e Computacional (ENIAC).

Teufel, S. and Moens, M. (2002). Summarizing scientific articles: experiments with relevance and rhetorical status. Computational linguistics, 28(4):409–445.

Zhou, J., Lu, T., Mishra, S., Brahma, S., Basu, S., Luan, Y., Zhou, D., and Hou, L. (2023). Instruction-following evaluation for large language models.