Evaluating Prompting Strategies for Clinical Information Extraction in Portuguese using LangExtract

• Bruno Rogério S. dos Santos UECE / ISCTE-IUL / INESC-ID
• Eugénio Ribeiro ISCTE-IUL / INESC-ID
• Leonardo S. Rocha UECE

DOI: https://doi.org/10.5753/eniac.2026.31511

Resumo

This work presents an initial systematic evaluation of LangExtract, an LLM-based extraction library, applied to LusoClin, a dataset of clinical notes in European Portuguese. The study compares zero-shot and few-shot prompt strategies for NER, using prompts adapted to the clinical domain and evaluating performance with Precision, Recall, and F1-score in Strict, Exact, Partial Match and Entity Type. When compared to a fine-tuned model and a rule-based model, the results show that LangExtract achieves performance close to that of mmBERT (F1-score up to 0.79), suggesting that LLM-based extraction via LangExtract can be a viable alternative for information extraction without the need to train on large volumes of data.

Referências

Al Nazi, Z., Hossain, M. R., and Al Mamun, F. (2025). Evaluation of open and closedsource llms for low-resource language with zero-shot, few-shot, and chain-of-thought prompting. Natural Language Processing Journal, 10:100124.

Almeida, T. S., Abonizio, H., Nogueira, R., and Pires, R. (2024). Sabiá-2: A new generation of portuguese large language models. arXiv preprint arXiv:2403.09887.

Anjos de Almeida, V., Saorin da Silva, S., Chire, J., Vicenzi, L., Borges, N. H., Kociolek, H., Miriã de Castro Rocha, S., Nassif Gomes, F., Ferreira, J. C., Marques, O., et al. (2026). Clinical named entity recognition in the portuguese language: a benchmark of modern bert models and llms. arXiv preprint arXiv:22603.26510.

Bender, E. M. (2019). The bender rule: On naming the languages we study and why it matters. The Gradient.

Cunha, L. F., Campos, R., and Jorge, A. (2023). Event extraction for portuguese: a qadriven approach using ace-2005. In EPIA Conference on Artificial Intelligence, pages 402–414. Springer.

Cunha, L. F., Silvano, P., Campos, R., and Jorge, A. (2024). Ace-2005-pt: Corpus for event extraction in portuguese. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 661–666.

Da Rocha, N. C., Barbosa, A. M. P., Schnr, Y. O., Machado-Rugolo, J., de Andrade, L. G. M., Corrente, J. E., and de Arruda Silveira, L. V. (2022). Natural language processing to extract information from portuguese-language medical records. Data, 8(1):11.

Da Silva, D. P., da Rosa Fröhlich, W., De Mello, B. H., Vieira, R., and Rigo, S. J. (2023). Exploring named entity recognition and relation extraction for ontology and medical records integration. Informatics in medicine unlocked, 43:101381.

Félix, D. and Teixeira Lopes, C. (2025). Dataset of synthetic clinical notes in european portuguese generated using an open-source large language model, along with prompting and evaluation data [data set]. inesc tec.

Goel, A. (2025). LangExtract: Llm-powered structured information extraction from text with source grounding. [link].

Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al. (2024). Gpt-4o system card. arXiv preprint arXiv:2410.21276.

Leite, M. A. A. (2024). Ontology-based extraction and structuring of narrative elements from clinical texts. Master’s thesis, Universidade do Porto (Portugal).

Lopes, F., Teixeira, C., and Oliveira, H. G. (2019). Contributions to clinical named entity recognition in portuguese. In Proceedings of the 18th BioNLP Workshop and Shared Task, pages 223–233.

Marone, M., Weller, O., Fleshman, W., Yang, E., Lawrie, D., and Van Durme, B. (2025). mmbert: A modern multilingual encoder with annealed language learning. arXiv preprint arXiv:2509.06888.

Nunes, M., Boné, J., Ferreira, J. C., Chaves, P., and Elvas, L. B. (2024). Medialbertina: An european portuguese medical language model. Computers in Biology and Medicine, 182:109233.

Oliveira, L. E. S. e., Peters, A. C., Da Silva, A. M. P., Gebeluca, C. P., Gumiel, Y. B., Cintho, L. M. M., Carvalho, D. R., Al Hasan, S., and Moro, C. M. C. (2022). Semclinbr-a multi-institutional and multi-specialty semantically annotated corpus for portuguese clinical nlp tasks. Journal of Biomedical Semantics, 13(1):13.

Sivarajkumar, S., Kelley, M., Samolyk-Mazzanti, A., Visweswaran, S., and Wang, Y. (2024). An empirical evaluation of prompting strategies for large language models in zero-shot clinical natural language processing: algorithm development and validation study. JMIR Medical Informatics, 12:e55318.

Wang, S., Sun, X., Li, X., Ouyang, R., Wu, F., Zhang, T., Li, J., Wang, G., and Guo, C. (2025). Gpt-ner: Named entity recognition via large language models. In Findings of the association for computational linguistics: NAACL 2025, pages 4257–4275.