An Ultra-Fine Entity Typing Method Based on Ensemble of Large Language Models
Resumo
Ultra-fine Entity Typing (UFET) is an evolution of Named Entity Recognition (NER) that proposes the classification of entities at an ultra-fine level, based on a set of free-form phrases that adequately and comprehensively describe them. Increasing the number of labels while maintaining performance remains a challenge, and most current models are capable of classifying entities into a limited set of types. The proven efficiency of Large Language Models (LLMs) in improving performance in Natural Language Processing tasks presents an opportunity to enhance UFET results. In this sense, this paper proposes an ensemble method of fine-tuned public LLMs aiming to maximize the efficiency of entity classification at the ultra-fine level and expand the set of labels. Experiments show the effectiveness of the proposed method, which outperformed baselines metrics in almost all scenarios and improved the set of types.
Palavras-chave:
fine tuning, machine learning, NER, prompt learning
Referências
AI@Meta. Llama 3 model card. arXiv:2407.21783 , 2024.
Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., Hui, B., Ji, L., Li, M., Lin, J., Lin, R., Liu, D., Liu, G., Lu, C., Lu, K., Ma, J., Men, R., Ren, X., Ren, X., Tan, C., Tan, S., Tu, J., Wang, P., Wang, S., Wang, W., Wu, S., Xu, B., Xu, J., Yang, A., Yang, H., Yang, J., Yang, S., Yao, Y., Yu, B., Yuan, H., Yuan, Z., Zhang, J., Zhang, X., Zhang, Y., Zhang, Z., Zhou, C., Zhou, J., Zhou, X., and Zhu, T. Qwen technical report. arXiv preprint arXiv:2309.16609 , 2023.
Choi, E., Levy, O., Choi, Y., and Zettlemoyer, L. Ultra-Fine Entity Typing, 2018. arXiv:1807.04905 [cs].
Dai, H., Song, Y., and Wang, H. Ultra-Fine Entity Typing with Weak Supervision from a Masked Language Model. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, Online, pp. 1790–1799, 2021.
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, 2019. arXiv:1810.04805 [cs].
Ding, N., Hu, S., Zhao, W., Chen, Y., Liu, Z., Zheng, H., and Sun, M. OpenPrompt: An Open-source Framework for Prompt-learning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, V. Basile, Z. Kozareva, and S. Stajner (Eds.). Association for Computational Linguistics, Dublin, Ireland, pp. 105–113, 2022.
Feng, Y., Pratapa, A., and Mortensen, D. R. Calibrated Seq2seq Models for Efficient and Generalizable Ultra-fine Entity Typing, 2023. arXiv:2311.00835 [cs].
Gemma Team, T. M., Hardin, C., Dadashi, R., Bhupatiraju, S., Sifre, L., Rivière, M., Kale, M. S., Love, J., Tafti, P., Hussenot, L., and et al. Gemma: Open models based on gemini research and technology, 2024.
Gillick, D., Lazic, N., Ganchev, K., Kirchner, J., and Huynh, D. Context-Dependent Fine-Grained Entity Type Tagging, 2016. arXiv:1412.1820 [cs] version: 2.
Grishman, R. and Sundheim, B. Message Understanding Conference-6: A Brief History. In COLING 1996 Volume 1: The 16th International Conference on Computational Linguistics, 1996.
Hearst, M. A. Automatic acquisition of hyponyms from large text corpora. In COLING 1992 Volume 2: The 14th International Conference on Computational Linguistics, 1992.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. LoRA: Low-Rank Adaptation of Large Language Models, 2021. arXiv:2106.09685 [cs].
Jehangir, B., Radhakrishnan, S., and Agarwal, R. A survey on Named Entity Recognition — datasets, tools, and methodologies. Natural Language Processing Journal vol. 3, pp. 100017, June, 2023.
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M.-A., Stock, P., Scao, T. L., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mistral 7b, 2023.
Li, J., Sun, A., Han, J., and Li, C. A Survey on Deep Learning for Named Entity Recognition. IEEE Trans. Knowl. Data Eng. 34 (1): 50–70, 2020. arXiv:1812.09449 [cs].
Li, Z., Li, X., Liu, Y., Xie, H., Li, J., Wang, F.-l., Li, Q., and Zhong, X. Label Supervised LLaMA Finetuning, 2023. arXiv:2310.01208 [cs].
Ling, X. and Weld, D. Fine-Grained Entity Recognition. AAAI 26 (1): 94–100, 2012.
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., and Neubig, G. Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing. ACM Comput. Surv. 55 (9): 1–35, Sept., 2023.
Mai, K., Pham, T.-H., Nguyen, M. T., Nguyen, T. D., Bollegala, D., Sasano, R., and Sekine, S. An Empirical Study on Fine-Grained Named Entity Recognition. In Proceedings of the 27th International Conference on Computational Linguistics. Association for Computational Linguistics, Santa Fe, New Mexico, USA, pp. 711–722, 2018.
Qian, J., Liu, Y., Liu, L., Li, Y., Jiang, H., Zhang, H., and Shi, S. Fine-grained Entity Typing without Knowledge Base. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, pp. 5309–5319, 2021.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research 21 (140): 1–67, 2020.
Sekine, S. Extended Named Entity Ontology with Attribute Information. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08), N. Calzolari, K. Choukri, B. Maegaard, J. Mariani, J. Odijk, S. Piperidis, and D. Tapias (Eds.). European Language Resources Association (ELRA), Marrakech, Morocco, 2008.
Wang, Y., Xin, X., and Guo, P. An Empirical Study of Pre-trained Embedding on Ultra-Fine Entity Typing. In 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, Toronto, ON, Canada, pp. 2656–2661, 2020.
Xiong, W., Wu, J., Lei, D., Yu, M., Chang, S., Guo, X., and Wang, W. Y. Imposing Label-Relational Inductive Bias for Extremely Fine-Grained Entity Typing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, Minneapolis, Minnesota, pp. 773–784, 2019.
Yadav, V. and Bethard, S. A Survey on Recent Advances in Named Entity Recognition from Deep Learning models. In Proceedings of the 27th International Conference on Computational Linguistics, E. M. Bender, L. Derczynski, and P. Isabelle (Eds.). Association for Computational Linguistics, Santa Fe, New Mexico, USA, pp. 2145–2158, 2018.
Yosef, M. A., Bauer, S., Hoffart, J., Spaniol, M., and Weikum, G. HYENA: Hierarchical Type Classification for Entity Names. In Proceedings of COLING 2012: Posters. The COLING 2012 Organizing Committee, Mumbai, India, pp. 1361–1370, 2012.
Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., Hui, B., Ji, L., Li, M., Lin, J., Lin, R., Liu, D., Liu, G., Lu, C., Lu, K., Ma, J., Men, R., Ren, X., Ren, X., Tan, C., Tan, S., Tu, J., Wang, P., Wang, S., Wang, W., Wu, S., Xu, B., Xu, J., Yang, A., Yang, H., Yang, J., Yang, S., Yao, Y., Yu, B., Yuan, H., Yuan, Z., Zhang, J., Zhang, X., Zhang, Y., Zhang, Z., Zhou, C., Zhou, J., Zhou, X., and Zhu, T. Qwen technical report. arXiv preprint arXiv:2309.16609 , 2023.
Choi, E., Levy, O., Choi, Y., and Zettlemoyer, L. Ultra-Fine Entity Typing, 2018. arXiv:1807.04905 [cs].
Dai, H., Song, Y., and Wang, H. Ultra-Fine Entity Typing with Weak Supervision from a Masked Language Model. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, Online, pp. 1790–1799, 2021.
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, 2019. arXiv:1810.04805 [cs].
Ding, N., Hu, S., Zhao, W., Chen, Y., Liu, Z., Zheng, H., and Sun, M. OpenPrompt: An Open-source Framework for Prompt-learning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, V. Basile, Z. Kozareva, and S. Stajner (Eds.). Association for Computational Linguistics, Dublin, Ireland, pp. 105–113, 2022.
Feng, Y., Pratapa, A., and Mortensen, D. R. Calibrated Seq2seq Models for Efficient and Generalizable Ultra-fine Entity Typing, 2023. arXiv:2311.00835 [cs].
Gemma Team, T. M., Hardin, C., Dadashi, R., Bhupatiraju, S., Sifre, L., Rivière, M., Kale, M. S., Love, J., Tafti, P., Hussenot, L., and et al. Gemma: Open models based on gemini research and technology, 2024.
Gillick, D., Lazic, N., Ganchev, K., Kirchner, J., and Huynh, D. Context-Dependent Fine-Grained Entity Type Tagging, 2016. arXiv:1412.1820 [cs] version: 2.
Grishman, R. and Sundheim, B. Message Understanding Conference-6: A Brief History. In COLING 1996 Volume 1: The 16th International Conference on Computational Linguistics, 1996.
Hearst, M. A. Automatic acquisition of hyponyms from large text corpora. In COLING 1992 Volume 2: The 14th International Conference on Computational Linguistics, 1992.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. LoRA: Low-Rank Adaptation of Large Language Models, 2021. arXiv:2106.09685 [cs].
Jehangir, B., Radhakrishnan, S., and Agarwal, R. A survey on Named Entity Recognition — datasets, tools, and methodologies. Natural Language Processing Journal vol. 3, pp. 100017, June, 2023.
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M.-A., Stock, P., Scao, T. L., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mistral 7b, 2023.
Li, J., Sun, A., Han, J., and Li, C. A Survey on Deep Learning for Named Entity Recognition. IEEE Trans. Knowl. Data Eng. 34 (1): 50–70, 2020. arXiv:1812.09449 [cs].
Li, Z., Li, X., Liu, Y., Xie, H., Li, J., Wang, F.-l., Li, Q., and Zhong, X. Label Supervised LLaMA Finetuning, 2023. arXiv:2310.01208 [cs].
Ling, X. and Weld, D. Fine-Grained Entity Recognition. AAAI 26 (1): 94–100, 2012.
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., and Neubig, G. Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing. ACM Comput. Surv. 55 (9): 1–35, Sept., 2023.
Mai, K., Pham, T.-H., Nguyen, M. T., Nguyen, T. D., Bollegala, D., Sasano, R., and Sekine, S. An Empirical Study on Fine-Grained Named Entity Recognition. In Proceedings of the 27th International Conference on Computational Linguistics. Association for Computational Linguistics, Santa Fe, New Mexico, USA, pp. 711–722, 2018.
Qian, J., Liu, Y., Liu, L., Li, Y., Jiang, H., Zhang, H., and Shi, S. Fine-grained Entity Typing without Knowledge Base. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, pp. 5309–5319, 2021.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research 21 (140): 1–67, 2020.
Sekine, S. Extended Named Entity Ontology with Attribute Information. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08), N. Calzolari, K. Choukri, B. Maegaard, J. Mariani, J. Odijk, S. Piperidis, and D. Tapias (Eds.). European Language Resources Association (ELRA), Marrakech, Morocco, 2008.
Wang, Y., Xin, X., and Guo, P. An Empirical Study of Pre-trained Embedding on Ultra-Fine Entity Typing. In 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, Toronto, ON, Canada, pp. 2656–2661, 2020.
Xiong, W., Wu, J., Lei, D., Yu, M., Chang, S., Guo, X., and Wang, W. Y. Imposing Label-Relational Inductive Bias for Extremely Fine-Grained Entity Typing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, Minneapolis, Minnesota, pp. 773–784, 2019.
Yadav, V. and Bethard, S. A Survey on Recent Advances in Named Entity Recognition from Deep Learning models. In Proceedings of the 27th International Conference on Computational Linguistics, E. M. Bender, L. Derczynski, and P. Isabelle (Eds.). Association for Computational Linguistics, Santa Fe, New Mexico, USA, pp. 2145–2158, 2018.
Yosef, M. A., Bauer, S., Hoffart, J., Spaniol, M., and Weikum, G. HYENA: Hierarchical Type Classification for Entity Names. In Proceedings of COLING 2012: Posters. The COLING 2012 Organizing Committee, Mumbai, India, pp. 1361–1370, 2012.
Publicado
17/11/2024
Como Citar
FERREIRA, Carlos Alberto F.; FERREIRA, Anderson A..
An Ultra-Fine Entity Typing Method Based on Ensemble of Large Language Models. In: SYMPOSIUM ON KNOWLEDGE DISCOVERY, MINING AND LEARNING (KDMILE), 12. , 2024, Belém/PA.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2024
.
p. 153-160.
ISSN 2763-8944.
DOI: https://doi.org/10.5753/kdmile.2024.244760.
