A Comparative Study of BERT, LLM-Based, and LLM2Vec Embeddings for Text Classification

• Matheus Yasuo Ribeiro Utino USP
• Ricardo Marcondes Marcacini USP

DOI: https://doi.org/10.5753/eniac.2026.29699

Resumo

This work presents a systematic comparison of BERT, raw hidden-layer embeddings from decoder-only Large Language Models (LLMs), and LLM2Vec for text classification across 11 datasets. The evaluation uses K-Nearest Neighbors as the downstream classifier, with hyperparameters tuned through Bayesian optimization. The results show that LLM2Vec achieves the best overall performance, while LLM2Vec and BERT form a statistically comparable performance tier. In contrast, raw LLM embeddings present lower and more variable performance, with only LLM2Vec showing a statistically significant advantage over them. Within LLM2Vec, supervised variants consistently outperform unsupervised ones, highlighting the importance of supervised contrastive learning. Regarding prompt engineering, raw LLM embeddings benefit from task-aligned instructions, whereas LLM2Vec representations remain robust to prompt variation. These findings indicate that encoder-oriented architectures, including BERT and adapted LLM2Vec models, provide more reliable representations for text classification than raw hidden-layer embeddings from decoder-only LLMs.

Referências

Akiba, T., Sano, S., Yanase, T., Ohta, T., and Koyama, M. (2019). Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2623–2631.

Barman, K. G., Caron, S., Sullivan, E., de Regt, H. W., de Austri, R. R., Boon, M., Färber, M., Fröse, S., Golling, T., Lopez, L. G., et al. (2025). Large physics models: towards a collaborative approach with large language models and foundation models. The European Physical Journal C, 85(9):1066.

BehnamGhader, P., Adlakha, V., Mosbach, M., Bahdanau, D., Chapados, N., and Reddy, S. (2024). LLM2vec: Large language models are secretly powerful text encoders. In First Conference on Language Modeling.

Buckmann, M. and Hill, E. (2024). Logistic regression makes small llms strong and explainable" tens-of-shot" classifiers. arXiv preprint arXiv:2408.03414.

da Costa, L. S., Oliveira, I. L., and Fileto, R. (2023). Text classification using embeddings: a survey. Knowledge and Information Systems, 65(7):2761–2803.

Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Burstein, J., Doran, C., and Solorio, T., editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.

Ding, M., Zhou, C., Yang, H., and Tang, J. (2020). Cogltx: Applying bert to long texts. Advances in Neural Information Processing Systems, 33:12792–12804.

Gao, S., Alawad, M., Young, M. T., Gounley, J., Schaefferkoetter, N., Yoon, H. J., Wu, X.-C., Durbin, E. B., Doherty, J., Stroup, A., et al. (2021). Limitations of transformers on clinical text classification. IEEE journal of biomedical and health informatics, 25(9):3596–3607.

Henry, M. M., Hervanto, N. A., Isnan, M., Kurnianingrum, D., Ratnapuri, C. I., and Pardamean, B. (2024). Llm2vec sentence embeddings analysis in sentiment classification. In 2024 IEEE International Conference on Data and Software Engineering (ICoDSE), pages 161–165. IEEE.

Jiang, J., Wang, F., Shen, J., Kim, S., and Kim, S. (2026). A survey on large language models for code generation. ACM Trans. Softw. Eng. Methodol., 35(2).

Muennighoff, N., Tazi, N., Magne, L., and Reimers, N. (2023). MTEB: Massive text embedding benchmark. In Vlachos, A. and Augenstein, I., editors, Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2014–2037, Dubrovnik, Croatia. Association for Computational Linguistics.

Nair, A. R., M, S., and Gupta, D. (2023). Comparative analysis of word embeddings for text classification in spark nlp. In 2023 IEEE International Conference on Cloud Computing in Emerging Markets (CCEM), pages 130–136.

Reusens, M., Stevens, A., Tonglet, J., De Smedt, J., Verbeke, W., vanden Broucke, S., and Baesens, B. (2024). Evaluating text classification: A benchmark study. Expert Systems with Applications, 254:124302.

Roumeliotis, K. I., Tselikas, N. D., and Nasiopoulos, D. K. (2025). Fake news detection and classification: A comparative study of convolutional neural networks, large language models, and natural language processing models. Future Internet, 17(1).

Sharma, N. A., Ali, A. S., and Kabir, M. A. (2025). A review of sentiment analysis: tasks, applications, and deep learning techniques. International journal of data science and analytics, 19(3):351–388.

Shen, L., Wang, Y., Li, Z., and Ma, W. (2025). Sms spam detection using bert and multi-graph convolutional networks. International journal of intelligent networks, 6:79–88.

Tao, C., Shen, T., Gao, S., Zhang, J., Li, Z., Tao, Z., and Ma, S. (2024). Llms are also effective embedding models: An in-depth overview. arXiv preprint arXiv:2412.12591.

Tennenholtz, G., Chow, Y., Hsu, C.-W., Jeong, J., Shani, L., Tulepbergenov, A., Ramachandran, D., Mladenov, M., and Boutilier, C. (2023). Demystifying embedding spaces using large language models. arXiv preprint arXiv:2310.04475.

Utino, M. and Marcacini, R. (2025). Bert vs. llm2vec: A comparative study of embedding models for semantic information retrieval. In Anais do XXII Encontro Nacional de Inteligência Artificial e Computacional, pages 427–438, Porto Alegre, RS, Brasil. SBC.

Utino, M. Y. R., Matsushima, E. H., Paes, A., and Mann, P. (2026). Synthetic data for mental health: A comparative analysis of llms, bert, and copy-based augmentation. In de Freitas, R. and Furtado, D., editors, Intelligent Systems, pages 393–408, Cham. Springer Nature Switzerland.

Wang, C., Nulty, P., and Lillis, D. (2020). A comparative study on word embeddings in deep learning for text classification. In Proceedings of the 4th international conference on natural language processing and information retrieval, pages 37–46.

Wang, P.-Y., Liu, T.-S., Wang, C., Li, Z., Wang, Y., Yan, S., Jia, C., Liu, X.-H., Chen, X., Xu, J., and Yu, Y. (2026). A survey on large language models for mathematical reasoning. ACM Comput. Surv., 58(8).

Zhao, W., Yang, X., Lyu, Z., Xu, C., and Guan, Z. (2025). Road of large language model: Source, challenge, and future perspectives. Research, 8:0655.

Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026). A survey of large language models. Frontiers of Computer Science, 20(12):2012627.