The Impact of Text Representations on Classification Complexity and Predictive Performance

• Filipe Araújo Dantas UPE
• Eraylson Galdino da Silva UPE

DOI: https://doi.org/10.5753/eniac.2026.31503

Resumo

This work investigates the impact of text representations on data complexity and classification performance. Five representation techniques (TF-IDF, Word2Vec, BERT, SBERT, and E5) and five classifiers were evaluated on three Portuguese text classification datasets from distinct domains. Beyond traditional performance metrics, complexity measures based on Instance Hardness were employed to analyze the quality of the generated feature spaces. The results show a consistent relationship between lower complexity and higher predictive performance. TF-IDF achieved the best overall results in most scenarios, while the effectiveness of contextual embeddings varied according to the characteristics of the dataset. The findings indicate that complexity metrics provide valuable insights for understanding and evaluating text representations.

Referências

Almatrafi, O. and Johri, A. (2022). Improving moocs using information from discussion forums: An opinion summarization and suggestion mining approach. IEEE Access, 10:15565–15573.

Atanasova, P., Simonsen, J. G., Lioma, C., and Augenstein, I. (2022). Diagnostics-guided explanation generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 10445–10453.

Choi, M., Kim, D., and Kim, J. (2026). Defining abusive news categories: Proposing a detection model for digital media integrity. Applied Sciences, 16(7):3190.

Das, A. K., Agrahari, S. K., Bhagat, K., and Pati, P. B. (2025). Enhanced analysis of viva dialogues: Classification of questions, answers, and statements with bert and catboost. In 2025 5th Asian Conference on Innovation in Technology (ASIANCON), pages 1–7. IEEE.

Dhingra, M., Dhabliya, D., Dubey, M. K., Gupta, A., and Reddy, D. H. (2022). A review on comparison of machine learning algorithms for text classification. In 2022 5th International Conference on Contemporary Computing and Informatics (IC3I), pages 1818–1823. IEEE.

Domladovac, M. (2021). Comparison of neural network with gradient boosted trees, random forest, logistic regression and svm in predicting student achievement. In 2021 44th International Convention on Information, Communication and Electronic Technology (MIPRO), pages 211–216. IEEE.

Dong, Y., Cordonnier, J.-B., and Loukas, A. (2021). Attention is not all you need: Pure attention loses rank doubly exponentially with depth. In International conference on machine learning, pages 2793–2803. PMLR.

Ethayarajh, K. (2019). How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pages 55–65.

Farhangian, F., Ensina, L. A., Cavalcanti, G. D., and Cruz, R. M. (2025). Dres: Fake news detection by dynamic representation and ensemble selection. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 20034–20052.

Harsha, T. M., Moukthika, G. S., Sai, D. S., Pravallika, M. N. R., Anamalamudi, S., and Enduri, M. (2022). Automated resume screener using natural language processing (nlp). In 2022 6th international conference on trends in electronics and informatics (ICOEI), pages 1772–1777. IEEE.

Hazim, L. R. and Ata, O. (2024). Textual authenticity in the ai era: evaluating bert and roberta with logistic regression and neural networks for text classification. In 2024 International Symposium on Electronics and Telecommunications (ISETC), pages 1–6. IEEE.

Kumar, A., Sharma, R., and Bedi, P. (2024). Towards optimal nlp solutions: Analyzing gpt and llama-2 models across model scale, dataset size, and task diversity. Engineering, Technology & Applied Science Research, 14(3):14219–14224.

Lavi, T., Shapira, B., and Rappoport, N. (2026). Trajectory-based difficulty scoring for reliable learning on tabular data. arXiv preprint arXiv:2605.24680.

Lorena, A. C., Garcia, L. P., Lehmann, J., Souto, M. C., and Ho, T. K. (2019). How complex is your classification problem? a survey on measuring classification complexity. ACM Computing Surveys (CSUR), 52(5):1–34.

MadhuMohan, A., Raghu, H., Priyanka, R., Dsouza, S. M., Teena, K., and Shetty, S. (2024). Automated text extraction and classification from images using multi-layer perceptron (mlp). In 2024 International Conference on Knowledge Engineering and Communication Systems (ICKECS), volume 1, pages 1–6. IEEE.

Nam, J., Choi, J.-W., Shin, Y.-G., and Park, S. (2023). A bert-based artificial intelligence to analyze free-text clinical notes for binary classification in papillary thyroid carcinoma recurrence. In 2023 IEEE International Conference on Consumer Electronics (ICCE), pages 1–2. IEEE.

Noguti, M. Y., Vellasques, E., and Oliveira, L. S. (2020). Legal document classification: An application to law area prediction of petitions to public prosecution service. In 2020 International joint conference on neural networks (IJCNN), pages 1–8. IEEE.

Patil, R., Boit, S., Gudivada, V., and Nandigam, J. (2023). A survey of text representation and embedding techniques in nlp. IEEe Access, 11:36120–36146.

Reimers, N. and Gurevych, I. (2019). Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pages 3982–3992.

Scheible-Schmitt, R., He, H., and Mendes, A. B. (2025). Portbert: Navigating the depths of portuguese language models. In Proceedings of the Workshop on Beyond English: Natural Language Processing for all Languages in an Era of Large Language Models, pages 59–71.

Sha, L., Raković, M., Das, A., Gašević, D., and Chen, G. (2022). Leveraging class balancing techniques to alleviate algorithmic bias for predictive tasks in education. IEEE Transactions on Learning Technologies, 15(4):481–492.

Shao, Z., Ahmad, M. N., and Javed, A. (2024). Comparison of random forest and xgboost classifiers using integrated optical and sar features for mapping urban impervious surface. Remote Sensing, 16(4):665.

Shen, Y. and Zhang, P. K. (2024). Financial sentiment analysis on news and reports using large language models and finbert. In 2024 IEEE 6th International Conference on Power, Intelligent Computing and Systems (ICPICS), pages 717–721. IEEE.

Smith, M. R., Martinez, T., and Giraud-Carrier, C. (2014). An instance level analysis of data complexity. Machine learning, 95(2):225–256.

Song, S., Wang, Z., Xu, S., Ni, S., and Xiao, J. (2019). A novel text classification approach based on word2vec and textrank keyword extraction. In 2019 IEEE Fourth International Conference on Data Science in Cyberspace (DSC), pages 536–543. IEEE.

Souza, F., Nogueira, R., and Lotufo, R. (2020). Bertimbau: pretrained bert models for brazilian portuguese. In Brazilian conference on intelligent systems, pages 403–417. Springer.

Sparck Jones, K. (1972). A statistical interpretation of term specificity and its application in retrieval. Journal of documentation, 28(1):11–21.

Vaičiukynas, E., Danėnas, P., Ablonskis, L., Šukys, A., Dambrauskas, E., Žitkus, V., Butkienė, R., and Butleris, R. (2026). Comparison of modern multilingual text embedding techniques for hate speech detection task. Applied Sciences, 16(10):5099.