Comparing TF-IDF, Sentence-BERT, and LSTM for Self-Admitted Technical Debt Classification

• Rafael Santos Silva UPE
• Auston Ricardo Soares Costa UPE
• Cleyton Mário de Oliveira Rodrigues UPE
• Wylliams Barbosa Santos UPE

DOI: https://doi.org/10.5753/eniac.2026.31301

Resumo

Self-Admitted Technical Debt (SATD) is the explicit acknowledgment of technical debt in software artifacts. This study compared TF-IDF, Word2Vec, and Sentence-BERT representations for SATD classification using five combinations on 4,071 comments. TF-IDF + linear SVM achieved the best overall performance, with a macro-F1 of 0.7444 and low computational cost. Sentence-BERT-based approaches did not outperform lexical models, while Word2Vec + LSTM showed a strong tendency toward the majority class and substantially lower macro-F1. Under the evaluated conditions, TF-IDF combined with linear models provided the best overall results.

Referências

Cunningham, W. (1992). The wycash portfolio management system. ACM Sigplan Oops Messenger, 4(2):29–30.

da Silva Maldonado, E., Shihab, E., and Tsantalis, N. (2017). Using natural language processing to automatically detect self-admitted technical debt. IEEE Transactions on Software Engineering, 43(11):1044–1062.

Fontana, F. A., Di Rocco, J., Di Ruscio, D., Di Salle, A., and Nguyen, P. T. (2025). Binary and multi-class classification of self-admitted technical debt: How far can we go? Information and Software Technology, page 107862.

Gorripati, S. M. C., Altalbe, A., and Rangarajan, P. K. (2025). Automated classification of self-admitted technical debt using advanced word embedding techniques. Journal of Computational and Cognitive Engineering.

Gu, H., Zhang, S., Huang, Q., Liao, Z., Liu, J., and Lo, D. (2024). Self-admitted technical debts identification: How far are we? In 2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), pages 804–815. IEEE.

He, H., Bai, Y., Garcia, E. A., and Li, S. (2008). Adasyn: Adaptive synthetic sampling approach for imbalanced learning. In IEEE International Joint Conference on Neural Networks, pages 1322–1328.

He, H. and Garcia, E. A. (2009). Learning from imbalanced data. IEEE Transactions on Knowledge and Data Engineering, 21(9):1263–1284.

Huang, Q., Shihab, E., Xia, X., Lo, D., and Li, S. (2018). Identifying self-admitted technical debt in open source projects using text mining. Empirical Software Engineering, 23(1):418–451.

Joachims, T. (1998). Text categorization with support vector machines: Learning with many relevant features. In European Conference on Machine Learning, pages 137–142. Springer.

Li, Y., Soliman, M., and Avgeriou, P. (2023). Automatic identification of self-admitted technical debt from four different sources. Empirical Software Engineering, 28(3):65.

Potdar, A. and Shihab, E. (2014). An exploratory study on self-admitted technical debt. In 2014 IEEE International Conference on Software Maintenance and Evolution, pages 91–100. IEEE.

Reimers, N. and Gurevych, I. (2019). Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pages 3982–3992.

Ren, X., Xing, Z., Xia, X., Lo, D., Wang, X., and Grundy, J. (2019). Neural network-based detection of self-admitted technical debt: From performance to explainability. ACM transactions on software engineering and methodology (TOSEM), 28(3):1–45.

Santos, R. M., Santos, I. M., Júnior, M. C. R., and de Mendonça Neto, M. G. (2020). Long term-short memory neural networks and word2vec for self-admitted technical debt detection. In ICEIS (2), pages 157–165.

Selvanayagam, N., Ghaleb, T. A., and Abdellatif, M. (2026). Self-admitted technical debt in llm software: An empirical comparison with ml and non-ml software. arXiv preprint arXiv:2601.06266.

Sierra, G., Shihab, E., and Kamei, Y. (2019). A survey of self-admitted technical debt. Journal of Systems and Software, 152:70–82.