Classification of prostate histological images using Vision Transformers: an analysis with stain normalization and ensemble learning

• Betânia Caroline Silva de Albuquerque UNIFESP
• Hanna Beatriz Couto Monteiro Fernandes de Castro UNIFESP
• Leandro Alves Neves UNESP
• Marcelo Zanchetta do Nascimento UFU
• Thaína Aparecida Azevedo Tosta UNIFESP

Resumo

Prostate cancer is the second most common cancer in men worldwide, diagnosed via histopathological evaluation of H&E-stained images. Gleason grading, however, is subjective and prone to inter-observer variability. Deep learning-based computer-aided diagnosis systems offer promising support, but stain color variations pose a challenge, motivating normalization algorithms. This study evaluates color normalization on H&E prostate cancer image classification using a pre-trained ViT and eight classifiers, including a majority voting ensemble. Binary classification on a public dataset compared benign and malignant cases across two normalization methods (SW-CCN and BKSVD) and original images. Results showed original images yielded superior ViT and classifier performance, despite more malignant cases being misclassified as benign. SVM with ViT feature extraction achieved the best overall performance, surpassing both the ensemble and ViT classifier.

Referências

J. Ferlay, M. Colombet, I. Soerjomataram, D. M. Parkin, M. Piñeros, A. Znaor, and F. Bray, “Cancer statistics for the year 2020: An overview,” International journal of cancer, vol. 149, no. 4, pp. 778–789, 2021.

J. Ferlay, M. Ervik, F. Lam, M. Colombet, L. Mery, M. Piñeros, A. Znaor, I. Soerjomataram, and F. Bray, “Global cancer observatory: cancer today,” Lyon: International agency for research on cancer, vol. 20182020, 2020.

P. Rawla, “Epidemiology of prostate cancer,” World journal of oncology, vol. 10, no. 2, p. 63, 2019.

Instituto Nacional de Câncer (Brasil), Estimativa 2023: incidência de câncer no Brasil. Rio de Janeiro: INCA, 2022, il. color.

M. N. Gurcan, L. E. Boucheron, A. Can, A. Madabhushi, N. M. Rajpoot, and B. Yener, “Histopathological image analysis: A review,” IEEE reviews in biomedical engineering, vol. 2, pp. 147–171, 2009.

W. C. Allsbrook Jr, K. A. Mangold, M. H. Johnson, R. B. Lane, C. G. Lane, and J. I. Epstein, “Interobserver reproducibility of gleason grading of prostatic carcinoma: general pathologist,” Human pathology, vol. 32, no. 1, pp. 81–88, 2001.

S. M. Ayyad, N. B. Abdel-Hamid, H. A. Ali, and L. M. Labib, “Multimodality imaging in prostate cancer diagnosis using artificial intelligence: basic concepts and current state-of-the-art,” Multimedia Tools and Applications, pp. 1–30, 2025.

J. J. Twilt, K. G. van Leeuwen, H. J. Huisman, J. J. Fütterer, and M. de Rooij, “Artificial intelligence based algorithms for prostate cancer classification and detection on magnetic resonance imaging: a narrative review,” Diagnostics, vol. 11, no. 6, p. 959, 2021.

S. Wang, K. Burtt, B. Turkbey, P. Choyke, and R. M. Summers, “Computer aided-diagnosis of prostate cancer on multiparametric mri: a technical review of current research,” BioMed research international, vol. 2014, no. 1, p. 789561, 2014.

X. Jiang, Z. Hu, S. Wang, and Y. Zhang, “Deep learning for medical image-based cancer diagnosis. cancers, 15 (14), 3608,” 2023.

R. A. Hoffman, S. Kothari, and M. D. Wang, “Comparison of normalization algorithms for cross-batch color segmentation of histopathological images,” in 2014 36th annual international conference of the IEEE engineering in medicine and biology society. IEEE, 2014, pp. 194–197.

A. Sethi, L. Sha, A. R. Vahadane, R. J. Deaton, N. Kumar, V. Macias, and P. H. Gann, “Empirical comparison of color normalization methods for epithelial-stromal classification in h and e images,” Journal of pathology informatics, vol. 7, no. 1, p. 17, 2016.

F. Ciompi, O. Geessink, B. E. Bejnordi, G. S. De Souza, A. Baidoshvili, G. Litjens, B. Van Ginneken, I. Nagtegaal, and J. Van Der Laak, “The importance of stain normalization in colorectal tissue classification with convolutional networks,” in 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017). IEEE, 2017, pp. 160–163.

C. Hu, X. Sun, Z. Yuan, and Y. Wu, “Classification of breast cancer histopathological image with deep residual learning,” International Journal of Imaging Systems and Technology, vol. 31, no. 3, pp. 1583–1594, 2021.

M. Salvi, F. Molinari, U. R. Acharya, L. Molinaro, and K. M. Meiburger, “Impact of stain normalization and patch selection on the performance of convolutional neural networks in histological breast and prostate cancer classification,” Computer Methods and Programs in Biomedicine Update, vol. 1, p. 100004, 2021.

J. Boschman, H. Farahani, A. Darbandsari, P. Ahmadvand, A. Van Spankeren, D. Farnell, A. B. Levine, J. R. Naso, A. Churg, S. J. Jones et al., “The utility of color normalization for ai-based diagnosis of hematoxylin and eosin-stained pathology images,” The Journal of Pathology, vol. 256, no. 1, pp. 15–24, 2022.

F. Pérez-Bueno, J. G. Serra, M. Vega, J. Mateos, R. Molina, and A. K. Katsaggelos, “Bayesian k-svd for h and e blind color deconvolution. applications to stain normalization, data augmentation and cancer classification,” Computerized Medical Imaging and Graphics, vol. 97, p. 102048, 2022.

R. Bazargani, W. Chen, S. Sadeghian, M. Asadi, J. Boschman, A. Darbandsari, A. Bashashati, and S. Salcudean, “A novel h&e color augmentation for domain invariance classification of unannotated histopathology prostate cancer images,” in Medical Imaging 2023: Digital and Computational Pathology, vol. 12471. SPIE, 2023, pp. 224–229.

T. A. A. Tosta, A. D. Freitas, P. R. de Faria, L. A. Neves, A. S. Martins, and M. Z. do Nascimento, “A stain color normalization with robust dictionary learning for breast cancer histological images processing,” Biomedical Signal Processing and Control, vol. 85, p. 104978, 2023.

N. Altini, T. M. Marvulli, F. A. Zito, M. Caputo, S. Tommasi, A. Azzariti, A. Brunetti, B. Prencipe, E. Mattioli, S. De Summa et al., “The role of unpaired image-to-image translation for stain color normalization in colorectal cancer histology classification,” Computer Methods and Programs in Biomedicine, vol. 234, p. 107511, 2023.

V. B. Fernandes, A. B. Silva, D. C. Pereira, S. V. Cardoso, P. R. de Faria, A. M. Loyola, T. A. Tosta, L. A. Neves, and M. Z. do Nascimento, “Investigation of deep neural network compression based on tucker decomposition for the classification of lesions in cavity oral,” Proceedings Copyright, vol. 516, p. 523, 2024.

G. Lee, M. Bajger, and K. Clark, “Deep learning and color variability in breast cancer histopathological images: a preliminary study,” in 14th International Workshop on Breast Imaging (IWBI 2018), vol. 10718. SPIE, 2018, pp. 370–375.

A. Kumar, S. K. Singh, S. Saxena, K. Lakshmanan, A. K. Sangaiah, H. Chauhan, S. Shrivastava, and R. K. Singh, “Deep feature learning for histopathological image classification of canine mammary tumors and human breast cancer,” Information Sciences, vol. 508, pp. 405–421, 2020.

F. Bianconi, J. N. Kather, and C. C. Reyes-Aldasoro, “Experimental assessment of color deconvolution and color normalization for automated classification of histology images stained with hematoxylin and eosin,” Cancers, vol. 12, no. 11, p. 3337, 2020.

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929, 2020.

M. López-Pérez, A. Morquecho, A. Schmidt, F. Pérez-Bueno, A. Martín-Castro, J. Mateos, and R. Molina, “The crowdgleason dataset: Learning the gleason grade from crowds and experts,” Computer Methods and Programs in Biomedicine, vol. 257, p. 108472, 2024.

X. Li and K. N. Plataniotis, “A complete color normalization approach to histopathology images using color cues computed from saturation-weighted statistics,” IEEE Transactions on Biomedical Engineering, vol. 62, no. 7, pp. 1862–1873, 2015.

A. F. Q. da Silva, A. D. Freitas, P. R. de Faria, L. A. Neves, M. Z. Do Nascimento, and T. A. A. Tosta, “Color normalization by dictionary learning with nuclear segmentation evaluation in he histological images,” in 38th IEEE International Symposium on Computer-Based Medical Systems (CBMS). IEEE, 2025.

K. Taunk, S. De, S. Verma, and A. Swetapadma, “A brief review of nearest neighbor algorithm for learning and classification,” in 2019 international conference on intelligent computing and control systems (ICCS). IEEE, 2019, pp. 1255–1260.

T. M. Mitchell and T. M. Mitchell, Machine learning. McGraw-hill New York, 1997, vol. 1, no. 9.

T. G. Nick and K. M. Campbell, “Logistic regression,” Topics in biostatistics, pp. 273–301, 2007.

W. Chen, X. Xie, J. Wang, B. Pradhan, H. Hong, D. T. Bui, Z. Duan, and J. Ma, “A comparative study of logistic model tree, random forest, and classification and regression tree models for spatial prediction of landslide susceptibility,” Catena, vol. 151, pp. 147–160, 2017.

E. Y. Boateng, J. Otoo, and D. A. Abaye, “Basic tenets of classification algorithms k-nearest-neighbor, support vector machine, random forest and neural network: A review,” Journal of Data Analysis and Information Processing, vol. 8, no. 4, pp. 341–357, 2020.

Z.-H. Zhou and Z.-H. Zhou, Ensemble learning. Springer, 2021.

T. Hastie, R. Tibshirani, J. Friedman et al., “The elements of statistical learning,” 2009.