Comparative Analysis of Deep Learning Architectures and Morphological Pre-processing for Prostate Cancer Histopathology

• Maxwell Gomes da Silva UFU
• Bruno Augusto Nassif Travençolo UFU
• André R. Backes UFSCar

DOI: https://doi.org/10.5753/sbcas.2026.20574

Resumo

Prostate cancer remains one of the main causes of male mortality worldwide. The histopathological assessment based on hematoxylin-eosin (H&E) staining continues to be the diagnostic gold standard. However, the process is subjective and computationally demanding because of the size and complexity of Whole Slide Images (WSIs). This paper presents a comparative study of deep learning architectures and morphological pre-processing strategies for prostate cancer histopathology. The proposed approach combines a morphological pre-processing stage with a Mask R-CNN model implemented in Detectron2 and compares its performance with other benchmark architectures, including GAN-based segmentation, hierarchical transformers (HIPT), Multiple Instance Learning (MIL), and the CrowdGleason framework. Performance was evaluated using accuracy, F1-score, Intersection over Union (IoU), and Cohen’s Kappa. The integration of morphological operations with Mask R-CNN achieved the best results, reaching 97.87% accuracy and a 0.96 F1-score. These findings reinforce how domain-guided enhancement can significantly improve segmentation quality and generalization in digital histopathology

Referências

Abadi, M. et al. (2016). Tensorflow: A system for large-scale machine learning.

Arvaniti, E., Fricker, N., Moret, M., Rupp, N., Hermanns, T., Fankhauser, C., Wey, N., Wild, P. J., Rueschoff, J. H., and Claassen, M. (2018). Automated gleason grading of prostate cancer tissue microarrays via deep learning. Scientific reports, 8(1):1–11.

Brazil (2002). Programa nacional de controle de câncer da próstata: documento de consenso. [link]. Retrieved 04 24, 2022.

Brazil (2022). Câncer de próstata. [link]. Retrieved 06 04, 2022.

Bulten, W., Kartasalo, K., Chen, P.-H. C., Ström, P., Pinckaers, H., Nagpal, K., Cai, Y., Steiner, D. F., van Boven, H., Vink, R., et al. (2022). Artificial intelligence for diagnosis and gleason grading of prostate cancer: a retrospective, multicohort study. Nature Medicine, 28(1):154–163.

Chen, R. J., Chen, T., Li, Y., Wang, J., Williamson, D. F., Lipkova, J., and Mahmood, F. (2023). Scaling vision transformers to gigapixel images via hierarchical self-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4164–4175.

da Silva, M. G., Travençolo, B. A. N., and Backes, A. R. (2025). Deep learning for image analysis and diagnosis aid of prostate cancer. In 20th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, volume 3, pages 699–706.

Epstein, J. I., Egevad, L., Amin, M., Delahunt, B., Srigley, J. R., and Humphrey, P. A. (2016). The 2014 international society of urological pathology (isup) consensus conference on gleason grading of prostatic carcinoma: Definition of grading patterns and proposal for a new grading system. The American journal of surgical pathology, 40(2):244–252.

Fan, K. (2025). Machine Learning Techniques for Medical Image Analysis with Data Scarcity. PhD thesis, University of Southampton.

He, K., Gkioxari, G., Dollár, P., and Girshick, R. (2017). Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961–2969.

He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778.

Johnson, J. M. and Khoshgoftaar, T. M. (2023). A survey of deep learning with imbalanced data. Journal of Big Data, 10(1):82.

Kaggle (2023). kaggle.com. Retrieved from [link].

Landis, J. R. and Koch, G. G. (1977). The measurement of observer agreement for categorical data.

Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. (2014). Microsoft COCO: Common objects in context. In European Conference on Computer Vision (ECCV).

Loeb, S., Bjurlin, M. A., Nicholson, J., Tammela, T. L., Penson, D. F., Carrol, H. B., and Etzioni, R. (2014). Overdiagnosis and overtreatment of prostate cancer. European Urology, 65(6):10.

López-Pérez, Y., Pérez-Paredes, J., Villalobos-Quesada, J., Maroñas, O., Fernández-Berni, J., Carmona-Sáez, P., and Rodríguez, J. (2024). The crowdgleason dataset: Learning the gleason grade from crowds and experts. Computer Methods and Programs in Biomedicine, 257:108472.

Mai, C., Wang, Q., Mai, Z., Qin, C., Zeng, J., Xie, H., Xiao, Y., Huang, H., Chen, W., Yan, W., et al. (2024). The application of multi-instance learning based on feature reconstruction and cross-mixing in the gleason grading of prostate cancer from whole-slide images. Quantitative Imaging in Medicine and Surgery, 14(7):5076.

Mescher, A. L. (2021). Junqueira’s Basic Histology: Text and Atlas. MC Graw Hill, Indiana, USA, 16 edition.

Rodriguez, J. S., Colomer, A., Sales, M. A., Molina, R., and Naranjo, V. (2020). Going deeper through the gleason scoring scale: An automatic end-to-end system for histology prostate grading and cribriform pattern detection. Computer Methods and Programs in Biomedicine, 195:105637.

Silva, F. d. A., Nascimento, A. A. d., and Medeiros, L. M. d. (2022). Uso da morfologia matemática na segmentação de imagens médicas para identificar o miocárdio com redes neurais convolucionais. Proceeding Series of the Brazilian Society of Computational and Applied Mathematics, 9(1).

Society, A. C. (2023). Facts & figures 2023. [link]. Retrieved from cancer.org.

Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (2016). Rethinking the inception architecture for computer vision. pages 2818–2826.

Taha, A. A. and Hanbury, A. (2015). Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool. BMC Medical Imaging, 15(1):29.

Vats, S., Al-Heejawi, S. M. A., Kondejkar, T., Breggia, A., Ahmad, B., Christman, R., Ryan, S. T., and Amal, S. (2024). Segmenting tumor gleason pattern using generative ai and digital pathology: Use case of prostate cancer on miccai dataset. Preprints.org.

Viso.ai (2024). Intersection over union (iou) for object detection. [link]. Acessado em: 2025-10-09.

Wu, Y., Kirillov, A., Massa, F., Lo, W.-Y., and Girshick, R. (2019). Detectron2: A pytorch-based modular object detection library. arXiv preprint arXiv:1904.04514.