Parallel-Hierarchical Multi-Task Learning for Skin Lesion Classification with Multimodal Data
Abstract
This paper introduces a novel approach for skin cancer detection that leverages both smartphone-captured images and clinical data. The proposed method utilizes a parallel-hierarchical multi-task learning framework to classify skin lesion types and assess malignancy simultaneously. Compared to existing methods reporting false positive rates around 3.6% in the Melanoma Skin Cancer dataset, our approach achieves a false positive rate of 8.2% while maintaining a high true positive rate of 94.03% for malignant lesion detection. These results demonstrate the potential of the approach to enhance diagnostic accuracy and support more effective treatment decisions in teledermatology settings.References
Al Zegair, F., Naranpanawa, N., Betz-Stablein, B., Janda, M., Soyer, H. P., and Chandra, S. S. (2023). Application of machine learning in melanoma detection and the identification of ’ugly duckling’ and suspicious naevi: A review. arXiv.
Barata, C., Celebi, M. E., and Marques, J. S. (2021). Explainable skin lesion diagnosis using taxonomies. Pattern Recognition, 110:107413.
Bissoto, A., Valle, E., and Avila, S. (2020). Debiasing skin lesion datasets and models? not so fast. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 740–741.
Bissoto, A., Valle, E., and Avila, S. (2021). Gan-based data augmentation and anonymization for skin-lesion analysis: A critical review. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 1847–1856.
Combalia, M., Codella, N. C., Rotemberg, V., Helba, B., Vilaplana, V., Reiter, O., Carrera, C., Barreiro, A., Halpern, A. C., Puig, S., et al. (2019). Bcn20000: Dermoscopic lesions in the wild. arXiv preprint arXiv:1908.02288.
Combalia, M. et al. (2022). Validation of artificial intelligence prediction models for skin cancer diagnosis using dermoscopy images: the 2019 international skin imaging collaboration grand challenge. The Lancet Digital Health, 4(5):e330–e339.
Coustasse, A., Sarkar, R., Abodunde, B., Metzger, B. J., and Slater, C. M. (2019). Use of teledermatology to improve dermatological access in rural areas. Telemedicine and e-Health, 25(11):1022–1032.
Cubuk, E. D., Zoph, B., Shlens, J., and Le, Q. V. (2020). Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 702–703.
Demyanov, S., Chakravorty, R., Ge, Z., Bozorgtabar, S., Pablo, M., Bowling, A., and Garnavi, R. (2017). Tree-loss function for training neural networks on weakly-labelled datasets. In 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017), pages 287–291. IEEE.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. (2021). An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR).
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., and Thrun, S. (2017). Dermatologist-level classification of skin cancer with deep neural networks. Nature, 542(7639):115–118.
Gessert, N., Nielsen, M., Shaikh, M., Werner, R., and Schlaefer, A. (2019). Skin lesion classification using ensembles of multi-resolution efficientnets with metadata. In Proceedings of the International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE.
Groh, M., Harris, C., Soenksen, L., Lau, F., Han, R., Kim, A., Koochek, A., and Badri, O. (2021). Evaluating deep neural networks trained on clinical images in dermatology with the fitzpatrick 17k dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1820–1828.
Harun, M. F., Samah, A. A., Shabuli, M. I. A., Majid, H. A., Hashim, H., Ismail, N. A., Abdullah, S. M., and Alias, A. (2022). Incisor malocclusion using cut-out method and convolutional neural network. Progress in Microbes and Molecular Biology.
He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778.
Hu, J., Shen, L., and Sun, G. (2018). Squeeze-and-excitation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132–7141.
Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K. Q. (2017). Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700–4708.
Huang, X. and Belongie, S. (2017). Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE International Conference on Computer Vision, pages 1501–1510.
Kawahara, J., Daneshvar, S., Argenziano, G., and Hamarneh, G. (2018). Seven-point checklist and skin lesion classification using multitask multimodal neural nets. IEEE journal of biomedical and health informatics, 23(2):538–546.
Khan, M. A., Muhammad, K., Sharif, M., Akram, T., and de Albuquerque, V. H. C. (2021). Multi-class skin lesion detection and classification via teledermatology. IEEE journal of biomedical and health informatics, 25(12):4267–4275.
Kinyanjui, N. M., Odonga, T., Cintas, C., Codella, N. C., Panda, R., Sattigeri, P., and Varshney, K. R. (2020). Fairness of classifiers across skin tones in dermatology. In Medical Image Computing and Computer Assisted Intervention–MICCAI 2020: 23rd International Conference, Lima, Peru, October 4–8, 2020, Proceedings, Part VI, pages 320–329. Springer.
Kowacz, , Janusz, A., Świderska Chadaj, Z., Kleczyński, P., Krecicki, T., Kruk, M., Szczepaniak, K., Korbicz, J., and Wróbel, Z. (2023). Assessing smartphone-based image acquisition for skin cancer classification using convolutional neural networks. Biomedical Signal Processing and Control, 83:104626.
Lanjewar, M. G., Panchbhai, K. G., and Charanarur, P. (2023). Lung cancer detection from ct scans using modified densenet with feature selection methods and ml classifiers. Expert Systems with Applications, 224:119961.
Li, W., Zhuang, J., Wang, R., Zhang, J., and Zheng, W.-S. (2020). Fusing metadata and dermoscopy images for skin disease diagnosis. In 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI), pages 1996–2000. IEEE.
Liu, Y., Jain, A., Eng, C., Way, D. H., Lee, K., Bui, P., Kanada, K., de Oliveira Marinho, G., Gallegos, J., Gabriele, S., et al. (2020). A deep learning system for differential diagnosis of skin diseases. Nature medicine, 26(6):900–908.
Maier, K., Zaniolo, L., and Marques, O. (2022). Image quality issues in teledermatology: A comparative analysis of artificial intelligence solutions. Journal of the American Academy of Dermatology, 87(1):240–242.
Pacheco, A. G., Lima, G. R., Salomão, A. S., Krohling, B., Biral, I. P., de Angelo, G. G., Alves Jr, F. C., Esgario, J. G., Simora, A. C., Castro, P. B., et al. (2020). Pad-ufes-20: A skin lesion dataset composed of patient data and clinical images collected from smartphones. Data in brief, 32:106221.
Pacheco, A. G. C. and Krohling, R. A. (2020). The impact of patient clinical information on automated skin cancer detection. Computers in Biology and Medicine, 116:103545.
Pacheco, A. G. C. and Krohling, R. A. (2021). An attention-based mechanism to combine images and metadata in deep learning models applied to skin cancer classification. IEEE Journal of Biomedical and Health Informatics.
Pasquali, P., Sonthalia, S., Moreno-Ramirez, D., Sharma, P., Agrawal, M., Gupta, S., Kumar, D., and Arora, D. (2020). Teledermatology and its current perspective. Indian dermatology online journal, 11(1):12.
Roh, Y.-S., Kim, C.-W., Kim, N.-H., Suh, K.-M., Park, J.-I., and Lee, J.-H. (2021). Feasibility of a deep learning–based smartphone application for mobile dermoscopic melanoma detection. Archives of Dermatological Research, 313(9):743–749.
Selvaraj, K. M., Gnanagurusubbiah, S., Roy, R. R. R., Balu, S., et al. (2024). Enhancing skin lesion classification with advanced deep learning ensemble models: a path towards accurate medical diagnostics. Current Problems in Cancer, 49:101077.
Tan, M. and Le, Q. (2019). Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pages 6105–6114. PMLR.
Tang, P., Liang, Q., Yan, X., Xiang, S., Sun, W., Zhang, D., and Coppola, G. (2019). Efficient skin lesion segmentation using separable-unet with stochastic weight averaging. Computer methods and programs in biomedicine, 178:289–301.
Tschandl, P., Rosendahl, C., and Kittler, H. (2018). The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data, 5(1):1–9.
Wall, C., Young, F., Zhang, L., Phillips, E.-J., Jiang, R., and Yu, Y. (2020). Deep learning based melanoma diagnosis using dermoscopic images. In Developments of Artificial Intelligence Technologies in Computation and Robotics: Proceedings of the 14th International FLINS Conference (FLINS 2020), pages 907–914. World Scientific.
Wang, S., Yin, Y., Wang, D., Wang, Y., and Jin, Y. (2021). Interpretability-based multi-modal convolutional neural networks for skin lesion diagnosis. IEEE Transactions on Cybernetics.
Wilcoxon, F. (1945). Individual comparisons by ranking methods. Biometrics Bulletin, 1(6):80–83.
Yang, J., Wu, X., Liang, J., Sun, X., Cheng, M.-M., Rosin, P. L., and Wang, L. (2019). Self-paced balance learning for clinical skin disease recognition. IEEE transactions on neural networks and learning systems, 31(8):2832–2846.
Yun, S., Han, D., Oh, S. J., Chun, S., Choe, J., and Yoo, Y. (2019). Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6023–6032.
Zhu, X., Liu, F., Zhu, M., and Zhang, J. (2020). Skin lesion classification using deep learning with data augmentation. Computer Methods and Programs in Biomedicine, 195:105591.
Barata, C., Celebi, M. E., and Marques, J. S. (2021). Explainable skin lesion diagnosis using taxonomies. Pattern Recognition, 110:107413.
Bissoto, A., Valle, E., and Avila, S. (2020). Debiasing skin lesion datasets and models? not so fast. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 740–741.
Bissoto, A., Valle, E., and Avila, S. (2021). Gan-based data augmentation and anonymization for skin-lesion analysis: A critical review. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 1847–1856.
Combalia, M., Codella, N. C., Rotemberg, V., Helba, B., Vilaplana, V., Reiter, O., Carrera, C., Barreiro, A., Halpern, A. C., Puig, S., et al. (2019). Bcn20000: Dermoscopic lesions in the wild. arXiv preprint arXiv:1908.02288.
Combalia, M. et al. (2022). Validation of artificial intelligence prediction models for skin cancer diagnosis using dermoscopy images: the 2019 international skin imaging collaboration grand challenge. The Lancet Digital Health, 4(5):e330–e339.
Coustasse, A., Sarkar, R., Abodunde, B., Metzger, B. J., and Slater, C. M. (2019). Use of teledermatology to improve dermatological access in rural areas. Telemedicine and e-Health, 25(11):1022–1032.
Cubuk, E. D., Zoph, B., Shlens, J., and Le, Q. V. (2020). Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 702–703.
Demyanov, S., Chakravorty, R., Ge, Z., Bozorgtabar, S., Pablo, M., Bowling, A., and Garnavi, R. (2017). Tree-loss function for training neural networks on weakly-labelled datasets. In 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017), pages 287–291. IEEE.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. (2021). An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR).
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., and Thrun, S. (2017). Dermatologist-level classification of skin cancer with deep neural networks. Nature, 542(7639):115–118.
Gessert, N., Nielsen, M., Shaikh, M., Werner, R., and Schlaefer, A. (2019). Skin lesion classification using ensembles of multi-resolution efficientnets with metadata. In Proceedings of the International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE.
Groh, M., Harris, C., Soenksen, L., Lau, F., Han, R., Kim, A., Koochek, A., and Badri, O. (2021). Evaluating deep neural networks trained on clinical images in dermatology with the fitzpatrick 17k dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1820–1828.
Harun, M. F., Samah, A. A., Shabuli, M. I. A., Majid, H. A., Hashim, H., Ismail, N. A., Abdullah, S. M., and Alias, A. (2022). Incisor malocclusion using cut-out method and convolutional neural network. Progress in Microbes and Molecular Biology.
He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778.
Hu, J., Shen, L., and Sun, G. (2018). Squeeze-and-excitation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132–7141.
Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K. Q. (2017). Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700–4708.
Huang, X. and Belongie, S. (2017). Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE International Conference on Computer Vision, pages 1501–1510.
Kawahara, J., Daneshvar, S., Argenziano, G., and Hamarneh, G. (2018). Seven-point checklist and skin lesion classification using multitask multimodal neural nets. IEEE journal of biomedical and health informatics, 23(2):538–546.
Khan, M. A., Muhammad, K., Sharif, M., Akram, T., and de Albuquerque, V. H. C. (2021). Multi-class skin lesion detection and classification via teledermatology. IEEE journal of biomedical and health informatics, 25(12):4267–4275.
Kinyanjui, N. M., Odonga, T., Cintas, C., Codella, N. C., Panda, R., Sattigeri, P., and Varshney, K. R. (2020). Fairness of classifiers across skin tones in dermatology. In Medical Image Computing and Computer Assisted Intervention–MICCAI 2020: 23rd International Conference, Lima, Peru, October 4–8, 2020, Proceedings, Part VI, pages 320–329. Springer.
Kowacz, , Janusz, A., Świderska Chadaj, Z., Kleczyński, P., Krecicki, T., Kruk, M., Szczepaniak, K., Korbicz, J., and Wróbel, Z. (2023). Assessing smartphone-based image acquisition for skin cancer classification using convolutional neural networks. Biomedical Signal Processing and Control, 83:104626.
Lanjewar, M. G., Panchbhai, K. G., and Charanarur, P. (2023). Lung cancer detection from ct scans using modified densenet with feature selection methods and ml classifiers. Expert Systems with Applications, 224:119961.
Li, W., Zhuang, J., Wang, R., Zhang, J., and Zheng, W.-S. (2020). Fusing metadata and dermoscopy images for skin disease diagnosis. In 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI), pages 1996–2000. IEEE.
Liu, Y., Jain, A., Eng, C., Way, D. H., Lee, K., Bui, P., Kanada, K., de Oliveira Marinho, G., Gallegos, J., Gabriele, S., et al. (2020). A deep learning system for differential diagnosis of skin diseases. Nature medicine, 26(6):900–908.
Maier, K., Zaniolo, L., and Marques, O. (2022). Image quality issues in teledermatology: A comparative analysis of artificial intelligence solutions. Journal of the American Academy of Dermatology, 87(1):240–242.
Pacheco, A. G., Lima, G. R., Salomão, A. S., Krohling, B., Biral, I. P., de Angelo, G. G., Alves Jr, F. C., Esgario, J. G., Simora, A. C., Castro, P. B., et al. (2020). Pad-ufes-20: A skin lesion dataset composed of patient data and clinical images collected from smartphones. Data in brief, 32:106221.
Pacheco, A. G. C. and Krohling, R. A. (2020). The impact of patient clinical information on automated skin cancer detection. Computers in Biology and Medicine, 116:103545.
Pacheco, A. G. C. and Krohling, R. A. (2021). An attention-based mechanism to combine images and metadata in deep learning models applied to skin cancer classification. IEEE Journal of Biomedical and Health Informatics.
Pasquali, P., Sonthalia, S., Moreno-Ramirez, D., Sharma, P., Agrawal, M., Gupta, S., Kumar, D., and Arora, D. (2020). Teledermatology and its current perspective. Indian dermatology online journal, 11(1):12.
Roh, Y.-S., Kim, C.-W., Kim, N.-H., Suh, K.-M., Park, J.-I., and Lee, J.-H. (2021). Feasibility of a deep learning–based smartphone application for mobile dermoscopic melanoma detection. Archives of Dermatological Research, 313(9):743–749.
Selvaraj, K. M., Gnanagurusubbiah, S., Roy, R. R. R., Balu, S., et al. (2024). Enhancing skin lesion classification with advanced deep learning ensemble models: a path towards accurate medical diagnostics. Current Problems in Cancer, 49:101077.
Tan, M. and Le, Q. (2019). Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pages 6105–6114. PMLR.
Tang, P., Liang, Q., Yan, X., Xiang, S., Sun, W., Zhang, D., and Coppola, G. (2019). Efficient skin lesion segmentation using separable-unet with stochastic weight averaging. Computer methods and programs in biomedicine, 178:289–301.
Tschandl, P., Rosendahl, C., and Kittler, H. (2018). The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data, 5(1):1–9.
Wall, C., Young, F., Zhang, L., Phillips, E.-J., Jiang, R., and Yu, Y. (2020). Deep learning based melanoma diagnosis using dermoscopic images. In Developments of Artificial Intelligence Technologies in Computation and Robotics: Proceedings of the 14th International FLINS Conference (FLINS 2020), pages 907–914. World Scientific.
Wang, S., Yin, Y., Wang, D., Wang, Y., and Jin, Y. (2021). Interpretability-based multi-modal convolutional neural networks for skin lesion diagnosis. IEEE Transactions on Cybernetics.
Wilcoxon, F. (1945). Individual comparisons by ranking methods. Biometrics Bulletin, 1(6):80–83.
Yang, J., Wu, X., Liang, J., Sun, X., Cheng, M.-M., Rosin, P. L., and Wang, L. (2019). Self-paced balance learning for clinical skin disease recognition. IEEE transactions on neural networks and learning systems, 31(8):2832–2846.
Yun, S., Han, D., Oh, S. J., Chun, S., Choe, J., and Yoo, Y. (2019). Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6023–6032.
Zhu, X., Liu, F., Zhu, M., and Zhang, J. (2020). Skin lesion classification using deep learning with data augmentation. Computer Methods and Programs in Biomedicine, 195:105591.
Published
2025-09-29
How to Cite
DIAS, Camila Alves et al.
Parallel-Hierarchical Multi-Task Learning for Skin Lesion Classification with Multimodal Data. In: NATIONAL MEETING ON ARTIFICIAL AND COMPUTATIONAL INTELLIGENCE (ENIAC), 22. , 2025, Fortaleza/CE.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2025
.
p. 309-320.
ISSN 2763-9061.
DOI: https://doi.org/10.5753/eniac.2025.12418.
