Generalizability of CNN on Predicting COVID-19 from Chest X-ray Images

• Natalia de Sousa Freire UFAM
• Pedro Paulo de Souza Leão UFAM
• Leonardo Albuquerque Tiago UFAM
• Alberto de Almeida Campos Gonçalves UFAM
• Rafael Albuquerque Pinto UFAM
• Eulanda Miranda dos Santos UFAM
• Eduardo Souto UFAM

DOI: https://doi.org/10.5753/sbcas.2022.222442

Abstract

Machine learning methods have been applied to predict COVID-19 using chest X-ray images in several works. However, to be helpful, a machine learning model must be robust to give reliable predictions for any target population, rather than only for the population used to generate the training data. Despite such an important issue, testing the generalizability of machine learning models is frequently not performed in current works. To test the generalizability of three models of CNN, four different databases obtained from various data sources are investigated in this paper in an internal-and-external validation procedure. All models are trained considering lung segmentation as a pre-processing step and without lung segmentation. The results show how important an external evaluation is to avoid providing performance evaluations excessively optimistic and inaccurate.

References

Ahmed, K. B., Goldgof, G. M., Paul, R., Goldgof, D. B., and Hall, L. O. (2021). Discovery of a generalization gap of convolutional neural networks on covid-19 x-rays classification. IEEE Access, 9:72970-72979.

Arias-Garzón, D., Alzate-Grisales, J. A., Orozco-Arias, S., Arteaga-Arteaga, H. B., Bravo-Ortiz, M. A., Mora-Rubio, A., Saborit-Torres, J. M., Ángel Montell Serrano, J., de la Iglesia Vayá, M., Cardona-Morales, O., and Tabares-Soto, R. (2021). Covid-19 detection in x-ray images using convolutional neural networks. Machine Learning with Applications, 6:100138.

Chowdhury, M. E. H., Rahman, T., Khandakar, A., Mazhar, R., Kadir, M. A., Mahbub, Z. B., Islam, K. R., Khan, M. S., Iqbal, A., Emadi, N. A., Reaz, M. B. I., and Islam, M. T. (2020). Can AI help in screening viral and COVID-19 pneumonia? IEEE Access, 8:132665-132676.

Cohen, J. P., Morrison, P., Dao, L., Roth, K., Duong, T. Q., and Ghassemi, M. (2020). Covid-19 image data collection: Prospective predictions are the future.

DeGrave, A. J., Janizek, J. D., and Lee, S.-I. (2021). Ai for radiographic covid-19 detection selects shortcuts over signal. Nature Machine Intelligence, Epub May 31.

Elaziz, M. A., Hosny, K. M., Salah, A., Darwish, M. M., Lu, S., and Sahlol, A. T. (2020). New machine learning method for image-based diagnosis of COVID-19. PLOS ONE, 15(6):e0235187.

Frid-Adar, M., Amer, R., Gozes, O., Nassar, J., and Greenspan, H. (2021). COVID-19 in CXR: From detection and severity scoring to patient disease monitoring. IEEE Journal of Biomedical and Health Informatics, 25(6):1892-1903.

Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R. S., Brendel, W., Bethge, M., and Wichmann, F. A. (2020). Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665-673.

Kermany, D., Zhang, K., and Goldbaum, M. (2018). Labeled optical coherence tomography (oct) and chest x-ray images for classification. Applied Sciences.

Khan, S. H., Sohail, A., Khan, A., Hassan, M., Lee, Y. S., Alam, J., Basit, A., and Zubair, S. (2021). Covid-19 detection in chest x-ray images using deep boosted hybrid learning. Computers in Biology and Medicine, 137:104816.

Khasawneh, N., Fraiwan, M., Fraiwan, L., Khassawneh, B., and Ibnian, A. (2021). Detection of covid-19 from chest x-ray images using deep convolutional neural networks. Sensors, 21(17).

Li, M. D., Arun, N. T., Aggarwal, M., Gupta, S., Singh, P., Little, B. P., Mendoza, D. P., Corradi, G. C., Takahashi, M. S., Ferraciolli, S. F., Succi, M. D., Lang, M., Bizzo, B. C., Dayan, I., Kitamura, F. C., and Kalpathy-Cramer, J. (2020a). Improvement and multi-population generalizability of a deep learning-based chest radiograph severity score for covid-19. medRxiv.

Li, M. D., Arun, N. T., Gidwani, M., Chang, K., Deng, F., Little, B. P., Mendoza, D. P., Lang, M., Lee, S. I., O'Shea, A., Parakh, A., Singh, P., and Kalpathy-Cramer, J. (2020b). Automated assessment and tracking of COVID-19 pulmonary disease severity on chest radiographs using convolutional siamese neural networks. Radiology: Artificial Intelligence, 2(4):e200079.

López-Cabrera, J. D., Orozco-Morales, R., Portal-Díaz, J. A., Lovelle-Enríquez, O., and Pérez-Díaz, M. (2021). Current limitations to identify covid-19 using artificial intelligence with chest x-ray imaging (part ii). the shortcut learning problem. Health and Technology, 11(6):1331-1345.

Pereira, R. M., Bertolini, D., Teixeira, L. O., Silla, C. N., and Costa, Y. M. (2020). Covid-19 identification in chest x-ray images on flat and hierarchical classification scenarios. Computer Methods and Programs in Biomedicine, 194:105532.

Roberts, M., , Driggs, D., Thorpe, M., Gilbey, J., Yeung, M., Ursprung, S., Aviles-Rivero, A. I., Etmann, C., McCague, C., Beer, L., Weir-McCall, J. R., Teng, Z., Gkrania-Klotsas, E., Rudd, J. H. F., Sala, E., and Schönlieb, C.-B. (2021). Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans. Nature Machine Intelligence, 3(3):199-217.

Ronneberger, O., Fischer, P., and Brox, T. (2015). U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention - MICCAI 2015, pages 234-241.

Saha, P., Sadi, M. S., Aranya, O. R. R., Jahan, S., and Islam, F.-A. (2021). Cov-vgx: An automated covid-19 detection system using x-ray images and transfer learning. Informatics in Medicine Unlocked, 26:100741.

Tabik, S., Gómez-Ríos, A., Martín-Rodríguez, J. L., Sevillano-García, I., Rey-Area, M., Charte, D., Guirado, E., Suárez, J. L., Luengo, J., Valero-González, M. A., García-Villanova, P., Olmedo-Sánchez, E., and Herrera, F. (2020). Covidgr dataset and covidsdnet methodology for predicting covid-19 based on chest x-ray images. IEEE Journal of Biomedical and Health Informatics, 24(12):3595-3605.

Tartaglione, E., Barbano, C. A., Berzovini, C., Calandri, M., and Grangetto, M. (2020). Unveiling covid-19 from chest x-ray with deep learning: A hurdles race with small data. International Journal of Environmental Research and Public Health, 17(18).

Wang, L., Lin, Z. Q., and Wong, A. (2020). COVID-net: a tailored deep convolutional neural network design for detection of COVID-19 cases from chest x-ray images. Scientific Reports, 10(1).

Yeh, C.-F., Cheng, H.-T., Wei, A., Chen, H.-M., Kuo, P.-C., Liu, K.-C., Ko, M.-C., Chen, R.-J., Lee, P.-C., Chuang, J.-H., Chen, C.-M., Chen, Y.-C., Lee, W.-J., Chien, N., Chen, J.-Y., Huang, Y.-S., Chang, Y.-C., Huang, Y.-C., Chou, N.-K., Chao, K.-H., Tu, Y.-C., Chang, Y.-C., and Liu, T.-L. (2020). A cascaded learning strategy for robust covid-19 pneumonia chest x-ray screening.

Zu, Z., Jiang, M., Xu, P., Chen, W., Ni, Q., Lu, G., and Zhang, L. (2020). Coronavirus disease 2019 (covid-19): A perspective from china. Radiology, 296:200490.