Characterization of Pneumonia Diagnostic Uncertainty: A Case Study on The CheXpert Dataset
Resumo
Pneumonia is a serious respiratory infection that presents significant diagnostic challenges due to the variability in its symptoms and its overlap with other respiratory diseases. This study investigates the potential of diagnostic uncertainty labels to enhance CAD system's pneumonia classification. Specifically, it explores the feasibility of a ternary classification approach (classifying X-rays as positive, negative, or uncertain), introducing uncertainty as a distinct diagnostic category, aiming to provide a more nuanced and cautious classification of pneumonia. Data processing techniques, including undersampling to balance classes, image resizing, and data augmentation, were applied. Transfer learning with the CheXNet model was then employed in a Monte Carlo cross-validation framework across 16 random data splits. The ROC curves and the areas under the ROC curves for the uncertainty class were analyzed, challenging the notion that uncertainty cannot be effectively characterized. The results indicated a degree of class separation, indicating that the uncertainty carried enough information to be characterized and suggesting the viability of the envisioned ternary model. Additionally, due to the exclusive use of frontal view X-rays and application of undersampling, results are expected to be further improved in future research.
Palavras-chave:
Transfer Learning, CheXpert, CheXNet, Uncertainty, Pneumonia Classification
Referências
K. Thomas, “Global burden of pneumonia,” International Journal of Infectious Diseases, vol. 45, p. 1, Apr 2016. [Online]. Available: DOI: 10.1016/j.ijid.2016.02.027
D. Wootton and C. Feldman, “The diagnosis of pneumonia requires a chest radiograph (x-ray)—yes, no or sometimes?” Pneumonia, vol. 5, no. 1, pp. 1–7, Dec 2014. [Online]. Available: DOI: 10.15172/pneu.2014.5/464
O. Julie, “Pneumonia: challenges in the definition, diagnosis, and man- agement of disease.” The Surgical clinics of North America, 2014.
R. Najjar, “Redefining radiology: A review of artificial intelligence integration in medical imaging,” Diagnostics (Basel), vol. 13, no. 17, Aug. 2023.
J. Irvin, P. Rajpurkar, M. Ko, Y. Yu, S. Ciurea-Ilcus, C. Chute, H. Marklund, B. Haghgoo, R. Ball, K. Shpanskaya, J. Seekins, D. A. Mong, S. S. Halabi, J. K. Sandberg, R. Jones, D. B. Larson, C. P. Langlotz, B. N. Patel, M. P. Lungren, and A. Y. Ng, “Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison,” 2019. [Online]. Available: [link]
P. Rajpurkar, J. Irvin, K. Zhu, B. Yang, H. Mehta, T. Duan, D. Ding, A. Bagul, C. Langlotz, K. Shpanskaya, M. P. Lungren, and A. Y. Ng, “Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning,” 2017. [Online]. Available: [link]
Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel, “Backpropagation applied to handwritten zip code recognition,” Neural Computation, vol. 1, no. 4, pp. 541–551, 1989.
Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol. 521, pp. 436–44, 05 2015.
S. DR, “Convolutional neural networks in medical image understanding: a survey.” Evolutionary Intelligence, 2022.
S. Soffer, A. Ben-Cohen, O. Shimon, M. M. Amitai, H. Greenspan, and E. Klang, “Convolutional neural networks for radiologic images: A radiologist’s guide,” Radiology, vol. 290, no. 3, pp. 590–606, 2019, pMID: 30694159. [Online]. Available: DOI: 10.1148/radiol.2018180547
A. Krizhevsky, I. Sutskever, and G. Hinton, “Imagenet classification with deep convolutional neural networks,” Neural Information Processing Systems, vol. 25, 01 2012.
S. Sharma and R. Mehra, “Implications of pooling strategies in convolutional neural networks: A deep insight,” Foundations of Computing and Decision Sciences, vol. 44, pp. 303 – 330, 2019. [Online]. Available: [link]
N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: A simple way to prevent neural networks from overfitting,” Journal of Machine Learning Research, vol. 15, no. 56, pp. 1929–1958, 2014. [Online]. Available: [link]
I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning. MIT Press, 2016, [link].
F. Zhuang, Z. Qi, K. Duan, D. Xi, Y. Zhu, H. Zhu, H. Xiong, and Q. He, “A comprehensive survey on transfer learning,” CoRR, vol. abs/1911.02685, 2019. [Online]. Available: [link].
J. Quinn. (2020) Dive into deep learning, chapter 14.2. [Online]. Avail- able: [link]
Z. Li and D. Hoiem, “Learning without forgetting,” 2017. [Online]. Available: [link]
M. McCloskey and N. J. Cohen, “Catastrophic interference in connectionist networks: The sequential learning problem,” in Psychology of Learning and Motivation, ser. Psychology of Learning and Motivation, G. H. Bower, Ed. Academic Press, 1989, vol. 24, pp. 109– 165. [Online]. Available: [link]
F. Varno, L. M. Petry, L. D. Jorio, and S. Matwin, “Learn faster and forget slower via fast and stable task adaptation,” 2020. [Online]. Available: [link]
scikit learn. Multiclass receiver operating characteristic (roc) — scikit- learn 1.5.2 documentation.
J. Egan, Signal Detection Theory and ROC-analysis, ser. Academic Press series in cognition and perception. Academic Press, 1975. [Online]. Available: [link]
T. Fawcett, “An introduction to roc analysis,” Pattern Recognition Letters, vol. 27, no. 8, pp. 861–874, 2006, rOC Analysis in Pattern Recognition. [Online]. Available: [link]
P. Burman, “A comparative study of ordinary cross-validation, v-fold cross-validation and the repeated learning-testing methods,” Biometrika, vol. 76, pp. 503–514, 09 1989.
“Anaconda software distribution,” 2020. [Online]. Available: [link]
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vander- plas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, “Scikit-learn: Machine learning in Python,” Journal of Machine Learning Research, vol. 12, pp. 2825–2830, 2011.
G. Lemaître, F. Nogueira, and C. K. Aridas, “Imbalanced-learn: A python toolbox to tackle the curse of imbalanced datasets in machine learning,” Journal of Machine Learning Research, vol. 18, no. 17, pp. 1–5, 2017. [Online]. Available: [link]
G. Huang, Z. Liu, and K. Q. Weinberger, “Densely connected convolutional networks,” CoRR, vol. abs/1608.06993, 2016. [Online]. Available: [link]
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE Conference on Computer Vision and Pattern Recognition, 2009, pp. 248–255.
F. Chollet et al., “Keras,” [link], 2015.
D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” 2017. [Online]. Available: [link]
R. S. of North America (RSNA) and A. C. of Radiology (ACR). Image gallery.
M. Seeland and P. M¨ader, “Multi-view classification with convolutional neural networks,” PLOS ONE, vol. 16, no. 1, pp. 1–17, 01 2021. [Online]. Available: DOI: 10.1371/journal.pone.0245230
D. Wootton and C. Feldman, “The diagnosis of pneumonia requires a chest radiograph (x-ray)—yes, no or sometimes?” Pneumonia, vol. 5, no. 1, pp. 1–7, Dec 2014. [Online]. Available: DOI: 10.15172/pneu.2014.5/464
O. Julie, “Pneumonia: challenges in the definition, diagnosis, and man- agement of disease.” The Surgical clinics of North America, 2014.
R. Najjar, “Redefining radiology: A review of artificial intelligence integration in medical imaging,” Diagnostics (Basel), vol. 13, no. 17, Aug. 2023.
J. Irvin, P. Rajpurkar, M. Ko, Y. Yu, S. Ciurea-Ilcus, C. Chute, H. Marklund, B. Haghgoo, R. Ball, K. Shpanskaya, J. Seekins, D. A. Mong, S. S. Halabi, J. K. Sandberg, R. Jones, D. B. Larson, C. P. Langlotz, B. N. Patel, M. P. Lungren, and A. Y. Ng, “Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison,” 2019. [Online]. Available: [link]
P. Rajpurkar, J. Irvin, K. Zhu, B. Yang, H. Mehta, T. Duan, D. Ding, A. Bagul, C. Langlotz, K. Shpanskaya, M. P. Lungren, and A. Y. Ng, “Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning,” 2017. [Online]. Available: [link]
Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel, “Backpropagation applied to handwritten zip code recognition,” Neural Computation, vol. 1, no. 4, pp. 541–551, 1989.
Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol. 521, pp. 436–44, 05 2015.
S. DR, “Convolutional neural networks in medical image understanding: a survey.” Evolutionary Intelligence, 2022.
S. Soffer, A. Ben-Cohen, O. Shimon, M. M. Amitai, H. Greenspan, and E. Klang, “Convolutional neural networks for radiologic images: A radiologist’s guide,” Radiology, vol. 290, no. 3, pp. 590–606, 2019, pMID: 30694159. [Online]. Available: DOI: 10.1148/radiol.2018180547
A. Krizhevsky, I. Sutskever, and G. Hinton, “Imagenet classification with deep convolutional neural networks,” Neural Information Processing Systems, vol. 25, 01 2012.
S. Sharma and R. Mehra, “Implications of pooling strategies in convolutional neural networks: A deep insight,” Foundations of Computing and Decision Sciences, vol. 44, pp. 303 – 330, 2019. [Online]. Available: [link]
N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: A simple way to prevent neural networks from overfitting,” Journal of Machine Learning Research, vol. 15, no. 56, pp. 1929–1958, 2014. [Online]. Available: [link]
I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning. MIT Press, 2016, [link].
F. Zhuang, Z. Qi, K. Duan, D. Xi, Y. Zhu, H. Zhu, H. Xiong, and Q. He, “A comprehensive survey on transfer learning,” CoRR, vol. abs/1911.02685, 2019. [Online]. Available: [link].
J. Quinn. (2020) Dive into deep learning, chapter 14.2. [Online]. Avail- able: [link]
Z. Li and D. Hoiem, “Learning without forgetting,” 2017. [Online]. Available: [link]
M. McCloskey and N. J. Cohen, “Catastrophic interference in connectionist networks: The sequential learning problem,” in Psychology of Learning and Motivation, ser. Psychology of Learning and Motivation, G. H. Bower, Ed. Academic Press, 1989, vol. 24, pp. 109– 165. [Online]. Available: [link]
F. Varno, L. M. Petry, L. D. Jorio, and S. Matwin, “Learn faster and forget slower via fast and stable task adaptation,” 2020. [Online]. Available: [link]
scikit learn. Multiclass receiver operating characteristic (roc) — scikit- learn 1.5.2 documentation.
J. Egan, Signal Detection Theory and ROC-analysis, ser. Academic Press series in cognition and perception. Academic Press, 1975. [Online]. Available: [link]
T. Fawcett, “An introduction to roc analysis,” Pattern Recognition Letters, vol. 27, no. 8, pp. 861–874, 2006, rOC Analysis in Pattern Recognition. [Online]. Available: [link]
P. Burman, “A comparative study of ordinary cross-validation, v-fold cross-validation and the repeated learning-testing methods,” Biometrika, vol. 76, pp. 503–514, 09 1989.
“Anaconda software distribution,” 2020. [Online]. Available: [link]
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vander- plas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, “Scikit-learn: Machine learning in Python,” Journal of Machine Learning Research, vol. 12, pp. 2825–2830, 2011.
G. Lemaître, F. Nogueira, and C. K. Aridas, “Imbalanced-learn: A python toolbox to tackle the curse of imbalanced datasets in machine learning,” Journal of Machine Learning Research, vol. 18, no. 17, pp. 1–5, 2017. [Online]. Available: [link]
G. Huang, Z. Liu, and K. Q. Weinberger, “Densely connected convolutional networks,” CoRR, vol. abs/1608.06993, 2016. [Online]. Available: [link]
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE Conference on Computer Vision and Pattern Recognition, 2009, pp. 248–255.
F. Chollet et al., “Keras,” [link], 2015.
D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” 2017. [Online]. Available: [link]
R. S. of North America (RSNA) and A. C. of Radiology (ACR). Image gallery.
M. Seeland and P. M¨ader, “Multi-view classification with convolutional neural networks,” PLOS ONE, vol. 16, no. 1, pp. 1–17, 01 2021. [Online]. Available: DOI: 10.1371/journal.pone.0245230
Publicado
27/11/2024
Como Citar
ALLAN, Amyr; DOS SANTOS, Gilmário Barbosa.
Characterization of Pneumonia Diagnostic Uncertainty: A Case Study on The CheXpert Dataset. In: CONGRESSO LATINO-AMERICANO DE SOFTWARE LIVRE E TECNOLOGIAS ABERTAS (LATINOWARE), 21. , 2024, Foz do Iguaçu/PR.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2024
.
p. 124-129.
DOI: https://doi.org/10.5753/latinoware.2024.245719.
