PathoSpotter-Search: A Content-Based Image Retrieval Tool for Nephropathology
Resumo
Nephropathologists typically organizes their repository of digital images of kidney biopsies in such a way that it is difficult to retrieve cases that have images similar to a picture under analysis. Having this in mind, we initiated the development of PathoSpotter-Search, a Content-Based Image Retrieval system for images of kidney biopsies. The system operates as a cloud service to avoid the need to install any software on the pathologist’s computer. Our approach combines a feature extractor followed by a similarity score calculator. We evaluated convolutional network (CN) architectures (VGG-16 (original and fine-tuned) and Inception-ResNet, and a network used in the proprietary classifier for glomerular hypercellularity), combined with Cosine and Euclidean distances as similarity scores. The first results have shown that the CN of the VGG- 16 combined with cosine distance yielded the best performance (precision 53%). To assess the usability and functionality of the PathoSpotter-Search as a cloud service, the system was tested by nephropathologists and proved to be useful as a tool for retrieving similar images from their local repositories. Currently, we are working to improve the system precision to at least 70%, and evaluating strategies to retrieve similar images based on segments or tiles of the query image.
Referências
N. A. M. Zin, R. Yusof, S. A. Lashari, A. Mustapha, N. Senan, and R. Ibrahim, “Content-based image retrieval in medical domain: A review,” Journal of Physics: Conference Series, vol. 1019, p. 012044, jun 2018. [Online]. Available: https://doi.org/10.1088/1742-6596/1019/1/012044
S. K. Sundararajan, B. Sankaragomathi, and D. S. Priya, “Deep belief cnn feature representation based content based image retrieval for medical images,” Journal of medical systems, vol. 43, no. 6, pp. 1–9, 2019.
D. A. Kumar and J. Esther, “Comparative study on cbir based by color histogram, gabor and wavelet transform,” International Journal of Computer Applications, vol. 17, no. 3, pp. 37–44, 2011.
A. Qayyum, S. M. Anwar, M. Awais, and M. Majid, “Medical image retrieval using deep convolutional neural network,” Neurocomputing, vol. 266, pp. 8–20, 2017.
K. Zheng, “Content-based image retrieval for medical image,” in 2015 11th International Conference on Computational Intelligence and Security (CIS), 2015, pp. 219–222.
A. H. Pilevar, “CBMIR: Content-based Image Retrieval Algorithm for Medical Image Databases,” Journal of medical signals and sensors, vol. 1, no. 1, pp. 12–18, jan 2011. [Online]. Available: https://pubmed.ncbi.nlm.nih.gov/22606654 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3317765/
L. R. Long, S. Antani, T. M. Deserno, and G. R. Thoma, “Content- Based Image Retrieval in Medicine: Retrospective Assessment, State of the Art, and Future Directions,” International journal of healthcare information systems and informatics : official publication of the Information Resources Management Association, vol. 4, no. 1, pp. 1–16, jan 2009. [Online]. Available: https://pubmed.ncbi.nlm.nih.gov/20523757 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2879660/
X. S. Zhou and T. S. Huang, “Cbir: from low-level features to high-level semantics,” in Image and Video Communications and Processing 2000, vol. 3974. International Society for Optics and Photonics, 2000, pp. 426–431.
K. Mikolajczyk and C. Schmid, “Scale & affine invariant interest point detectors,” International journal of computer vision, vol. 60, no. 1, pp. 63–86, 2004.
H. Bay, A. Ess, T. Tuytelaars, and L. Van Gool, “Speeded-up robust features (surf),” Computer vision and image understanding, vol. 110, no. 3, pp. 346–359, 2008.
N. Dalal and B. Triggs, “Histograms of oriented gradients for human detection,” in 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR’05), vol. 1. Ieee, 2005, pp. 886–893.
A. Kumar, J. Kim, W. Cai, M. Fulham, and D. Feng, “Content-based medical image retrieval: a survey of applications to multidimensional and multimodality data,” Journal of digital imaging, vol. 26, no. 6, pp. 1025–1039, 2013.
A. M. Rinaldi and C. Russo, “A content based image retrieval approach based on multiple multimedia features descriptors in e-health environment,” in 2020 IEEE International Symposium on Medical Measurements and Applications (MeMeA). IEEE, 2020, pp. 1–6.
Y. LeCun, K. Kavukcuoglu, and C. Farabet, “Convolutional networks and applications in vision,” in Proceedings of 2010 IEEE international symposium on circuits and systems. IEEE, 2010, pp. 253–256.
A. Shah, R. Naseem, S. Iqbal, M. A. Shah et al., “Improving cbir accuracy using convolutional neural network for feature extraction,” in 2017 13th International Conference on Emerging Technologies (ICET). IEEE, 2017, pp. 1–5.
H. Azizpour, A. Sharif Razavian, J. Sullivan, A. Maki, and S. Carlsson, “From generic to specific deep representations for visual recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition workshops, 2015, pp. 36–45.
K. Kruthika, H. Maheshappa, A. D. N. Initiative et al., “Cbir system using capsule networks and 3d cnn for alzheimer’s disease diagnosis,” Informatics in Medicine Unlocked, vol. 14, pp. 59–68, 2019.
A. Sezavar, H. Farsi, and S. Mohamadzadeh, “Content-based image retrieval by combining convolutional neural networks and sparse representation,” Multimedia Tools and Applications, vol. 78, no. 15, pp. 20 895–20 912, 2019.
Z. N. K. Swati, Q. Zhao, M. Kabir, F. Ali, Z. Ali, S. Ahmed, and J. Lu, “Content-based brain tumor retrieval for mr images using transfer learning,” IEEE Access, vol. 7, pp. 17 809–17 822, 2019.
K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014.
C. Szegedy, S. Ioffe, V. Vanhoucke, and A. Alemi, “Inception-v4, inception-resnet and the impact of residual connections on learning,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 31, no. 1, 2017.
P. Chagas, L. Souza, I. Araújo, N. Aldeman, A. Duarte, M. Angelo, W. L. Dos-Santos, and L. Oliveira, “Classification of glomerular hypercellularity using convolutional features and support vector machine,” Artificial intelligence in medicine, vol. 103, p. 101808, 2020.
N. Tajbakhsh, J. Y. Shin, S. R. Gurudu, R. T. Hurst, C. B. Kendall, M. B. Gotway, and J. Liang, “Convolutional neural networks for medical image analysis: Full training or fine tuning?” IEEE Transactions on Medical Imaging, vol. 35, no. 5, pp. 1299–1312, 2016.
M. D. Kumar, M. Babaie, S. Zhu, S. Kalra, and H. R. Tizhoosh, “A comparative study of cnn, bovw and lbp for classification of histopathological images,” in 2017 IEEE Symposium Series on Computational Intelligence (SSCI). IEEE, 2017, pp. 1–7.
S. K. Sundararajan, B. Sankaragomathi, and D. S. Priya, “Deep belief cnn feature representation based content based image retrieval for medical images,” Journal of medical systems, vol. 43, no. 6, pp. 1–9, 2019.
D. A. Kumar and J. Esther, “Comparative study on cbir based by color histogram, gabor and wavelet transform,” International Journal of Computer Applications, vol. 17, no. 3, pp. 37–44, 2011.
A. Qayyum, S. M. Anwar, M. Awais, and M. Majid, “Medical image retrieval using deep convolutional neural network,” Neurocomputing, vol. 266, pp. 8–20, 2017.
K. Zheng, “Content-based image retrieval for medical image,” in 2015 11th International Conference on Computational Intelligence and Security (CIS), 2015, pp. 219–222.
A. H. Pilevar, “CBMIR: Content-based Image Retrieval Algorithm for Medical Image Databases,” Journal of medical signals and sensors, vol. 1, no. 1, pp. 12–18, jan 2011. [Online]. Available: https://pubmed.ncbi.nlm.nih.gov/22606654 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3317765/
L. R. Long, S. Antani, T. M. Deserno, and G. R. Thoma, “Content- Based Image Retrieval in Medicine: Retrospective Assessment, State of the Art, and Future Directions,” International journal of healthcare information systems and informatics : official publication of the Information Resources Management Association, vol. 4, no. 1, pp. 1–16, jan 2009. [Online]. Available: https://pubmed.ncbi.nlm.nih.gov/20523757 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2879660/
X. S. Zhou and T. S. Huang, “Cbir: from low-level features to high-level semantics,” in Image and Video Communications and Processing 2000, vol. 3974. International Society for Optics and Photonics, 2000, pp. 426–431.
K. Mikolajczyk and C. Schmid, “Scale & affine invariant interest point detectors,” International journal of computer vision, vol. 60, no. 1, pp. 63–86, 2004.
H. Bay, A. Ess, T. Tuytelaars, and L. Van Gool, “Speeded-up robust features (surf),” Computer vision and image understanding, vol. 110, no. 3, pp. 346–359, 2008.
N. Dalal and B. Triggs, “Histograms of oriented gradients for human detection,” in 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR’05), vol. 1. Ieee, 2005, pp. 886–893.
A. Kumar, J. Kim, W. Cai, M. Fulham, and D. Feng, “Content-based medical image retrieval: a survey of applications to multidimensional and multimodality data,” Journal of digital imaging, vol. 26, no. 6, pp. 1025–1039, 2013.
A. M. Rinaldi and C. Russo, “A content based image retrieval approach based on multiple multimedia features descriptors in e-health environment,” in 2020 IEEE International Symposium on Medical Measurements and Applications (MeMeA). IEEE, 2020, pp. 1–6.
Y. LeCun, K. Kavukcuoglu, and C. Farabet, “Convolutional networks and applications in vision,” in Proceedings of 2010 IEEE international symposium on circuits and systems. IEEE, 2010, pp. 253–256.
A. Shah, R. Naseem, S. Iqbal, M. A. Shah et al., “Improving cbir accuracy using convolutional neural network for feature extraction,” in 2017 13th International Conference on Emerging Technologies (ICET). IEEE, 2017, pp. 1–5.
H. Azizpour, A. Sharif Razavian, J. Sullivan, A. Maki, and S. Carlsson, “From generic to specific deep representations for visual recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition workshops, 2015, pp. 36–45.
K. Kruthika, H. Maheshappa, A. D. N. Initiative et al., “Cbir system using capsule networks and 3d cnn for alzheimer’s disease diagnosis,” Informatics in Medicine Unlocked, vol. 14, pp. 59–68, 2019.
A. Sezavar, H. Farsi, and S. Mohamadzadeh, “Content-based image retrieval by combining convolutional neural networks and sparse representation,” Multimedia Tools and Applications, vol. 78, no. 15, pp. 20 895–20 912, 2019.
Z. N. K. Swati, Q. Zhao, M. Kabir, F. Ali, Z. Ali, S. Ahmed, and J. Lu, “Content-based brain tumor retrieval for mr images using transfer learning,” IEEE Access, vol. 7, pp. 17 809–17 822, 2019.
K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014.
C. Szegedy, S. Ioffe, V. Vanhoucke, and A. Alemi, “Inception-v4, inception-resnet and the impact of residual connections on learning,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 31, no. 1, 2017.
P. Chagas, L. Souza, I. Araújo, N. Aldeman, A. Duarte, M. Angelo, W. L. Dos-Santos, and L. Oliveira, “Classification of glomerular hypercellularity using convolutional features and support vector machine,” Artificial intelligence in medicine, vol. 103, p. 101808, 2020.
N. Tajbakhsh, J. Y. Shin, S. R. Gurudu, R. T. Hurst, C. B. Kendall, M. B. Gotway, and J. Liang, “Convolutional neural networks for medical image analysis: Full training or fine tuning?” IEEE Transactions on Medical Imaging, vol. 35, no. 5, pp. 1299–1312, 2016.
M. D. Kumar, M. Babaie, S. Zhu, S. Kalra, and H. R. Tizhoosh, “A comparative study of cnn, bovw and lbp for classification of histopathological images,” in 2017 IEEE Symposium Series on Computational Intelligence (SSCI). IEEE, 2017, pp. 1–7.
Publicado
18/10/2021
Como Citar
AGUIAR, Ellen; CALUMBY, Rodrigo; OLIVEIRA, Luciano; SANTOS, Washington dos; DUARTE, Angelo.
PathoSpotter-Search: A Content-Based Image Retrieval Tool for Nephropathology. In: WORKSHOP DE TRABALHOS EM ANDAMENTO - CONFERENCE ON GRAPHICS, PATTERNS AND IMAGES (SIBGRAPI), 34. , 2021, Online.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2021
.
p. 146-149.
DOI: https://doi.org/10.5753/sibgrapi.est.2021.20027.
