Retinal Images Registration via Unsupervised Deep Learning
Resumo
In ophthalmology and vision science applications, aligning a pair of retinal images is of paramount importance to support disease diagnosis and routine eye examinations. This paper introduces an end-to-end framework capable of learning the registration task in a fully unsupervised manner. The proposed approach combines Convolutional Neural Networks and Spatial Transformer Network into a unified pipeline that incorporates a similarity metric to gauge the difference between the images, enabling image alignment without requiring any ground-truth data. The validation study demonstrates that the model can successfully deal with several categories of fundus images, surpassing other recent techniques for retinal registration.
Referências
R. N. Weinreb, T. Aung, and F. A. Medeiros, “The pathophysiology and treatment of glaucoma: a review,” The Journal of the American Medical Association (JAMA), vol. 311, no. 18, pp. 1901–1911, 2014.
K. M. Kim, T.-Y. Heo, A. Kim, J. Kim, K. J. Han, J. Yun, and J. K. Min, “Development of a fundus image-based deep learning diagnostic tool for various retinal diseases,” Journal of Personalized Medicine, vol. 11, no. 5, p. 321, 2021.
G. Litjens, T. Kooi, B. E. Bejnordi, A. A. A. Setio, F. Ciompi, M. Ghafoorian, J. A. van der Laak, B. van Ginneken, and C. I. Sánchez, “A survey on deep learning in medical image analysis,” Medical Image Analysis, vol. 42, pp. 60–88, 2017. [Online]. Available: [link].
G. Haskins, U. Kruger, and P. Yan, “Deep learning in medical image registration: a survey,” Machine Vision and Applications, vol. 31, no. 1, pp. 1–18, 2020.
Y. Fu, Y. Lei, T. Wang, W. J. Curran, T. Liu, and X. Yang, “Deep learning in medical image registration: a review,” Physics in Medicine & Biology, vol. 65, no. 20, p. 20TR01, 2020.
D. Mahapatra, B. Antony, S. Sedai, and R. Garnavi, “Deformable medical image registration using generative adversarial networks,” in IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), 2018, pp. 1449–1453.
Y. Wang, J. Zhang, C. An, M. Cavichini, M. Jhingan, M. J. Amador-Patarroyo, C. P. Long, D.-U. G. Bartsch, W. R. Freeman, and T. Q. Nguyen, “A segmentation based robust deep learning framework for multimodal retinal image registration,” in ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020, pp. 1369–1373.
D. Rivas-Villar, A´ lvaro S. Hervella, J. Rouco, and J. Novo, “Color fundus image registration using a learning-based domainspecific landmark detection methodology,” Computers in Biology and Medicine, vol. 140, p. 105101, 2022. [Online]. Available: [link].
L. He, X. Ren, Q. Gao, X. Zhao, B. Yao, and Y. Chao, “The connectedcomponent labeling problem: A review of state-of-the-art algorithms,” Pattern Recognition, vol. 70, pp. 25–43, 2017. [Online]. Available: [link].
P. Bankhead, C. Scholfield, J. McGeown, and T. Curtis, “Fast retinal vessel detection and measurement using wavelets and edge location refinement,” PloS One, vol. 7, no. 3, p. e32435, 2012.
M. Jaderberg, K. Simonyan, A. Zisserman, and K. Kavukcuoglu, “Spatial transformer networks,” in NIPS, 2015.
A. Kaso, “Computation of the normalized cross-correlation by fast fourier transform,” PLOS ONE, vol. 13, no. 9, pp. 1–16, 09 2018. [Online]. Available: https://doi.org/10.1371/journal.pone.0203434
M. Hisham, S. N. Yaakob, R. Raof, A. A. Nazren, and N. Wafi, “Template matching using sum of squared difference and normalized cross correlation,” in 2015 IEEE Student Conference on Research and Development (SCOReD), 2015, pp. 100–104.
Z. Cui, W. Qi, and Y. Liu, “A fast image template matching algorithm based on normalized cross correlation,” Journal of Physics: Conference Series, vol. 1693, no. 1, p. 012163, dec 2020. [Online]. Available: https://doi.org/10.1088/1742-6596/1693/1/012163
C. Hernandez-Matas, X. Zabulis, A. Triantafyllou, P. Anyfanti, S. Douma, and A. Argyros, “Fire: Fundus image registration dataset,” Journal for Modeling in Ophthalmology, vol. 1, no. 4, pp. 16–28, 2017, source code: [link].
T. Köhler, A. Budai, M. F. Kraus, J. Odstrčilik, G. Michelson, and J. Hornegger, “Automatic no-reference quality assessment for retinal fundus images using vessel segmentation,” in Proceedings of the 26th IEEE International Symposium on Computer-Based Medical Systems, 2013, pp. 95–100.
A. Kori and G. Krishnamurthi, “Zero Shot Learning for Multi-Modal Real Time Image Registration,” arXiv e-prints, p. arXiv:1908.06213, 08 2019.
J. Fan, X. Cao, P.-T. Yap, and D. Shen, “Birnet: Brain image registration using dual-supervised fully convolutional networks,” Medical Image Analysis, vol. 54, pp. 193–206, 2019. [Online]. Available: [link].
B. D. De Vos, F. F. Berendsen, M. A. Viergever, H. Sokooti, M. Staring, and I. Išgum, “A deep learning framework for unsupervised affine and deformable image registration,” Medical Image Analysis, vol. 52, pp. 128–143, 2019. [Online]. Available: [link].
C. WANG, G. Papanastasiou, A. Chartsias, G. Jacenkow, S. A. Tsaftaris, and H. Zhang, “FIRE: Unsupervised bi-directional inter-modality registration using deep networks,” arXiv e-prints, p. arXiv:1907.05062, Jul. 2019.
T. Che, Y. Zheng, J. Cong, Y. Jiang, Y. Niu, W. Jiao, B. Zhao, and Y. Ding, “Deep group-wise registration for multi-spectral images from fundus images,” IEEE Access, vol. 7, pp. 27 650–27 661, 2019.
D. Motta, W. Casaca, and A. Paiva, “Fundus image transformation revisited: Towards determining more accurate registrations,” in IEEE International Symposium on Computer-Based Medical Systems (CBMS), 2018, pp. 227–232.
——, “Vessel optimal transport for automated alignment of retinal fundus images,” IEEE Transactions on Image Processing, vol. 28, no. 12, pp. 6154–6168, 2019.
G. A. Benvenuto, “Registro de imagens de retina via aprendizado profundo não-supervisionado,” Master’s thesis, Universidade Estadual Paulista (Unesp), 2022, disponível em: [link].
G. A. Benvenuto, M. Colnago, and W. Casaca, “Unsupervised deep learning network for deformable fundus image registration,” in ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022, pp. 1281–1285.
G. A. Benvenuto, M. Colnago, M. A. Dias, R. G. Negri, E. A. Silva, and W. Casaca, “A fully unsupervised deep learning framework for non-rigid fundus image registration,” Bioengineering, vol. 9, no. 8, 2022. [Online]. Available: [link]
J. Wang, J. Chen, H. Xu, S. Zhang, X. Mei, J. Huang, and J. Ma, “Gaussian field estimator with manifold regularization for retinal image registration,” Signal Processing, vol. 157, pp. 225–235, 2019. [Online]. Available: [link]
B. D. de Vos, F. F. Berendsen, M. A. Viergever, M. Staring, and I. Išgum, “End-to-end unsupervised deformable image registration with a convolutional neural network,” in Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, M. J. Cardoso, T. Arbel, G. Carneiro, T. Syeda-Mahmood, J. M. R. Tavares, M. Moradi, A. Bradley, H. Greenspan, J. P. Papa, A. Madabhushi, J. C. Nascimento, J. S. Cardoso, V. Belagiannis, and Z. Lu, Eds. Cham: Springer International Publishing, 2017, pp. 204–212.
Y. Hu, M. Modat, E. Gibson, W. Li, N. Ghavami, E. Bonmati, G. Wang, S. Bandula, C. M. Moore, M. Emberton, S. Ourselin, J. A. Noble, D. C. Barratt, and T. Vercauteren, “Weakly-supervised convolutional neural networks for multimodal image registration,” Medical Image Analysis, vol. 49, pp. 1–13, 2018. [Online]. Available: [link].
M. Colnago, C. F. A. Lages, H. S. Picoli, G. A. Benvenuto, T. B. Ghetti, and W. C. d. O. Casaca, “Um estudo de gênero a partir da distribuição de bolsas do programa universidade para todos,” 2022.
M. Colnago, G. A. Benvenuto, W. Casaca, R. G. Negri, E. G. Fernandes, and J. A. Cuminato, “Risk factors associated with mortality in hospitalized patients with covid-19 during the omicron wave in brazil,” Bioengineering, vol. 9, no. 10, 2022. [Online]. Available: [link]
K. M. Kim, T.-Y. Heo, A. Kim, J. Kim, K. J. Han, J. Yun, and J. K. Min, “Development of a fundus image-based deep learning diagnostic tool for various retinal diseases,” Journal of Personalized Medicine, vol. 11, no. 5, p. 321, 2021.
G. Litjens, T. Kooi, B. E. Bejnordi, A. A. A. Setio, F. Ciompi, M. Ghafoorian, J. A. van der Laak, B. van Ginneken, and C. I. Sánchez, “A survey on deep learning in medical image analysis,” Medical Image Analysis, vol. 42, pp. 60–88, 2017. [Online]. Available: [link].
G. Haskins, U. Kruger, and P. Yan, “Deep learning in medical image registration: a survey,” Machine Vision and Applications, vol. 31, no. 1, pp. 1–18, 2020.
Y. Fu, Y. Lei, T. Wang, W. J. Curran, T. Liu, and X. Yang, “Deep learning in medical image registration: a review,” Physics in Medicine & Biology, vol. 65, no. 20, p. 20TR01, 2020.
D. Mahapatra, B. Antony, S. Sedai, and R. Garnavi, “Deformable medical image registration using generative adversarial networks,” in IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), 2018, pp. 1449–1453.
Y. Wang, J. Zhang, C. An, M. Cavichini, M. Jhingan, M. J. Amador-Patarroyo, C. P. Long, D.-U. G. Bartsch, W. R. Freeman, and T. Q. Nguyen, “A segmentation based robust deep learning framework for multimodal retinal image registration,” in ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020, pp. 1369–1373.
D. Rivas-Villar, A´ lvaro S. Hervella, J. Rouco, and J. Novo, “Color fundus image registration using a learning-based domainspecific landmark detection methodology,” Computers in Biology and Medicine, vol. 140, p. 105101, 2022. [Online]. Available: [link].
L. He, X. Ren, Q. Gao, X. Zhao, B. Yao, and Y. Chao, “The connectedcomponent labeling problem: A review of state-of-the-art algorithms,” Pattern Recognition, vol. 70, pp. 25–43, 2017. [Online]. Available: [link].
P. Bankhead, C. Scholfield, J. McGeown, and T. Curtis, “Fast retinal vessel detection and measurement using wavelets and edge location refinement,” PloS One, vol. 7, no. 3, p. e32435, 2012.
M. Jaderberg, K. Simonyan, A. Zisserman, and K. Kavukcuoglu, “Spatial transformer networks,” in NIPS, 2015.
A. Kaso, “Computation of the normalized cross-correlation by fast fourier transform,” PLOS ONE, vol. 13, no. 9, pp. 1–16, 09 2018. [Online]. Available: https://doi.org/10.1371/journal.pone.0203434
M. Hisham, S. N. Yaakob, R. Raof, A. A. Nazren, and N. Wafi, “Template matching using sum of squared difference and normalized cross correlation,” in 2015 IEEE Student Conference on Research and Development (SCOReD), 2015, pp. 100–104.
Z. Cui, W. Qi, and Y. Liu, “A fast image template matching algorithm based on normalized cross correlation,” Journal of Physics: Conference Series, vol. 1693, no. 1, p. 012163, dec 2020. [Online]. Available: https://doi.org/10.1088/1742-6596/1693/1/012163
C. Hernandez-Matas, X. Zabulis, A. Triantafyllou, P. Anyfanti, S. Douma, and A. Argyros, “Fire: Fundus image registration dataset,” Journal for Modeling in Ophthalmology, vol. 1, no. 4, pp. 16–28, 2017, source code: [link].
T. Köhler, A. Budai, M. F. Kraus, J. Odstrčilik, G. Michelson, and J. Hornegger, “Automatic no-reference quality assessment for retinal fundus images using vessel segmentation,” in Proceedings of the 26th IEEE International Symposium on Computer-Based Medical Systems, 2013, pp. 95–100.
A. Kori and G. Krishnamurthi, “Zero Shot Learning for Multi-Modal Real Time Image Registration,” arXiv e-prints, p. arXiv:1908.06213, 08 2019.
J. Fan, X. Cao, P.-T. Yap, and D. Shen, “Birnet: Brain image registration using dual-supervised fully convolutional networks,” Medical Image Analysis, vol. 54, pp. 193–206, 2019. [Online]. Available: [link].
B. D. De Vos, F. F. Berendsen, M. A. Viergever, H. Sokooti, M. Staring, and I. Išgum, “A deep learning framework for unsupervised affine and deformable image registration,” Medical Image Analysis, vol. 52, pp. 128–143, 2019. [Online]. Available: [link].
C. WANG, G. Papanastasiou, A. Chartsias, G. Jacenkow, S. A. Tsaftaris, and H. Zhang, “FIRE: Unsupervised bi-directional inter-modality registration using deep networks,” arXiv e-prints, p. arXiv:1907.05062, Jul. 2019.
T. Che, Y. Zheng, J. Cong, Y. Jiang, Y. Niu, W. Jiao, B. Zhao, and Y. Ding, “Deep group-wise registration for multi-spectral images from fundus images,” IEEE Access, vol. 7, pp. 27 650–27 661, 2019.
D. Motta, W. Casaca, and A. Paiva, “Fundus image transformation revisited: Towards determining more accurate registrations,” in IEEE International Symposium on Computer-Based Medical Systems (CBMS), 2018, pp. 227–232.
——, “Vessel optimal transport for automated alignment of retinal fundus images,” IEEE Transactions on Image Processing, vol. 28, no. 12, pp. 6154–6168, 2019.
G. A. Benvenuto, “Registro de imagens de retina via aprendizado profundo não-supervisionado,” Master’s thesis, Universidade Estadual Paulista (Unesp), 2022, disponível em: [link].
G. A. Benvenuto, M. Colnago, and W. Casaca, “Unsupervised deep learning network for deformable fundus image registration,” in ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022, pp. 1281–1285.
G. A. Benvenuto, M. Colnago, M. A. Dias, R. G. Negri, E. A. Silva, and W. Casaca, “A fully unsupervised deep learning framework for non-rigid fundus image registration,” Bioengineering, vol. 9, no. 8, 2022. [Online]. Available: [link]
J. Wang, J. Chen, H. Xu, S. Zhang, X. Mei, J. Huang, and J. Ma, “Gaussian field estimator with manifold regularization for retinal image registration,” Signal Processing, vol. 157, pp. 225–235, 2019. [Online]. Available: [link]
B. D. de Vos, F. F. Berendsen, M. A. Viergever, M. Staring, and I. Išgum, “End-to-end unsupervised deformable image registration with a convolutional neural network,” in Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, M. J. Cardoso, T. Arbel, G. Carneiro, T. Syeda-Mahmood, J. M. R. Tavares, M. Moradi, A. Bradley, H. Greenspan, J. P. Papa, A. Madabhushi, J. C. Nascimento, J. S. Cardoso, V. Belagiannis, and Z. Lu, Eds. Cham: Springer International Publishing, 2017, pp. 204–212.
Y. Hu, M. Modat, E. Gibson, W. Li, N. Ghavami, E. Bonmati, G. Wang, S. Bandula, C. M. Moore, M. Emberton, S. Ourselin, J. A. Noble, D. C. Barratt, and T. Vercauteren, “Weakly-supervised convolutional neural networks for multimodal image registration,” Medical Image Analysis, vol. 49, pp. 1–13, 2018. [Online]. Available: [link].
M. Colnago, C. F. A. Lages, H. S. Picoli, G. A. Benvenuto, T. B. Ghetti, and W. C. d. O. Casaca, “Um estudo de gênero a partir da distribuição de bolsas do programa universidade para todos,” 2022.
M. Colnago, G. A. Benvenuto, W. Casaca, R. G. Negri, E. G. Fernandes, and J. A. Cuminato, “Risk factors associated with mortality in hospitalized patients with covid-19 during the omicron wave in brazil,” Bioengineering, vol. 9, no. 10, 2022. [Online]. Available: [link]
Publicado
06/11/2023
Como Citar
BENVENUTO, Giovana Augusta; CASACA, Wallace.
Retinal Images Registration via Unsupervised Deep Learning. In: WORKSHOP DE TESES E DISSERTAÇÕES - CONFERENCE ON GRAPHICS, PATTERNS AND IMAGES (SIBGRAPI), 36. , 2023, Rio Grande/RS.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2023
.
p. 42-48.
DOI: https://doi.org/10.5753/sibgrapi.est.2023.27450.
