Neonatal Face Segmentation with and without Clinical Devices using SAM
Abstract
Facial expression analysis has been widely used as one of the main approaches for pain diagnosis, both by humans and computing systems. However, in clinical practice, newborns who remain hospitalized in Neonatal Intensive Care Units often have devices connected to their faces, such as enteral/gastric probes, orotracheal intubation tubes, and phototherapy goggles, which hinder the visualization of facial regions making the proper diagnosis of pain much harder in practice. Therefore, to address this issue, we have evaluated the state-of-the-art Segment Anything Model (SAM) tool combined with RetinaFace for segmentation of 2D face images of neonates, including free faces and the ones with devices connected, against a simple and traditional landmark method, and a recently proposed deep neural network fine-tuned for face segmentation under occlusion. SAM performed comparatively better than the other two models for both no occlusion and high occlusion 2D face images, scoring on average impressively 0.98 and 0.91 at the standard dice similarity coefficient respectively.
References
M. García-Rodríguez, S. Bujan-Bravo, R. Seijo-Bestilleiro, and C. Gonzalez, “Pain assessment and management in the newborn: A systematized review,” World journal of clinical cases, vol. 9, pp. 5921–5931, 07 2021.
T. M. Heiderich, M. C. d. M. Barros, and R. Guinsburg, “Concordância interavaliadores na identificação de faces de dor de recém-nascidos a termo e pré-termo tardio: estudo transversal,” BrJP, vol. 3, pp. 348–353, 2020.
G. De Clifford Faugère, M. Aita, N. Feeley, S. Colson et al., “Nurses’ perception of preterm infants’ pain and the factors of their pain assessment and management,” The Journal of Perinatal & Neonatal Nursing, vol. 36, no. 3, pp. 312–326, 2022.
G. V. T. d. Silva, M. C. d. M. Barros, J. d. C. A. Soares, L. P. Carlini, T. M. Heiderich, R. N. Orsi, R. d. C. X. Balda, C. E. Thomaz, and R. Guinsburg, “What facial features does the pediatrician look to decide that a newborn is feeling pain?” American Journal of Perinatology, vol. 40, no. 08, pp. 851–857, 2021.
J. d. C. A. Soares, M. C. d. M. Barros, G. V. T. d. Silva, L. P. Carlini, T. M. Heiderich, R. N. Orsi, R. d. C. X. Balda, P. A. S. O. Silva, C. E. Thomaz, and R. Guinsburg, “Looking at neonatal facial features of pain: do health and non-health professionals differ?” Jornal de Pediatria, vol. 98, pp. 406–412, 2022.
M. C. d. M. Barros, C. E. Thomaz, G. V. T. da Silva, J. do Carmo Azevedo Soares, L. P. Carlini, T. M. Heiderich, R. N. Orsi, R. d. C. X. Balda, P. A. S. O. Silva, A. Sanudo et al., “Identification of pain in neonates: the adults’ visual perception of neonatal facial features,” Journal of Perinatology, vol. 41, no. 9, pp. 2304–2308, 2021.
R. Orsi, L. Carlini, T. Heiderich, G. Silva, J. Soares, R. Balda, M. Barros, R. Guinsburg, and C. Thomaz, “Visual attention during neonatal pain assessment: A 2-second exposure to a facial expression is sufficient,” Authorea Preprints, 2022.
A. Llerena, K. Tran, D. Choudhary, J. Hausmann, D. Goldgof, Y. Sun, and S. Prescott, “Neonatal pain assessment: Do we have the right tools?” Frontiers in Pediatrics, vol. 10, 02 2023.
S. Gkikas and M. Tsiknakis, “Automatic assessment of pain based on deep learning methods: A systematic review,” Computer Methods and Programs in Biomedicine, vol. 231, p. 107365, 2023. [Online]. Available: [link].
R. V. Grunau and K. D. Craig, “Pain expression in neonates: facial action and cry,” Pain, vol. 28, no. 3, pp. 395–410, 1987. [Online]. Available: [link].
B. J. Stevens, S. Gibbins, J. Yamada, K. Dionne, G. Lee, C. Johnston, and A. Taddio, “The premature infant pain profile-revised (pipp-r): initial validation and feasibility,” The Clinical journal of pain, vol. 30, no. 3, pp. 238–243, 2014.
G. Zamzmi, R. Paul, M. S. Salekin, D. Goldgof, R. Kasturi, T. Ho, and Y. Sun, “Convolutional neural networks for neonatal pain assessment,” IEEE Transactions on Biometrics, Behavior, and Identity Science, vol. 1, no. 3, pp. 192–200, 2019.
L. P. Carlini, L. A. Ferreira, G. A. Coutrin, V. V. Varoto, T. M. Heiderich, R. C. Balda, M. C. Barros, R. Guinsburg, and C. E. Thomaz, “A convolutional neural network-based mobile application to bedside neonatal pain assessment,” in 2021 34th SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI). IEEE, 2021, pp. 394–401.
P. H. S. Domingues, R. M. M. da Silva, I. J. Orra, M. E. Cruz, T. M. Heiderich, and C. E. Thomaz, “Neonatal face mosaic: An areas-ofinterest segmentation method based on 2d face images,” in Anais do XVII Workshop de Visão Computacional. SBC, 2021, pp. 201–205.
Y. S. Dosso, D. Kyrollos, K. J. Greenwood, J. Harrold, and J. R. Green, “Nicuface: Robust neonatal face detection in complex nicu scenes,” IEEE Access, vol. 10, pp. 62 893–62 909, 2022.
A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo et al., “Segment anything,” arXiv preprint arXiv:2304.02643, 2023.
J. Deng, J. Guo, E. Ververas, I. Kotsia, and S. Zafeiriou, “Retinaface: Single-shot multi-level face localisation in the wild,” in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 5202–5211.
Y. Liu, H. Shen, Y. Si, X. Wang, X. Zhu, H. Shi, Z. Hong, H. Guo, Z. Guo, Y. Chen, B. Li, T. Xi, J. Yu, H. Xie, G. Xie, M. Li, Q. Lu, Z. Wang, S. Lai, Z. Chai, and X. Wei, “Grand challenge of 106-point facial landmark localization,” 2019 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), pp. 613–616, 2019. [Online]. Available: [link]
K. T. R. Voo, L. Jiang, and C. C. Loy, “Delving into high-quality synthetic face occlusion segmentation datasets,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2022.
T. Marcondes Heiderich and A. Leslie, “Neonatal procedural pain can be assessed by computer software that has good sensitivity and specificity to detect facial movements,” Acta Paediatrica, vol. 104, 11 2014.
L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam, “Encoderdecoder with atrous separable convolution for semantic image segmentation,” in Computer Vision – ECCV 2018, V. Ferrari, M. Hebert, C. Sminchisescu, and Y. Weiss, Eds. Cham: Springer International Publishing, 2018, pp. 833–851.
T. M. Heiderich, M. C. d. M. Barros, and R. Guinsburg, “Concordância interavaliadores na identificação de faces de dor de recém-nascidos a termo e pré-termo tardio: estudo transversal,” BrJP, vol. 3, pp. 348–353, 2020.
G. De Clifford Faugère, M. Aita, N. Feeley, S. Colson et al., “Nurses’ perception of preterm infants’ pain and the factors of their pain assessment and management,” The Journal of Perinatal & Neonatal Nursing, vol. 36, no. 3, pp. 312–326, 2022.
G. V. T. d. Silva, M. C. d. M. Barros, J. d. C. A. Soares, L. P. Carlini, T. M. Heiderich, R. N. Orsi, R. d. C. X. Balda, C. E. Thomaz, and R. Guinsburg, “What facial features does the pediatrician look to decide that a newborn is feeling pain?” American Journal of Perinatology, vol. 40, no. 08, pp. 851–857, 2021.
J. d. C. A. Soares, M. C. d. M. Barros, G. V. T. d. Silva, L. P. Carlini, T. M. Heiderich, R. N. Orsi, R. d. C. X. Balda, P. A. S. O. Silva, C. E. Thomaz, and R. Guinsburg, “Looking at neonatal facial features of pain: do health and non-health professionals differ?” Jornal de Pediatria, vol. 98, pp. 406–412, 2022.
M. C. d. M. Barros, C. E. Thomaz, G. V. T. da Silva, J. do Carmo Azevedo Soares, L. P. Carlini, T. M. Heiderich, R. N. Orsi, R. d. C. X. Balda, P. A. S. O. Silva, A. Sanudo et al., “Identification of pain in neonates: the adults’ visual perception of neonatal facial features,” Journal of Perinatology, vol. 41, no. 9, pp. 2304–2308, 2021.
R. Orsi, L. Carlini, T. Heiderich, G. Silva, J. Soares, R. Balda, M. Barros, R. Guinsburg, and C. Thomaz, “Visual attention during neonatal pain assessment: A 2-second exposure to a facial expression is sufficient,” Authorea Preprints, 2022.
A. Llerena, K. Tran, D. Choudhary, J. Hausmann, D. Goldgof, Y. Sun, and S. Prescott, “Neonatal pain assessment: Do we have the right tools?” Frontiers in Pediatrics, vol. 10, 02 2023.
S. Gkikas and M. Tsiknakis, “Automatic assessment of pain based on deep learning methods: A systematic review,” Computer Methods and Programs in Biomedicine, vol. 231, p. 107365, 2023. [Online]. Available: [link].
R. V. Grunau and K. D. Craig, “Pain expression in neonates: facial action and cry,” Pain, vol. 28, no. 3, pp. 395–410, 1987. [Online]. Available: [link].
B. J. Stevens, S. Gibbins, J. Yamada, K. Dionne, G. Lee, C. Johnston, and A. Taddio, “The premature infant pain profile-revised (pipp-r): initial validation and feasibility,” The Clinical journal of pain, vol. 30, no. 3, pp. 238–243, 2014.
G. Zamzmi, R. Paul, M. S. Salekin, D. Goldgof, R. Kasturi, T. Ho, and Y. Sun, “Convolutional neural networks for neonatal pain assessment,” IEEE Transactions on Biometrics, Behavior, and Identity Science, vol. 1, no. 3, pp. 192–200, 2019.
L. P. Carlini, L. A. Ferreira, G. A. Coutrin, V. V. Varoto, T. M. Heiderich, R. C. Balda, M. C. Barros, R. Guinsburg, and C. E. Thomaz, “A convolutional neural network-based mobile application to bedside neonatal pain assessment,” in 2021 34th SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI). IEEE, 2021, pp. 394–401.
P. H. S. Domingues, R. M. M. da Silva, I. J. Orra, M. E. Cruz, T. M. Heiderich, and C. E. Thomaz, “Neonatal face mosaic: An areas-ofinterest segmentation method based on 2d face images,” in Anais do XVII Workshop de Visão Computacional. SBC, 2021, pp. 201–205.
Y. S. Dosso, D. Kyrollos, K. J. Greenwood, J. Harrold, and J. R. Green, “Nicuface: Robust neonatal face detection in complex nicu scenes,” IEEE Access, vol. 10, pp. 62 893–62 909, 2022.
A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo et al., “Segment anything,” arXiv preprint arXiv:2304.02643, 2023.
J. Deng, J. Guo, E. Ververas, I. Kotsia, and S. Zafeiriou, “Retinaface: Single-shot multi-level face localisation in the wild,” in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 5202–5211.
Y. Liu, H. Shen, Y. Si, X. Wang, X. Zhu, H. Shi, Z. Hong, H. Guo, Z. Guo, Y. Chen, B. Li, T. Xi, J. Yu, H. Xie, G. Xie, M. Li, Q. Lu, Z. Wang, S. Lai, Z. Chai, and X. Wei, “Grand challenge of 106-point facial landmark localization,” 2019 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), pp. 613–616, 2019. [Online]. Available: [link]
K. T. R. Voo, L. Jiang, and C. C. Loy, “Delving into high-quality synthetic face occlusion segmentation datasets,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2022.
T. Marcondes Heiderich and A. Leslie, “Neonatal procedural pain can be assessed by computer software that has good sensitivity and specificity to detect facial movements,” Acta Paediatrica, vol. 104, 11 2014.
L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam, “Encoderdecoder with atrous separable convolution for semantic image segmentation,” in Computer Vision – ECCV 2018, V. Ferrari, M. Hebert, C. Sminchisescu, and Y. Weiss, Eds. Cham: Springer International Publishing, 2018, pp. 833–851.
Published
2023-11-06
How to Cite
DOMINGUES, Pedro Henrique Silva; HEIDERICH, Tatiany Marcondes; BARROS, Marina Carvalho de Moraes; GUINSBURG, Ruth; THOMAZ, Carlos Eduardo.
Neonatal Face Segmentation with and without Clinical Devices using SAM. In: WORKSHOP OF WORKS IN PROGRESS - CONFERENCE ON GRAPHICS, PATTERNS AND IMAGES (SIBGRAPI), 36. , 2023, Rio Grande/RS.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2023
.
p. 101-104.
DOI: https://doi.org/10.5753/sibgrapi.est.2023.27459.
