YOLOv8 Object Detection on HoloLens 2: Navigating the Accuracy–Efficiency Trade-Off
Resumo
Object detection is a relevant capability in augmented reality (AR) systems, enabling applications to understand physical environments. However, implementing deep learning-based object detectors like the YOLO (You Only Look Once) family on devices like the Microsoft HoloLens 2 (HL2) is challenging due to limited processing power. This paper investigates this trade-off regarding deploying YOLOv8 variants on the HL2 for real-time object detection in AR scenarios. A two-phase methodology is proposed: theoretical benchmarking using the COCO dataset and empirical validation on the device. Using a multi-criteria decision-making framework based on Euclidean distance optimization, the study identifies optimal configurations balancing inference speed and detection quality. Results show that while larger YOLOv8 models achieve higher accuracy, only lightweight configurations maintain processing rates above 15 FPS with latencies below 100 ms — thresholds necessary for perceptual synchronization in AR. The findings provide practical guidelines and performance metrics for selecting YOLO-based models in resource-constrained AR platforms, contributing to the development of efficient and interactive AR systems.
Palavras-chave:
YOLO, HoloLens2, Accuracy, Efficiency, Object Detection, Augmented Reality
Referências
R. T. Azuma, "A survey of augmented reality," Presence: Teleoperators and Virtual Environments, vol. 6, no. 4, pp. 355–385, 1997. DOI: 10.1162/pres.1997.6.4.355
D. Wu, Z. Li, M. H. D. Ansari, X. T. Ha, M. Ourak, J. Dankelman, A. Menciassi, E. De Momi, and E. V. Poorten, "Comparative analysis of interactive modalities for intuitive endovascular interventions," IEEE Transactions on Visualization and Computer Graphics, vol. 31, no. 2, pp. 1371–1388, 2025.
M. Benmahdjoub, A. Thabit, M.-L. C. van Veelen, W. J. Niessen, E. B. Wolvius, and T. v. Walsum, "Evaluation of AR visualization approaches for catheter insertion into the ventricle cavity," IEEE Transactions on Visualization and Computer Graphics, vol. 29, no. 5, pp. 2434–2445, 2023.
T. Bui, M. A. Ruiz-Cardozo, H. S. Dave, K. Barot, M. R. Kann, K. Joseph, S. Lopez-Alviar, G. Trevino, S. Brehm, A. T. Yahanda, and C. A. Molina, "Virtual, augmented, and mixed reality applications for surgical rehearsal, operative execution, and patient education in spine surgery: A scoping review," Medicina (Lithuania), vol. 60, no. 2, 2024.
X. Min, W. Zhang, S. Sun, N. Zhao, S. Tang, and Y. Zhuang, "Vpmodel: High-fidelity product simulation in a virtual-physical environment," IEEE Transactions on Visualization and Computer Graphics, vol. 25, pp. 3083–3093, 2019.
J. S. Devagiri, S. Paheding, Q. Niyaz, X. Yang, and S. Smith, "Augmented reality and artificial intelligence in industry: Trends, tools, and future challenges," Expert Systems with Applications, vol. 207, p. 118002, 2022.
M. Safi and J. Chung, "Augmented reality uses and applications in aerospace and aviation," in Springer Handbook of Augmented Reality, Springer International Publishing, 2023, p. 473–494.
F. Zulfiqar, R. Raza, M. O. Khan, M. Arif, A. Alvi, and T. Alam, "Augmented reality and its applications in education: A systematic survey," IEEE Access, vol. 11, pp. 143250–143271, 2023.
R. Suzuki, A. Karim, T. Xia, H. Hedayati, and N. Marquardt, "Augmented reality and robotics: A survey and taxonomy for AR-enhanced human-robot interaction and robotic interfaces," in CHI Conference on Human Factors in Computing Systems, 2022.
M. Sohan, T. Sai Ram, and C. V. Rami Reddy, "A review on YOLOv8 and its advancements," in Data Intelligence and Cognitive Informatics, Springer Nature Singapore, 2024, pp. 529–545.
Y. Ghasemi, H. Jeong, S. H. Choi, K.-B. Park, and J. Y. Lee, "Deep learning-based object detection in augmented reality: A systematic review," Computers in Industry, vol. 139, p. 103661, 2022.
Microsoft, "Microsoft HoloLens," [link], 2016.
Microsoft, "About the HoloLens 2," [link], 2024.
T. Tene, D. F. Vique Lopez, P. E. Valverde Aguirre, L. M. Orna Puente, and C. Vacacela Gomez, "Virtual reality and augmented reality in medical education: An umbrella review," Frontiers in Digital Health, vol. 6, 2024.
C.-L. Hwang and K. Yoon, "Methods for multiple attribute decision making," Multiple Attribute Decision Making: Methods and Applications, pp. 58–191, 1981.
E. Triantaphyllou, "Multi-Criteria Decision Making Methods: A Comparative Study," Springer, vol. 44, 2000.
D. Ungureanu, F. Bogo, S. Galliani, P. Sama, X. Duan, C. Meekhof, J. Stuhmer, T. J. Cashman, B. Tekin, J. L. Schonberger, P. Olszta, and M. Pollefeys, "HoloLens 2 research mode as a tool for computer vision research," 2020.
J. S. Rieder, D. H. Van Tol, and D. Aschenbrenner, "Effective close-range accuracy comparison of Microsoft HoloLens generation one and two using Vuforia ImageTargets," in 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, 2021.
I. Soares, R. B. Sousa, M. Petry, and A. P. Moreira, "Accuracy and repeatability tests on HoloLens 2 and HTC Vive," Multimodal Technologies and Interaction, vol. 5, no. 8, 2021.
I. Matyash, R. Kutzner, T. Neumuth, and M. Rockstroh, "Accuracy measurement of HoloLens2 IMUs in medical environments," Current Directions in Biomedical Engineering, vol. 7, no. 2, pp. 633–636, 2021.
M. Lysakowski, K. Zywanowski, A. Banaszczyk, M. R. Nowicki, P. Skrzypczynski, and S. K. Tadeja, "Real-time onboard object detection for augmented reality: Enhancing head-mounted display with YOLOv8," 2023.
A. Farasin, F. Peciarolo, M. Grangetto, E. Gianaria, and P. Garza, "Real-time object detection and tracking in mixed reality using Microsoft HoloLens," in Proceedings of the 15th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2020), pp. 165–172, 2020.
L. Perez-Pachon, P. Sharma, H. Brech, J. Gregory, T. Lowe, M. Poyade, and F. Groning, "Augmented reality headsets for surgical guidance: The impact of holographic model positions on user localisation accuracy," Virtual Reality, vol. 28, no. 2, 2024.
C. Scherl, J. Stratemeier, C. Karle, N. Rotter, J. Hesser, L. Huber, A. Dias, O. Hoffmann, P. Riffel, S. O. Schoenberg, A. Schell, A. Lammert, A. Affolter, and D. Mannle, "Augmented reality with HoloLens in parotid surgery: how to assess and to improve accuracy," European Archives of Oto-Rhino-Laryngology, vol. 278, no. 7, pp. 2473–2483, 2021.
F. Van Gestel, T. Frantz, C. Vannerom, A. Verhellen, A. G. Gallagher, S. A. Elprama, A. Jacobs, R. Buyl, M. Bruneau, B. Jansen, J. Vandemeulebroucke, T. Scheerlinck, and J. Duerinck, "The effect of augmented reality on the accuracy and learning curve of external ventricular drain placement," Neurosurgical Focus, vol. 51, pp. 1–9, 2021.
L. C. Peixe, S. Agati, and M. D. S. Hounsell, "Manual assembly augmented reality systems implementation: A systematic literature mapping," in Proceedings of the 25th Symposium on Virtual and Augmented Reality, SVR ’23, 2024, pp. 17–25.
J.-P. Stauffert, F. Niebling, and M. Latoschik, "Latency and cybersickness: Impact, causes, and measures. A review," Frontiers in Virtual Reality, vol. 1, 2020.
N. Elmqvist, A. Vande Moere, H.-C. Jetter, D. Cernea, H. Reiterer, and T. J. Jankun-Kelly, "Fluid interaction for information visualization," Information Visualization, vol. 10, no. 4, pp. 327–340, 2011.
D. Wu, Z. Li, M. H. D. Ansari, X. T. Ha, M. Ourak, J. Dankelman, A. Menciassi, E. De Momi, and E. V. Poorten, "Comparative analysis of interactive modalities for intuitive endovascular interventions," IEEE Transactions on Visualization and Computer Graphics, vol. 31, no. 2, pp. 1371–1388, 2025.
M. Benmahdjoub, A. Thabit, M.-L. C. van Veelen, W. J. Niessen, E. B. Wolvius, and T. v. Walsum, "Evaluation of AR visualization approaches for catheter insertion into the ventricle cavity," IEEE Transactions on Visualization and Computer Graphics, vol. 29, no. 5, pp. 2434–2445, 2023.
T. Bui, M. A. Ruiz-Cardozo, H. S. Dave, K. Barot, M. R. Kann, K. Joseph, S. Lopez-Alviar, G. Trevino, S. Brehm, A. T. Yahanda, and C. A. Molina, "Virtual, augmented, and mixed reality applications for surgical rehearsal, operative execution, and patient education in spine surgery: A scoping review," Medicina (Lithuania), vol. 60, no. 2, 2024.
X. Min, W. Zhang, S. Sun, N. Zhao, S. Tang, and Y. Zhuang, "Vpmodel: High-fidelity product simulation in a virtual-physical environment," IEEE Transactions on Visualization and Computer Graphics, vol. 25, pp. 3083–3093, 2019.
J. S. Devagiri, S. Paheding, Q. Niyaz, X. Yang, and S. Smith, "Augmented reality and artificial intelligence in industry: Trends, tools, and future challenges," Expert Systems with Applications, vol. 207, p. 118002, 2022.
M. Safi and J. Chung, "Augmented reality uses and applications in aerospace and aviation," in Springer Handbook of Augmented Reality, Springer International Publishing, 2023, p. 473–494.
F. Zulfiqar, R. Raza, M. O. Khan, M. Arif, A. Alvi, and T. Alam, "Augmented reality and its applications in education: A systematic survey," IEEE Access, vol. 11, pp. 143250–143271, 2023.
R. Suzuki, A. Karim, T. Xia, H. Hedayati, and N. Marquardt, "Augmented reality and robotics: A survey and taxonomy for AR-enhanced human-robot interaction and robotic interfaces," in CHI Conference on Human Factors in Computing Systems, 2022.
M. Sohan, T. Sai Ram, and C. V. Rami Reddy, "A review on YOLOv8 and its advancements," in Data Intelligence and Cognitive Informatics, Springer Nature Singapore, 2024, pp. 529–545.
Y. Ghasemi, H. Jeong, S. H. Choi, K.-B. Park, and J. Y. Lee, "Deep learning-based object detection in augmented reality: A systematic review," Computers in Industry, vol. 139, p. 103661, 2022.
Microsoft, "Microsoft HoloLens," [link], 2016.
Microsoft, "About the HoloLens 2," [link], 2024.
T. Tene, D. F. Vique Lopez, P. E. Valverde Aguirre, L. M. Orna Puente, and C. Vacacela Gomez, "Virtual reality and augmented reality in medical education: An umbrella review," Frontiers in Digital Health, vol. 6, 2024.
C.-L. Hwang and K. Yoon, "Methods for multiple attribute decision making," Multiple Attribute Decision Making: Methods and Applications, pp. 58–191, 1981.
E. Triantaphyllou, "Multi-Criteria Decision Making Methods: A Comparative Study," Springer, vol. 44, 2000.
D. Ungureanu, F. Bogo, S. Galliani, P. Sama, X. Duan, C. Meekhof, J. Stuhmer, T. J. Cashman, B. Tekin, J. L. Schonberger, P. Olszta, and M. Pollefeys, "HoloLens 2 research mode as a tool for computer vision research," 2020.
J. S. Rieder, D. H. Van Tol, and D. Aschenbrenner, "Effective close-range accuracy comparison of Microsoft HoloLens generation one and two using Vuforia ImageTargets," in 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, 2021.
I. Soares, R. B. Sousa, M. Petry, and A. P. Moreira, "Accuracy and repeatability tests on HoloLens 2 and HTC Vive," Multimodal Technologies and Interaction, vol. 5, no. 8, 2021.
I. Matyash, R. Kutzner, T. Neumuth, and M. Rockstroh, "Accuracy measurement of HoloLens2 IMUs in medical environments," Current Directions in Biomedical Engineering, vol. 7, no. 2, pp. 633–636, 2021.
M. Lysakowski, K. Zywanowski, A. Banaszczyk, M. R. Nowicki, P. Skrzypczynski, and S. K. Tadeja, "Real-time onboard object detection for augmented reality: Enhancing head-mounted display with YOLOv8," 2023.
A. Farasin, F. Peciarolo, M. Grangetto, E. Gianaria, and P. Garza, "Real-time object detection and tracking in mixed reality using Microsoft HoloLens," in Proceedings of the 15th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2020), pp. 165–172, 2020.
L. Perez-Pachon, P. Sharma, H. Brech, J. Gregory, T. Lowe, M. Poyade, and F. Groning, "Augmented reality headsets for surgical guidance: The impact of holographic model positions on user localisation accuracy," Virtual Reality, vol. 28, no. 2, 2024.
C. Scherl, J. Stratemeier, C. Karle, N. Rotter, J. Hesser, L. Huber, A. Dias, O. Hoffmann, P. Riffel, S. O. Schoenberg, A. Schell, A. Lammert, A. Affolter, and D. Mannle, "Augmented reality with HoloLens in parotid surgery: how to assess and to improve accuracy," European Archives of Oto-Rhino-Laryngology, vol. 278, no. 7, pp. 2473–2483, 2021.
F. Van Gestel, T. Frantz, C. Vannerom, A. Verhellen, A. G. Gallagher, S. A. Elprama, A. Jacobs, R. Buyl, M. Bruneau, B. Jansen, J. Vandemeulebroucke, T. Scheerlinck, and J. Duerinck, "The effect of augmented reality on the accuracy and learning curve of external ventricular drain placement," Neurosurgical Focus, vol. 51, pp. 1–9, 2021.
L. C. Peixe, S. Agati, and M. D. S. Hounsell, "Manual assembly augmented reality systems implementation: A systematic literature mapping," in Proceedings of the 25th Symposium on Virtual and Augmented Reality, SVR ’23, 2024, pp. 17–25.
J.-P. Stauffert, F. Niebling, and M. Latoschik, "Latency and cybersickness: Impact, causes, and measures. A review," Frontiers in Virtual Reality, vol. 1, 2020.
N. Elmqvist, A. Vande Moere, H.-C. Jetter, D. Cernea, H. Reiterer, and T. J. Jankun-Kelly, "Fluid interaction for information visualization," Information Visualization, vol. 10, no. 4, pp. 327–340, 2011.
Publicado
30/09/2025
Como Citar
SILVA, Edvar; LIMA, Bruno; ALVES, Renalvo; GEORGEVICH, Bruno; VIEIRA, Tiago.
YOLOv8 Object Detection on HoloLens 2: Navigating the Accuracy–Efficiency Trade-Off. In: SIMPÓSIO DE REALIDADE VIRTUAL E AUMENTADA (SVR), 27. , 2025, Salvador/BA.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2025
.
p. 90-98.
