Challenges for XR in Games
Resumo
Extended Reality as a consolidated game platform was always a dream for both final consumers and game producers. If for one side this technology had enchanted and called the attention due its possibilities, for other side many challenges and difficulties had delayed its proliferation and massification. This paper intends to rise and discuss aspects and considerations related to these challenges and solutions. We try to bring the most relevant research topics and try to guess how XR games should look in the near future. We divide the challenges into 7 topics, based on extensive literature reviews: Cybersickness, User Experience, Displays, Rendering, Movements, Body Tracking and External World Information.
Palavras-chave:
Extended Reality, Virtual Reality, Digital Entertainment, Head-mounted Displays, UX
Referências
L. Valente, B. Feijo, J. C. S. do Prado Leite, and E. Clua, “A method to assess pervasive qualities in mobile games,” Personal and Ubiquitous Computing, vol. 22, no. 4, pp. 647–670, 2018.
A. Kemeny, J.-R. Chardonnet, and F. Colombet, Getting Rid of Cybersickness: In Virtual Reality, Augmented Reality, and Simulators. Springer Nature, 2020.
E. M. Kolasinski, “Simulator sickness in virtual environments.” DTIC Document, Tech. Rep., 1995
M. S. Dennison and M. D’Zmura, “Cybersickness without the wobble: experimental results speak against postural instability theory,” Applied ergonomics, vol. 58, pp. 215–223, 2017.
D. Jeong, S. Yoo, and J. Yun, “Cybersickness analysis with EEG using deep learning algorithms,” in 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, 2019, pp. 827–835
J. Kim, W. Kim, H. Oh, S. Lee, and S. Lee, “A deep cybersickness predictor based on brain signal analysis for virtual reality contents,” in Proceedings of the IEEE International Conference on Computer Vision, 2019, pp. 10 580–10 589
N. Xie, G. Ras, M. van Gerven, and D. Doran, “Explainable deep learning: A field guide for the uninitiated,” arXiv:2004.14545, 2020.
T. Porcino, E. O. Rodrigues, A. Silva, E. Clua, and D. Trevisan, “Using the gameplay and user data to predict and identify causes of cybersickness manifestation in virtual reality games,” in 2020 IEEE 8th International Conference on Serious Games and Applications for Health (SeGAH). IEEE, 2020, pp. 1–8.
S. Grassini and K. Laumann, “Are modern head-mounted displays sexist? a systematic review on gender differences in hmd-mediated virtual reality,” Frontiers in Psychology, vol. 11, 2020
C. Curry, R. Li, N. Peterson, and T. A. Stoffregen, “Cybersickness in virtual reality head-mounted displays: Examining the influence of sex differences and vehicle control,” International Journal of Human– Computer Interaction, pp. 1–7, 2020.
C. Hodent, The Gamer’s Brain: How Neuroscience and UX Can Impact Video Game Design. CRC Press, 2017. [Online]. Available: https://books.google.com.br/books?id=JzyhDwAAQBAJ
E. de Oliveira, E. W. G. Clua, C. N. Vasconcelos, B. A. D. Marques, D. G. Trevisan, and L. C. de Castro Salgado, “Fpvrgame: Deep learning for hand pose recognition in real-time using low-end hmd,” in Joint International Conference on Entertainment Computing and Serious Games. Springer, 2019, pp. 70–84.
R. Albert, A. Patney, D. Luebke, and J. Kim, “Latency requirements for foveated rendering in virtual reality,” ACM Transactions on Applied Perception (TAP), vol. 14, no. 4, pp. 1–13, 2017.
N. T. Swafford, J. A. Iglesias-Guitian, C. Koniaris, B. Moon, D. Cosker, and K. Mitchell, “User, metric, and computational evaluation of foveated rendering methods,” in Proceedings of the ACM Symposium on Applied Perception, 2016, pp. 7–14
Fove’s $250,000 kickstarter campaign wants to bring eye-tracking control to virtual reality. Apr 2021, accessed: 2021-07-28. [Online]. Available: [link].
M. Weier, T. Roth, E. Kruijff, A. Hinkenjann, A. Perard-Gayot, P. Slusallek, and Y. Li, “Foveated real-time ray tracing for head-mounted displays,” in Computer Graphics Forum, vol. 35, no. 7. Wiley Online Library, 2016, pp. 289–298.
A. Cannavo, D. Calandra, F. G. Pratticò, V. Gatteschi, and F. Lamberti, “An evaluation testbed for locomotion in virtual reality,” IEEE Transactions on Visualization and Computer Graphics, vol. 27, no. 3, pp. 1871–1889, 2021.
E. Jonathan, C. Roberts, S. Presentation, S. Razzaque, Z. Kohn, and M. Whitton, “Redirected walking,” Proc. of Eurographics, 08 2001.
Q. Sun, A. Patney, L.-Y. Wei, O. Shapira, J. Lu, P. Asente, S. Zhu, M. McGuire, D. Luebke, and A. Kaufman, “Towards virtual reality infinite walking: dynamic saccadic redirection,” ACM Transactions on Graphics (TOG), vol. 37, no. 4, p. 67, 2018.
K. Matsumoto, Y. Ban, T. Narumi, Y. Yanase, T. Tanikawa, and M. Hirose, “Unlimited corridor: Redirected walking techniques using visuo haptic interaction,” in ACM SIGGRAPH 2016 Emerging Technologies, ser. SIGGRAPH ’16. New York, NY, USA: Association for Computing Machinery, 2016. [Online]. Available: https://doi.org/10.1145/2929464.2929482
M. Rietzler, J. Gugenheimer, T. Hirzle, M. Deubzer, E. Langbehn, and E. Rukzio, “Rethinking redirected walking: On the use of curvature gains beyond perceptual limitations and revisiting bending gains,” 10 2018, pp. 115–122
S. Kasahara, K. Konno, R. Owaki, T. Nishi, A. Takeshita, T. Ito, S. Kasuga, and J. Ushiba, “Malleable embodiment: changing sense of embodiment by spatial-temporal deformation of virtual human body,” in Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, 2017, pp. 6438–6448.
F. Jiang, X. Yang, and L. Feng, “Real-time full-body motion reconstruction and recognition for off-the-shelf vr devices,” in Proceedings of the 15th ACM SIGGRAPH Conference on Virtual-Reality Continuum and Its Applications in Industry-Volume 1, 2016, pp. 309–318.
M. Slater and S. Wilbur, “A framework for immersive virtual environments (five): Speculations on the role of presence in virtual environments,” Presence: Teleoperators & Virtual Environments, vol. 6, no. 6, pp. 603–616, 1997.
P. Caserman, A. Garcia-Agundez, and S. Göbel, “A survey of full-body motion reconstruction in immersive virtual reality applications,” IEEE transactions on visualization and computer graphics, vol. 26, no. 10, pp. 3089–3108, 2019.
J. Taylor, V. Tankovich, D. Tang, C. Keskin, D. Kim, P. Davidson, A. Kowdle, and S. Izadi, “Articulated distance fields for ultra-fast tracking of hands interacting,” ACM Transactions on Graphics (TOG), vol. 36, no. 6, pp. 1–12, 2017.
S. Han, B. Liu, R. Cabezas, C. D. Twigg, P. Zhang, J. Petkau, T.-H. Yu, C.-J. Tai, M. Akbay, Z. Wang et al., “Megatrack: monochrome egocentric articulated hand-tracking for virtual reality.” ACM Trans. Graph., vol. 39, no. 4, p. 87, 2020.
R. Pandey, S. Orts-Escolano, C. LeGendre, C. Haene, S. Bouaziz, C. Rhemann, P. Debevec, and S. Fanello, “Total relighting: Learning to relight portraits for background replacement,” vol. 40, no. 4, 2021.
B. A. D. Marques, E. W. G. Clua, and C. N. Vasconcelos, “Deep spherical harmonics light probe estimator for mixed reality games,” Computers Graphics, vol. 76, pp. 96–106, 2018.
A. Kemeny, J.-R. Chardonnet, and F. Colombet, Getting Rid of Cybersickness: In Virtual Reality, Augmented Reality, and Simulators. Springer Nature, 2020.
E. M. Kolasinski, “Simulator sickness in virtual environments.” DTIC Document, Tech. Rep., 1995
M. S. Dennison and M. D’Zmura, “Cybersickness without the wobble: experimental results speak against postural instability theory,” Applied ergonomics, vol. 58, pp. 215–223, 2017.
D. Jeong, S. Yoo, and J. Yun, “Cybersickness analysis with EEG using deep learning algorithms,” in 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, 2019, pp. 827–835
J. Kim, W. Kim, H. Oh, S. Lee, and S. Lee, “A deep cybersickness predictor based on brain signal analysis for virtual reality contents,” in Proceedings of the IEEE International Conference on Computer Vision, 2019, pp. 10 580–10 589
N. Xie, G. Ras, M. van Gerven, and D. Doran, “Explainable deep learning: A field guide for the uninitiated,” arXiv:2004.14545, 2020.
T. Porcino, E. O. Rodrigues, A. Silva, E. Clua, and D. Trevisan, “Using the gameplay and user data to predict and identify causes of cybersickness manifestation in virtual reality games,” in 2020 IEEE 8th International Conference on Serious Games and Applications for Health (SeGAH). IEEE, 2020, pp. 1–8.
S. Grassini and K. Laumann, “Are modern head-mounted displays sexist? a systematic review on gender differences in hmd-mediated virtual reality,” Frontiers in Psychology, vol. 11, 2020
C. Curry, R. Li, N. Peterson, and T. A. Stoffregen, “Cybersickness in virtual reality head-mounted displays: Examining the influence of sex differences and vehicle control,” International Journal of Human– Computer Interaction, pp. 1–7, 2020.
C. Hodent, The Gamer’s Brain: How Neuroscience and UX Can Impact Video Game Design. CRC Press, 2017. [Online]. Available: https://books.google.com.br/books?id=JzyhDwAAQBAJ
E. de Oliveira, E. W. G. Clua, C. N. Vasconcelos, B. A. D. Marques, D. G. Trevisan, and L. C. de Castro Salgado, “Fpvrgame: Deep learning for hand pose recognition in real-time using low-end hmd,” in Joint International Conference on Entertainment Computing and Serious Games. Springer, 2019, pp. 70–84.
R. Albert, A. Patney, D. Luebke, and J. Kim, “Latency requirements for foveated rendering in virtual reality,” ACM Transactions on Applied Perception (TAP), vol. 14, no. 4, pp. 1–13, 2017.
N. T. Swafford, J. A. Iglesias-Guitian, C. Koniaris, B. Moon, D. Cosker, and K. Mitchell, “User, metric, and computational evaluation of foveated rendering methods,” in Proceedings of the ACM Symposium on Applied Perception, 2016, pp. 7–14
Fove’s $250,000 kickstarter campaign wants to bring eye-tracking control to virtual reality. Apr 2021, accessed: 2021-07-28. [Online]. Available: [link].
M. Weier, T. Roth, E. Kruijff, A. Hinkenjann, A. Perard-Gayot, P. Slusallek, and Y. Li, “Foveated real-time ray tracing for head-mounted displays,” in Computer Graphics Forum, vol. 35, no. 7. Wiley Online Library, 2016, pp. 289–298.
A. Cannavo, D. Calandra, F. G. Pratticò, V. Gatteschi, and F. Lamberti, “An evaluation testbed for locomotion in virtual reality,” IEEE Transactions on Visualization and Computer Graphics, vol. 27, no. 3, pp. 1871–1889, 2021.
E. Jonathan, C. Roberts, S. Presentation, S. Razzaque, Z. Kohn, and M. Whitton, “Redirected walking,” Proc. of Eurographics, 08 2001.
Q. Sun, A. Patney, L.-Y. Wei, O. Shapira, J. Lu, P. Asente, S. Zhu, M. McGuire, D. Luebke, and A. Kaufman, “Towards virtual reality infinite walking: dynamic saccadic redirection,” ACM Transactions on Graphics (TOG), vol. 37, no. 4, p. 67, 2018.
K. Matsumoto, Y. Ban, T. Narumi, Y. Yanase, T. Tanikawa, and M. Hirose, “Unlimited corridor: Redirected walking techniques using visuo haptic interaction,” in ACM SIGGRAPH 2016 Emerging Technologies, ser. SIGGRAPH ’16. New York, NY, USA: Association for Computing Machinery, 2016. [Online]. Available: https://doi.org/10.1145/2929464.2929482
M. Rietzler, J. Gugenheimer, T. Hirzle, M. Deubzer, E. Langbehn, and E. Rukzio, “Rethinking redirected walking: On the use of curvature gains beyond perceptual limitations and revisiting bending gains,” 10 2018, pp. 115–122
S. Kasahara, K. Konno, R. Owaki, T. Nishi, A. Takeshita, T. Ito, S. Kasuga, and J. Ushiba, “Malleable embodiment: changing sense of embodiment by spatial-temporal deformation of virtual human body,” in Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, 2017, pp. 6438–6448.
F. Jiang, X. Yang, and L. Feng, “Real-time full-body motion reconstruction and recognition for off-the-shelf vr devices,” in Proceedings of the 15th ACM SIGGRAPH Conference on Virtual-Reality Continuum and Its Applications in Industry-Volume 1, 2016, pp. 309–318.
M. Slater and S. Wilbur, “A framework for immersive virtual environments (five): Speculations on the role of presence in virtual environments,” Presence: Teleoperators & Virtual Environments, vol. 6, no. 6, pp. 603–616, 1997.
P. Caserman, A. Garcia-Agundez, and S. Göbel, “A survey of full-body motion reconstruction in immersive virtual reality applications,” IEEE transactions on visualization and computer graphics, vol. 26, no. 10, pp. 3089–3108, 2019.
J. Taylor, V. Tankovich, D. Tang, C. Keskin, D. Kim, P. Davidson, A. Kowdle, and S. Izadi, “Articulated distance fields for ultra-fast tracking of hands interacting,” ACM Transactions on Graphics (TOG), vol. 36, no. 6, pp. 1–12, 2017.
S. Han, B. Liu, R. Cabezas, C. D. Twigg, P. Zhang, J. Petkau, T.-H. Yu, C.-J. Tai, M. Akbay, Z. Wang et al., “Megatrack: monochrome egocentric articulated hand-tracking for virtual reality.” ACM Trans. Graph., vol. 39, no. 4, p. 87, 2020.
R. Pandey, S. Orts-Escolano, C. LeGendre, C. Haene, S. Bouaziz, C. Rhemann, P. Debevec, and S. Fanello, “Total relighting: Learning to relight portraits for background replacement,” vol. 40, no. 4, 2021.
B. A. D. Marques, E. W. G. Clua, and C. N. Vasconcelos, “Deep spherical harmonics light probe estimator for mixed reality games,” Computers Graphics, vol. 76, pp. 96–106, 2018.
Publicado
18/10/2021
Como Citar
CLUA, Esteban W. G. et al.
Challenges for XR in Games. In: WORKSHOP GRANDGAMESBR - SIMPÓSIO BRASILEIRO DE JOGOS E ENTRETENIMENTO DIGITAL (SBGAMES), 20. , 2021, Online.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2021
.
p. 1017-1020.
DOI: https://doi.org/10.5753/sbgames_estendido.2021.19750.
