A mixed path tracing and NeRF approach for optimizing rendering in XR Displays
Resumo
Real-time effects achieved by path tracing are essential for creating highly accurate illumination effects in interactive environments. However, due to its computational complexity, it is essential to explore optimization techniques like Foveated Rendering. Recent advancements in the field of Neural Radiance Fields allow for reconstructing scenes while maintaining adequate frame rates and keeping global and dynamic illumination while in Virtual Reality. The present paper proposes the usage of hybrid rendering: real path tracing at the fovea region and NeRF baked from offline path-traced images at the peripheries. The main idea is to reduce the size of the path-traced rendered image only for a small amount of pixels related to the fovea area, while at the same time providing less costly reconstructions in the periphery. To assess the effectiveness of our method, we presented users with images of a particular scene generated using both conventional rendering and our unique approach. We then asked for their evaluation regarding any perceived degradation in the images. Our work contributes to the development of rendering techniques for XR experiences that demand low latency and high visual quality through path traced effects.
Palavras-chave:
perception-based rendering, neural 3d rendering, real-time raytracing, virtual reality
Referências
[n. d.]. AR & VR - Worldwide | Statista Market Forecast. [link] [link] (accessed Jun. 02, 2023).
[n. d.]. "Getting started with ray tracing". Unity Technologies. [link]. (accessed May 12, 2023).
[n. d.]. Unity Real-Time Development Platform | 3D, 2D, VR & AR Engine. Unity Technologies. [link] (accessed Dec. 1, 2022).
Rachel Albert, Anjul Patney, David Luebke, and Joohwan Kim. 2017. Latency requirements for foveated rendering in virtual reality. ACM Transactions on Applied Perception (TAP) 14, 4 (2017), 1–13. https://doi.org/10.1145/3127589
Colin Barré-Brisebois, Henrik Halén, Graham Wihlidal, Andrew Lauritzen, Jasper Bekkers, Tomasz Stachowiak, and Johan Andersson. 2019. Hybrid rendering for real-time ray tracing. In Ray Tracing Gems. Springer, 437–473. https://doi.org/10.1007/978-1-4842-4427-2
Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. 2022. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5470–5479. https://doi.org/10.1109/CVPR52688.2022.00539
Brian Caulfield. 2022. "What Is Path Tracing?". [link]. NVIDIA Blog. [link]. (accessed Dec. 26, 2022.).
Rama Chellappa and Sergios Theodoridis. 2017. Academic Press Library in Signal Processing, Volume 6: Image and Video Processing and Analysis and Computer Vision. Elsevier Science & Technology, Saint Louis. https://doi.org/10.1016/C2016-0-00726-X
Nianchen Deng, Zhenyi He, Jiannan Ye, Budmonde Duinkharjav, Praneeth Chakravarthula, Xubo Yang, and Qi Sun. 2022. Fov-nerf: Foveated neural radiance fields for virtual reality. IEEE Transactions on Visualization and Computer Graphics 28, 11 (2022), 3854–3864. https://doi.org/10.1109/TVCG.2022.3203102
Sara Fridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. 2022. Plenoxels: Radiance fields without neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5501–5510. https://doi.org/10.1109/CVPR52688.2022.00542
Brian Guenter, Mark Finch, Steven Drucker, Desney Tan, and John Snyder. 2012. Foveated 3D graphics. ACM Transactions on Graphics (TOG) 31, 6 (2012), 1–10. https://doi.org/10.1145/2366145.2366183
Susmija Jabbireddy, Xuetong Sun, Xiaoxu Meng, and Amitabh Varshney. 2022. Foveated Rendering: Motivation, Taxonomy, and Research Directions. arXiv e-prints (2022), 1–16. https://doi.org/10.48550/arXiv.2205.04529
James T Kajiya. 1986. The rendering equation. In Proceedings of the 13th annual conference on Computer graphics and interactive techniques. 143–150. https://doi.org/10.1145/15922.15902
Emmett Kilgariff, Henry Moreton, Nick Stam, and Brandon Bell. 2018. NVIDIA Turing Architecture In-Depth | NVIDIA Technical Blog. [link]. [link]. (accessed Dec. 19, 2022).
Jihwan Kim. 2022. Individualized Foveated Rendering. (2022), 60.
Jihwan Kim. 2022. Individualized Foveated Rendering. Master’s thesis. Hanyang University.
Matias Koskela, Atro Lotvonen, Markku Mäkitalo, Petrus Kivi, Timo Viitanen, and Pekka Jääskeläinen. 2019. Foveated Real-Time Path Tracing in Visual-Polar Space. In Eurographics Symposium on Rendering - DL-only and Industry Track, Tamy Boubekeur and Pradeep Sen (Eds.). The Eurographics Association, 1–12. https://doi.org/10.2312/sr.20191219
Marc Levoy and Ross Whitaker. 1990. Gaze-directed volume rendering. In Proceedings of the 1990 symposium on interactive 3d graphics. 217–223. https://doi.org/10.1145/91385.91449
Yiming Li, Zhiding Yu, Christopher Choy, Chaowei Xiao, Jose M Alvarez, Sanja Fidler, Chen Feng, and Anima Anandkumar. 2023. Voxformer: Sparse voxel transformer for camera-based 3d semantic scene completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 9087–9098.
Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. 2020. Neural Sparse Voxel Fields. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (Eds.). Vol. 33. Curran Associates, Inc., 15651–15663. [link].
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. 2021. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. Commun. ACM 65, 1 (dec 2021), 99–106. https://doi.org/10.1145/3503250
Bipul Mohanto, ABM Tariqul Islam, Enrico Gobbetti, and Oliver Staadt. 2022. An integrative view of foveated rendering. Computers & Graphics 102 (2022), 474–501. https://doi.org/10.1016/j.cag.2021.10.010
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. 2022. Instant Neural Graphics Primitives with a Multiresolution Hash Encoding. ACM Trans. Graph. 41, 4, Article 102 (July 2022), 15 pages. https://doi.org/10.1145/3528223.3530127
Thomas Müller, Fabrice Rousselle, Jan Novák, and Alexander Keller. 2021. Real-time neural radiance caching for path tracing. ACM Transactions on Graphics 40, 4 (Aug. 2021), 1–16. https://doi.org/10.1145/3450626.3459812
Thiago Porcino, Daniela Trevisan, and Esteban Clua. 2020. Minimizing cybersickness in head-mounted display systems: causes and strategies review. In 2020 22nd Symposium on Virtual and Augmented Reality (SVR). IEEE, 154–163. https://doi.org/10.1109/SVR51698.2020.00035
Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. 2021. Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. arXiv e-prints (2021), 1–11. https://doi.org/10.48550/arXiv.2103.13744
Katja Schwarz, Axel Sauer, Michael Niemeyer, Yiyi Liao, and Andreas Geiger. 2022. Voxgraf: Fast 3d-aware image synthesis with sparse voxel grids. arXiv preprint arXiv:2206.07695 (2022), 1–22. https://doi.org/10.48550/arXiv.2206.07695
BT Series. 2012. Methodology for the subjective assessment of the quality of television pictures. Recommendation ITU-R BT 500 (2012), 1–46.
Cheng Sun, Min Sun, and Hwann-Tzong Chen. 2022. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5459–5469. https://doi.org/10.1109/CVPR52688.2022.00538
Nicholas T. Swafford, José A. Iglesias-Guitian, Charalampos Koniaris, Bochang Moon, Darren Cosker, and Kenny Mitchell. 2016. User, metric, and computational evaluation of foveated rendering methods. In Proceedings of the ACM Symposium on Applied Perception. ACM, Anaheim California, 7–14. https://doi.org/10.1145/2931002.2931011
Lili Wang, Xuehuai Shi, and Yi Liu. 2023. Foveated rendering: A state-of-the-art survey. Computational Visual Media 9, 2 (2023), 195–228. https://doi.org/10.1007/s41095-022-0306-4
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing 13, 4 (2004), 600–612. https://doi.org/10.1109/TIP.2003.819861
Martin Weier, Thorsten Roth, Ernst Kruijff, André Hinkenjann, Arsène Pérard-Gayot, Philipp Slusallek, and Yongmin Li. 2016. Foveated Real-Time Ray Tracing for Head-Mounted Displays. Computer Graphics Forum, 289–298. https://doi.org/10.1111/cgf.13026
Turner Whitted. 1979. An Improved Illumination Model for Shaded Display. In Proceedings of the 6th Annual Conference on Computer Graphics and Interactive Techniques (Chicago, Illinois, USA) (SIGGRAPH ’79). Association for Computing Machinery, New York, NY, USA, 14. https://doi.org/10.1145/800249.807419
Turner Whitted. 1980. An Improved Illumination Model for Shaded Display. Commun. ACM 23, 6 (jun 1980), 343–349. https://doi.org/10.1145/358876.358882
Jiannan Ye. 2022. Rectangular-Mapping-based-Foveated-Rendering. [link]. original-date: 2022-02-11T07:56:42Z.
Jiannan Ye, Anqi Xie, Susmija Jabbireddy, Yunchuan Li, Xubo Yang, and Xiaoxu Meng. 2022. Rectangular Mapping-based Foveated Rendering. In 2022 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, 756–764. https://doi.org/10.1109/VR51125.2022.00097
[n. d.]. "Getting started with ray tracing". Unity Technologies. [link]. (accessed May 12, 2023).
[n. d.]. Unity Real-Time Development Platform | 3D, 2D, VR & AR Engine. Unity Technologies. [link] (accessed Dec. 1, 2022).
Rachel Albert, Anjul Patney, David Luebke, and Joohwan Kim. 2017. Latency requirements for foveated rendering in virtual reality. ACM Transactions on Applied Perception (TAP) 14, 4 (2017), 1–13. https://doi.org/10.1145/3127589
Colin Barré-Brisebois, Henrik Halén, Graham Wihlidal, Andrew Lauritzen, Jasper Bekkers, Tomasz Stachowiak, and Johan Andersson. 2019. Hybrid rendering for real-time ray tracing. In Ray Tracing Gems. Springer, 437–473. https://doi.org/10.1007/978-1-4842-4427-2
Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. 2022. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5470–5479. https://doi.org/10.1109/CVPR52688.2022.00539
Brian Caulfield. 2022. "What Is Path Tracing?". [link]. NVIDIA Blog. [link]. (accessed Dec. 26, 2022.).
Rama Chellappa and Sergios Theodoridis. 2017. Academic Press Library in Signal Processing, Volume 6: Image and Video Processing and Analysis and Computer Vision. Elsevier Science & Technology, Saint Louis. https://doi.org/10.1016/C2016-0-00726-X
Nianchen Deng, Zhenyi He, Jiannan Ye, Budmonde Duinkharjav, Praneeth Chakravarthula, Xubo Yang, and Qi Sun. 2022. Fov-nerf: Foveated neural radiance fields for virtual reality. IEEE Transactions on Visualization and Computer Graphics 28, 11 (2022), 3854–3864. https://doi.org/10.1109/TVCG.2022.3203102
Sara Fridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. 2022. Plenoxels: Radiance fields without neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5501–5510. https://doi.org/10.1109/CVPR52688.2022.00542
Brian Guenter, Mark Finch, Steven Drucker, Desney Tan, and John Snyder. 2012. Foveated 3D graphics. ACM Transactions on Graphics (TOG) 31, 6 (2012), 1–10. https://doi.org/10.1145/2366145.2366183
Susmija Jabbireddy, Xuetong Sun, Xiaoxu Meng, and Amitabh Varshney. 2022. Foveated Rendering: Motivation, Taxonomy, and Research Directions. arXiv e-prints (2022), 1–16. https://doi.org/10.48550/arXiv.2205.04529
James T Kajiya. 1986. The rendering equation. In Proceedings of the 13th annual conference on Computer graphics and interactive techniques. 143–150. https://doi.org/10.1145/15922.15902
Emmett Kilgariff, Henry Moreton, Nick Stam, and Brandon Bell. 2018. NVIDIA Turing Architecture In-Depth | NVIDIA Technical Blog. [link]. [link]. (accessed Dec. 19, 2022).
Jihwan Kim. 2022. Individualized Foveated Rendering. (2022), 60.
Jihwan Kim. 2022. Individualized Foveated Rendering. Master’s thesis. Hanyang University.
Matias Koskela, Atro Lotvonen, Markku Mäkitalo, Petrus Kivi, Timo Viitanen, and Pekka Jääskeläinen. 2019. Foveated Real-Time Path Tracing in Visual-Polar Space. In Eurographics Symposium on Rendering - DL-only and Industry Track, Tamy Boubekeur and Pradeep Sen (Eds.). The Eurographics Association, 1–12. https://doi.org/10.2312/sr.20191219
Marc Levoy and Ross Whitaker. 1990. Gaze-directed volume rendering. In Proceedings of the 1990 symposium on interactive 3d graphics. 217–223. https://doi.org/10.1145/91385.91449
Yiming Li, Zhiding Yu, Christopher Choy, Chaowei Xiao, Jose M Alvarez, Sanja Fidler, Chen Feng, and Anima Anandkumar. 2023. Voxformer: Sparse voxel transformer for camera-based 3d semantic scene completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 9087–9098.
Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. 2020. Neural Sparse Voxel Fields. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (Eds.). Vol. 33. Curran Associates, Inc., 15651–15663. [link].
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. 2021. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. Commun. ACM 65, 1 (dec 2021), 99–106. https://doi.org/10.1145/3503250
Bipul Mohanto, ABM Tariqul Islam, Enrico Gobbetti, and Oliver Staadt. 2022. An integrative view of foveated rendering. Computers & Graphics 102 (2022), 474–501. https://doi.org/10.1016/j.cag.2021.10.010
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. 2022. Instant Neural Graphics Primitives with a Multiresolution Hash Encoding. ACM Trans. Graph. 41, 4, Article 102 (July 2022), 15 pages. https://doi.org/10.1145/3528223.3530127
Thomas Müller, Fabrice Rousselle, Jan Novák, and Alexander Keller. 2021. Real-time neural radiance caching for path tracing. ACM Transactions on Graphics 40, 4 (Aug. 2021), 1–16. https://doi.org/10.1145/3450626.3459812
Thiago Porcino, Daniela Trevisan, and Esteban Clua. 2020. Minimizing cybersickness in head-mounted display systems: causes and strategies review. In 2020 22nd Symposium on Virtual and Augmented Reality (SVR). IEEE, 154–163. https://doi.org/10.1109/SVR51698.2020.00035
Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. 2021. Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. arXiv e-prints (2021), 1–11. https://doi.org/10.48550/arXiv.2103.13744
Katja Schwarz, Axel Sauer, Michael Niemeyer, Yiyi Liao, and Andreas Geiger. 2022. Voxgraf: Fast 3d-aware image synthesis with sparse voxel grids. arXiv preprint arXiv:2206.07695 (2022), 1–22. https://doi.org/10.48550/arXiv.2206.07695
BT Series. 2012. Methodology for the subjective assessment of the quality of television pictures. Recommendation ITU-R BT 500 (2012), 1–46.
Cheng Sun, Min Sun, and Hwann-Tzong Chen. 2022. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5459–5469. https://doi.org/10.1109/CVPR52688.2022.00538
Nicholas T. Swafford, José A. Iglesias-Guitian, Charalampos Koniaris, Bochang Moon, Darren Cosker, and Kenny Mitchell. 2016. User, metric, and computational evaluation of foveated rendering methods. In Proceedings of the ACM Symposium on Applied Perception. ACM, Anaheim California, 7–14. https://doi.org/10.1145/2931002.2931011
Lili Wang, Xuehuai Shi, and Yi Liu. 2023. Foveated rendering: A state-of-the-art survey. Computational Visual Media 9, 2 (2023), 195–228. https://doi.org/10.1007/s41095-022-0306-4
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing 13, 4 (2004), 600–612. https://doi.org/10.1109/TIP.2003.819861
Martin Weier, Thorsten Roth, Ernst Kruijff, André Hinkenjann, Arsène Pérard-Gayot, Philipp Slusallek, and Yongmin Li. 2016. Foveated Real-Time Ray Tracing for Head-Mounted Displays. Computer Graphics Forum, 289–298. https://doi.org/10.1111/cgf.13026
Turner Whitted. 1979. An Improved Illumination Model for Shaded Display. In Proceedings of the 6th Annual Conference on Computer Graphics and Interactive Techniques (Chicago, Illinois, USA) (SIGGRAPH ’79). Association for Computing Machinery, New York, NY, USA, 14. https://doi.org/10.1145/800249.807419
Turner Whitted. 1980. An Improved Illumination Model for Shaded Display. Commun. ACM 23, 6 (jun 1980), 343–349. https://doi.org/10.1145/358876.358882
Jiannan Ye. 2022. Rectangular-Mapping-based-Foveated-Rendering. [link]. original-date: 2022-02-11T07:56:42Z.
Jiannan Ye, Anqi Xie, Susmija Jabbireddy, Yunchuan Li, Xubo Yang, and Xiaoxu Meng. 2022. Rectangular Mapping-based Foveated Rendering. In 2022 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, 756–764. https://doi.org/10.1109/VR51125.2022.00097
Publicado
06/11/2023
Como Citar
HENRIQUES, Horácio; OLIVEIRA, Eder; CLUA, Esteban; TREVISAN, Daniela.
A mixed path tracing and NeRF approach for optimizing rendering in XR Displays. In: SIMPÓSIO DE REALIDADE VIRTUAL E AUMENTADA (SVR), 25. , 2023, Rio Grande/RS.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2023
.
p. 123–130.
