A Study on Gaussian Splatting and Neural Fields

• Daniel Perazzo IMPA
• Tiago Novello IMPA
• Joao Paulo Lima IMPA
• Luiz Velho IMPA

Resumo

3D reconstruction from posed images is undergoing a fundamental transformation, driven by continuous advances in Neural Radiance Fields (NeRFs) and 3D Gaussian Splatting (3DGS), where scenes are volumetrically represented and rendered using volume rendering techniques. However, extracting detailed surfaces from these fuzzy, density-based representations remains a challenge due to the lack of effective surface regularization. To address this, recent work has explored alternative representations such as 2D Gaussian Splatting (2DGS), which enables stronger regularization and has shown promising results for surface extraction. Although 2DGS is a powerful representation, it suffers from certain drawbacks, such as high memory consumption. Motivated by these limitations, this work presents a study of two increasingly popular methodologies: Implicit Neural Representations (INRs) and 2DGS. First, we demonstrate that augmenting 2DGS with depth information can significantly improve geometric accuracy—by up to 10% compared to models without depth. Importantly, depth is relatively inexpensive to obtain, as it can be predicted using state-of-the-art depth estimation methods. Next, we introduce an application that extracts an INR from 2D Gaussian representations. However, previous efforts have focused primarily on isolated objects of interest. In practice, scenes are often more complex and contain both foreground and background elements. To tackle this, we propose a method to segment foreground and background Gaussians. Leveraging this segmentation, we further demonstrate the extraction of a 360° panorama of the scene.

Palavras-chave: Gaussian Splatting, 3D Reconstruction, Machine Learning

Referências

Z. Ma and S. Liu, “A review of 3d reconstruction techniques in civil engineering and their applications,” Advanced Engineering Informatics, vol. 37, pp. 163–174, 2018.

L. Gomes, O. R. P. Bellon, and L. Silva, “3d reconstruction methods for digital preservation of cultural heritage: A survey,” Pattern Recognition Letters, vol. 50, pp. 3–14, 2014.

K. Gao, Y. Gao, H. He, D. Lu, L. Xu, and J. Li, “Nerf: Neural radiance field in 3d vision, a comprehensive review,” arXiv preprint arXiv:2210.00379, 2022.

M. Faraday, “Liv. thoughts on ray-vibrations,” The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, vol. 28, no. 188, pp. 345–350, 1846.

A. Gershun, “The light field,” Journal of Mathematics and Physics, vol. 18, no. 1-4, pp. 51–151, 1939.

M. Levoy and P. Hanrahan, “Light field rendering,” in Seminal Graphics Papers: Pushing the Boundaries, Volume 2, 2023, pp. 441–452.

S. J. Gortler, R. Grzeszczuk, R. Szeliski, and M. F. Cohen, “The lumigraph,” in Seminal Graphics Papers: Pushing the Boundaries, Volume 2, 2023, pp. 453–464.

J. J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove, “Deepsdf: Learning continuous signed distance functions for shape representation,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 165–174.

L. Schirmer, T. Novello, V. da Silva, G. Schardong, D. Perazzo, H. Lopes, N. Gonçalves, and L. Velho, “Geometric implicit neural representations for signed distance functions,” Computers & Graphics, vol. 125, p. 104085, 2024.

B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis, “3d gaussian splatting for real-time radiance field rendering.” ACM Trans. Graph., vol. 42, no. 4, pp. 139–1, 2023.

B. Huang, Z. Yu, A. Chen, A. Geiger, and S. Gao, “2d gaussian splatting for geometrically accurate radiance fields,” in ACM SIGGRAPH 2024 Conference Papers, 2024, pp. 1–11.

T. Novello, V. da Silva, G. Schardong, L. Schirmer, H. Lopes, and L. Velho, “Neural implicit surface evolution,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 14 279–14 289.

L. Yang, B. Kang, Z. Huang, X. Xu, J. Feng, and H. Zhao, “Depth anything: Unleashing the power of large-scale unlabeled data,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 10 371–10 381.

D. Azinović, R. Martin-Brualla, D. B. Goldman, M. Nießner, and J. Thies, “Neural rgb-d surface reconstruction,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 6290–6301.

R. Jensen, A. Dahl, G. Vogiatzis, E. Tola, and H. Aanæs, “Large scale multi-view stereopsis evaluation,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2014, pp. 406–413.

W. E. Lorensen and H. E. Cline, “Marching cubes: A high resolution 3d surface construction algorithm,” in Seminal graphics: pioneering efforts that shaped the field, 1998, pp. 347–353.

M. Turkulainen, X. Ren, I. Melekhov, O. Seiskari, E. Rahtu, and J. Kannala, “Dn-splatter: Depth and normal priors for gaussian splatting and meshing,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2025.

T. Novello, G. Schardong, L. Schirmer, V. da Silva, H. Lopes, and L. Velho, “Exploring differential geometry in neural implicits,” Computers & Graphics, vol. 108, pp. 49–60, 2022.

V. d. Silva, T. Novello, G. Schardong, L. Schirmer, H. Lopes, and L. Velho, “Neural implicit mapping via nested neighborhoods,” arXiv preprint arXiv:2201.09147, 2022.

J. L. Schönberger and J.-M. Frahm, “Structure-from-motion revisited,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2016.

D. Perazzo, J. P. Lima, L. Velho, and V. Teichrieb, “Directvoxgo++: Grid-based fast object reconstruction using radiance fields,” Computers & Graphics, vol. 114, pp. 96–104, 2023.

K. Zhang, G. Riegler, N. Snavely, and V. Koltun, “Nerf++: Analyzing and improving neural radiance fields,” arXiv preprint arXiv:2010.07492, 2020.

J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman, “Mip-nerf 360: Unbounded anti-aliased neural radiance fields,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 5470–5479.

T. L. da Silveira and C. R. Jung, “Omnidirectional visual computing: Foundations, challenges, and applications,” Computers & Graphics, vol. 113, pp. 89–101, 2023.

M. Zwicker, H. Pfister, J. Van Baar, and M. Gross, “Ewa splatting,” IEEE Transactions on Visualization and Computer Graphics, vol. 8, no. 3, pp. 223–238, 2002.

H. Paz, D. Perazzo, T. Novello, G. Schardong, L. Schirmer, V. da Silva, D. Yukimura, F. Chagas, H. Lopes, and L. Velho, “Mr-net: Multiresolution sinusoidal neural networks,” Computers & Graphics, 2023.

A. Kopiler, T. Novello, G. Schardong, L. Schirmer, D. Perazzo, and L. Velho, “Interact-net: An interactive interface for multimedia machine learning,” in 2024 37th SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI). IEEE, 2024, pp. 1–6.