Directional Light Vector Estimation From a Virtual AR Object and its 2D Shadow Mask
Resumo
This work presents a novel method for inferring the main directional light source in a 3D scene, given only 2D inputs, namely a camera image and a rough shadow mask. A two stage algorithm is proposed, in which the first stage handles the inputs and makes an initial estimation for the light source. The second stage refines this first estimate to find a new vector assumed to be closer to the real directional light vector. Both stages are performed by using one virtual object rendered into the real scene. In the first stage, an external shadow estimator produces a coarse shadow for the virtual object, enabling the computation of an initial directional light vector. Also, it is possible to compare the coarse shadow with a virtual shadow computed from the object. In the second stage, we seek to maximize the intersection over union (IoU) between both shadows. We assume that the best directional light vector provides the best shadow matching. The experiments are made both in virtual and real environments, in scenes with different levels of control and known data. Results show that our method is capable of finding the 3D light vector from the 2D scene, enhancing the initial rough shadow input.
Palavras-chave:
Directional Light estimation, Shadow Estimation, 3D lighting, Augmented Reality
Referências
Y. Chen, Q. Wang, H. Chen, X. Song, H. Tang, and M. Tian. An overview of augmented reality technology. Journal of Physics Conference Series, 1237:022082, 2019.
R. Azuma. A survey of augmented reality. Presence Teleoperators and Virtual Environments, 6:355–385, 1997.
J. Xiong, E. Hsiang, Z. He, T. Zhan, and S. Wu. Augmented reality and virtual reality displays emerging technologies and future perspectives. Light Science and Applications, 10:216, 2021.
P. Kan and H. Kafumann. Deeplight light source estimation for augmented reality using deep learning. The Visual Computer, 35:873–883, 2019.
J. Frahm, K. Koeser, D. Grest, and R. Koch. Markerless augmented reality with light source estimation for direct illumination. Conference on Visual Media Production CVMP, London, 211–220, 2005.
T. Whelan, R. Salas-Moreno, B. Glocker, A. Davison, and S. Leutenegger. ElasticFusion Real-time dense SLAM and light source estimation. The International Journal of Robotics Research, 35:1697–1716, 2016.
M. Meilland, C. Barat, and A. Comport. 3D high dynamic range dense visual SLAM and its application to real-time object relighting. 2013 IEEE International Symposium on Mixed and Augmented Reality ISMAR, 143–152, 2013.
M. Fiala. ARTag a fiducial marker system using digital techniques. 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition CVPR05, 2:590–596, 2005.
D. Liu, C. Long, H. Zhang, H. Yu, X. Dong, and C. Xiao. ARShadowGAN Shadow generative adversarial network for augmented reality in single light scenes. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 8139–8148, 2020.
S. Khan, M. Bennamoun, F. Sohel, and R. Togneri. Automatic feature learning for robust shadow detection. 2014 IEEE Conference on Computer Vision and Pattern Recognition, 1939–1946, 2014.
S. Khan, M. Bennamoun, F. Sohel, and R. Togneri. Automatic shadow detection and removal from a single image. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38:431–446, 2015.
L. Hou, T. Vicente, M. Hoai, and D. Samaras. Large scale shadow annotation and detection using lazy annotation and stacked CNNs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43:1337–1351, 2019.
E. Kurbatova and V. Lyalina. Shadow detection on color images. Journal of Physics Conference Series, 1368:032018, 2019.
N. Otsu. A threshold selection method from gray-level histograms. IEEE Transactions on Systems Man and Cybernetics, 9:62–66, 1979.
S. Suzuki and T. Topological structural analysis of digitized binary images by border following. Computer Vision Graphics and Image Processing, 30:32–46, 1985.
O. Rodrigues. Des lois geometriques qui regissent les deplacements d un systeme solide dans l espace et de la variation des coordonnees provenant de ces deplacements consideres independamment des causes qui peuvent les produire. Journal de Mathematiques Pures et Appliquees, 5:380–440, 1840.
C. Urena and I. Georgiev. Stratified sampling of projected spherical caps. Computer Graphics Forum, 37:13–20, 2018.
J. Stauder. Point light source estimation from two images and its limits. International Journal of Computer Vision, 36:195–220, 2000.
B. Marques, R. Drumond, C. Vasconcelos, and E. Clua. Deep Light Source Estimation for Mixed Reality. Visigrapp 1 grapp, 303–311, 2018.
D. Liu and S. Wu. Light direction estimation and hand touchable interaction for augmented reality. Virtual Reality, 26:1155–1172, 2022.
C. LeGendre, W. Ma, G. Fyffe, J. Flynn, L. Charbonnel, J. Busch, and P. Debevec. Deeplight Learning illumination for unconstrained mobile mixed reality. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 5918–5928, 2019.
Y. Sun, D. Li, S. Liu, T. Cao, and Y. Hu. Learning illumination from a limited field-of-view image. 2020 IEEE International Conference on Multimedia and Expo Workshops ICMEW, 1–6, 2020.
Z. Wang, W. Chen, D. Acuna, J. Kautz, and S. Fidler. Neural light field estimation for street scenes with differentiable virtual object insertion. European Conference on Computer Vision, 380–397, 2022.
C. Liu, L. Wang, Z. Li, S. Quan, and Y. Xu. Real-time lighting estimation for augmented reality via differentiable screen-space rendering. IEEE Transactions on Visualization and Computer Graphics, 29:2132–2145, 2022.
B. Marques, E. Clua, A. Montenegro, and C. Vasconcelos. Spatially and color consistent environment lighting estimation using deep neural networks for mixed reality. Computers and Graphics, 102:257–268, 2022.
X. Cao and H. Foroosh. Camera calibration and light source orientation from solar shadows. Computer Vision and Image Understanding, 105:60–72, 2007.
E. Koc and S. Balcisoy. Estimation of environmental lighting from known geometries for mobile augmented reality. 2013 International Conference on Cyberworlds, 132–139, 2013.
Y. Wang and D. Samaras. Estimation of multiple directional light sources for synthesis of mixed reality images. 10th Pacific Conference on Computer Graphics and Applications, 38–47, 2002.
B. Boom, S. Orts-Escolano, X. Ning, S. McDonagh, P. Sandilands, and R. Fisher. Interactive light source position estimation for augmented reality with an RGB-D camera. Computer Animation and Virtual Worlds, 28:e1686, 2017.
R. Nguyen and M. Le. Light source estimation from a single image. 2012 12th International Conference on Control Automation Robotics and Vision ICARCV, 1358–1363, 2012.
J. Lopez-Moreno, E. Garces, S. Hadap, E. Reinhard, and D. Gutierrez. Multiple light source estimation in a single image. Computer Graphics Forum, 32:170–182, 2013.
N. Chotikakamthorn. Near point light source location estimation from shadow edge correspondence. 2015 IEEE 7th International Conference on Cybernetics and Intelligent Systems CIS and IEEE Conference on Robotics Automation and Mechatronics RAM, 30–35, 2015.
L. Gruber, T. Richter-Trummer, and D. Schmalstieg. Real-time photometric registration from arbitrary geometry. 2012 IEEE International Symposium on Mixed and Augmented Reality ISMAR, 119–128, 2012.
Y. Yu and W. Smith. Inverserendernet Learning single image inverse rendering. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3155–3164, 2019.
F. Crow. Summed-area tables for texture mapping. Proceedings of the 11th Annual Conference on Computer Graphics and Interactive Techniques, 207–212, 1984.
A. Colaitis, R. Follett, J. Palastro, I. Igumenschev, and V. Goncharov. Adaptive inverse ray-tracing for accurate and efficient modeling of cross beam energy transfer in hydrodynamics simulations. Physics of Plasmas, 26, 2019.
G. Patow and X. Pueyo. A survey of inverse rendering problems. Computer Graphics Forum, 22:663–687, 2003.
H. Kato, D. Beker, M. Morariu, T. Ando, T. Matsuoka, W. Kehl, and A. Gaidon. Differentiable rendering a survey. arXiv preprint arXiv 2006.12057, 2020.
D. Azinovic, T. Li, A. Kaplanyan, and M. Niessner. Inverse path tracing for joint material and lighting estimation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2447–2456, 2019.
R. Azuma. A survey of augmented reality. Presence Teleoperators and Virtual Environments, 6:355–385, 1997.
J. Xiong, E. Hsiang, Z. He, T. Zhan, and S. Wu. Augmented reality and virtual reality displays emerging technologies and future perspectives. Light Science and Applications, 10:216, 2021.
P. Kan and H. Kafumann. Deeplight light source estimation for augmented reality using deep learning. The Visual Computer, 35:873–883, 2019.
J. Frahm, K. Koeser, D. Grest, and R. Koch. Markerless augmented reality with light source estimation for direct illumination. Conference on Visual Media Production CVMP, London, 211–220, 2005.
T. Whelan, R. Salas-Moreno, B. Glocker, A. Davison, and S. Leutenegger. ElasticFusion Real-time dense SLAM and light source estimation. The International Journal of Robotics Research, 35:1697–1716, 2016.
M. Meilland, C. Barat, and A. Comport. 3D high dynamic range dense visual SLAM and its application to real-time object relighting. 2013 IEEE International Symposium on Mixed and Augmented Reality ISMAR, 143–152, 2013.
M. Fiala. ARTag a fiducial marker system using digital techniques. 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition CVPR05, 2:590–596, 2005.
D. Liu, C. Long, H. Zhang, H. Yu, X. Dong, and C. Xiao. ARShadowGAN Shadow generative adversarial network for augmented reality in single light scenes. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 8139–8148, 2020.
S. Khan, M. Bennamoun, F. Sohel, and R. Togneri. Automatic feature learning for robust shadow detection. 2014 IEEE Conference on Computer Vision and Pattern Recognition, 1939–1946, 2014.
S. Khan, M. Bennamoun, F. Sohel, and R. Togneri. Automatic shadow detection and removal from a single image. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38:431–446, 2015.
L. Hou, T. Vicente, M. Hoai, and D. Samaras. Large scale shadow annotation and detection using lazy annotation and stacked CNNs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43:1337–1351, 2019.
E. Kurbatova and V. Lyalina. Shadow detection on color images. Journal of Physics Conference Series, 1368:032018, 2019.
N. Otsu. A threshold selection method from gray-level histograms. IEEE Transactions on Systems Man and Cybernetics, 9:62–66, 1979.
S. Suzuki and T. Topological structural analysis of digitized binary images by border following. Computer Vision Graphics and Image Processing, 30:32–46, 1985.
O. Rodrigues. Des lois geometriques qui regissent les deplacements d un systeme solide dans l espace et de la variation des coordonnees provenant de ces deplacements consideres independamment des causes qui peuvent les produire. Journal de Mathematiques Pures et Appliquees, 5:380–440, 1840.
C. Urena and I. Georgiev. Stratified sampling of projected spherical caps. Computer Graphics Forum, 37:13–20, 2018.
J. Stauder. Point light source estimation from two images and its limits. International Journal of Computer Vision, 36:195–220, 2000.
B. Marques, R. Drumond, C. Vasconcelos, and E. Clua. Deep Light Source Estimation for Mixed Reality. Visigrapp 1 grapp, 303–311, 2018.
D. Liu and S. Wu. Light direction estimation and hand touchable interaction for augmented reality. Virtual Reality, 26:1155–1172, 2022.
C. LeGendre, W. Ma, G. Fyffe, J. Flynn, L. Charbonnel, J. Busch, and P. Debevec. Deeplight Learning illumination for unconstrained mobile mixed reality. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 5918–5928, 2019.
Y. Sun, D. Li, S. Liu, T. Cao, and Y. Hu. Learning illumination from a limited field-of-view image. 2020 IEEE International Conference on Multimedia and Expo Workshops ICMEW, 1–6, 2020.
Z. Wang, W. Chen, D. Acuna, J. Kautz, and S. Fidler. Neural light field estimation for street scenes with differentiable virtual object insertion. European Conference on Computer Vision, 380–397, 2022.
C. Liu, L. Wang, Z. Li, S. Quan, and Y. Xu. Real-time lighting estimation for augmented reality via differentiable screen-space rendering. IEEE Transactions on Visualization and Computer Graphics, 29:2132–2145, 2022.
B. Marques, E. Clua, A. Montenegro, and C. Vasconcelos. Spatially and color consistent environment lighting estimation using deep neural networks for mixed reality. Computers and Graphics, 102:257–268, 2022.
X. Cao and H. Foroosh. Camera calibration and light source orientation from solar shadows. Computer Vision and Image Understanding, 105:60–72, 2007.
E. Koc and S. Balcisoy. Estimation of environmental lighting from known geometries for mobile augmented reality. 2013 International Conference on Cyberworlds, 132–139, 2013.
Y. Wang and D. Samaras. Estimation of multiple directional light sources for synthesis of mixed reality images. 10th Pacific Conference on Computer Graphics and Applications, 38–47, 2002.
B. Boom, S. Orts-Escolano, X. Ning, S. McDonagh, P. Sandilands, and R. Fisher. Interactive light source position estimation for augmented reality with an RGB-D camera. Computer Animation and Virtual Worlds, 28:e1686, 2017.
R. Nguyen and M. Le. Light source estimation from a single image. 2012 12th International Conference on Control Automation Robotics and Vision ICARCV, 1358–1363, 2012.
J. Lopez-Moreno, E. Garces, S. Hadap, E. Reinhard, and D. Gutierrez. Multiple light source estimation in a single image. Computer Graphics Forum, 32:170–182, 2013.
N. Chotikakamthorn. Near point light source location estimation from shadow edge correspondence. 2015 IEEE 7th International Conference on Cybernetics and Intelligent Systems CIS and IEEE Conference on Robotics Automation and Mechatronics RAM, 30–35, 2015.
L. Gruber, T. Richter-Trummer, and D. Schmalstieg. Real-time photometric registration from arbitrary geometry. 2012 IEEE International Symposium on Mixed and Augmented Reality ISMAR, 119–128, 2012.
Y. Yu and W. Smith. Inverserendernet Learning single image inverse rendering. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3155–3164, 2019.
F. Crow. Summed-area tables for texture mapping. Proceedings of the 11th Annual Conference on Computer Graphics and Interactive Techniques, 207–212, 1984.
A. Colaitis, R. Follett, J. Palastro, I. Igumenschev, and V. Goncharov. Adaptive inverse ray-tracing for accurate and efficient modeling of cross beam energy transfer in hydrodynamics simulations. Physics of Plasmas, 26, 2019.
G. Patow and X. Pueyo. A survey of inverse rendering problems. Computer Graphics Forum, 22:663–687, 2003.
H. Kato, D. Beker, M. Morariu, T. Ando, T. Matsuoka, W. Kehl, and A. Gaidon. Differentiable rendering a survey. arXiv preprint arXiv 2006.12057, 2020.
D. Azinovic, T. Li, A. Kaplanyan, and M. Niessner. Inverse path tracing for joint material and lighting estimation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2447–2456, 2019.
Publicado
30/09/2025
Como Citar
YACOVENCO, Aleksander; VIEIRA, Marcelo Bernardes; MACIEL, Luiz Maurílio Da Silva; DA SILVA, Rodrigo Luis De Souza.
Directional Light Vector Estimation From a Virtual AR Object and its 2D Shadow Mask. In: SIMPÓSIO DE REALIDADE VIRTUAL E AUMENTADA (SVR), 27. , 2025, Salvador/BA.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2025
.
p. 179-188.
