Digital Video Stabilization: Algorithms and Evaluation
Resumo
Several devices have allowed the acquisition and editing of videos in various circumstances, such as digital cameras, smartphones and other mobile devices. However, the use of cameras under adverse conditions usually results in non-precise motion and occurrence of shaking, which may compromise the stability of the obtained videos. To overcome such problem, digital stabilization aims to correct camera motion oscillations that occur in the acquisition process, particularly when the cameras are mobile and handled in adverse conditions, through software techniques - without the use of specific hardware - to enhance visual quality either with the intention of enhancing human perception or improving final applications, such as detection and tracking of objects. This is important in order to avoid hardware cost and indispensable for videos already recorded. This work proposed three methods to perform digital video stabilization and two other techniques to evaluate video stabilization quality.
Referências
Z. Zhao and X. Ma, “Video Stabilization based on Local Trajectories and Robust Mesh Transformation,” in IEEE International Conference on Image Processing. IEEE, 2016, pp. 4092–4096. https://doi.org/10.1109/ICIP.2016.7533129
B.-Y. Chen, K.-Y. Lee, W.-T. Huang, and J.-S. Lin, “Capturing Intention-based Full-Frame Video Stabilization,” Computer Graphics Forum, vol. 27, no. 7, pp. 1805–1814, 2008. https://doi.org/10.1111/j.1467-8659.2008.01326.x
H. Guo, S. Liu, S. Zhu, and B. Zeng, “Joint Bundled Camera Paths for Stereoscopic Video Stabilization,” in IEEE International Conference on Image Processing. IEEE, 2016, pp. 1071–1075. https://doi.org/10.1109/ICIP.2016.7532522
S. Liu, B. Xu, C. Deng, S. Zhu, B. Zeng, and M. Gabbouj, “A Hybrid Approach for Near-Range Video Stabilization,” IEEE Transactions on Circuits and Systems for Video Technology, 2016. https://doi.org/10.1109/TCSVT.2016.2556587
B.-H. Chen, A. Kopylov, S.-C. Huang, O. Seredin, R. Karpov, S.-Y. Kuo, K. R. Lai, T.-H. Tan, M. Gochoo, and D. Bayanduuren, “Improved Global Motion Estimation via Motion Vector Clustering for Video Stabilization,” Engineering Applications of Artificial Intelligence, vol. 54, pp. 39–48, 2016. https://doi.org/10.1016/j.engappai.2016.05.004
B. Chen, J. Zhao, and Y. Wang, “Research on Evaluation Method of Video Stabilization,” in International Conference on Advanced Material Science and Environmental Engineering. Atlantis Press, 2016, pp. 253–258. https://dx.doi.org/10.2991/amsee-16.2016.67
J. Kopf, “360 Video Stabilization,” ACM Transactions on Graphics, vol. 35, no. 6, p. 195, 2016. https://doi.org/10.1145/2980179.2982405
C. Buehler, M. Bosse, and L. McMillan, “Non-Metric Image-based Rendering for Video Stabilization,” in IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2. IEEE, 2001. https://doi.org/10.1109/CVPR.2001.991019
Y. Matsushita, E. Ofek, W. Ge, X. Tang, and H.-Y. Shum, “Full-Frame Video Stabilization with Motion Inpainting,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 28, no. 7, pp. 1150–1163, 2006. https://doi.org/10.1109/TPAMI.2006.141
L. Marcenaro, G. Vernazza, and C. S. Regazzoni, “Image Stabilization Algorithms for Video-Surveillance Applications,” in International Conference on Image Processing, vol. 1. IEEE, 2001, pp. 349–352. https://doi.org/10.1109/ICIP.2001.959025
A. A. Amanatiadis and I. Andreadis, “Digital Image Stabilization by Independent Component Analysis,” IEEE Transactions on Instrumentation and Measurement, vol. 59, no. 7, pp. 1755–1763, 2010. https://doi.org/10.1109/TIM.2009.2028216
D. Shukla and R. K. Jha, “A Robust Video Stabilization Technique using Integral Frame Projection Warping,” Signal, Image and Video Processing, vol. 9, no. 6, pp. 1287–1297, 2015. https://doi.org/10.1007/s11760-013-0584-5
H. Qu, L. Song, and G. Xue, “Shaking Video Synthesis for Video Stabilization Performance Assessment,” in Visual Communications and Image Processing. IEEE, 2013, pp. 1–6. https://doi.org/10.1109/VCIP.2013.6706422
A. Hamza, R. Hafiz, M. M. Khan, Y. Cho, and J. Cha, “Stabilization of Panoramic Videos from Mobile Multi-camera Platforms,” Image and Vision Computing, vol. 37, pp. 20–30, 2015. https://doi.org/10.1016/j.imavis.2015.02.002
J. Bai, A. Agarwala, M. Agrawala, and R. Ramamoorthi, “User-Assisted Video Stabilization,” in Computer Graphics Forum, vol. 33, no. 4. Wiley Online Library, 2014, pp. 61–70. https://doi.org/10.1111/cgf.12413
M. Okade and P. K. Biswas, “Video Stabilization using Maximally Stable Extremal Region Features,” Multimedia Tools and Applications, vol. 68, no. 3, pp. 947–968, 2014. https://doi.org/10.1007/s11042-012-1095-z
S. W. Kim, S. Yin, K. Yun, and J. Y. Choi, “Spatio-temporal Weighting in Local Patches for Direct Estimation of Camera Motion in Video Stabilization,” Computer Vision and Image Understanding, vol. 118, pp. 71–83, 2014. https://doi.org/10.1016/j.cviu.2013.09.005
S. Liu, L. Yuan, P. Tan, and J. Sun, “Steadyflow: Spatially Smooth Optical Flow for Video Stabilization,” in IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 4209–4216. https://doi.org/10.1109/CVPR.2014.536
M. Grundmann, V. Kwatra, and I. Essa, “Auto-Directed Video Stabilization with Robust L1 Optimal Camera Paths,” in IEEE Conference on Computer Vision and Pattern Recognition. IEEE, Jun. 2011, pp. 225–232. https://doi.org/10.1109/CVPR.2011.5995525
J.-Y. Chang, W.-F. Hu, M.-H. Cheng, and B.-S. Chang, “Digital Image Translational and Rotational Motion Stabilization using Optical Flow Technique,” IEEE Transactions on Consumer Electronics, vol. 48, no. 1, pp. 108–115, 2002. https://doi.org/10.1109/TCE.2002.1010098
S. Ertürk, “Real-Time Digital Image Stabilization using Kalman Filters,” Real-Time Imaging, vol. 8, no. 4, pp. 317–328, 2002. https://doi.org/10.1006/rtim.2001.0278
R. Jia, H. Zhang, L. Wang, and J. Li, “Digital Image Stabilization based on Phase Correlation,” in International Conference on Artificial Intelligence and Computational Intelligence, vol. 3. IEEE, 2009, pp. 485–489. https://doi.org/10.1109/AICI.2009.489
S.-J. Ko, S.-H. Lee, and K.-H. Lee, “Digital Image Stabilizing Algorithms based on Bit-Plane Matching,” IEEE Transactions on Consumer Electronics, vol. 44, no. 3, pp. 617–622, 1998. https://doi.org/10.1109/30.713172
S. Kumar, H. Azartash, M. Biswas, and T. Nguyen, “Real-Time Affine Global Motion Estimation using Phase Correlation and its Application for Digital Image Stabilization,” IEEE Transactions on Image Processing, vol. 20, no. 12, pp. 3406–3418, 2011. https://doi.org/10.1109/TIP.2011.2156420
C.-T. Lin, C.-T. Hong, and C.-T. Yang, “Real-Time Digital Image Stabilization System using Modified Proportional Integrated Controller,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 19, no. 3, pp. 427–431, 2009. https://doi.org/10.1109/TCSVT.2009.2013508
C. Morimoto and R. Chellappa, “Fast Electronic Digital Image Stabilization,” in 13th International Conference on Pattern Recognition, vol. 3. IEEE, 1996, pp. 284–288. https://doi.org/10.1109/ICPR.1996.546956
Y. G. Ryu and M. J. Chung, “Robust Online Digital Image Stabilization based on Point-Feature Trajectory without Accumulative Global Motion Estimation,” IEEE Signal Processing Letters, vol. 19, no. 4, pp. 223–226, 2012. https://doi.org/10.1109/LSP.2012.2188286
J. Yang, D. Schonfeld, C. Chen, and M. Mohamed, “Online Video Stabilization based on Particle Filters,” in International Conference on Image Processing. IEEE, 2006, pp. 1545–1548. https://doi.org/10.1109/ICIP.2006.312645
Q. Zheng and M. Yang, “A Video Stabilization Method Based on Interframe Image Matching Score,” Global Journal of Computer Science and Technology, vol. 17, no. 1-F, 2017.
N. Bhowmik, V. Gouet-Brunet, L. Wei, and G. Bloch, “Adaptive and Optimal Combination of Local Features for Image Retrieval,” in International Conference on Multimedia Modeling. Springer, 2017,pp. 76–88. https://doi.org/10.1007/978-3-319-51814-5_7
C. Jia and B. L. Evans, “Online Motion Smoothing for Video Stabilization via Constrained Multiple-model Estimation,” EURASIP Journal on Image and Video Processing, vol. 2017, no. 1, p. 25, 2017. https://doi.org/10.1186/s13640-017-0171-8
S. Liu, M. Li, S. Zhu, and B. Zeng, “CodingFlow: Enable Video Coding for Video Stabilization,” IEEE Transactions on Image Processing, vol. 26, no. 7, pp. 3291–3302, 2017. https://doi.org/10.1109/TIP.2017.2697759
M. R. Souza, “Digital Video Stabilization: Algorithms and Evaluation,” Master’s thesis, Institute of Computing, University of Campinas, 2018.
R. Stephens, Essential Algorithms: A Practical Approach to Computer Algorithms. John Wiley & Sons, 2013.
Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image Quality Assessment: From Error Visibility to Structural Similarity,” IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600–612, 2004. https://doi.org/10.1109/TIP.2003.819861
M. J. Powell, “An Efficient Method for Finding the Minimum of a Function of Several Variables Without Calculating Derivatives,” The Computer Journal, vol. 7, no. 2, pp. 155–162, 1964. https://doi.org/10.1093/comjnl/7.2.155
K. Zuiderveld, “Contrast Limited Adaptive Histogram Equalization,” in Graphics Gems IV. Academic Press Professional, Inc., 1994, pp. 474–485.
S. Liu, L. Yuan, P. Tan, and J. Sun, “Bundled Camera Paths for Video Stabilization,” ACM Transactions on Graphics, vol. 32, no. 4, p. 78, 2013. https://doi.org/10.1145/2461912.2461995
M. R. Souza and H. Pedrini, “Combination of Local Feature Detection Methods for Digital Video Stabilization,” Signal, Image and Video Processing, vol. 12, no. 8, pp. 1513–1521, 2018. https://doi.org/10.1007/s11760-018-1307-8
M. R. Souza, L. F. R. da Fonseca, and H. Pedrini, “Improvement of Global Motion Estimation in Two-Dimensional Digital Video Stabilisation Methods,” IET Image Processing, vol. 12, no. 12, pp. 2204–2211, 2018. https://doi.org/10.1049/iet-ipr.2018.5445
M. R. Souza and H. Pedrini, “Digital Video Stabilization Based on Adaptive Camera Trajectory Smoothing,” EURASIP Journal on Image and Video Processing, vol. 2018, no. 1, p. 37, 2018. https://doi.org/10.1186/s13640-018-0277-7
M. R. Souza and H. Pedrini, “Motion Energy Image for Evaluation of Video Stabilization,” The Visual Computer, pp. 1–13, 2017. https://doi.org/10.1007/s00371-018-1572-0
M. R. Souza and H. Pedrini, “Visual Rhythms for Qualitative Evaluation of Video Stabilization,” EURASIP Journal on Image and Video Processing, 2018, (submitted).
B.-Y. Chen, K.-Y. Lee, W.-T. Huang, and J.-S. Lin, “Capturing Intention-based Full-Frame Video Stabilization,” Computer Graphics Forum, vol. 27, no. 7, pp. 1805–1814, 2008. https://doi.org/10.1111/j.1467-8659.2008.01326.x
H. Guo, S. Liu, S. Zhu, and B. Zeng, “Joint Bundled Camera Paths for Stereoscopic Video Stabilization,” in IEEE International Conference on Image Processing. IEEE, 2016, pp. 1071–1075. https://doi.org/10.1109/ICIP.2016.7532522
S. Liu, B. Xu, C. Deng, S. Zhu, B. Zeng, and M. Gabbouj, “A Hybrid Approach for Near-Range Video Stabilization,” IEEE Transactions on Circuits and Systems for Video Technology, 2016. https://doi.org/10.1109/TCSVT.2016.2556587
B.-H. Chen, A. Kopylov, S.-C. Huang, O. Seredin, R. Karpov, S.-Y. Kuo, K. R. Lai, T.-H. Tan, M. Gochoo, and D. Bayanduuren, “Improved Global Motion Estimation via Motion Vector Clustering for Video Stabilization,” Engineering Applications of Artificial Intelligence, vol. 54, pp. 39–48, 2016. https://doi.org/10.1016/j.engappai.2016.05.004
B. Chen, J. Zhao, and Y. Wang, “Research on Evaluation Method of Video Stabilization,” in International Conference on Advanced Material Science and Environmental Engineering. Atlantis Press, 2016, pp. 253–258. https://dx.doi.org/10.2991/amsee-16.2016.67
J. Kopf, “360 Video Stabilization,” ACM Transactions on Graphics, vol. 35, no. 6, p. 195, 2016. https://doi.org/10.1145/2980179.2982405
C. Buehler, M. Bosse, and L. McMillan, “Non-Metric Image-based Rendering for Video Stabilization,” in IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2. IEEE, 2001. https://doi.org/10.1109/CVPR.2001.991019
Y. Matsushita, E. Ofek, W. Ge, X. Tang, and H.-Y. Shum, “Full-Frame Video Stabilization with Motion Inpainting,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 28, no. 7, pp. 1150–1163, 2006. https://doi.org/10.1109/TPAMI.2006.141
L. Marcenaro, G. Vernazza, and C. S. Regazzoni, “Image Stabilization Algorithms for Video-Surveillance Applications,” in International Conference on Image Processing, vol. 1. IEEE, 2001, pp. 349–352. https://doi.org/10.1109/ICIP.2001.959025
A. A. Amanatiadis and I. Andreadis, “Digital Image Stabilization by Independent Component Analysis,” IEEE Transactions on Instrumentation and Measurement, vol. 59, no. 7, pp. 1755–1763, 2010. https://doi.org/10.1109/TIM.2009.2028216
D. Shukla and R. K. Jha, “A Robust Video Stabilization Technique using Integral Frame Projection Warping,” Signal, Image and Video Processing, vol. 9, no. 6, pp. 1287–1297, 2015. https://doi.org/10.1007/s11760-013-0584-5
H. Qu, L. Song, and G. Xue, “Shaking Video Synthesis for Video Stabilization Performance Assessment,” in Visual Communications and Image Processing. IEEE, 2013, pp. 1–6. https://doi.org/10.1109/VCIP.2013.6706422
A. Hamza, R. Hafiz, M. M. Khan, Y. Cho, and J. Cha, “Stabilization of Panoramic Videos from Mobile Multi-camera Platforms,” Image and Vision Computing, vol. 37, pp. 20–30, 2015. https://doi.org/10.1016/j.imavis.2015.02.002
J. Bai, A. Agarwala, M. Agrawala, and R. Ramamoorthi, “User-Assisted Video Stabilization,” in Computer Graphics Forum, vol. 33, no. 4. Wiley Online Library, 2014, pp. 61–70. https://doi.org/10.1111/cgf.12413
M. Okade and P. K. Biswas, “Video Stabilization using Maximally Stable Extremal Region Features,” Multimedia Tools and Applications, vol. 68, no. 3, pp. 947–968, 2014. https://doi.org/10.1007/s11042-012-1095-z
S. W. Kim, S. Yin, K. Yun, and J. Y. Choi, “Spatio-temporal Weighting in Local Patches for Direct Estimation of Camera Motion in Video Stabilization,” Computer Vision and Image Understanding, vol. 118, pp. 71–83, 2014. https://doi.org/10.1016/j.cviu.2013.09.005
S. Liu, L. Yuan, P. Tan, and J. Sun, “Steadyflow: Spatially Smooth Optical Flow for Video Stabilization,” in IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 4209–4216. https://doi.org/10.1109/CVPR.2014.536
M. Grundmann, V. Kwatra, and I. Essa, “Auto-Directed Video Stabilization with Robust L1 Optimal Camera Paths,” in IEEE Conference on Computer Vision and Pattern Recognition. IEEE, Jun. 2011, pp. 225–232. https://doi.org/10.1109/CVPR.2011.5995525
J.-Y. Chang, W.-F. Hu, M.-H. Cheng, and B.-S. Chang, “Digital Image Translational and Rotational Motion Stabilization using Optical Flow Technique,” IEEE Transactions on Consumer Electronics, vol. 48, no. 1, pp. 108–115, 2002. https://doi.org/10.1109/TCE.2002.1010098
S. Ertürk, “Real-Time Digital Image Stabilization using Kalman Filters,” Real-Time Imaging, vol. 8, no. 4, pp. 317–328, 2002. https://doi.org/10.1006/rtim.2001.0278
R. Jia, H. Zhang, L. Wang, and J. Li, “Digital Image Stabilization based on Phase Correlation,” in International Conference on Artificial Intelligence and Computational Intelligence, vol. 3. IEEE, 2009, pp. 485–489. https://doi.org/10.1109/AICI.2009.489
S.-J. Ko, S.-H. Lee, and K.-H. Lee, “Digital Image Stabilizing Algorithms based on Bit-Plane Matching,” IEEE Transactions on Consumer Electronics, vol. 44, no. 3, pp. 617–622, 1998. https://doi.org/10.1109/30.713172
S. Kumar, H. Azartash, M. Biswas, and T. Nguyen, “Real-Time Affine Global Motion Estimation using Phase Correlation and its Application for Digital Image Stabilization,” IEEE Transactions on Image Processing, vol. 20, no. 12, pp. 3406–3418, 2011. https://doi.org/10.1109/TIP.2011.2156420
C.-T. Lin, C.-T. Hong, and C.-T. Yang, “Real-Time Digital Image Stabilization System using Modified Proportional Integrated Controller,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 19, no. 3, pp. 427–431, 2009. https://doi.org/10.1109/TCSVT.2009.2013508
C. Morimoto and R. Chellappa, “Fast Electronic Digital Image Stabilization,” in 13th International Conference on Pattern Recognition, vol. 3. IEEE, 1996, pp. 284–288. https://doi.org/10.1109/ICPR.1996.546956
Y. G. Ryu and M. J. Chung, “Robust Online Digital Image Stabilization based on Point-Feature Trajectory without Accumulative Global Motion Estimation,” IEEE Signal Processing Letters, vol. 19, no. 4, pp. 223–226, 2012. https://doi.org/10.1109/LSP.2012.2188286
J. Yang, D. Schonfeld, C. Chen, and M. Mohamed, “Online Video Stabilization based on Particle Filters,” in International Conference on Image Processing. IEEE, 2006, pp. 1545–1548. https://doi.org/10.1109/ICIP.2006.312645
Q. Zheng and M. Yang, “A Video Stabilization Method Based on Interframe Image Matching Score,” Global Journal of Computer Science and Technology, vol. 17, no. 1-F, 2017.
N. Bhowmik, V. Gouet-Brunet, L. Wei, and G. Bloch, “Adaptive and Optimal Combination of Local Features for Image Retrieval,” in International Conference on Multimedia Modeling. Springer, 2017,pp. 76–88. https://doi.org/10.1007/978-3-319-51814-5_7
C. Jia and B. L. Evans, “Online Motion Smoothing for Video Stabilization via Constrained Multiple-model Estimation,” EURASIP Journal on Image and Video Processing, vol. 2017, no. 1, p. 25, 2017. https://doi.org/10.1186/s13640-017-0171-8
S. Liu, M. Li, S. Zhu, and B. Zeng, “CodingFlow: Enable Video Coding for Video Stabilization,” IEEE Transactions on Image Processing, vol. 26, no. 7, pp. 3291–3302, 2017. https://doi.org/10.1109/TIP.2017.2697759
M. R. Souza, “Digital Video Stabilization: Algorithms and Evaluation,” Master’s thesis, Institute of Computing, University of Campinas, 2018.
R. Stephens, Essential Algorithms: A Practical Approach to Computer Algorithms. John Wiley & Sons, 2013.
Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image Quality Assessment: From Error Visibility to Structural Similarity,” IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600–612, 2004. https://doi.org/10.1109/TIP.2003.819861
M. J. Powell, “An Efficient Method for Finding the Minimum of a Function of Several Variables Without Calculating Derivatives,” The Computer Journal, vol. 7, no. 2, pp. 155–162, 1964. https://doi.org/10.1093/comjnl/7.2.155
K. Zuiderveld, “Contrast Limited Adaptive Histogram Equalization,” in Graphics Gems IV. Academic Press Professional, Inc., 1994, pp. 474–485.
S. Liu, L. Yuan, P. Tan, and J. Sun, “Bundled Camera Paths for Video Stabilization,” ACM Transactions on Graphics, vol. 32, no. 4, p. 78, 2013. https://doi.org/10.1145/2461912.2461995
M. R. Souza and H. Pedrini, “Combination of Local Feature Detection Methods for Digital Video Stabilization,” Signal, Image and Video Processing, vol. 12, no. 8, pp. 1513–1521, 2018. https://doi.org/10.1007/s11760-018-1307-8
M. R. Souza, L. F. R. da Fonseca, and H. Pedrini, “Improvement of Global Motion Estimation in Two-Dimensional Digital Video Stabilisation Methods,” IET Image Processing, vol. 12, no. 12, pp. 2204–2211, 2018. https://doi.org/10.1049/iet-ipr.2018.5445
M. R. Souza and H. Pedrini, “Digital Video Stabilization Based on Adaptive Camera Trajectory Smoothing,” EURASIP Journal on Image and Video Processing, vol. 2018, no. 1, p. 37, 2018. https://doi.org/10.1186/s13640-018-0277-7
M. R. Souza and H. Pedrini, “Motion Energy Image for Evaluation of Video Stabilization,” The Visual Computer, pp. 1–13, 2017. https://doi.org/10.1007/s00371-018-1572-0
M. R. Souza and H. Pedrini, “Visual Rhythms for Qualitative Evaluation of Video Stabilization,” EURASIP Journal on Image and Video Processing, 2018, (submitted).
Publicado
28/10/2019
Como Citar
SOUZA, Marcos Roberto; PEDRINI, Helio.
Digital Video Stabilization: Algorithms and Evaluation. In: WORKSHOP DE TESES E DISSERTAÇÕES - CONFERENCE ON GRAPHICS, PATTERNS AND IMAGES (SIBGRAPI), 32. , 2019, Rio de Janeiro.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2019
.
p. 35-41.
DOI: https://doi.org/10.5753/sibgrapi.est.2019.8299.
