Novel Light Field Encoding Framework Based on Optical Flow and Phase Correlation
Resumo
This work proposes a novel Light Field encoding framework based on optical flow and phase correlation. This encoder works by com bining one reference sub-aperture image (SAI) with optical flow and phase correlation information of a few SAIs to synthesize all SAIs of the Light Field. In order to improve the quality of the synthesized Light Field, an HEVC encoder is modified to encode synthesis error and add a quality enhancement layer. The proposed encoder out performs JPEG Pleno 4D Transform Mode encoder and the HEVC anchor in lower bitrates. Also, it can be easily adapted to the existing multimedia infrastructure.
Palavras-chave:
pseudovideo sequences, light field compression, sub-aperture image synthesis, low-bitrate encoding
Referências
H. Amirpour, M. Pereira, and A. Pinheiro. 2018. High efficient snake order pseudo-sequence based light field image compression. In 2018 Data Compression Conference. IEEE, 397–397.
Pekka Astola, Luis A da Silva Cruz, Eduardo AB da Silva, Touradj Ebrahimi, Pedro Garcia Freitas, Antonin Gilles, Kwan-Jung Oh, Carla Pagliari, Fernando Pereira, Cristian Perra, et al. 2020. JPEG Pleno: Standardizing a coding framework and tools for plenoptic imaging modalities. ITU Journal: ICT Discoveries (2020).
P. Astola and I. Tabus. 2018. Wasp: Hierarchical warping, merging, and sparse prediction for light field image compression. In 2018 7th European Workshop on Visual Information Processing (EUVIP). IEEE, 1–6.
J. R Bergen and E. H. Adelson. 1991. The plenoptic function and the elements of early vision. Computational models of visual processing 1 (1991), 8.
Thiago Bubolz, Ruhan Conceição, Heitor Almeida, Érick Moreira, Bruno Zatt, Tatiana Tavares, Marcelo Porto, and Guilherme Corrêa. 2017. Video Quality Assessment of Early SKIP/DIS for 3D-HEVC Complexity Reduction. In Proceedings of the 23rd Brazillian Symposium on Multimedia and the Web (Gramado, RS, Brazil) (WebMedia ’17). Association for Computing Machinery, New York, NY, USA, 73–79. https://doi.org/10.1145/3126858.3126898
R. Conceicao, M. Porto, B. Zatt, and L. Agostini. 2018. LF-CAE: Context-adaptive encoding for lenslet light fields using HEVC. In 2018 25th IEEE International Conference on Image Processing (ICIP). IEEE, 3174–3178.
M. B. de Carvalho, M. P. Pereira, G. Alves, EAB da Silva, C. L. Pagliari, F. Pereira, and V. Testoni. 2018. A 4D DCT-based lenslet light field codec. In 2018 25th IEEE International Conference on Image Processing (ICIP). IEEE, 435–439.
E. Dib, M. L. Pendu, and C. Guillemot. 2019. Light Field Compression Using Fourier Disparity Layers. In 2019 IEEE International Conference on Image Processing (ICIP). 3751–3755. https://doi.org/10.1109/ICIP.2019.8803756
FFMPEG. 2022. Ffmpeg software. ffmpeg. https://ffmpeg.org/ 1 de junho 2022
S. J. Gortler, R. Grzeszczuk, R. Szeliski, and M. F. Cohen. 1996. The lumigraph. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques. 43–54.
D. M. Hoffman, A. R. Girshick, K. Akeley, and M. S. Banks. 2008. Vergence–accommodation conflicts hinder visual performance and cause visual fatigue. Journal of vision 8, 3 (2008), 33–33.
A. Holst. 2021. Immersive technology consumer market revenue worldwide from 2018 to 2023, by segment. Statista. [link]. fev de 2022
BKP Horn and B. G. Schunck. 1981. Determining optical flow. Artificial intelligence 17, 1-3 (1981), 185–203.
J. Hou, J. Chen, and L.-P. Chau. 2019. Light Field Image Compression Based on Bi-Level View Compensation With Rate-Distortion Optimization. IEEE Transactions on Circuits and Systems for Video Technology 29, 2 (2019), 517–530. https://doi.org/10.1109/TCSVT.2018.2802943
Ronaldo Husemann, Jose Valdeni Lima, and Valter Roesler. 2015. Experimentation of Motion Estimation Algorithms in GPU. In Proceedings of the 21st Brazilian Symposium on Multimedia and the Web (Manaus, Brazil) (WebMedia ’15). Association for Computing Machinery, New York, NY, USA, 161–164. https://doi.org/10.1145/2820426.2820454
Ronaldo Husemann, Valter Roesler, and Jose Valdeni de Lima. 2016. Experimental Investigation of the SHVC Scalable Video Encoder Architecture. In Proceedings of the 22nd Brazilian Symposium on Multimedia and the Web (Teresina, Piauí State, Brazil) (Webmedia ’16). Association for Computing Machinery, New York, NY, USA, 211–214. https://doi.org/10.1145/2976796.2988172
I. Ihrke, J. Restrepo, and L. Mignard-Debise. 2016. Principles of light field imaging: Briefly revisiting 25 years of research. IEEE Signal Processing Magazine 33, 5 (2016), 59–69.
JCT-VC. 2022. High Efficiency Video Coding Test Model 16.20. JCT-VC. https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware/tags/HM-16.20/ 1 de junho de 2022
JPEG. 2022. JPEG Pleno Reference Software. JPEG. https://https://gitlab.com/wg1/jpeg-pleno-refsw/ 1 de junho de 2022
C.D. Kuglin and D. C. Hinne. 1975. The phase correlation image alignment method. In International Conference on Cybernetics and Society. 163–165.
F. Pereira, C. Pagliari, EAB da Silva, I. Tabus, H. Amirpour, M. Bernardo, and A. Pinheiro. 2019. JPEG Pleno light field coding common test conditions v3. 2. Doc.ISO/IEC JTC 1 (2019).
D. Rüefenacht, A. T. Naman, R. Mathew, and D. Taubman. 2019. Base-Anchored Model for Highly Scalable and Accessible Compression of Multiview Imagery. IEEE Transactions on Image Processing 28, 7 (2019), 3205–3218. https://doi.org/10.1109/TIP.2019.2894968
Cristiano Santos, Ruhan Conceição, Luciano Agostini, Guilherme Corrêa, Bruno Zatt, and Marcelo Porto. 2017. Rate and Complexity-Aware Coding Scheme for Fixed-Camera Videos Based on Region-of-Interest Detection. In Proceedings of the 23rd Brazillian Symposium on Multimedia and the Web (Gramado, RS, Brazil) (WebMedia ’17). Association for Computing Machinery, New York, NY, USA, 409–412. https://doi.org/10.1145/3126858.3131599
G. J. Sullivan, J.-R. Ohm, W.-J. Han, and T. Wiegand. 2012. Overview of the high efficiency video coding (HEVC) standard. IEEE Transactions on circuits and systems for video technology 22, 12 (2012), 1649–1668.
VideoLan. 2022. x265 Encoder. VideoLAN. https://www.videolan.org/developers/x265.html 1 de junho de 2022
Pekka Astola, Luis A da Silva Cruz, Eduardo AB da Silva, Touradj Ebrahimi, Pedro Garcia Freitas, Antonin Gilles, Kwan-Jung Oh, Carla Pagliari, Fernando Pereira, Cristian Perra, et al. 2020. JPEG Pleno: Standardizing a coding framework and tools for plenoptic imaging modalities. ITU Journal: ICT Discoveries (2020).
P. Astola and I. Tabus. 2018. Wasp: Hierarchical warping, merging, and sparse prediction for light field image compression. In 2018 7th European Workshop on Visual Information Processing (EUVIP). IEEE, 1–6.
J. R Bergen and E. H. Adelson. 1991. The plenoptic function and the elements of early vision. Computational models of visual processing 1 (1991), 8.
Thiago Bubolz, Ruhan Conceição, Heitor Almeida, Érick Moreira, Bruno Zatt, Tatiana Tavares, Marcelo Porto, and Guilherme Corrêa. 2017. Video Quality Assessment of Early SKIP/DIS for 3D-HEVC Complexity Reduction. In Proceedings of the 23rd Brazillian Symposium on Multimedia and the Web (Gramado, RS, Brazil) (WebMedia ’17). Association for Computing Machinery, New York, NY, USA, 73–79. https://doi.org/10.1145/3126858.3126898
R. Conceicao, M. Porto, B. Zatt, and L. Agostini. 2018. LF-CAE: Context-adaptive encoding for lenslet light fields using HEVC. In 2018 25th IEEE International Conference on Image Processing (ICIP). IEEE, 3174–3178.
M. B. de Carvalho, M. P. Pereira, G. Alves, EAB da Silva, C. L. Pagliari, F. Pereira, and V. Testoni. 2018. A 4D DCT-based lenslet light field codec. In 2018 25th IEEE International Conference on Image Processing (ICIP). IEEE, 435–439.
E. Dib, M. L. Pendu, and C. Guillemot. 2019. Light Field Compression Using Fourier Disparity Layers. In 2019 IEEE International Conference on Image Processing (ICIP). 3751–3755. https://doi.org/10.1109/ICIP.2019.8803756
FFMPEG. 2022. Ffmpeg software. ffmpeg. https://ffmpeg.org/ 1 de junho 2022
S. J. Gortler, R. Grzeszczuk, R. Szeliski, and M. F. Cohen. 1996. The lumigraph. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques. 43–54.
D. M. Hoffman, A. R. Girshick, K. Akeley, and M. S. Banks. 2008. Vergence–accommodation conflicts hinder visual performance and cause visual fatigue. Journal of vision 8, 3 (2008), 33–33.
A. Holst. 2021. Immersive technology consumer market revenue worldwide from 2018 to 2023, by segment. Statista. [link]. fev de 2022
BKP Horn and B. G. Schunck. 1981. Determining optical flow. Artificial intelligence 17, 1-3 (1981), 185–203.
J. Hou, J. Chen, and L.-P. Chau. 2019. Light Field Image Compression Based on Bi-Level View Compensation With Rate-Distortion Optimization. IEEE Transactions on Circuits and Systems for Video Technology 29, 2 (2019), 517–530. https://doi.org/10.1109/TCSVT.2018.2802943
Ronaldo Husemann, Jose Valdeni Lima, and Valter Roesler. 2015. Experimentation of Motion Estimation Algorithms in GPU. In Proceedings of the 21st Brazilian Symposium on Multimedia and the Web (Manaus, Brazil) (WebMedia ’15). Association for Computing Machinery, New York, NY, USA, 161–164. https://doi.org/10.1145/2820426.2820454
Ronaldo Husemann, Valter Roesler, and Jose Valdeni de Lima. 2016. Experimental Investigation of the SHVC Scalable Video Encoder Architecture. In Proceedings of the 22nd Brazilian Symposium on Multimedia and the Web (Teresina, Piauí State, Brazil) (Webmedia ’16). Association for Computing Machinery, New York, NY, USA, 211–214. https://doi.org/10.1145/2976796.2988172
I. Ihrke, J. Restrepo, and L. Mignard-Debise. 2016. Principles of light field imaging: Briefly revisiting 25 years of research. IEEE Signal Processing Magazine 33, 5 (2016), 59–69.
JCT-VC. 2022. High Efficiency Video Coding Test Model 16.20. JCT-VC. https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware/tags/HM-16.20/ 1 de junho de 2022
JPEG. 2022. JPEG Pleno Reference Software. JPEG. https://https://gitlab.com/wg1/jpeg-pleno-refsw/ 1 de junho de 2022
C.D. Kuglin and D. C. Hinne. 1975. The phase correlation image alignment method. In International Conference on Cybernetics and Society. 163–165.
F. Pereira, C. Pagliari, EAB da Silva, I. Tabus, H. Amirpour, M. Bernardo, and A. Pinheiro. 2019. JPEG Pleno light field coding common test conditions v3. 2. Doc.ISO/IEC JTC 1 (2019).
D. Rüefenacht, A. T. Naman, R. Mathew, and D. Taubman. 2019. Base-Anchored Model for Highly Scalable and Accessible Compression of Multiview Imagery. IEEE Transactions on Image Processing 28, 7 (2019), 3205–3218. https://doi.org/10.1109/TIP.2019.2894968
Cristiano Santos, Ruhan Conceição, Luciano Agostini, Guilherme Corrêa, Bruno Zatt, and Marcelo Porto. 2017. Rate and Complexity-Aware Coding Scheme for Fixed-Camera Videos Based on Region-of-Interest Detection. In Proceedings of the 23rd Brazillian Symposium on Multimedia and the Web (Gramado, RS, Brazil) (WebMedia ’17). Association for Computing Machinery, New York, NY, USA, 409–412. https://doi.org/10.1145/3126858.3131599
G. J. Sullivan, J.-R. Ohm, W.-J. Han, and T. Wiegand. 2012. Overview of the high efficiency video coding (HEVC) standard. IEEE Transactions on circuits and systems for video technology 22, 12 (2012), 1649–1668.
VideoLan. 2022. x265 Encoder. VideoLAN. https://www.videolan.org/developers/x265.html 1 de junho de 2022
Publicado
07/11/2022
Como Citar
STORCH, Iago; CORRÊA, Douglas; ROSLER, Igor; PALOMINO, Daniel; BAMPI, Sergio; ZATT, Bruno.
Novel Light Field Encoding Framework Based on Optical Flow and Phase Correlation. In: SIMPÓSIO BRASILEIRO DE SISTEMAS MULTIMÍDIA E WEB (WEBMEDIA), 28. , 2022, Curitiba.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2022
.
p. 311-317.
