A Machine Learning-Based Solution to Accelerate the Intra Mode Decision for the VVC Standard
Resumo
The VVC video coding standard achieves high compression rates due to innovative tools that were introduced mainly in the intra prediction. However, the high computational effort associated with the intra mode decision poses a challenge for real-time video coding applications. In this paper, we propose a machine learning-based solution to accelerate the intra mode decision of VVC. The intra modes are organized in three classes (Planar/DC, Angular and MIP) and a Decision Tree model is developed to predict the class of modes more likely to be chosen, avoiding the evaluation of the classes of modes with less chance to be the optimal. As a result, the proposed solution can reduce the total encoding time in 15.67% on average with only 0.80% of coding efficiency loss. When compared with related works, our solution presents good results.
Palavras-chave:
VVC, Intra Mode Decision, Machine Learning
Referências
James Bergstra and Yoshua Bengio. 2012. Random search for hyper-parameter optimization.Journal of machine learning research 13, 2 (2012), 281–305. [link].
Gisle Bjontegaard. 2001. Calculation of average PSNR differences between RD-curves. . VCEG Meeting [link].
Frank Bossen, Jill Boyce, Karsten Sühring, Xiang Li, and Vadim Seregin. 2020. VTM common test conditions and software reference configurations for SDR video. [link]. JVET-T2010-v1
Benjamin Bross, Jianle Chen, Shan Liu, and Ye-Kui Wang. 2020. Versatile Video Coding Editorial Refinements on Draft 10. JVET-T2001-v2 [link].
Yao-Jen Chang, Hong-Jheng Jhu, Hui-Yu Jiang, Liang Zhao, Xin Zhao, Xiang Li, Shan Liu, Benjamin Bross, Paul Keydel, Heiko Schwarz, Detlev Marpe, and Thomas Wiegand. 2019. Multiple Reference Line Coding for Most Probable Modes in Intra Prediction. In 2019 Data Compression Conference (DCC). IEEE, Snowbird, UT, USA, 559–559. https://doi.org/10.1109/DCC.2019.00071
Thomas Daede, Andrey Norkin, and Ilya Brailovkkiy. 2018. Video Codec Testing and Quality Measurement. [link].
Santiago De-Luxán-Hernández, Valeri George, Jackie Ma, Tung Nguyen, Heiko Schwarz, Detlev Marpe, and Thomas Wiegand. 2019. An Intra Subpartition Coding Mode for VVC. In 2019 IEEE International Conference on Image Processing (ICIP). IEEE, Taipei, Taiwan, 1203–1207. https://doi.org/10.1109/ICIP.2019.8803777
Adson Duarte, Paulo Gonçalves, Luciano Agostini, Bruno Zatt, Guilherme Correa, Marcelo Porto, and Daniel Palomino. 2022. Fast Affine Motion Estimation for VVC using Machine-Learning-Based Early Search Termination. In 2022 IEEE International Symposium on Circuits and Systems (ISCAS). IEEE, Austin, TX, USA, 1–5. https://doi.org/10.1109/ISCAS48785.2022.9937973
ITU-T. 2022. Subjective video quality assessment methods for multimedia applications. [link]. P.910
Zheng Liu, Tianyi Li, Ying Chen, Kaijin Wei, Mai Xu, and Honggang Qi. 2023. Deep Multi-task Learning based Fast Intra-mode Decision for Versatile Video Coding. IEEE Transactions on Circuits and Systems for Video Technology (2023), 1–1. https://doi.org/10.1109/TCSVT.2023.3262733
Alexandre Mercat, Marko Viitanen, and Jarno Vanne. 2020. UVG Dataset: 50/120fps 4K Sequences for Video Codec Analysis and Development. In Proceedings of the 11th ACM Multimedia Systems Conference (Istanbul, Turkey) (MMSys ’20). Association for Computing Machinery, New York, NY, USA, 297–302. https://doi.org/10.1145/3339825.3394937
Chi-Ting Ni, Shih-Hsiang Lin, Pei-Yin Chen, and Yu-Ting Chu. 2022. High Efficiency Intra CU Partition and Mode Decision Method for VVC. IEEE Access 10 (2022), 77759–77771. https://doi.org/10.1109/ACCESS.2022.3193401
Jeeyoon Park, Bumyoon Kim, and Byeungwoo Jeon. 2022. Fast VVC Intra Subpartition based on Position of Reference Pixels. In 2022 International Conference on Electronics, Information, and Communication (ICEIC). IEEE, Jeju, Korea, Republic of, 1–2. https://doi.org/10.1109/ICEIC54506.2022.9748554
Jeeyoon Park, Bumyoon Kim, Jeehwan Lee, and Byeungwoo Jeon. 2022. Machine Learning-Based Early Skip Decision for Intra Subpartition Prediction in VVC. IEEE Access 10 (2022), 111052–111065. https://doi.org/10.1109/ACCESS.2022.3215163
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay. 2011. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research 12, 85 (2011), 2825–2830. [link]
Jonathan Pfaff, Alexey Filippov, Shan Liu, Xin Zhao, Jianle Chen, Santiago De-Luxán-Hernández, Thomas Wiegand, Vasily Rufitskiy, Adarsh Krishnan Ramasubramonian, and Geert Van der Auwera. 2021. Intra Prediction and Mode Coding in VVC. IEEE Transactions on Circuits and Systems for Video Technology 31, 10 (2021), 3834–3847. https://doi.org/10.1109/TCSVT.2021.3072430
Mário Saldanha, Gustavo Sanchez, César Marcon, and Luciano Agostini. 2020. Complexity Analysis Of VVC Intra Coding. In 2020 IEEE International Conference on Image Processing (ICIP). IEEE, Abu Dhabi, United Arab Emirates, 3119–3123. https://doi.org/10.1109/ICIP40778.2020.9190970
Mário Saldanha, Gustavo Sanchez, César Marcon, and Luciano Agostini. 2021. Learning-Based Complexity Reduction Scheme for VVC Intra-Frame Prediction. In 2021 International Conference on Visual Communications and Image Processing (VCIP). IEEE, Munich, Germany, 1–5. https://doi.org/10.1109/VCIP53242.2021.9675394
Mário Saldanha, Gustavo Sanchez, César Marcon, and Luciano Agostini. 2022. Fast Transform Decision Scheme for VVC Intra-Frame Prediction Using Decision Trees. In 2022 IEEE International Symposium on Circuits and Systems (ISCAS). IEEE, Austin, TX, USA, 1948–1952. https://doi.org/10.1109/ISCAS48785.2022.9938000
Michael Schäfer, Björn Stallenberger, Jonathan Pfaff, Philipp Helle, Heiko Schwarz, Detlev Marpe, and Thomas Wiegand. 2019. An Affine-Linear Intra Prediction With Complexity Constraints. In 2019 IEEE International Conference on Image Processing (ICIP). IEEE, Taipei, Taiwan, 1089–1093. https://doi.org/10.1109/ICIP.2019.8803724
Julia Stoll. 2023. Number of digital video viewers worldwide from 2019 to 2023. Statista. Retrieved Jun 06, 2023 from [link].
Gary Sullivan and Thomas Wiegand. 1998. Rate-distortion optimization for video compression. IEEE Signal Processing Magazine 15, 6 (1998), 74–90. https://doi.org/10.1109/79.733497
Liang Zhao, Li Zhang, Siwei Ma, and Debin Zhao. 2011. Fast mode decision algorithm for intra prediction in HEVC. In 2011 Visual Communications and Image Processing (VCIP). IEEE, Tainan, Taiwan, 1–4. https://doi.org/10.1109/VCIP.2011.6115979
Naima Zouidi, Amina Kessentini, Wassim Hamidouche, Nouri Masmoudi, and Daniel Menard. 2023. Complexity assessment of the intra prediction in Versatile Video Coding. Multimedia Tools and Applications (2023), 1–20
Gisle Bjontegaard. 2001. Calculation of average PSNR differences between RD-curves. . VCEG Meeting [link].
Frank Bossen, Jill Boyce, Karsten Sühring, Xiang Li, and Vadim Seregin. 2020. VTM common test conditions and software reference configurations for SDR video. [link]. JVET-T2010-v1
Benjamin Bross, Jianle Chen, Shan Liu, and Ye-Kui Wang. 2020. Versatile Video Coding Editorial Refinements on Draft 10. JVET-T2001-v2 [link].
Yao-Jen Chang, Hong-Jheng Jhu, Hui-Yu Jiang, Liang Zhao, Xin Zhao, Xiang Li, Shan Liu, Benjamin Bross, Paul Keydel, Heiko Schwarz, Detlev Marpe, and Thomas Wiegand. 2019. Multiple Reference Line Coding for Most Probable Modes in Intra Prediction. In 2019 Data Compression Conference (DCC). IEEE, Snowbird, UT, USA, 559–559. https://doi.org/10.1109/DCC.2019.00071
Thomas Daede, Andrey Norkin, and Ilya Brailovkkiy. 2018. Video Codec Testing and Quality Measurement. [link].
Santiago De-Luxán-Hernández, Valeri George, Jackie Ma, Tung Nguyen, Heiko Schwarz, Detlev Marpe, and Thomas Wiegand. 2019. An Intra Subpartition Coding Mode for VVC. In 2019 IEEE International Conference on Image Processing (ICIP). IEEE, Taipei, Taiwan, 1203–1207. https://doi.org/10.1109/ICIP.2019.8803777
Adson Duarte, Paulo Gonçalves, Luciano Agostini, Bruno Zatt, Guilherme Correa, Marcelo Porto, and Daniel Palomino. 2022. Fast Affine Motion Estimation for VVC using Machine-Learning-Based Early Search Termination. In 2022 IEEE International Symposium on Circuits and Systems (ISCAS). IEEE, Austin, TX, USA, 1–5. https://doi.org/10.1109/ISCAS48785.2022.9937973
ITU-T. 2022. Subjective video quality assessment methods for multimedia applications. [link]. P.910
Zheng Liu, Tianyi Li, Ying Chen, Kaijin Wei, Mai Xu, and Honggang Qi. 2023. Deep Multi-task Learning based Fast Intra-mode Decision for Versatile Video Coding. IEEE Transactions on Circuits and Systems for Video Technology (2023), 1–1. https://doi.org/10.1109/TCSVT.2023.3262733
Alexandre Mercat, Marko Viitanen, and Jarno Vanne. 2020. UVG Dataset: 50/120fps 4K Sequences for Video Codec Analysis and Development. In Proceedings of the 11th ACM Multimedia Systems Conference (Istanbul, Turkey) (MMSys ’20). Association for Computing Machinery, New York, NY, USA, 297–302. https://doi.org/10.1145/3339825.3394937
Chi-Ting Ni, Shih-Hsiang Lin, Pei-Yin Chen, and Yu-Ting Chu. 2022. High Efficiency Intra CU Partition and Mode Decision Method for VVC. IEEE Access 10 (2022), 77759–77771. https://doi.org/10.1109/ACCESS.2022.3193401
Jeeyoon Park, Bumyoon Kim, and Byeungwoo Jeon. 2022. Fast VVC Intra Subpartition based on Position of Reference Pixels. In 2022 International Conference on Electronics, Information, and Communication (ICEIC). IEEE, Jeju, Korea, Republic of, 1–2. https://doi.org/10.1109/ICEIC54506.2022.9748554
Jeeyoon Park, Bumyoon Kim, Jeehwan Lee, and Byeungwoo Jeon. 2022. Machine Learning-Based Early Skip Decision for Intra Subpartition Prediction in VVC. IEEE Access 10 (2022), 111052–111065. https://doi.org/10.1109/ACCESS.2022.3215163
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay. 2011. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research 12, 85 (2011), 2825–2830. [link]
Jonathan Pfaff, Alexey Filippov, Shan Liu, Xin Zhao, Jianle Chen, Santiago De-Luxán-Hernández, Thomas Wiegand, Vasily Rufitskiy, Adarsh Krishnan Ramasubramonian, and Geert Van der Auwera. 2021. Intra Prediction and Mode Coding in VVC. IEEE Transactions on Circuits and Systems for Video Technology 31, 10 (2021), 3834–3847. https://doi.org/10.1109/TCSVT.2021.3072430
Mário Saldanha, Gustavo Sanchez, César Marcon, and Luciano Agostini. 2020. Complexity Analysis Of VVC Intra Coding. In 2020 IEEE International Conference on Image Processing (ICIP). IEEE, Abu Dhabi, United Arab Emirates, 3119–3123. https://doi.org/10.1109/ICIP40778.2020.9190970
Mário Saldanha, Gustavo Sanchez, César Marcon, and Luciano Agostini. 2021. Learning-Based Complexity Reduction Scheme for VVC Intra-Frame Prediction. In 2021 International Conference on Visual Communications and Image Processing (VCIP). IEEE, Munich, Germany, 1–5. https://doi.org/10.1109/VCIP53242.2021.9675394
Mário Saldanha, Gustavo Sanchez, César Marcon, and Luciano Agostini. 2022. Fast Transform Decision Scheme for VVC Intra-Frame Prediction Using Decision Trees. In 2022 IEEE International Symposium on Circuits and Systems (ISCAS). IEEE, Austin, TX, USA, 1948–1952. https://doi.org/10.1109/ISCAS48785.2022.9938000
Michael Schäfer, Björn Stallenberger, Jonathan Pfaff, Philipp Helle, Heiko Schwarz, Detlev Marpe, and Thomas Wiegand. 2019. An Affine-Linear Intra Prediction With Complexity Constraints. In 2019 IEEE International Conference on Image Processing (ICIP). IEEE, Taipei, Taiwan, 1089–1093. https://doi.org/10.1109/ICIP.2019.8803724
Julia Stoll. 2023. Number of digital video viewers worldwide from 2019 to 2023. Statista. Retrieved Jun 06, 2023 from [link].
Gary Sullivan and Thomas Wiegand. 1998. Rate-distortion optimization for video compression. IEEE Signal Processing Magazine 15, 6 (1998), 74–90. https://doi.org/10.1109/79.733497
Liang Zhao, Li Zhang, Siwei Ma, and Debin Zhao. 2011. Fast mode decision algorithm for intra prediction in HEVC. In 2011 Visual Communications and Image Processing (VCIP). IEEE, Tainan, Taiwan, 1–4. https://doi.org/10.1109/VCIP.2011.6115979
Naima Zouidi, Amina Kessentini, Wassim Hamidouche, Nouri Masmoudi, and Daniel Menard. 2023. Complexity assessment of the intra prediction in Versatile Video Coding. Multimedia Tools and Applications (2023), 1–20
Publicado
23/10/2023
Como Citar
DUARTE, Adson; OLIVEIRA, Anna; ZATT, Bruno; CORREA, Guilherme; PALOMINO, Daniel.
A Machine Learning-Based Solution to Accelerate the Intra Mode Decision for the VVC Standard. In: BRAZILIAN SYMPOSIUM ON MULTIMEDIA AND THE WEB (WEBMEDIA), 29. , 2023, Ribeirão Preto/SP.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2023
.
p. 73–81.
