A Machine Learning-Based Solution to Accelerate the Intra Mode Decision for the VVC Standard

  • Adson Duarte UFPel
  • Anna Oliveira UFPel
  • Bruno Zatt UFPel
  • Guilherme Correa UFPel
  • Daniel Palomino UFPel

Resumo


The VVC video coding standard achieves high compression rates due to innovative tools that were introduced mainly in the intra prediction. However, the high computational effort associated with the intra mode decision poses a challenge for real-time video coding applications. In this paper, we propose a machine learning-based solution to accelerate the intra mode decision of VVC. The intra modes are organized in three classes (Planar/DC, Angular and MIP) and a Decision Tree model is developed to predict the class of modes more likely to be chosen, avoiding the evaluation of the classes of modes with less chance to be the optimal. As a result, the proposed solution can reduce the total encoding time in 15.67% on average with only 0.80% of coding efficiency loss. When compared with related works, our solution presents good results.

Palavras-chave: VVC, Intra Mode Decision, Machine Learning

Referências

James Bergstra and Yoshua Bengio. 2012. Random search for hyper-parameter optimization.Journal of machine learning research 13, 2 (2012), 281–305. [link].

Gisle Bjontegaard. 2001. Calculation of average PSNR differences between RD-curves. . VCEG Meeting [link].

Frank Bossen, Jill Boyce, Karsten Sühring, Xiang Li, and Vadim Seregin. 2020. VTM common test conditions and software reference configurations for SDR video. [link]. JVET-T2010-v1

Benjamin Bross, Jianle Chen, Shan Liu, and Ye-Kui Wang. 2020. Versatile Video Coding Editorial Refinements on Draft 10. JVET-T2001-v2 [link].

Yao-Jen Chang, Hong-Jheng Jhu, Hui-Yu Jiang, Liang Zhao, Xin Zhao, Xiang Li, Shan Liu, Benjamin Bross, Paul Keydel, Heiko Schwarz, Detlev Marpe, and Thomas Wiegand. 2019. Multiple Reference Line Coding for Most Probable Modes in Intra Prediction. In 2019 Data Compression Conference (DCC). IEEE, Snowbird, UT, USA, 559–559. https://doi.org/10.1109/DCC.2019.00071

Thomas Daede, Andrey Norkin, and Ilya Brailovkkiy. 2018. Video Codec Testing and Quality Measurement. [link].

Santiago De-Luxán-Hernández, Valeri George, Jackie Ma, Tung Nguyen, Heiko Schwarz, Detlev Marpe, and Thomas Wiegand. 2019. An Intra Subpartition Coding Mode for VVC. In 2019 IEEE International Conference on Image Processing (ICIP). IEEE, Taipei, Taiwan, 1203–1207. https://doi.org/10.1109/ICIP.2019.8803777

Adson Duarte, Paulo Gonçalves, Luciano Agostini, Bruno Zatt, Guilherme Correa, Marcelo Porto, and Daniel Palomino. 2022. Fast Affine Motion Estimation for VVC using Machine-Learning-Based Early Search Termination. In 2022 IEEE International Symposium on Circuits and Systems (ISCAS). IEEE, Austin, TX, USA, 1–5. https://doi.org/10.1109/ISCAS48785.2022.9937973

ITU-T. 2022. Subjective video quality assessment methods for multimedia applications. [link]. P.910

Zheng Liu, Tianyi Li, Ying Chen, Kaijin Wei, Mai Xu, and Honggang Qi. 2023. Deep Multi-task Learning based Fast Intra-mode Decision for Versatile Video Coding. IEEE Transactions on Circuits and Systems for Video Technology (2023), 1–1. https://doi.org/10.1109/TCSVT.2023.3262733

Alexandre Mercat, Marko Viitanen, and Jarno Vanne. 2020. UVG Dataset: 50/120fps 4K Sequences for Video Codec Analysis and Development. In Proceedings of the 11th ACM Multimedia Systems Conference (Istanbul, Turkey) (MMSys ’20). Association for Computing Machinery, New York, NY, USA, 297–302. https://doi.org/10.1145/3339825.3394937

Chi-Ting Ni, Shih-Hsiang Lin, Pei-Yin Chen, and Yu-Ting Chu. 2022. High Efficiency Intra CU Partition and Mode Decision Method for VVC. IEEE Access 10 (2022), 77759–77771. https://doi.org/10.1109/ACCESS.2022.3193401

Jeeyoon Park, Bumyoon Kim, and Byeungwoo Jeon. 2022. Fast VVC Intra Subpartition based on Position of Reference Pixels. In 2022 International Conference on Electronics, Information, and Communication (ICEIC). IEEE, Jeju, Korea, Republic of, 1–2. https://doi.org/10.1109/ICEIC54506.2022.9748554

Jeeyoon Park, Bumyoon Kim, Jeehwan Lee, and Byeungwoo Jeon. 2022. Machine Learning-Based Early Skip Decision for Intra Subpartition Prediction in VVC. IEEE Access 10 (2022), 111052–111065. https://doi.org/10.1109/ACCESS.2022.3215163

Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay. 2011. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research 12, 85 (2011), 2825–2830. [link]

Jonathan Pfaff, Alexey Filippov, Shan Liu, Xin Zhao, Jianle Chen, Santiago De-Luxán-Hernández, Thomas Wiegand, Vasily Rufitskiy, Adarsh Krishnan Ramasubramonian, and Geert Van der Auwera. 2021. Intra Prediction and Mode Coding in VVC. IEEE Transactions on Circuits and Systems for Video Technology 31, 10 (2021), 3834–3847. https://doi.org/10.1109/TCSVT.2021.3072430

Mário Saldanha, Gustavo Sanchez, César Marcon, and Luciano Agostini. 2020. Complexity Analysis Of VVC Intra Coding. In 2020 IEEE International Conference on Image Processing (ICIP). IEEE, Abu Dhabi, United Arab Emirates, 3119–3123. https://doi.org/10.1109/ICIP40778.2020.9190970

Mário Saldanha, Gustavo Sanchez, César Marcon, and Luciano Agostini. 2021. Learning-Based Complexity Reduction Scheme for VVC Intra-Frame Prediction. In 2021 International Conference on Visual Communications and Image Processing (VCIP). IEEE, Munich, Germany, 1–5. https://doi.org/10.1109/VCIP53242.2021.9675394

Mário Saldanha, Gustavo Sanchez, César Marcon, and Luciano Agostini. 2022. Fast Transform Decision Scheme for VVC Intra-Frame Prediction Using Decision Trees. In 2022 IEEE International Symposium on Circuits and Systems (ISCAS). IEEE, Austin, TX, USA, 1948–1952. https://doi.org/10.1109/ISCAS48785.2022.9938000

Michael Schäfer, Björn Stallenberger, Jonathan Pfaff, Philipp Helle, Heiko Schwarz, Detlev Marpe, and Thomas Wiegand. 2019. An Affine-Linear Intra Prediction With Complexity Constraints. In 2019 IEEE International Conference on Image Processing (ICIP). IEEE, Taipei, Taiwan, 1089–1093. https://doi.org/10.1109/ICIP.2019.8803724

Julia Stoll. 2023. Number of digital video viewers worldwide from 2019 to 2023. Statista. Retrieved Jun 06, 2023 from [link].

Gary Sullivan and Thomas Wiegand. 1998. Rate-distortion optimization for video compression. IEEE Signal Processing Magazine 15, 6 (1998), 74–90. https://doi.org/10.1109/79.733497

Liang Zhao, Li Zhang, Siwei Ma, and Debin Zhao. 2011. Fast mode decision algorithm for intra prediction in HEVC. In 2011 Visual Communications and Image Processing (VCIP). IEEE, Tainan, Taiwan, 1–4. https://doi.org/10.1109/VCIP.2011.6115979

Naima Zouidi, Amina Kessentini, Wassim Hamidouche, Nouri Masmoudi, and Daniel Menard. 2023. Complexity assessment of the intra prediction in Versatile Video Coding. Multimedia Tools and Applications (2023), 1–20
Publicado
23/10/2023
DUARTE, Adson; OLIVEIRA, Anna; ZATT, Bruno; CORREA, Guilherme; PALOMINO, Daniel. A Machine Learning-Based Solution to Accelerate the Intra Mode Decision for the VVC Standard. In: BRAZILIAN SYMPOSIUM ON MULTIMEDIA AND THE WEB (WEBMEDIA), 29. , 2023, Ribeirão Preto/SP. Anais [...]. Porto Alegre: Sociedade Brasileira de Computação, 2023 . p. 73–81.