Neural-Based Image and Video Coding: When Machines Learn to Compress
Resumo
Neural image and video coding have emerged as promising alternatives to traditional compression approaches by replacing hand-crafted coding tools with data-driven models. This tutorial article provides an introductory overview of the main concepts behind neural compression. We first revisit fundamental deep learning concepts required to understand neural codecs and then discuss neural image coding. For neural video coding, particular attention is given to different inter-frame coding schemes and temporal buffering strategies. Finally, we present major challenges that still limit practical deployment, including computational complexity, energy consumption, real-time operation, and cross-platform consistency. The article aims to provide readers with a conceptual foundation for understanding current neural image and video coding research and its open challenges.
Palavras-chave:
neural compression, video coding, image coding, deep learning
Referências
Eirikur Agustsson, David Minnen, Nick Johnston, Johannes Ballé, Sung Jin Hwang, and George Toderici. 2020. Scale-Space Flow for End-to-End Optimized Video Compression. In 2020 IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR). [link]
Johannes Ballé, Valero Laparra, and Eero P. Simoncelli. 2016. Density Modeling of Images using a Generalized Normalization Transformation. In International Conference on Learning Representations (ICLR). [link]
Johannes Ballé, Nick Johnston, and David Minnen. 2019. Integer Networks for Data Compression with Latent-Variable Models. In International Conference on Learning Representations. [link]
Johannes Ballé, Valero Laparra, and Eero P. Simoncelli. 2017. End-to-end Optimized Image Compression. arXiv:1611.01704 [cs.CV] [link]
Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston. 2018. Variational image compression with a scale hyperprior. arXiv:1802.01436 [eess.IV] [link]
Jean Bégaint, Fabien Racapé, Simon Feltman, and Akshay Pushparaja. 2020. CompressAI: a PyTorch library and evaluation platform for end-to-end compression research. arXiv preprint arXiv:2011.03029 (2020).
Fabian Brand, Jürgen Seiler, and André Kaup. 2024. Conditional Residual Coding: A Remedy for Bottleneck Problems in Conditional Inter Frame Coding. IEEE Transactions on Circuits and Systems for Video Technology (TCSVT) 34, 7 (2024), 6445–6459.
Benjamin Bross, Ye-Kui Wang, Yaowu Ye, Shan Liu, Jianle Chen, Gary J Sullivan, and Jens-Rainer Ohm. 2021. Overview of the versatile video coding (VVC) standard and its applications. IEEE Transactions on Circuits and Systems for Video Technology 31, 10 (2021), 3736–3764.
Yi-Hsin Chen, Hong-Sheng Xie, Cheng-Wei Chen, Zong-Lin Gao, Martin Benjak, Wen-Hsiao Peng, and Jörn Ostermann. 2024. MaskCRT: Masked Conditional Residual Transformer for Learned Video Compression. IEEE Transactions on Circuits and Systems for Video Technology (TCSVT) 34, 11 (2024), 11980–11992.
Yi-Hsin Chen, Yi-Chen Yao, Kuan-Wei Ho, Chun-Hung Wu, Huu-Tai Phung, Martin Benjak, Jörn Ostermann, and Wen-Hsiao Peng. 2025. HyTIP: Hybrid Temporal Information Propagation for Masked Conditional Residual Video Coding. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV).
Zhengxue Cheng, Heming Sun, Masaru Takeuchi, and Jiro Katto. 2020. Learned Image Compression With Discretized Gaussian Mixture Likelihoods and Attention Modules. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 7936–7945. DOI: 10.1109/CVPR42600.2020.00796
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. MIT Press. [link].
Zhaoyang Jia, Bin Li, Jiahao Li, Wenxuan Xie, Linfeng Qi, Houqiang Li, and Yan Lu. 2025. Towards Practical Real-Time Neural Video Compression. arXiv preprint arXiv:2502.20762 (2025). [link]
Diederik P Kingma and Max Welling. 2014. Auto-Encoding Variational Bayes. In 2nd International Conference on Learning Representations, ICLR 2014. [link]
Théo Ladune, Pierrick Philippe, Félix Henry, Gordon Clare, and Thomas Leguay. 2023. COOL-CHIC: Coordinate-based Low Complexity Hierarchical Image Codec. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 13515–13522.
Jiahao Li, Bin Li, and Yan Lu. 2021. Deep Contextual Video Compression. Advances in Neural Information Processing Systems 34 (2021).
Jiahao Li, Bin Li, and Yan Lu. 2022. Hybrid Spatial-Temporal Entropy Modelling for Neural Video Compression. In Proceedings of the 30th ACM International Conference on Multimedia (MM ’22). DOI: 10.1145/3503161.3547845
Jiahao Li, Bin Li, and Yan Lu. 2023. Neural Video Compression with Diverse Contexts. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, Canada, June 18-22, 2023.
Jiahao Li, Bin Li, and Yan Lu. 2024. Neural Video Compression with Feature Modulation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 17-21, 2024.
Guo Lu, Wanli Ouyang, Dong Xu, Xiaoyun Zhang, Chunlei Cai, and Zhiyong Gao. 2019. DVC: An End-To-End Deep Video Compression Framework. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 10998–11007. DOI: 10.1109/CVPR.2019.01126
David Minnen, Johannes Ballé, and George Toderici. 2018. Joint Autoregressive and Hierarchical Priors for Learned Image Compression. arXiv:1809.02736 [cs.CV] [link]
Xihua Sheng, Jiahao Li, Bin Li, Li Li, Dong Liu, and Yan Lu. 2022. Temporal context mining for learned video compression. IEEE Transactions on Multimedia (2022).
Johannes Ballé, Valero Laparra, and Eero P. Simoncelli. 2016. Density Modeling of Images using a Generalized Normalization Transformation. In International Conference on Learning Representations (ICLR). [link]
Johannes Ballé, Nick Johnston, and David Minnen. 2019. Integer Networks for Data Compression with Latent-Variable Models. In International Conference on Learning Representations. [link]
Johannes Ballé, Valero Laparra, and Eero P. Simoncelli. 2017. End-to-end Optimized Image Compression. arXiv:1611.01704 [cs.CV] [link]
Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston. 2018. Variational image compression with a scale hyperprior. arXiv:1802.01436 [eess.IV] [link]
Jean Bégaint, Fabien Racapé, Simon Feltman, and Akshay Pushparaja. 2020. CompressAI: a PyTorch library and evaluation platform for end-to-end compression research. arXiv preprint arXiv:2011.03029 (2020).
Fabian Brand, Jürgen Seiler, and André Kaup. 2024. Conditional Residual Coding: A Remedy for Bottleneck Problems in Conditional Inter Frame Coding. IEEE Transactions on Circuits and Systems for Video Technology (TCSVT) 34, 7 (2024), 6445–6459.
Benjamin Bross, Ye-Kui Wang, Yaowu Ye, Shan Liu, Jianle Chen, Gary J Sullivan, and Jens-Rainer Ohm. 2021. Overview of the versatile video coding (VVC) standard and its applications. IEEE Transactions on Circuits and Systems for Video Technology 31, 10 (2021), 3736–3764.
Yi-Hsin Chen, Hong-Sheng Xie, Cheng-Wei Chen, Zong-Lin Gao, Martin Benjak, Wen-Hsiao Peng, and Jörn Ostermann. 2024. MaskCRT: Masked Conditional Residual Transformer for Learned Video Compression. IEEE Transactions on Circuits and Systems for Video Technology (TCSVT) 34, 11 (2024), 11980–11992.
Yi-Hsin Chen, Yi-Chen Yao, Kuan-Wei Ho, Chun-Hung Wu, Huu-Tai Phung, Martin Benjak, Jörn Ostermann, and Wen-Hsiao Peng. 2025. HyTIP: Hybrid Temporal Information Propagation for Masked Conditional Residual Video Coding. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV).
Zhengxue Cheng, Heming Sun, Masaru Takeuchi, and Jiro Katto. 2020. Learned Image Compression With Discretized Gaussian Mixture Likelihoods and Attention Modules. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 7936–7945. DOI: 10.1109/CVPR42600.2020.00796
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. MIT Press. [link].
Zhaoyang Jia, Bin Li, Jiahao Li, Wenxuan Xie, Linfeng Qi, Houqiang Li, and Yan Lu. 2025. Towards Practical Real-Time Neural Video Compression. arXiv preprint arXiv:2502.20762 (2025). [link]
Diederik P Kingma and Max Welling. 2014. Auto-Encoding Variational Bayes. In 2nd International Conference on Learning Representations, ICLR 2014. [link]
Théo Ladune, Pierrick Philippe, Félix Henry, Gordon Clare, and Thomas Leguay. 2023. COOL-CHIC: Coordinate-based Low Complexity Hierarchical Image Codec. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 13515–13522.
Jiahao Li, Bin Li, and Yan Lu. 2021. Deep Contextual Video Compression. Advances in Neural Information Processing Systems 34 (2021).
Jiahao Li, Bin Li, and Yan Lu. 2022. Hybrid Spatial-Temporal Entropy Modelling for Neural Video Compression. In Proceedings of the 30th ACM International Conference on Multimedia (MM ’22). DOI: 10.1145/3503161.3547845
Jiahao Li, Bin Li, and Yan Lu. 2023. Neural Video Compression with Diverse Contexts. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, Canada, June 18-22, 2023.
Jiahao Li, Bin Li, and Yan Lu. 2024. Neural Video Compression with Feature Modulation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 17-21, 2024.
Guo Lu, Wanli Ouyang, Dong Xu, Xiaoyun Zhang, Chunlei Cai, and Zhiyong Gao. 2019. DVC: An End-To-End Deep Video Compression Framework. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 10998–11007. DOI: 10.1109/CVPR.2019.01126
David Minnen, Johannes Ballé, and George Toderici. 2018. Joint Autoregressive and Hierarchical Priors for Learned Image Compression. arXiv:1809.02736 [cs.CV] [link]
Xihua Sheng, Jiahao Li, Bin Li, Li Li, Dong Liu, and Yan Lu. 2022. Temporal context mining for learned video compression. IEEE Transactions on Multimedia (2022).
Publicado
27/10/2026
Como Citar
CONCEIÇÃO, Ruhan; AGOSTINI, Luciano.
Neural-Based Image and Video Coding: When Machines Learn to Compress. In: TUTORIAIS - SIMPÓSIO BRASILEIRO DE SISTEMAS MULTIMÍDIA E WEB (WEBMEDIA), 32. , 2026, Lavras/MG.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2026
.
p. 11-15.
ISSN 2596-1683.
DOI: https://doi.org/10.5753/webmedia_estendido.2026.31950.
