Dictionary Learning based High Frequencynon-linear Prediction for Video Coding
Final Report Abstract
Dictionary Learning (DL)/Sparse Coding (SC)-based image processing is known to outperform classical signal processing, when applied to typical inverse problems like image denoising, inpainting, or Super-Resolution (SR). Thus, the general assumption for this project was that state-of-the art video compression standards can be improved by DL/SC. Consequently, the potential entry points for DL/SC-based image processing into the hybrid video coding scheme and higher level concepts were evaluated. The research focused on experimental evaluations. For this, the state-of-the-art reference software of Versatile Video Coding (VVC) was extended by DL/SC-based algorithms, and the achievable bit rate savings were measured in terms of Bjøntegaard Delta (BD)-rate savings. The results indicate that bit rate reductions are possible, when the algorithms are applied to in-loop filtering, dynamic resolution video coding, and intra prediction. Generally, these results are promising with respect to the currently very active research area of machine learning-based image or video compression. For this reason, it is also expected that the results could serve as reference for future research projects or standardization activities. Over the course of the project, the question how coding artefacts influence the behavior of DL/SC-based algorithms arose frequently. This was to some extent unexpected, since at the time writing the proposal for the project, our general assumption was that coding artefact removal should be achievable by DL/SC. However, our experiments, especially in regard to in-loop filtering in VVC, showed that the capabilities of the conventional DL-based image denoising algorithms are limited with respect to the refinement of coded images or video sequences. Consequently, the research item of coding artefact removal appeared to be more challenging than initially expected. In summary, the project delivered a deep insight into the link between DL/SC-based image processing and state-of-the-art video coding. From our perspective, the implemented methods and results are valuable explorations on the pursuit towards machine learning-based image or video compression.
Publications
- "Adaptive Resolution Change Using Uncoded Areas and Dictionary Learning-Based Super-Resolution in Versatile Video Coding”. In: ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2020, pages 2203–2207
J. Schneider, J. Sauer, and C. Rohlfing
(See online at https://doi.org/10.1109/ICASSP40776.2020.9054522) - “Sparse Coding-based Intra Prediction in VVC”. in: 2021 Picture Coding Symposium (PCS). Bristol, UK, June 2021
J. Schneider, D. Mehlem, M. Meyer, and C. Rohlfing
(See online at https://doi.org/10.1109/PCS50896.2021.9477474)
