HFS-HNeRV: High-Frequency Spectrum Hybrid Neural Representation for Videos

Abstract

Implicit neural representations have recently demonstrated considerable potential in various applications, including video compression and reconstruction, owing to their rapid decoding speed and high adaptability. Based on the most advanced neural representation for Videos (NeRV), Expedite Neural Representation for Videos (E-NeRV) and Hybrid Neural Representation for Videos (H-NeRV) primarily boost performance by enhancing and broadening the NeRV network’s embedded input, whereas the NeRV module—the central component involved in video reconstruction—has attracted less attention. With a focus on high-frequency data in the frequency domain, this paper proposes a High-frequency Spectrum Hybrid Network (HFS-HNeRV), which adopts effective high-frequency data from the frequency domain to generate image details. Its core, HFS-HNeRV block, is a novel NeRV module, which adds the high-frequency spectrum convolution module (HFSCM) to the original one. This module extracts and emphasizes high-frequency features through the frequency domain attention mechanism, which not only provides superior performance, but also enhances the local detail recovery in video images. As an upgrade of the NeRV module, it has exceptional performance in terms of adaptability and versatility. It can conveniently substitute in a variety of current NeRV designs without requiring significant alterations to attain enhanced performance. Furthermore, this paper also introduces the High-frequency Spectrum (HFS) loss function to further mitigate the blurriness issue caused by the loss of high-frequency information during image generation. In the video compression task, the proposed HFS-HNeRV network outperformed NeRV, E-NeRV and HNeRV with an improvement of +5.68 dB, +4.46 dB, and +0.98 dB in reconstruction quality (PSNR), respectively.