The capability of video super-resolution (VSR) to synthesize high-resolution (HR) video from ideal datasets has been demonstrated in many works. However, applying the VSR model to real-world video with unknown and complex degradation remains a challenging task. First, existing degradation metrics in most VSR methods are not able to effectively simulate real-world noise and blur. On the contrary, simple combinations of classical degradation are used for real-world noise modeling, which led to the VSR model often being violated by out-of-distribution noise. Second, many SR models focus on noise simulation and transfer. Nevertheless, the sampled noise is monotonous and limited. To address the aforementioned problems, we propose a Negatives augmentation strategy for generalized noise modeling in Video Super-Resolution (NegVSR) task. Specifically, we first propose sequential noise generation toward real-world data to extract practical noise sequences. Then, the degeneration domain is widely expanded by negative augmentation to build up various yet challenging real-world noise sets. We further propose the augmented negative guidance loss to learn robust features among augmented negatives effectively. Extensive experiments on real-world datasets (e.g., VideoLQ and FLIR) show that our method outperforms state-of-the-art methods with clear margins, especially in visual quality.
Keywords: real-world video super-resolution, noise sequences sampling, negative augmentation, augmented negative guidance

The overview of the proposed NegVSR. (a) Our approach initially extracts noise sequence N_sq through window sequence C in an unsupervised manner. The motion of C occurs within the OOD video noise dataset V_od. Subsequently, it mixes N_sq and LR video V_lr to create novel training input V_lr^N. (b) V_lr^N is applied with a patch-based random central rotation to derive V_neg. (c) Both V_neg and V_lr are fed into the VSR model to generate Ŷ and Y, respectively. And L_Aug-P enables the model to recover more effective pixels from the V_lr. L_Aug-N drives Y to learn the robust features present in the negative output Ŷ.

Method

The figure depicts the process of our Augmented Negative Guidance approach. We obtain the positive output Ŷ by passing V_hr sequential through the degeneration model D and VSR. Then we inject noise sequence N_sq into the degraded video and apply the video with negative augmentation. Finally, we encourage the model to learn robust features from the augmented noise and video by L_Aug-N and L_Aug-P.

Results: compare with other methods

More details: