GenDeg

Abstract

Deep learning–based models for All-In-One image Restoration (AIOR) have achieved significant advancements in recent years. However, their practical applicability is limited by poor generalization to samples outside the training distribution. This limitation arises primarily from insufficient diversity in degradation variations and scenes within existing datasets, resulting in inadequate representations of real-world scenarios. Additionally, capturing large-scale real-world paired data for degradations such as haze, low-light, and raindrops is often cumbersome and sometimes infeasible. In this paper, we leverage the generative capabilities of latent diffusion models to synthesize high-quality degraded images from their clean counterparts. Specifically, we introduce GenDeg, a degradation and intensity-aware conditional diffusion model, capable of producing diverse degradation patterns on clean images. Using GenDeg, we synthesize over 550k samples across six degradation types: haze, rain, snow, motion blur, low-light, and raindrops. These generated samples are integrated with existing datasets to form the GenDS dataset, comprising over 750k samples. Our experiments reveal that image restoration models trained on GenDS dataset exhibit significant improvements in out-of-distribution performance as compared to when trained solely on existing datasets. Furthermore, we provide comprehensive analyses on implications of diffusion model-based synthetic degradations for AIOR.

Out-of-distribution performance of image restoration models when trained solely using existing datasets and our proposed GenDS dataset. Significant improvements can be observed across all degradations. Metric values reduce outward.

Motivation

Existing datasets are limited in size and lack scene diversity. Our GenDS dataset overcomes these challenges, as demonstrated in (a). Furthermore, the degradation patterns in GenDS dataset are highly diverse, bridging the gap between the degradation patterns in existing datasets and OoD datasets, as illustrated in (b).

Proposed Approach

Illustration of our GenDeg framework: (a) Describes the training stage of our pipeline. (b) Shows the inference process for degradation generation. (c) Depicts the architecture of our Swin transformer-based restoration model.

Quantitative Results

Quantitative comparisons of restoration models using LPIPS and FID metrics (lower is better), trained with and without our GenDS dataset. Performance is evaluated on OoD test sets. The table also includes the performance of existing state-of-the-art (SOTA) approaches. Training with the GenDS dataset significantly enhances OoD performance. (R) indicates real images and (S) indicates synthetic images.

Qualitative Results

Qualitative comparisons of image restoration models trained with and without our GenDS dataset. The suffix GD represents training with the GenDS dataset. Zoomed-in patches are provided for viewing fine details.

Analysis

(a) Shows the effect of scaling number of synthetic samples augmented with real data on OoD performance (LPIPS and FID). (b) The table presents LPIPS/FID scores for analyzing the performance difference between training on solely existing data, solely GenDeg data, and both real and GenDeg data (GenDS data).

BibTeX

@misc{rajagopalan2024gendegdiffusionbaseddegradationsynthesis,
      title={GenDeg: Diffusion-Based Degradation Synthesis for Generalizable All-in-One Image Restoration}, 
      author={Sudarshan Rajagopalan and Nithin Gopalakrishnan Nair and Jay N. Paranjape and Vishal M. Patel},
      year={2024},
      eprint={2411.17687},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2411.17687}, 
}

GenDeg: Diffusion-Based Degradation Synthesis for Generalizable All-in-One Image Restoration