AWRaCLe : All-Weather Image Restoration using Visual In-Context Learning

Sudarshan Rajagopalan, Vishal M. Patel
Johns Hopkins University
Paper Arxiv Code (Coming Soon!)

Illustration of AWRaCLe: Our visual in-context learning approach for all-weather image restoration. Given a context pair (first two rows), AWRaCLe extracts relevant degradation context from it to restore a query image. Our method also performs selective removal of haze and snow from an image containing their mixture as shown in (d) and (e).

Abstract

We tackle the challenging task of All-Weather Image Restoration (AWIR) under adverse weather conditions. Prior works do not utilize degradation-specific visual contextual guidance during training. Consequently, their performance is limited by the degradation cues that are learnt from individual training samples. We propose All-Weather Image Restoration using Visual In-Context Learning (AWRaCLe), a novel approach for AWIR which leverages degradation-specific visual context information as a prior to enhance the restoration performance. Our approach incorporates Degradation Context Extraction (DCE) and Context Fusion (CF) to seamlessly integrate degradation-specific features from the context into an image restoration network. The proposed DCE and CF modules leverage CLIP features and incorporate attention mechanisms to adeptly learn and fuse contextual information. AWRaCLe achieves state-of-the-art performance on standard all-weather image restoration benchmarks.

Proposed Approach

AWRaCLe integrates degradation-specific information from a context pair to facilitate the image restoration process. Initially, CLIP features are extracted from the context pair and fed into Degradation Context Extraction (DCE) blocks at various levels of the decoder within the image restoration network. The Context Fusion (CF) blocks then fuse the degradation information obtained from the DCE blocks with the decoder features of the query image requiring restoration. Finally, the restored image is generated.

Results and Comparison

Quantitative comparisons of AWRaCLe with state-of-the-art approaches on all-weather image restoration benchmarks. AWRaCLe achieves a significant improvement in performance over SOTA.

Qualitative comparisons of AWRaCLe with state-of-the-art-approaches for dehazing, desnowing and deraining tasks.

Analysis

DCE block activations overlayed on the degraded and clean image of the context pair. The DCE blocks capture attributes of degradations such as the spatially varying characteristics of haze and sparseness of snow. Yellow-High, Blue-Low

t-SNE plot of DCE block outputs for hazy, rainy and snowy context pairs. The clusters are well separated.

Comparison of activations of the restoration network prior to CF and after CF. After CF, the degradation information in the activation is significantly enhanced.

BibTeX

@article{rajagopalan2024awracle,
      title={AWRaCLe: All-Weather Image Restoration using Visual In-Context Learning}, 
      author={Sudarshan Rajagopalan and Vishal M. Patel},
      year={2024},
      eprint={2409.00263},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2409.00263}, 
}
Acknowledgement: The website template is taken from Nerfies