1 Introduction

The light disperses into the surroundings, or it gets absorbed by the fragments present in the airspace. The total amount of fragments in the region is proportional to the scattering of light. This scattering reduces the clarity and colour in that region. Different particles and the environment they are in impact the scattering. Such as humidity and particles like sulphur cause a hazier environment. Most of these particles are present due to air pollution from industry. Some exist because of the natural world, such as windblown dust and forest fire. This haze, coupled with smoke, dust and low light condition, makes the haze removal task particularly difficult. This haze makes visibility extremely difficult for both humans and computer vision. e.g. the analysis from Wang et al. [1] concluded, the presence of haze can drastically effects tourist arrival. For better and more effective accuracy, vision algorithms expect clean, clear, and visible input. Since real-world places have haze environment tasks like object detection [2], object segmentation [3] becomes difficult due to a lack of information in captured images. To get the best out of these algorithms, we need to process real-world data to have the minimum possible haze and other artefacts. We can fix this via either pre-processing, post-processing or both. Haze removal could be achievable in two ways physical and non-physical. The physical method analyses the atmospheric transmission that degrades the image and regenerates the image by inversing the existing transmission. The physical-based methods take the path of approximation and assumption about the prior condition. The non-physical-based techniques solve this problem through image enhancement. Popular approaches like Histogram equalization [4], logarithmic transformation [5], various filters [6–9] in the frequency domain and Retinex theory [10] are very robust and achieve remarkable results. These methods are not robust to the change and thus struggle with noise, different exposure settings and artefacts in the image. The core computer vision task needs clean and visible input to give a decent performance. Tasks like object detection, segmentation, and depth estimation deal with real-world data, these tasks struggle in performance when the images are hazy. We present a simple and miniature method to pre-process the input image, such that the performance of these tasks will improve.
Recently, Deep learning-based methods perform exceptionally on the image enhancement task. These methods provide robust solutions to haze removal better than the quaint algorithm results. In this paper, we remove haze from both indoor and outdoor images. Our methods fix the different difficulties in the real world of hazy images like noise, challenging light and image artefacts. We constructed U-Net-like CNN [11] with an encoder, bottleneck and decoder. Each of these consists of blocks of layers carefully designed for haze removal. These blocks consist of downsampling and upsampling layers with skip connections. We used a combination of loss functions to enhance different aspects of the images like the structural and perceptual quality of the image.
The outline of the paper is summarized as follows. In this Section 1, we described haze and the difficulties faced in the haze removal process. Also, we explained the goal of the suggested method. In Section 2, we will discuss existing literature, the popular methods and their approach to solving the haze removal problem. The proposed technique and details about the model architecture and implementation are in Section 3. Training structure and comparison of our results with different methods are illustrated Section 4. Finally in Section 6, we concluded the findings and summarised our approach of single image haze removal.

2 Related Work

We can categorise haze removal tasks into prior-knowledge and learning-based techniques. The prior-knowledge approaches take advantage of statistical hypotheses. In opposition, learning-based methods learn the nonlinear mapping in the haze-free and between the given haze-full image. This section, we will embrace the difference between these two techniques in detail. In the prior-based algorithms [9, 12], physical assumptions such as atmospheric scattering technique (ATM) [13–15] and statistical hypotheses are used to expect the outcome to be a haze-free image. In the prior-based methods [9, 12, 16–18], authors investigate the factors affecting the image and minimize those factors to get a haze-free outcome. The transmission map with colour attenuation prior is calculated to manipulate the discrepancy between RGB channels. However, these techniques only reach sub-optimal performance in haze removal. Other approaches include polarized filters, multiview images of the common scenery and prior-based hazy methods. He et al. demonstrated dark channels prior (DHCP) [19] to improve visibility in haze-full images. This approach struggles with edge structure and undergoes a halo effect [20]. Further approaches Lai et al. [4] proposed two prior-based transmission maps for the hazy scene, a multi-scale retinex-based algorithm [21] to estimate luminous detail in the image, colour attenuation prior (CAP) [12], and an HSV colourspace method to extract haze-related features.
In learning-based methods, the first few techniques estimated transmission maps using end-to-end CNN like Cai et al.[20] and Ren et al. [22] Zhang et al. [23] suggested an approach to get transmission maps and atmospheric light with atmospheric scattering to estimate haze-free images. A variety of methods used CNN [2, 18, 20, 24] to get a direct clean haze-free image. The result of the investigation from Chen et al. [25] using dilated convolutional for haze-free image reconstruction showed better unit performance than the previous existing methods.The approaches like AOD-net [26] analyzes the contradiction between the quaint atmospheric models and the haze removal techniques. Later Swami et al. [27] proposed conditional GAN to remove haze from the degraded hazy images. Now, researchers [28, 29] use an unpaired training approach for a variety of computer vision tasks. Various methods used this unpaired training technique some even used this for motion object segmentation [3]. Another application-specific task on haze removal includes Chen et al. [30] proposed nonuniform remote sensing technique for removing the haze. Zheng et al. [31] performed dehazing on a Ultra-high definition image using multi-guided bilateral learning. Illumination adjustment for haze [32], smoke and fire removal from a single input image. VRHI [33] is used for visibility restoration in hazy images by incorporating haze density model.
In GAN-based techniques [34], BPPnet [35] used a pyramid-based network that gives state-of-the-art results with only 20 pairs of training images. EDN-GTM [36] use dark channel prior in their approach with a guided transmission map that outperforms the traditional methodology. DW-GAN [37] used discrete wavelet transform with two-branch GAN. RI-GAN [38] with residual inception modules in both generator and discriminator. All of the above-mentioned methods works decently on haze removal task, but the GANs are computationally expensive and difficult to implement. Therefore, our model is CNN-based, easy to implement and computationally inexpensive.

3 Proposed Method

3.1 Colour space

We compared RGB and YCbCr colourspace as training inputs to our neural network. We determined that the YCbCr image representation works very well on image haze removal. The YCbCr colourspace is mainly used in image enhancement and image super-resolution tasks because it represents information better than RGB and is noise free compared to RGB colourspace. Recently, YCbCr image representation has shown exceptional outcomes in haze removal tasks. As Bianco et al. [39] showed in their experiment, the mean squared error (MSE) (4) is qualitatively better in YCbCr colourspace than RGB because atmospheric illumination in hazy images affects more Y channel more. A better MSE score in YCbCr colourspace is the direct reason for better peak-signal-to-noise ratio (PSNR). Therefore to evaluate the methodology, we will be using YCbCr colourspace as our primary image input representation.

3.2 Model overview

We prioritise performance, efficiency and simple architecture for our neural network model. The proposed technique is miniature and straightforward to conduct yet achieves better performance than the traditional methodology. We experimented with different approaches that will advance our expectations from our model. The proposed architecture is a U-Net-based formation of encoder, bottleneck and decoder. Each of the sub-formations is a set of cautiously selected blocks of layers that meet close to our prioritise.
Our end-to-end network’s encoder and decoder are composed of convolutional blocks. Each of these blocks contains manually designed layers. Such as the encoder is constructed of downsampling layers and convolutional layers with pre-defined pooling. Whereas the decoder is created using upsampling layers and convolutional layers. These blocks coupled with the skip connection make our network end-to-end. We avoided the batch-normalization layer because of its poor performance on small batches, expensive computation and dissimilarity between training and testing. We used weight normalization because of is computationally inexpensive and gives better weight initialization. We used parametric PReLU as our activation function since PReLU’s learn parameters on how activation should be permitted for negative values.The bottleneck is created with only convolutional layers with fixed weights and biases.

3.3 Model architecture

As outlined in Fig (1), the encoder is stacked with multiple convolutional blocks. Each of these blocks contains a set of convolutional layers, average pooling, max pooling, weight normalization and parametric ReLU. We downsampled the input by half i.e. the stride of two and raise the weight by a factor of two. The feature extracted in the block passed to the next block till they reach the bottleneck. The encoder is directly connected to the decoder via skip connection. Such that, the initially learned parameter could directly transfer to the decoder. The bottleneck is the only phase without any skip connection and sampling. The input and output size of this phase is equal and the weight is the same for each block.
The decoder is composed of downsampling convolutional blocks. The decoder has the same set of layers as the encoder except for the downsampling layers such as average pooling and max pooling decoder has an upsampling layer. we used stride two with decreasing weight by a factor of two. For upsampling, we used bilinear interpolation. The decoder is directly connected to encode via skip connections from which the decoder gets learned parameters from an encoder.

Ground-truth	Input	Our
(a)	(b)	(c)
(d)	(e)	(f)

3.4 Losses and metrics

The loss function associated with the trained model is stated as follows:

• Frequency-domain loss [43] ($L_{\mathit{FFT}}$) transforms output images into frequency signals. To increase sharpness and reduced noise and other artifacts. Let $m$, $n$ dimension of images $I_1$ and $I_2$ , where $K$ is scaling factor FFT stands for Fast Fourier Transform of Image $I$. The Fast Fourier transform is

  $$L_{\mathit{FFT}}=L^{\frac{m}{k}\times \frac{n}{k}}=\frac{k^2}{m\times n}{\left|\mathit{FFT}(I_1)-\mathit{FFT}(I_2)\right|}^{\frac{m}{k}\times \frac{n}{k}}$$

  (2)

  fast algorithm to compute $\mathit{DFT}$ i.e. Discrete Fourier Transform. By transforming images into frequency, we can learn to see the differences in higher dimensional region. By transforming image signals into a frequency domain for Fourier transform such that we can enhance the higher frequencies while keeping lower frequencies which will sharpen the images.

• Smooth $L_1$ loss ($L_1$) is amalgamation of $L_1$ and $L_2$ losses, when the respective absolute error is high it behaves like $L_1$ and when the absolute error is close to zero it behaves like an $L_2$. We used Huber loss [44] as smooth $L_1$ loss because of its similarity

  $$L_{\delta }\left(a\right)=\{\begin{array}{cc}\hfill \frac{a^2}{2}\hfill & \hfill \mathit{for}\left|a\right|\le \delta \hfill \\ \hfill \hfill \\ \hfill \delta \left(\left|a\right|-\frac{\delta }{2}\right)\hfill & \hfill \mathit{otherwise}\hfill \end{array}$$

  (3)

• Mean Square Error (MSE) is used as $L_2$ loss function as follows:
  

  $$L_2=\mathit{MSE}=\frac{1}{\mathit{mn}}\sum _1^m\sum _1^n{\parallel \widehat{I}(n,m)-I^{\ast }(n,m)\parallel }^2$$

  (4)

  Where, $\widehat{I}$ is predicted image and $I^{\ast }$ is ground-truth image.

• PSNR is used as a metric for image quality. The higher value indicates better quality. The unit of PSNR is dB. We calcuate the PSNR as following equation.

  $$\mathit{PSNR}=20{\log <!--\; FUNCTION\; APPLICATION\; -->}_{10}\left(\frac{\mathit{MA}{X}_{\widehat{I}}}{\sqrt{\mathit{MSE}}}\right)$$

  (5)

  Where, $\widehat{I}$ is predicted image based on MSE (4).

• Structural Similarity (SSIM) is used as a metric to improve the structural accuracy of predicted output.

  $$\mathit{SSIM}(x,y)=\frac{\left(2{\mu }_x{\mu }_y+C_1\right)\left(2{\sigma }_{\mathit{xy}}+C_2\right)}{\left({\mu }_x^2+{\mu }_y^2+C_1\right)\left({\sigma }_x^2+{\sigma }_y^2+C_2\right)}$$

  (6)

  whereas, ${\mu }_x$ the pixel sample mean of x , ${\mu }_y$ the pixel sample mean of y, ${\sigma }_x^2$ and the change in variance of x, ${\sigma }_y^2$ can effect the variance of y, ${\sigma }_{\mathit{xy}}$ which is associated with the covariance of x,

• Structural Similarity loss ($L_{\mathit{SSIM}}$) is used retain structural similarity of reconstructed image we minimized SSIM loss. The loss function is stated as below,

  $$L_{\mathit{SSIM}}=\frac{\left(1-\mathit{SSIM}\right)}{2}$$

  (7)

  where as SSIM is from (6)

4 Experiments

4.1 Dataset

To train the model, we incorporated two different datasets Realistic Single Image Dehazing (RESIDE) dataset [45] with total number 18,200 images for training are consider and additionally the Real-world Video Dehazing dataset (REVIDE-indoor) [46] with a total of 1697 images for training and 282 for testing. To infer the proposed model, we used various benchmarking datasets like Dense haze [41], NH haze [47], I haze [42] and O haze [40]. We split the training dataset into 80% training and 20% validation. In pre-processing, we resize input images into $128\times 128$ and change the colour space to YCbCr. We used batch size $64$ for all of the images.

4.2 Training setup

We used AdamW [48] rather than Adam optimizer, as we used weight normalization weight decay from AdamW will be a reasonable regularizer. The weights are initialized in uniform distribution as proposed by He et al. [49] initial biases were set to zero. The learning rate we consider for training the model is 0.0001 and the weight decay is $10^{-5}$. If the loss doesn’t change for 7 epochs, then dynamically the learning rate will be reduced by a factor of 2.

After testing both $L_1$ (3) and $L_2$ losses (4), we choose $L_2$ as our primary loss. The final loss function includes $L_2$ loss, Frequency domain loss (2), and structural similarity loss (7), and metrics include PSNR (5) and SSIM (6) as shown in Fig (3).

a) Loss	b) PSNR	c) SSIM

4.3 Comparing results with other methods

We computed metrics PSNR (5) and SSIM (6) on various real-world and synthetic images. We presented our results in Table. (1) and Table. (2) with a detail analysis with existing methods.

As exhibited in Table (1), the model performs near state-of-the-art in both PSNR and SSIM metrics on both datasetes [40, 42]. Whereas from Table (2), our model outperforms existing methods in both PSNR and SSIM with the Dense-haze dataset [41]. For the NH-Haze [47] dataset, our model outperforms existing models in SSIM and the PSNR difference between our model and the existing model is 1.31dB. In both cases, EDN-GTM [36] struggles in the SSIM metric, but our method gives a precise SSIM score because of the Structural similarity loss (7), see results in Fig (2).

5 Ablation study

We conducted an ablation study on I-Haze, O-Haze, Dense-Haze and NH-haze datasets. We evaluated ablation with loss components in Section 5.1 and colourspace image representation in training images in Section 5.2.

5.1 Loss ablation

In section 3.4, we used three types of losses to train our neural network. In this section, we will assess the result if we change the loss function $L_2$ with $L_1$, i.e. Huber loss. Our observation is presented in Table (3), which shows the qualitative comparison between these losses.

5.2 Use of YCbCr versus RGB space

In Section 3.1, we discussed the use of YCbCr colourspace over RGB. In this section, we will examine the qualitative results of both colourspace image representations for training the proposed network. Our findings reveal that the YCbCr colourspace images give slightly better results than the RGB colourspace images in most of the datasets. On some input images, RGB-based images were perceptually better than the YCbCr. Also, the metric-wise difference is diminutive for all datasets.

6 Conclusion

We presented the coherent architecture train on sufficiently large and diverse data that can give near state-of-the-art results on indoor, outdoor, non-homogenous, dense and synthetic haze environments on qualitative metrics PSNR and SSIM. Our model gives structurally enhanced output on poorly visible input images. We used a combination of loss functions to generate a haze-free image without colour degradation, noise and structural loss. Our method demonstrated by choosing adequate qualitative and quantitative data miniature and straightforward network can effective as the existing methods. In future, our model can be used to get haze-free images for vision tasks such as object detection, segmentation and human action recognition if needed. Also, we will improve our model on low-light hazy images and colour restoration from densely hazy images.

References