Deeplearning method for single image dehazing based on HSI colour space

CHEN Yong， TAO Meifeng， GUO Hongguang

(School of Electronics and Information Engineering, Lanzhou Jiaotong University, Lanzhou 730070, China)

Abstract： The traditional single image dehazing algorithm is susceptible to the prior knowledge of hazy image and colour distortion. A new method of deep learning multi-scale convolution neural network based on HSI colour space for single image dehazing is proposed in this paper, which directly learns the mapping relationship between hazy image and corresponding clear image in colour, saturation and brightness by the designed structure of deep learning network to achieve haze removal. Firstly, the hazy image is transformed from RGB colour space to HSI colour space. Secondly, an end-to-end multi-scale full convolution neural network model is designed. The multi-scale extraction is realized by three different dehazing sub-networks: hue H, saturation S and intensity I, and the mapping relationship between hazy image and clear image is obtained by deep learning. Finally, the model was trained and tested with hazy data set. The experimental results show that this method can achieve good dehazing effect for both synthetic hazy images and real hazy images, and is superior to other contrast algorithms in subjective and objective evaluations.

Key words： image processing; image dehazing; HSI colour space; multi-scale convolution neural network

0 Introduction

Haze is a common natural weather phenomenon. A large number of tiny water droplets are suspended in the air under haze weather. The scattering and refraction of these water droplets will lead to the gray-white images, and the colour saturation and contrast of the image will decrease, thus losing a lot of important details. Hazy images are not conducive to the extraction of image features, and the difficulty of subsequent image processing is increased, which makes the application of various security monitoring systems and target detection systems extremely limited. Therefore, it is of great practical significance and great application prospects to study how to effectively process degraded images in hazy scenes, and to restore degraded atmospheric images and enhance the details of scenery information[1-2]. In addition, with the deepening of the research in the field of computer vision, people have higher requirements for image sharpness collected by imaging equipment, and sharpness of hazy image has become an important research content of computer vision[3-8].

Among the numerous dehazing methods, there are mainly three kinds of methods.

The first kind of method is image enhancement based on non-physical model[9-15]. This kind of method does not consider the fundamental reason of image degradation. It mainly enhances the contrast and saturation of hazy image by image enhancement processing to highlight the valuable information in the image, so as to improve the quality of the image. This method is not a real method of haze removal, compared with the method based on Retinex theory, the method based on wavelet decomposition transform and the method based on histogram equalization.

The second kind of method is image restoration based on physical model[16-24]. This method is based on atmospheric scattering physical model, and completes parameter estimation in the process of inversion of physical model, so as to achieve the clarity of hazy image. For example, the dark channel prior method proposed by He et al.[20-21]solved the atmospheric scattering model by using the dark channel prior information in hazy images, and achieved a good dehazing effect. However, the algorithm used soft matting method to refine the transmission and had a large amount of computation. Rasmita et al.[22]proposed an improved dark channel prior dehazing algorithm considering edge protection filtering. The edge information of the dehazed image was improved, but the colour distortion still occurred in the sky with higher brightness. Deepa et al.[23]estimated the transmission by combining the central filter with the dark channel. This method can preserve the detail information of the image better, but it has the problem of colour distortion of dehazed image. Meng et al.[24]proposed a dehazing method based on boundary constraints. The dehazed image was obtained by sacrificing part of the boundary information. This method solved the problem of low brightness of the dehazed image, but colour distortion still occurred in some areas of the dehazed image. Although the traditional image haze removal methods have made great progress, the above methods mostly rely on a variety of prior information or hypothetical conditions, and extract the relevant features of haze manually. In some methods, the hypothetical conditions can not be established in many cases, so there are some limitations.

The third kind of method is image dehazing algorithm based on deep learning, which utilizes deep learning to estimate transmission map directly from a single hazy image[25-27]. In summary, in view of the dependence of traditional dehazing algorithms on prior information and the loss detail information of dehazing images by deep learning algorithm, a single image dehazing method based on HSI (hue, saturation and intersity) colour space is proposed in this paper, which learns the mapping relationship between hazy images and clear images to realize image dehazing. In the process of model building, according to the mechanism of atmospheric scattering model forming hazy image, a multi-scale full-convolution neural network model is designed, which includes three dehazing sub-networks: H, S and I. Firstly, the hazy image is transformed from RGB colour space to HSI colour space and decomposed into H sub-image, S sub-image and I sub-image. Then, the mapping relationship between hazy image and clear image is obtained by paralleling extraction of three dehazing sub-networks. Then, the sub-images of H, S and I obtained from the model are merged and inversely transformed to obtain clear haze-free images. Compared with the experimental results of the other algorithms, the proposed method can achieve good haze removal effect.

1 Basic principles

1.1 Atmospheric scattering model

In computer vision and computer image, atmospheric scattering physical model is usually used to simulate the degradation process of hazy image[28-29]. The model considers that the main reasons for the degradation of the observed image are the background light formed by the scattering of the sunlight and other environmental light by the scattering medium in the atmosphere, and the absorption of the suspended particles in the atmosphere and the scattering of the object itself which lead to the decrease of the brightness and contrast of the imaging system. The obtained image is blurred, thus forming a hazy image. The spatial representation of the physical model of atmospheric scattering is shown in Fig.1.

Fig.1 Physical model of atmospheric scattering

The model describes the degradation mechanism of hazy images, and can be expressed as

I(x)=J(x)t(x)+A[1-t(x)],

(1)

whereI(x) is the observed intensity;J(x) is the scene radiance;Ais the transmission of the medium; andt(x) is the global atmospheric light value. In addition,t(x) can be expressed as

t(x)=exp[-βd(x)],

(2)

whereβis the extinction coefficient of the medium;d(x) is the distance from the object to the imaging system, which is the depth of the scene.

Deforming Eq.(1), it could be expressed as

(3)

In Eq.(3), the known condition is hazy imageI(x), and haze-free imageJ(x) needs to be obtained. According to the basic algebraic knowledge, both transmission and global atmospheric light value can be changed simultaneously. This is an equation with innumerable solutions.

1.2 HSI colour space

As colour model or system, colour space is used to describe colour in a generally acceptable way under certain standards. There are many methods to describe colour images, such as RGB (red, green and blue), CMYK, HSI and so on. RGB is a hardware-oriented colour model. In RGB model, each colour appears in the spectral components of red, green and blue primary colours. At this time, the colour image is represented by RGB three colours. When RGB light of various intensities is mixed together, various colours will be produced. These colours are called primary colours of mixed colours. HSI colour model describes colour characteristics with three parameters: H, S and I. H defines the wavelength of colour; S represents the depth of colour; I represents intensity or brightness. RGB model can adapt to the fact of colour very well, but it can not adapt to the human interpretation of colour very well. It reflects the way that people’s visual system perceives colour. This colour description is intuitive and natural for people. Generally speaking, RGB model is more suitable for image colour generation. For example, when three primary colour components of RGB are fed into RGB monitor, the three images are mixed on the screen to generate a composite colour image. HSI model is suitable for describing images. HSI model fully reflects the basic attributes of human perception of colour, and corresponds to the results of human perception of colour one by one. In hazy environment, haze and dust will blur people’s vision, and greatly reduce the visibility of the scenery. There are some problems in hazy images, such as hue distortion, saturation reduction and low brightness. It’s hardly to reflect the above characteristics by using RGB colour space model. HSI colour space model is more in line with the characteristics of hazy scenery observed by people. For the above reasons, HSI colour space model is used to conduct deep learning haze removal for hazy images in this paper.

1.3 Convolutional neural network

Convolutional neural network (CNN) is a kind of feedforward neural network with deep structure and convolution calculation, which is a typical deep learning. CNN is essentially a multi-layer perceptron, which uses local connection and weight sharing. On the one hand, it reduces the number of weights of traditional neural networks, which makes the network easy to optimize. On the other hand, it reduces the complexity of the model and the risk of over-fitting. It can directly input images into the network, avoiding the complex process of feature extraction and data reconstruction in traditional image processing algorithms.

In recent years, with the rapid development of deep learning and its wide application in image processing, more and more researchers use deep learning to deal with image dehazing. For example, Tang et al.[25]restored haze-free images by studying the dark channel priority, maximum contrast and colour attenuation prior of hazy image in random forest. However, when the number of decision trees in the haze removal algorithm is large, the training complexity is large. In addition, the model of haze removal in random forest is easy to fall into over-fitting problem.Cai et al.[26]proposed an end-to-end dehazing method based on deep learning. The haze features were learned through feature extraction, multi-scale mapping and maximum pooling operations, and the transmission map of hazy images was obtained to achieve the clarity of hazy images. Ren et al.[27]learned the mapping relationship between hazy image and its corresponding transmission by establishing coarse-scale and fine-scale network models. Due to the max-pooling operation in the network model of the deep learning image dehazing algorithm mentioned above, it is easy to cause the loss of some details in the transmission learning process, which will affect the dehazing effect.

2 Proposed method

In view of the loss detail information of dehazing images by deep learning algorithm, a single image dehazing method based on HSI colour space is proposed in this paper, which directly learns the mapping relationship between hazy image and clear image in the three channels of hue, saturation and brightness, and overcomes the dependence of traditional image dehazing algorithm on prior knowledge.

2.1 Design of network model

In order to overcome the restriction of prior knowledge in traditional atmospheric scattering model dehazing algorithm, a multi-scale convolution neural network dehazing model based on HSI colour space is proposed in this paper, which takes full account of the relationship between hazy images and HSI channels of clear images. Firstly, hazy images are transformed from RGB colour space to HSI colour space and decomposed into three sub-images. Then, multi-scale extraction is carried out through three different sub-networks of H, S and I. The mapping relationship between hazy images and HSI channels of clear images is obtained by deep learning. Finally, the sub-images of H, S and I obtained by model learning are merged and inversely transformed into RGB colour space, and the clear image after haze removal is obtained. The model structure of multi-scale convolution neural network dehazing network based on HSI colour space is presented in this paper as shown in Fig.2.

Fig.2 Deep learning dehazing network model diagram based on HSI colour space

Traditional convolutional neural networks usually contain pooling layers, which compresses feature maps by Max-pooling or Mean-pooling. This operation may result in information loss, while image dehazing needs to minimize the loss of detail information in feature learning process. Therefore, the full convolution method is adopted to avoid information loss in feature learning caused by pooling operation. The model input is H sub-image, S sub-image and I sub-image of hazy image, and output is H sub-image, S sub-image and I sub-image of clear image obtained by training and learning.

In this paper, convolution kernels of different scales are used to extract the features of hazy images according to their H, S and I characteristics in HSI colour space deep learning dehazing model. The choice of convolution kernel size combines the characteristics of hazy image imaging. From the hazy image formation mechanism, it can be found that hazy environment has a greater impact on image colour distortion. In order to extract the fine features of mist image colour distortion information, a smaller 3×3 convolution kernel is used in the model’s H-channel haze removal sub-network to extract the features, which can extract more mist image colour feature information. For the overall feature of local whiteness in hazy images, the sensitivity field of convolution kernel feature extraction can be increased. Therefore, a relatively large 5×5 convolution kernel is used to extract the feature of local whiteness in hazy images in the S dehaze sub-network. For the problem of low overall brightness in the hazy image, a larger 7×7 convolution kernel is used to obtain the characteristics of the overall brightness of the hazy image.

When the model was established in this paper, convolution kernels of size 3×3 were used for each layer of the H dehazing sub-network, and the number was set as 10. Multiple smaller 3×3 convolution kernels could be used to extract more refined colour feature information of hazy image. The S dehazing network adopts the full convolution method, and gradually extracts the features of haze saturation by means of convolution kernel with the size of 5×5 in 4 consecutive layers. Finally, the convolution kernel with the size of 1×1 is used for nonlinear regression to obtain the saturation feature map of the hazy image. The I dehazing network also adopts the full convolution method to gradually extract the brightness features of the hazy image by means of convolution kernel with a continuous size of 7×7 for 4 layers. At last, the convolution kernel with a size of 1×1 is used for nonlinear regression to obtain the brightness features of the hazy image. In the dehazing model proposed in this paper, convolution kernels of different sizes are used in H, S and I dehazing sub-networks for parallel convolution operation to extract the feature information of hazy image. The convolution kernels are 3×3, 5×5 and 7×7, respectively. Specific multi-scale feature extraction convolution kernel parameters at each layer of HSI neural network are shown in Table 1.

Table 1 Multi-scale feature extraction parameter table

In order to ensure that the size of the feature map after convolution does not change with that of the original image, zero filling operation is used in convolution. The 10 convolution kernels in each layer are operated on the input image through local connection and weight sharing to realize feature learning. The convolution equation is

(4)

(a) ReLU activation function

ThePReLUfunction expression is

PReLU=max(xi,0)+aimin(0,xi),

(5)

wherexiis the positive interval input signal of layeri;aiis the weight coefficient of the negative interval in the layeri, which is set as 0 inReLU.

However, as a learnable parameter inPReLU, it overcomes the shortcoming of “feature death” of theReLUactivation function. The activation function in the multi-scale deep learning model based on HSI colour space designed in this paper adoptsPReLUfunction. In the dehazing network model in this paper, the hazy image is firstly converted from RGB colour space to HSI colour space, and then three multi-scale dehazing networks of hue, saturation and brightness are respectively used to learn the characteristics of hue, saturation and brightness of the hazy image. Finally, the inverse transformation of HSI sub-images obtained by the model is carried out to obtain the clear RGB image after haze removal.

2.2 Loss function

In this paper, the deep learning dehazing network model based on HSI colour space is based on the parallel deep learning of three different sub-networks to obtain the mapping relationship between hazy images and clear haze-free images. Mean square error is used as a loss function to realize the training of the model by minimizing the loss function. Loss function can be expressed as

(6)

whereI(xi) is the clear haze-free image learned by the network model;J(xi) is the clear haze-free image label corresponding to the data set;nis the number of training samples. The back propagation algorithm and the gradient descent method of 0.9 momentum are used to minimize the loss. The model training parameter batchsize is 64, the initial learning rate is 0.001, and the decay rate is 0.99 after every 10 rounds, the iteration period epoch is 100 times.

In order to enhance the generalization ability of the model, it is necessary to avoid the over-fitting phenomenon. Therefore, dropout method is used in the training process of the network model, and the probability of randomly discarding neurons is set as 0.5.

2.3 Algorithm steps

The steps of HSI colour space deep learning dehazing algorithm proposed are as follows:

1) The deep learning dehazing model training based on HSI colour space established in this paper is adopted to obtain the mapping relationship between hazy image and clear image;

2) Testing the network model, inputing the hazy image and getting the corresponding clear haze-free image, so as to realize the recovery of haze-free image. The algorithm steps are shown in Fig.4.

Fig.4 Algorithmic steps

3 Results and discussion

In order to verify the effectiveness of the method proposed in this paper, experiments are carried out on synthetic hazy images and real hazy images. The results of haze-removal are compared with the existing methods[20,24,26-27]. The experiments are done under the Ubuntu 18.04 system, using Python programming language and Tensorflow deep learning framework to build the network and implement the algorithm in this paper. The hardware environment was Intel (R) Core (TM) i7-9700K CPU @3.60 GHz, 16.0 GB RAM, NVIDIA GeForce GTX 1660. The comparison experiments were completed under Windows 10 system with the same hardware configuration environment.

3.1 Training dataset

In this paper, the OTS (outdoor training set) sub-set of RESIDE-beta dataset of hazy image is selected to train and test the network model[30]. OTS dataset collected 2 061 real complex outdoor scenes and corresponding depth maps, including 72 135 synthetic outdoor hazy images and standard clear images in the same scene. In the training process, 70% of the data in OTS datasets are selected as training set and 30% of that are used as test set.

3.2 Experimental results of synthetic hazy images

In the experiment of synthetic hazy images, some experimental results are shown in Fig.5.

The results show that the method proposed in the Ref.[20] can effectively remove most of the haze in the images, but the overall brightness of the images after haze removal is low and the visual effect is not good. The brightness of the haze removal images in Ref.[24] is higher than that in Ref.[20], but some regions have different degrees of colour distortion, such as the floor part in the third image and the wall part in the fifth image as shown in Fig.5(d). The deep learning method in Ref.[26] has an overall better dehazing effect than the traditional dehazing method in Refs.[20] and [24], but there is a phenomenon of incomplete dehazing, such as the red area in the fourth image and the pillow area in the fifth image shown in Fig.5(e). The overall effect of the method proposed in Ref.[27] is good, but there is also the problem of incomplete dehazing in some areas, such as the cabinet area in the first image and the table and chair area in the second image shown in Fig.5(f). Compared with other dehazing methods, the method proposed in this paper has better dehazing effect and higher image quality.

Fig.5 Comparison of the fourth group of experimental results

Since subjective judgment cannot prove the validity of the method in this paper, structural similarity index measurement(SSIM) and peak signal-to-noise ratio (PSNR) are further selected for data analysis of experimental results[31].

SSIM is an index to measure the similarity of two images. Comparing the dehazing image with the real standard haze-free image, the higher the value of SSIM is, the smaller the distortion degree of dehazing image is. PSNR is an important index to measure the image quality, and it is the ratio of the maximum semaphore to the noise intensity. The larger the PSNR value is, the closer the dehazing image is to the standard haze-free image. Data analysis results of the synthetic hazy images is shown in Table 2.

Table 2 Analysis results of synthetic hazy images

It shows that the PSNR and SSIM values of the dehazing image in this paper are both higher than those in the Refs.[20,24,26,27]. The dehazing image in this paper is closer to the standard haze-free image with relatively high recovery quality, which verifies the effectiveness of the proposed method.

3.3 Experimental results of real hazy images

In order to verify the validity of the model in restoring real hazy images, 6 real outdoor hazy images are selected for haze removal experiments. The experimental results are shown in Fig.6.

Fig.6 Experimental results of real hazy images

As can be seen from Fig.6, for the method in the Ref.[20], when there is no sky area or a small sky area in the hazy image, the method can achieve good dehazing effect, but the brightness is low, which affects the overall visual effect. When there is a large area of sky in hazy image, colour distortion occurs in the sky part of hazy image, such as the sky part in the sixth image shown in Fig.6(b). The brightness of Ref.[24] is higher than that of Ref.[20], but colour distortion still occurs when there is a large area of sky in hazy image, such as the sky region of the first image shown in Fig.6(c). Compared with the traditional haze removal methods in Refs.[20] and [24], the deep learning method in Ref.[26] has no sky distortion, but the haze removal is not complete in the far area of depth of field. As shown in Fig.6(d), haze still remains in the red area of the fourth image. The overall effect using the method in Ref.[27] is better than that in Ref.[26], but there is still residual haze in some areas, such as the residual haze in the automobile area in the sixth image shown in Fig.6(e). The method proposed in this paper effectively removes the haze of the image without sky distortion. The overall brightness is moderate, and the visual effect is great.

Similarly, in order to further verify the validity of the model in restoring real hazy images, information entropy (IE) and average gradient (AG) are selected to compare and analyse the experimental results. IE reflects the amount of information contained in an image. It is an important index to measure the richness of image information. The larger the IE value is, the clearer the image is. AG reflects the rate of contrast change in minute details of the image, and represents the relative clarity of the image. The larger the average gradient and the more the image level, the clearer the image will be. The experimental data of hazy outdoor images are analysed as shown in Table 3.

Table 3 Analysis of experimental data of real hazy images

From the data in the table, it can be seen that the IE value of the dehazing image in this method is higher than other methods. In terms of AG index, the AG value of the third dehazing image in this paper is slightly smaller than that of Ref.[24]. Data comparison shows that this method can obtain clearer dehazing images, which verifies the effectiveness of this method for natural real hazy images.

4 Conclusions

In this paper, a single image dehazing method based on HSI multi-scale convolution neural network is proposed to solve the problem of poor stability of traditional dehazing methods, which is constrained by prior conditions, and relies on manual extraction of haze-related features, resulting in colour distortion and low contrast. Firstly, the hazy image is transformed from RGB colour space to HSI colour space. Then a multi-scale full-convolution neural network model with three sub-networks is designed. The hazy image and clear image are extracted by three different dehazing sub-networks: H, S and I. And the mapping relationship between them is obtained to restore the haze-free image. The network model is trained with hazy image dataset. Finally, the trained model is used to test hazy images in different scenarios, and clear haze-free images are obtained. The experimental results show that the proposed method overcomes the shortcomings of the traditional single image dehazing algorithm, such as being susceptible to the prior knowledge of hazy images and low contrast. It can effectively recognize the relevant features of haze, enhance the visual contrast, effectively obtain clear images, and improve the image quality and visual effect. It can be applied in practice, and provide a good basis for subsequent image processing. The next step is how to optimize the dehazing model to get better dehazing effect.