Detection of broken manhole coverusing improved Hough and image contrast

Zhang Fengyan　 Chen Rongbao　 Li Yang　 Guo Xiucheng

(1School of Transportation, Southeast University, Nanjing 210096, China)(2School of Electrical Engineering and Automation, Hefei University of Technology, Hefei 230009, China)(3Jiangsu Transportation Institute Co., Ltd., Nanjing 210017, China)(4Department of Civil Engineering, Anhui Communications Vocational and Technical College, Hefei 230051, China)



Detection of broken manhole coverusing improved Hough and image contrast

Zhang Fengyan1,3,4Chen Rongbao2Li Yang2Guo Xiucheng1

(1School of Transportation, Southeast University, Nanjing 210096, China)(2School of Electrical Engineering and Automation, Hefei University of Technology, Hefei 230009, China)(3Jiangsu Transportation Institute Co., Ltd., Nanjing 210017, China)(4Department of Civil Engineering, Anhui Communications Vocational and Technical College, Hefei 230051, China)

Abstract：The damage or loss of urban road manhole covers may cause great risk to residents’ lives and property if they cannot be discovered in time. Most existing research recommendations for solving this problem are difficult to implement. This paper proposes an algorithm that combines the improved Hough transform and image comparison to identify the damage or loss of the manhole covers in complicated surface conditions by using existing urban road video images. Focusing on the pre-processed images, the edge contour tracking algorithm is applied to find all of the edges. Then with the improved Hough transformation, color recognition and image matching algorithm, the manhole cover area is found and the change rates of the manhole cover area are calculated. Based on the threshold of the change rates, it can be determined whether there is potential damage or loss in the manhole cover. Compared with the traditional Hough transform, the proposed method can effectively improve the processing speed and reduce invalid sampling and accumulation. Experimental results indicate that the proposed algorithm has the functions of effective positioning and early warning in the conditions of complex background, different perspectives, and different videoing time and conditions, such as when the target is partially covered.

Key words：manhole cover; edge tracking; improved Hough transform; shape detection; image contrast

Received 2015-01-20.

Biographies：Zhang Fengyan(1972—),doctor, male, zhangleike@126.com; Guo Xiucheng (corresponding author), male, doctor, professor, seuguo@163.com.

Foundation item：The Natural Science Fundation of Education Department of Anhui Province (No.KJ2012B051).

Citation：Zhang Fengyan, Chen Rongbao, Li Yang, et al. Detection of broken manhole cover using improved Hough and image contrast[J].Journal of Southeast University (English Edition),2015,31(4):553-558.[doi:10.3969/j.issn.1003-7985.2015.04.021]

The manhole cover is essential to any city. There are typically a large number of manhole covers in a city which are difficult to manage and monitor. Once a manhole cover is damaged but not dealt with timely, it will bring great hidden dangers to the lives, property, or safety of people. Prasad et al.[1]used the edge curvature to determine the edge based on the traditional Hough transform method, which searches each edge profile in the area and makes a grouping calculation based on the conditions of the edge contour to determine the elliptical shape. However, this method is limited by the standard elliptical contour; therefore, it cannot be used on the shape detection of broken and damaged elliptical manhole covers. Kasemir et al.[2]proposed a detection algorithm for the elliptical shape in a high noise level image. The algorithm combines the improved Hough transform and the genetic algorithm. However, we need to design a novel algorithm that synthesizes the improved Hough transform and image comparison for solving different technical problems; so it is difficult to generalize this algorithm for the detection of the broken manhole covers. Wei et al.[3]developed a detection method on the incomplete elliptical image in strong noise environments based on the iterative randomized Hough transform (IRHT). The IRHT method iteratively applies the randomized Hough transforms, constantly adjusting the iterative parameter and repeatedly amplifying the target curve by using image space and parameter space. Due to the fact that the core of this method is the iterative computation, it is time and computer memory demanding, and thus its adaptability for moving vehicles is poor in dynamic environments. Much research has been done on the manhole cover, but the research mainly focuses on manufacturing intelligence, by which a timely alarm can be set if the manhole cover is damaged. Although this method can effectively monitor the manhole cover, the implementation process of these measures requires extensive human and material resources, which will be very expense. Since a large number of manhole covers need to be maintained, there will be a great resistance for the application of manufacturing intelligence[4]. In recent years, many cities have established municipal video monitoring networks. Therefore, using existing video surveillance networks, and the image acquisition and processing technology of computer vision, we develop a method to monitor the manhole covers and raise alarm in time when there are missing and broken manhole covers. This can achieve the aim of monitoring manhole covers without any hardware investment and repetitive construction.

Usually, very complex images can be taken by cameras with a complex background, but people are often interested in a particular region of the image with special characteristics, which is the target area. It is necessary to extract the target area from the original image to further identify and analyze the target. This paper focuses on identifying the manhole cover region with more clear geometric features and computational geometry characteristics. Hence, compared with other spatial invariance features, calculating geometric features is more simple and convenient[5], and with the help of the colour recognition algorithm, it can better meet the requirements of accuracy and real-time. Therefore, we select the geometry features and colors of the manhole cover as the positioning method to contrast the images and detect manhole cover breakage.

1Overall Scheme

The manhole cover in the image can be viewed approximately as the distortion of ellipses. The recognition of the manhole cover is to determine the region[4]and position of the manhole cover with edge curves calculated by the image processing algorithm. The Hough transform is not sensitive to image noise, and it easily conducts parallel processing and convenient for handling characteristics when images are covered locally. Therefore, it is more suitable for the elliptical extraction of the manhole cover image with complex backgrounds[5-6]. However, the traditional Hough transform has several limitations:

1) It requires a large amount of computation. For linear, the algorithm complexity isO(n2); for circle and ellipse, the algorithm complexities areO(n3) andO(n5), respectively.

2) It needs a large CPU memory.

3) The parameters extracted by quantization have interval constraints of parameter space, so it is difficult to meet high precision and real-time requirements.

Therefore, conducting an elliptical traditional Hough transform directly in the image with complicated background is not feasible[7]. In order to overcome the limitations of the traditional Hough transform, this paper first conducts the edge search to find the edges above all the image and stores them in the image list. Then we divide them into each sub image which is transformed by Hough. This can reduce the noise influence to a great extent and reduce invalid image accumulation. Also, the space of the Hough transform and time consumption are reduced to meet the system’s high precision and real-time requirements. Due to the interference from other elliptic objects[8-10], the road condition is very complex. So this paper combines the color recognition and image alignment algorithm to recognize and localize the manhole cover, which can greatly improve the identification accuracy of the manhole cover.

As shown in Fig.1, the method mainly consists of four parts: pre-processing, contour extraction, target localization and identification, and detecting suspected damage. In order to ensure an accurate extraction of the target region, before positioning, it is necessary to do some pre-processing operations such as brightness and contrast adjustment, smoothing, and denoising and edge detection. Then we search the edge, delete the useless contour, extract the useful contour, and recognize and localize the manhole cover by the improved Hough transform and the color recognition algorithm. Finally, the image matching algorithm is used to calculate the suspected discrimination by the change rate.

Fig.1　The process of the proposed method

2Identification and Localization of the Manhole Cover

Since images are affected by acquisition conditions such as illumination uniformity and the time period, we need to carry out image preprocessing, which mainly includes the adjustment of brightness, equalization and smoothing of the contrast histogram, the denoising method of mathematical morphology and the detection of Canny edge[11]. These preprocessing algorithms are classic approaches, which are not focused on in this paper.

2.1　Edge tracking

In order to analyze the shape of the collected image, we need to find edge points of the target and then connect the edge points into one continuous boundary for boundary tracking. As shown in Fig.2, the specific steps are as follows:

1) Set a discontinuous reference distance “gap”, establish a two-dimensional image arrayI(store two images after binaryzation) and a dynamic two-dimensional array “cur” (coordinate storage of edge point).

2) Scan a two-dimensional arrayIof an image from top to bottom and from left to right, save the point on the left side ofA0after locating the edge pointA0(x0,y0) in the array “cur”, and setA0(x0,y0) to correspond to the coordinate pointI(x,y)=0 in theIarray, showing that the point has been recorded.

3) Scan the local window [x-gap,x+gap]×[y-gap,y+gap] centered by coordinates (x,y) in the imageIfrom top to bottom and from left to right, and seek the nearest edge pointA1to point (x,y). IfA1exists, then recordA1for coordinates (x,y) and repeat Step 3), until no edge points can be found. Denote edge pointsAe(xe,ye). It should be noted that in Step 3) the tracking edge contour is carried out only in one direction.

4)A0(x0,y0) is regarded as the starting point coordinates. Set (x,y)=(x0,y0), repeat Step 3), and the end point coordinate is denoted asAs(xs,ys).

5) If there arenlines of continuous edge in the images, repeat Steps 2) to 4) and produce a closed edge point set. The corresponding edge point set is {cur(i),i=1,2,…,n}.

Fig.2　The schematic of edge tracking

2.2　Elliptic automatic recognition

Among a plurality of edge point set boundaries calculated by the edge tracking algorithm, there is the elliptic boundary of the manhole cover and other interfering boundaries. To distinguish the target boundary from the boundaries, we need to recognize the elliptical objects. The camera angle is different, so the manhole cover in the image will stretch and distort to a different elliptic according to a certain angle, that is, an oblique ellipse with a certain included angle existing in the direction of the main shaft and theXshaft. The standard equation of the ellipse is

(1)

In the actual situation, the axis of the elliptical image of the manhole cover is not parallel to the coordinate axes and can be approximately seen as translation and rotation.

Also, the major and minor axis of ellipse may not be parallel to the coordinate axes, which can be viewed as the translation and rotation of the coordinates. As shown in Fig.3, thexoyaxis is translated tox′o′y′, and thenx′o′y′is rotated tox″o″y″. The ellipse in the coordinatesx″o″y″ is finished. Supposing that the rotation angle isθ, the elliptic equations in thexoycoordinates can be written as

(2)

The expression of the short half axisbis

(3)

Fig.3　The schematic of elliptic rotation and translation

Based on contour tracking, each profile in the list is processed in turn. The steps are as follows:

1) For each profile in the list, initialize the horizontal and vertical coordinates of the initial and end points from the neighborhood control structure.

2) Do circulation processing of each pixel point in the current contour neighborhood supposing that the point is the center pointCof the ellipse. Calculate the Euclidean distance of every point in the current profile to pointC, find the maximum distance value, set it as the semi major axis of the elliptic, and preserve the corresponding pointP.

3) If the maximum distance value found in Step 2) is less than the specified threshold, then switch to Step 2) and treat the next pixel of neighborhood structure, otherwise, calculate the angle between pointPandCconnection andxaxis.

4) Do circular processing of every pixel in the current profile. According to Eq.(3), calculate the ellipse semi minor axisb, and plus 1 to the value of each pixel in the accumulator of the semi minor axis.

5) Carry out circular treatment for each element of the accumulator of the semi minor axis. If its value is between the upper and lower bounds of the prescribed threshold, set the currenta,b,C,θ, respectively, as the semi major axis, ellipse semi minor axis, center, rotation angle of coordinate value, to obtain all the parameters of elliptic equations, and add them to the list of ellipses.

6) Empty the semi minor axis accumulator array. If there are any pixels of the current neighborhood profile not disposed of, turn to Step 2), otherwise, turn to Step 7).

7) If there are some outlines left in the list, turn to Step 1), otherwise, the algorithm terminates.

Performing the Hough transform to each scattered contour field and reducing the processing range can improve the processing speed effectively, so the system can satisfy the real-time requirement[12].

2.3　The image matching algorithm

Although the improved Hough transform algorithm can recognize ellipses on the surface of the road, the average case is very complex due to many other interfering elliptic objects; therefore, the manhole cover cannot be distinguished from the elliptical interference. To solve this problem, this paper adopts the color recognition and image alignment algorithm after staining the manhole cover at the same time. Besides the improved Hough method, the comprehensive discrimination method combining these two algorithms can improve the accuracy of the recognition. At the same time, the image matching algorithm can distinguish between suspect broken or missing manhole covers by different rates[13].

For the image matching algorithm,it requires a stored image of the standard pavement of the manhole cover part as a standard reference, and then the image is pretreated. Since the detected pavement image and standard image are different, after pretreatment, a registration of the standard pavement image and the detected pavement image is required. In the process of detecting, compared with the input image and the standard image, if the difference is greater than a certain threshold, the manhole cover is suspected to be broken or missing. As shown in Fig.4, the image in Fig.4 is the manhole cover pavement image.

Fig.4　The process of inspecting manhole cover’s breakage and loss

Image registration is a fundamental task in image processing and is used to match two or more images acquired at different time, different sensors, different angles and different shooting conditions[13]. Considering the actual situation of the defective detection system, the difference between the collected detected image and standard image is rigid deformation, so we can use the registration method based on gray level information.

Assume that the standard reference image isR; the detecting image isS, the size ofRism×n, the size ofSisM×N, as shown in Fig.5. The basic process of the image registration method based on gray level information is as follows: superimpose the reference imageRon the imageSand translate; let the area covered by the reference image on the searching imageSto be the subgraph ofSij; (i,j) is the coordinates of the upper left corner of subgraph in the image on theS. The search range is

1≤i≤M-m, 1≤j≤N-n

(4)

Fig.5　The basic process of image registration method

The registration process is completed by analyzing the similarity betweenRandSij.

According to the similarity measure function, the registration methods based on gray level information can be classified into the cross-correlation algorithm, the maximum mutual information registration method etc. The cross-correlation algorithm is used to realize the registration of two images, which is the most basic image registration method based on gray level statistics. It requires the reference image and the matching image with a similar scale and gray information. The cross-correlation algorithm is used to match the selected region of the image on the reference template and to calculate the cross correlation coefficient for each position on the reference image and the matching image. The choice of the method is due to that before calculating the cross correlation coefficient of the reference image and the test image, a rotating operation is needed to ensure that the maximum cross correlation coefficient is achieved when the axes of two plates are parallel. To maintain an unchanged standard plate direction, we only move the test plate, which can reduce error and improve registration accuracy. The registration process is shown in Fig.6.

1) According to the preset step value, rotate the measured pavement image within a certain angle range (including the forward rotation and the backward rotation). After each rotation, conduct a cross correlation calculation of the selected area of two pictures.

2) Choose the corresponding rotation angle when the

Fig.6　The process of image registration by cross-correlation algorithm

cross-correlation coefficient reaches the maximum; modify the measured pavement image according to the angle and the axial of two images will become parallel.

3) Select some area on the revised image of the pavement surface, and then calculate the cross-correlation coefficient. This is mainly in order to obtain the corresponding position when the coefficient reaches its maximum.

4) Using the position of the maximum value, we can deduce the relationship between pixel values of the corresponding points on the two images. Through translation and a cutting operation, complete the registration.

Conduct the XOR Boolean calculation after matching the detecting manhole cover image with the standard image of the two values, and then create the outline of image change. After some morphological treatment, we can obtain the suspected damaged area of the manhole cover. We find the manhole cover area through the method of Hough and color recognition and the alarm will sound when the manhole cover area is beyond a certain threshold by calculating the suspected damage zone size in its region.

3Program Verification

A simulation platform was developed based on the Matlab. The experimental image is obtained by the color video camera. Figs.7 to 9 show the key steps of image processing. In each of Figs.7 to 9, the subfigure (a) is the original image of a manhole cover, subfigure (b) is a manhole cover image after dyeing, subfigure (c) is an image of the manhole cover after damage or loss, subfigure (d) is the preprocessed image of a manhole cover after damage or loss, subfigure (e) is an alarming manhole cover image of the suspected defect. Fig.7 shows a manhole cover with a small area of the damage; Fig.8 presents a manhole cover with a large area of damage; and Fig.9 shows the pictures of a completely lost manhole cover. It is seen that this system can accurately sound the alarm for manhole cover damage or loss in the above three situations.

4Conclusion

In this paper, the edge contour tracking algorithm is first used to find all the edges of broken manhole cover based on the pre-processing image. Then, the improved Hough transform is adopted to find all the elliptic regions

(a) (b) (c) (d) (e)

Fig.7The manhole cover with a small area of damage. (a) Original image; (b) Dyed image; (c) Image of damaged cover; (d) Preprocessed image; (e) Alarm of damage

(a) (b) (c) (d) (e)

Fig.8The manhole cover with a large area of damage. (a) Original image; (b) Dyed image; (c) Image of damaged cover; (d) Preprocessed image; (e) Alarm of damage

(a) (b) (c) (d) (e)

Fig.9The completely lost manhole cover. (a) Original image; (b) Dyed image; (c) Image of damaged cover; (d) Preprocessed image; (e) Alarm of damage

and the color recognition algorithm is adopted to distinguish the manhole cover area from the elliptic regions. The proposed synthesis algorithm can not only eliminate the interference from other elliptic objects, but also exclude the interference of other color approximate regions. Therefore, it can accurately determine the manhole cover area. After the manhole cover area is identified, the damage rate can be calculated by the image matching algorithm. Whether the change rate reaches the threshold value can be used to determine whether it is a suspected damaged manhole cover or not. This synthesis algorithm can effectively improve the processing speed of the traditional Hough transform and reduce inefficient sampling accumulation. The experimental results show that the synthesis algorithm has a good early warning effect in complex backgrounds from different perspectives, and in different videoing time and conditions, such as when the target is partially covered.

References

[1]Prasad D K, Leung M K H, Cho S Y. Edge curvature and convexity based ellipse detection method[J].PatternRecognition, 2012, 45(9): 3204-3221.

[2]Kasemir K U, Betzler K. Detecting ellipses of limited eccentricity in images with high noise levels[J].ImageandVisionComputing, 2003, 21(2): 221-227.

[3]Wei H, Tan J L. Detection of incomplete ellipse in images with strong noise by iterative randomized Hough transform (IRHT) [J].PatternRecognition, 2007, 41(4): 1268-1279.

[4]Chen Ying, Xiong Zhang, Wang Yunhong. Location and target detection of manhole cover based on improved Hough[J].OpticalTechnique, 2006, 8(4): 463-467. (in Chinese)

[5]Liu Jianhua. Automatic location and recognition of manhole cover based on driving on the road with sensor[J].ApplicationResearchofComputers, 2011, 28(8): 3138-3140. (in Chinese)

[6]Wang Zheshen, Li Cuihua. Building object search and recognition based on improved Hough[J].JournalofImageandGraphics, 2005, 10(4): 463-468. (in Chinese)

[7]Zhou Xiang, Kong Xiaodong, Zeng Guihua. A new ellipse contour detection method based on Hough[J].ComputerEngineering, 2007, 33(16): 166-171. (in Chinese)

[8]Li Ziqiang, Teng Hongfei. Generalized Hough fast randomized oval detection[J].JournalofComputer-AidedDesign&ComputerGraphics, 2006, 18(1): 27-33. (in Chinese)

[9]Chen L, Chen H J, Pan Y, et al. A fast efficient parallel Hough algorithm on LARPBS[J].TheJournalofSuperComputing, 2004, 32(5): 759-762.

[10]Ou Xiangyin. Image identification and feature point calculation based on Hough to the known figure[D]. Changchun: School of Computer Science and Technology, Jilin University, 2005. (in Chinese)

[11]Chen R B, Wei H, Li Y, et al. Recognition of lanes on complex roads based on video image[J].AdvancedMaterialsResearch, 2014, 950: 298-305.

[12]Li Q, Xie Y. Randomised Hough transform with error propagation for line and circle detection[J].PatternAnalysis&Applications, 2003, 6(1): 55-64.

[13]Yang Fan.Digitalimageprocessingandapplication(Matlab) [M]. Beijing: Chemical Industry Press, 2003. (in Chinese)

doi:10.3969/j.issn.1003-7985.2015.04.021