Real-time vehicle tracking for traffic monitoring systems①

Hu Shuo (胡 硕):Zhang Xuguang:Wu Na

(*Key Laboratory of Industrial Computer Control Engineering of Hebei Province:Institute of Electrical Engineering,Yanshan University:Qinhuangdao 066004:P.R.China) (**Key Laboratory of Measurement Technology & Instrumentation of Hebei Province:Institute of Electrical Engineering,Yanshan University:Qinhuangdao 066004:P.R.China)



Real-time vehicle tracking for traffic monitoring systems①

Hu Shuo (胡 硕)*To whom correspondence should be addressed.E-mail:hus@ysu.edu.cnReceived on Sep.7，2015*:Zhang Xuguang*:Wu Na**

(*Key Laboratory of Industrial Computer Control Engineering of Hebei Province:Institute of Electrical Engineering,Yanshan University:Qinhuangdao 066004:P.R.China) (**Key Laboratory of Measurement Technology & Instrumentation of Hebei Province:Institute of Electrical Engineering,Yanshan University:Qinhuangdao 066004:P.R.China)

A real-time vehicle tracking method is proposed for traffic monitoring system at road intersections:and the vehicle tracking module consists of an initialization stage and a tracking stage.License plate location based on edge density and color analysis is used to detect the license plate region for tracking initialization.In the tracking stage:covariance matching is employed to track the license plate.Genetic algorithm is used to reduce the computational cost.Real-time image tracking of multi-lane vehicles is achieved.In the experiment:test videos are recorded in advance by recorders of actual E-police systems at several different city intersections.In the tracking module:the average false detection rate and missed plates rate are 1.19%:and 1.72%:respectively.

traffic monitoring system:covariance matching:genetic algorithms:vehicle tracking

0 Introduction

Intelligent transportation systems (ITS) play an important role in human life:and traffic monitoring has been a popular research focus for the development of ITS[1,2].The task of ITS is to ensure that traffic information can be collected online and distributed in real time.Recently:vision-based vehicle tracking has become the most popular method to collect traffic information:and many researchers proposed algorithms and methods to improve the performance of this tracking method[3].To achieve a robust solution for object matching:numerous features:such as color:edge:gradient:texture:and active contours have been selected[4,5].The contour is used to track the object.A real-time image tracking system of multi-lane vehicles is performed in Ref.[6].A new spatial color histogram model is presented in Ref.[7].With the use of this model:a voting method based on the generalized Hough transform is employed to estimate the object location for tracking.In general:utilizing a single cue to deal with a variety of environmental conditions robustly is a difficult task.A covariance descriptor was proposed to describe a target in Ref.[8].The covariance descriptor not only fuses multiple features but also finds a global optimal solution in the matching region.Furthermore:covariance matrices:which are scale and rotation independent:and robust against illumination change:filter the corruption of noises during the covariance computation[9].Covariance matching has been successfully used in many applications:such as object tracking[10,11]:human detection[12]:and image clustering[13,14].An effective framework for covariance tracking based on the genetic algorithm (GA) is proposed in Refs[15,16].Region covariance matrices are constructed to describe image regions robustly.GA is employed to find the optimal solution of the template in the scene image.

The purpose of this research is to design and implement a real-time vehicle tracking method for traffic monitoring system.A system architecture based on embedded platform is developed.The proposed tracker extracts traffic parameters online in real time.All the algorithms are performed in the embedded traffic monitor component based on the DM6446 chip.

The rest of this paper is organized as follows:Section 1 describes the real-time vehicle tracking approach in detail.Section 2 presents experimental results of the proposed method and provides several interesting and practical examples of estimating traffic parameters and detecting red light running accidents.Section 3 presents the conclusion.

1 Proposed vehicle tracking algorithm

In the proposed system:a vehicle tracking module consists of two stages:namely:the initialization stage and the tracking stage.In the initialization state:the location of the license plate (LP) is detected by edge density information[17]and color analysis[18].Feature detection is employed in the LP region for the construction of the covariance matrix.In the tracking stage:covariance-based object matching is used to locate the vehicles and the genetic algorithm is also used to reduce the computational cost of the covariance matching for real-time tracking.

In the tracking initialization:LP location based on the edge density and color analysis is performed to detect LP in the license plate detection (LPD) region:as shown in Fig.1.Multi-target tracking is performed in a multi-lane scene.In this study:the detected LP region is considered as the target template image for tracking.In the LPD region:all the LPs detected in the current frame are assumed asLc(n):wheren= 0,1,2,…:while all the LPs detected in the previous frame are denoted asLp(m).Tracking by covariance matching between theLc(n) andLp(m) is performed:wherem:n= 1,2,….IfLc(n) does not match successfully withLp(m):thenLc(n) is considered as a new target.Assuming that the number of LPs detected in current frame isN:and the number of LPs detected in the previous frame isM:Fig.2 shows the flowchart of the covariance matching and new target detection.The purpose of LP location is to extract candidate regions for matching in the LPD region.The proposed system performs vehicle tracking or new LPD based on the matching method shown in Fig.2.Furthermore:the LP location is employed only in the LPD region.Compared with the LPD region:the scale of the LP regions changes when the targets (vehicles) are far from the camera in the tracking region (see Fig.1):which affects the LP location accuracy.Hence:the method of tracking vehicles is different from the one employed in the LPD region and a target in the entire tracking region is searched by using covariance matching to find the most similar region to the template image.In addition:GA is employed to reduce the computational cost for real-time tracking.The system stops tracking when the vehicle leaves the tracking region:as shown in Fig.1.

1.1 License plate location

Generally:LP location methods are either texture based or color based[19].The texture-based methods analyze the texture characteristics in the LP region by using mathematical tools:such as edge density[20-22]and region connectivity[23,24]:and use the color information to detect the LP.Although the color characteristic in the LP region contains richer information:color-based methods have been paid less attention than texture-based ones because color is not stable in natural scenes[19].

Fig.1 LPD region and tracking region

Fig.2 Flowchart of matching and new target detection method

In the proposed system:the LP region is detected by edge density and color analysis.The proposed method consists of the following three steps:

Step 1 Vertical edge extraction.In general:the density of the vertical edges at the LP region is considerably higher than its neighborhood[17].The Sobel edge detection algorithm was used to extract the vertical edges to estimate their density in the detection region.In this paper:the Sobel kernel is defined in Eq.(1).The gradient image was computed by the Sobel convolution operation:and the binarized edge image was obtained through an adaptive threshold segmentation method.Fig.3(b) shows the binarized edge image.

Fig.3 LP location by edge density and color analysis

(1)

Step 2 Edge density estimation.To estimate the edge density:a common method is to use a 2D Gaussian filter.The size of the filter is assigned according to the size of the LP in the LPD region.In our system:the size of the filter was set as 130×40 based on the experimental statistic analysis.

Step 3 False object removing.The candidate regions for the LP are extracted by the filter described in Step 2.Some extracted candidate regions included the nose edge region [see Fig.3(c)]:which should be removed.We used the length-width ratio constraint and the color analysis described in Ref.[18] to remove the false LP region.Fig.3(d) shows the final detected LP region.

1.2 Covariance matching

fk=⎣Y(x,y)U(x,y)V(x,y)D(x,y)Yx(x,y)

Yy(x,y)」

(2)

The LP is selected as a target in tracking the vehicle.Therefore:the region of the LP can be represented as a covariance matrix CRas

(3)

where μRis the average of the vector that corresponds to the features of the points in the region R[10].

To match the LP between the template image and the object image:the similarity of the covariance matrices needs to be measured.Assume thatCtandCoare the two covariance matrices extracted from template and candidate image region.In the system:the covariance distance betweenCtandCois calculated based on Förstner’s contribution[25]

(4)

whereλk(C1:C2) are the generalized eigenvalues of C1and C2:computed from |λC1-C2|=0.Therefore:the tracking of LPs can be achieved from the location of the object image that has a minimum distance of covariance matrices.

1.3 Genetic algorithms

GA is an optimization method that is robust and useful in solving problems when the solution space has high dimensionality or contains discontinuities[26,27].Recently:GAs have been used in target tracking[15,16,27]because of their optimization property.

In this implementation:the N-points-sampling:which is at a constraint interval in both horizontal and vertical direction:is employed for initiation of the population:whereNis the population size.The selection of fitness function is crucial to a GA system.Based on the property of covariance matching described in Section 1.2:the fitness of an individual at the candidate position (x:y) is defined as

(5)

where ρ′=min(T:ρ):and ρ can be derived from Eq.(4).ThresholdTis used to restrict the fitness from 0 to 1.

Crossover operators generate offspring by exchanging information between the parents.The exchange of vertical coordinates of two parents is defined as the crossover operator.

A new mutation operator is also designed in this system and is employed by adding a random number into the horizontal or vertical coordinate of the individual as

It+1=It+mm∈[-M:M]

(6)

whereItis the individual selected to mutate in thetthgeneration:andmis a random value in the range from -MtoM.In this paper:the population is initiated as a constraint interval in both horizontal and vertical direction:andMis assigned as the interval in the horizontal or vertical direction.

2 Experimental results

Fig.4 shows the system architecture of the vehicle monitoring system developed in this study.All the algorithms are performed in the embedded traffic monitor component (ETMC) based on the DM6446 chip.Before the system starts working:the configuration file:which contains preset parameter information:is created using a configuration software and then downloaded to the ETMC.The ETMC processes the video from the HD camera and extracted the traffic parameters such as vehicle flow:vehicle speed:and traffic accident information:which are then transmitted to the information process center server.

Fig.4 Architecture of the proposed monitoring system

The experimental results of the proposed traffic-monitoring system are presented to show the real-time vehicle tracking performance.The test videos are recorded in advance by the recorders of actual E-police systems at several different city intersections.

Five videos recorded in three intersections are used to estimate the tracking performance.Intersection 1 includes daytime (shown in Fig.6):nighttime (shown in Fig.7):and rainy day (shown in Fig.8) observations:whereas intersections 2 and 3 are observed during daytime only.The results of the LPD are shown in Table 1.As observed from Table 1:the average false detection rate and missed plates rate are 1.19%:and 1.72%:respectively.

Table 1 LPD results

Fig.5 Time cost of tracking module

In the tracking module:GA is used to reduce the computational cost of covariance matching.In the experiment:the size of the initialization population is set to 50:and the number of iteration generations is 20 to balance the accuracy and speed.The crossover and mutation probabilities are both set to 0.6.To avoid missing the best matching position:and the individual is retained with the maximum fitness in all generations.Fig.5 shows the average runtime of the proposed tracking system.The average time cost is 20.75 ms in a 24-hour period.From Fig.5:it can be seen that from 7:00 to 9:00 and from 17:00 to 19:00:the runtime of the system is greater than the mean value because of the heavy traffic flow in morning and the evening.However,the peak value is still smaller than 40 ms:which shows that the proposed system satisfies the real-time requirement.

Fig.6 Vehicle tracking results in daytime scene

Fig.6 shows the experimental result images of vehicle tracking in a multiple lane scene.In Fig.6(a):three vehicles are detected in the LPD region:and each vehicle was assigned with a number as a tracking ID according to the detected order.The vehicle IDs are recorded in the traffic flow information:then the traffic flow status for a given period is obtained by counting the number of vehicle IDs.From Fig.6(b):a car and a van:which are labeled “0” and “2,” respectively:in the frame 1 has left the tracking region; their ID numbers are removed in the 56thframe:which indicates that tracking for these vehicles has ended.In the proposed system:each vehicle’s tracked distance is recorded by using the LP detection position and the track end position.The speed of a vehicle is calculated by dividing the tracked distance by the elapsed time.In addition:two cars are detected and labeled as “3” and “4” in the same frame in Fig.6(b).Fig.6(c)-16(f) illustrate the red light running detection at an intersection.In Fig.6(c):a van is detected and labeled as “45” in the 713thframe.In Fig.6(d):the red light is on:which signals the cars to stop.However:the van does not stop:so the system records the van as a red light runner.The scene image is recorded for law enforcement evidence.To distinguish the violating vehicles from other vehicles:a large red rectangle is used as the tracking window for the red light runners:as shown in Fig.6(d).

Fig.7 Red light running detection results in nighttime scene

Fig.7 shows a red light running accident detection in the nighttime scene.In Fig.7(a):two cars are detected and labeled as “48” and “49,” respectively.The car labeled as “49” does not stop at the stop line when the traffic light signaled a go-straight red light in Fig.7(b).The system detects the red light running accident as shown in Fig.7(c).In Fig.7(d):the peccant car leaves the tracking region:and the system stops the tracking:and the relevant detection information of the car is recorded.

Fig.8 Monitoring results in rainy day condition

Fig.8(a) shows the occlusion problem:which induces detection errors in the tracking module.In Fig.8(a):the LP of the front bus is occluded by the vehicle behind because the vehicles are too close to each other; hence:the detection accuracy is reduced.Fig.8(b) shows the detection result on a rainy day.The blue LP and yellow LP are detected successfully:thereby demonstrating the robustness of the system.

3 Conclusions

A real-time traffic-monitoring system is developed and tested in this research.The performance of the proposed system is evaluated.In the tracking module:the average false detection rate and missed plates rate of the LPD are 1.19%:and 1.72%:respectively.The run-time of the tracking module is also tested.The average time cost of the tracking module is 20.75 ms in a 24-hour period:and experimental results demonstrate the effectiveness and efficiency of the proposed system.Moreover:the applications of red light running detection of vehicles at an intersection are demonstrated.

The developed traffic monitoring system is useful for providing real-time online traffic parameters:such as number of vehicles:vehicle speed:and traffic flow:to a traffic control center.In the proposed system:LPD is performed to initialize the tracking:and the detected LP regions are considered as the template image to track the vehicles.Therefore:in the tracking module:one of the factors that cause detection error is the occlusion problem due to vehicles being too close to each other.Future work can focus on resolving the occlusion problem to improve the monitoring system further.

[1] Kastrinaki V:Zervakis M:Kalaitzakis K.A survey of video processing techniques for traffic applications.ImageandVisionComputing:2003:21(4):359-381

[2] Aoude G S :Desaraju V R :Stephens L H :et al.Driver behavior classification at intersections and validation on large naturalistic dataset.IEEETransactionsonIntelligentTransportationSystems:2012:13(2):724-736

[3] Yang H:Shao L:Zheng F:et al.Recent advances and trends in visual tracking:A review.Neurocomputing:2011:74(18):3823-3831

[4] Ma C:Liu C:Peng F:et al.Multi-feature Hashing tracking.PatternRecognitionLetters:2016:69(1):62-71

[5] Zhang X:Liu H:Wang Y.Feature fusion based object tracking for robot platforms.IndustrialRobot:2011:38(1):66-75

[6] Battiato S:Farinella G:Furnari A:et al.An integrated system for vehicle tracking and classification.ExpertSystemswithApplications:2015:42(11):7263-7275

[7] Suryanto D:Kim H:Kim S.Spatial color histogram based center voting method for subsequent object tracking and segmentation.ImageandVisionComputing:2011:29:850-860

[8] Tuzel O:Porikli F:Meer P.Region covariance:A fast descriptor for detection and for detection and classification.In:Proceedings of the European Conference on Computer Vision:Graz:Austria:2006.589-600

[9] Zhang X:Liu H:Li X.Target tracking for mobile robot platforms via object matching and background anti-matching.RoboticsandAutonomousSystems:2010:58(11):1197-1206

[10] Porikli F:Tuzel O:Meer P.Covariance tracking using model update based on Lie algebra.In:Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition:New York:USA:2006.728-735

[11] Zhang X:Li X:Liang M:et al.Covariance tracking with forgetting factor and random sampling.InternationalJournalofUncertainty:FuzzinessandKnowledge-BasedSystems:2011:19(3):547-558

[12] Tuzel O:Porikli F:Meer P.Pedestrian detection via classification on Riemannian manifolds.IEEETransPatternAnalMachIntell:2008:30(10):1713-1727

[13] Zhang X:Zhang Y:Zhang J:et al.Unsupervised clustering for logo images using singular values region covariance matrices on Lie groups.OpticalEngineering:2012:51(04):301-311

[14] Pang Y:Yuan Y:Li X.Gabor-based region covariance matrices for face recognition.IEEETransCircuitsSystVideoTechnol:2008:18(7):989-993

[15] Zhang X:Hu S:Zhang L:et al.Fast covariance matching based on genetic algorithm.In:Proceedings of the WiCOM:Chengdu:China:2010.1-4

[16] Zhang X:Hu S:Li X:et al.Fast covariance matching with fuzzy genetic algorithm.IEEETransactionsonIndustrialInformatics:2012:8(1):148-157

[17] Anagnostopoulos C:Anagnostopoulos I:Loumos V:et al.A license plate-recognition algorithm for intelligent transportation system applications.IEEETransactionsonIntelligentTransportationSystems:2006:7(3):377-392

[18] Abolghasemi V:Ahmadyfard A.An edge-based color-aided method for license plate detection.ImageandVisionComputing:2009:27(8):1134-1142

[19] Yang X:Hao X:Zhao G.License plate location based on trichromatic imaging and color-discrete characteristic.OptikIntJLightElectronOpt:2012:123(16):1486-1491

[20] Al-Hmouz R:Challa S.License plate localization based on a probabilistic model.MachineVisionandApplications:2010:21(3):319-330

[21] Huang Y:Chang T :Chen Y:et al.A back propagation based realtime license plate recognition system.IntJPatternRecognArtifIntel:2008:22 (2):233-251

[22] Guo J:Liu Y.License plate localization and character segmentation with feedback self-learning and hybrid binarization techniques.IEEETransVehTechnol:2008:57 (3):1417-1724

[23] Jiao J:Ye X:Huang Q.A configurable method for multi-style license plate recognition.PatternRecognition:2009:42(3):358-369

[24] Conci A:Carvalho J:Rauber T.A complete system for vehicle plate localization:segmentation and recognition in real life scene.IEEELatAmTrans:2009:7(5):497-506

[25] Förstner W:Moonen B.A Metric for Covariance Matrices:[Technical Report].Stuttgart:Department of Geodesy and Geoinformatics:Stuttgart University.1999.299-309

[26] Goldberg D E.Genetic Algorithm in Search:Optimization and Machine Learning.Boston:Addison-Wesley:1989.1-10

[27] Karmali F:Shelhamer M.Compensating for camera translation in video eye-movement recordings by tracking a representative landmark selected automatically by a genetic algorithm.JournalofNeuroscienceMethods:2009,176(2):157-165

Hu Shuo:born in 1976.He received a B.Sc.in Electronics and Information System and M.Sc.in Circuits and Systems from the Northeast Normal University in 2000 and 2003:respectively.He received a Ph.D in Optical Engineering from Institute of Optics Fine Mechanics and Physics:Chinese Academy of Sciences:in 2006.His research interests include activity recognition:video surveillance and time series analysis.

10.3772/j.issn.1006-6748.2016.03.003

①Supported by the National Natural Science Foundation of China (No.61005034):China Postdoctoral Science Foundation and under Grant (No.2012M510768) and the Science Foundation of Hebei Province under Grant (No.F2012203182).