VIETNAM NATIONAL UNIVERSITY HOCHIMINH CITY INTERNATIONAL UNIVERSITY SCHOOL OF ELECTRICAL ENGINEERING SPEED LIMIT TRAFFIC SIGN DETECTION & RECOGNITION

Transcription

1 VIETNAM NATIONAL UNIVERSITY HOCHIMINH CITY INTERNATIONAL UNIVERSITY SCHOOL OF ELECTRICAL ENGINEERING SPEED LIMIT TRAFFIC SIGN DETECTION & RECOGNITION By Nguyen Quang Do Advisor Dao Thi Phuong

2 VIETNAM NATIONAL UNIVERSITY HOCHIMINH CITY INTERNATIONAL UNIVERSITY SCHOOL OF ELECTRICAL ENGINEERING SPEED LIMIT TRAFFIC SIGN DETECTION & RECOGNITION By Nguyen Quang Do Advisor Dao Thi Phuong A senior submitted to the School of Electrical Engineering in partial fulfillment of the requirements for the degree of Bachelor of Electrical Engineering

3 SPEED LIMIT TRAFFIC SIGN DETECTION & RECOGNITION APPROVED BY: Advisor: M.Eng. Dao Thi Phuong Reviewer: Prof. Huynh Huu Tue Dr. Nguyen Tuan Đuc Dr. Udo Klein M.Eng. Vo Minh Thanh

4 ACKNOWLEGMENTS It is with deep gratitude and appreciation that I acknowledge the professional guidance of MEng. Dao Thi Phuong. Her constant encouragement and support helped me to achieve my goal. I am grateful to the faculty of the School of Electronic and Telecommunication Engineering of the International University for supporting since the beginning of my studies. Finally, I am really grateful to my friends and family for their support and encouragement throughout the year. Their guidance allowed me to complete this work.

5 ABSTRACT Intelligent Transport Systems (ITS) have great potential to save time, to save money. ITS s have considerable potential to be a future commercial success. Automatic road sign recognition method is one of the importance fields in the ITS. This is due to the importance of the road signs and traffic signals in daily life. In this work, Speed Limit Traffic Sign Detection & Recognition will be studied. The system works in two layers: Sign detection and sign recognition. Sign detection finds signs in images using color thresholding and sign recognition reads the digits on the sign. After this study, the object detection and recognition methods are known. They can be applied in other fields in Image Processing.

6 Contents LIST OF FIGURE LIST OF TABLE ABBREVIATION AND NOTATION Chapter 1: INTRODUCTION Overview Thesis objectives Scope Limitation Structure of project....3 Chapter 2: LITERATURE REVIEW Traffic signs in Vietnam The traffic sign system in Vietnam The speed limit sign in Vietnam Color segmentation RGB color space HSV color space Binarization Circular sign detection Speed limit sign recognition Artificial Neural Networks Fuzzy template & local feature matching Scan-line based digit recognition..18 Chapter 3: METHODOLOGY Database Methods description Methods specification Color segmentation Speed Limit Sign Detection Speed Limit Sign Recognition...34 Chapter 4: RESULTS AND DISCUSSION Color segmentation Speed Limit Sign Detection

7 4.3. Speed Limit Sign Recognition...39 Chapter 5: CONCLUSION AND FUTURE WORKS Conclusion Future works LIST OF REFERENCES....43

8 LIST OF FIGURE Figure 2.1: Some speed limit signs in Vietnam. Figure 2.2: Schematic of the RGB color cube. Figure 2.3: RGB color model. Figure 2.4: Color gamut produced by RGB monitors. Figure 2.5: Value of Hue in HSV color space. Figure 2.6: a. RGB original image. b. HSV image. c. Binary image extracts red color from HSV image. Figure 2.7: a. Gray scale image. b. Binarized image by adaptive thresholding. Figure 2.8: ANN structure for speed limit sign reognition. Figure 2.9: Two local feature vectors for digit 6 and 4. Figure 2.10: The distribution of critical points the vertical scan-line encountered, and two horizontal scan-lines for digit 8 and 2. Figure 3.1: a. The input image. b. The HSV image. Figure 3.2: a. The HSV image.

9 b. The binarized image. Figure 3.3: Image enhancements by median filter and then dilation. Figure 3.4: a. Inverting image. b. Locating the candidate region. Figure 3.5: The speed limit sign and some traffic signs have similar features with speed limit sign. Figure 3.6: a. Candidate circles in gray-scale. b. Binarized image by adaptive thresholding. Figure 3.7: a. Candidate circles in gray-scale. b. Binarized image by adaptive thresholding. Figure 3.8: Speed Limit Sign ( km/h). Figure 3.9: Speed Limit Sign (5 km/h). Figure 3.10: Three pattern speed limit sign after being scanned by scan-lines. Figure 4.1: The error in color segmentation stage. Figure 4.2: The mistake in speed limit detection stage.

10 LIST OF TABLE Table 2.1: Show the circularity value. Table 3.1: Show the structure of database. Table 3.2: Show the threshold values used in color segmentation stage to extract the red color. Table 3.3: The number of points for each digit. Table 4.1: The result of color segmentation. Table 4.2: The result of speed limit detection. Table 4.3: The result of recognition stage.

11 ABBREVIATION AND NOTATION ANN : Artificial Neural Networks. ITS : Intelligent Transport Systems. OCR : Optical character recognition. # CP : No. of points in middle line. (Vertical axis) # LLP : No. of points in left line. (Vertical axis) # RLP : No. of points in right line. (Vertical axis) # MP : No. of points in middle line. (Horizontal axis) # ULP : No. of points in upper line. (Horizontal axis) # LLP : No. of points in lower line. (Horizontal axis)

12 Chapter I INTRODUCTION 1.1. Overview of thesis Traffic Sign Recognition becomes one of research subjects aiming to improve safety today. The automatic traffic sign recognition is developed to warn drivers some important traffic signs, in the future, this system may take control of the vehicle under some circumstances. Especially, speed limit sign recognition is an important task for a driver support system. The speed limit signs are used to guide the drivers to drive under the maximum speed. This helps to reduce number of traffic accidents caused by over-speed. However, a driver may not notice a particular speed limit sign due to tiredness, distraction or lack of concentration. Therefore, an automatic speed limit sign recognition system may be helpful to make drivers aware of speed limit information they have missed Thesis objectives The main objectives of this research are as follows: Red color traffic signs location from color image by using color segmentation. Speed limit sign detection by using shapes identification technique. Speed limit sign recognition by using scan-line technique Scope of thesis In this thesis, detection and recognition of speed limit signs in static images are studied. Traffic signs in each set of test image have some adverse effects, which enables providing more realistic result. The database uses image with size of 640x480 and has moderate image quality. The distance from camera to traffic signs is about 9 11 meters.

13 1.4. Limitation of thesis At first glance, although the traffic sign detection and recognition system has some beneficial characteristics, some inevitable effects make this task quite complex and challenging. The main problems arising can be listed as follows: Illumination of signs varies continuously during day. Especially in bad lighting conditions, it is harder to gather color and contour information of traffic signs. The color of the sign fades with time. That is a result of the long exposure to the sun light, and the reaction of the paint with the pollutants in the air. Some bad weather conditions such as rain, fog reduce the ease of getting some sign characteristics. Traffic signs may partially be occluded by other objects. This may render a sign very hard, even impossible to detect. Installation of signs and surface material of signs may physically be damaged or changed. Camera mounted on the car will always suffer from vibrations Structure of thesis Some of the major algorithms, used for Speed Limit Sign Recognition in previous researches are mentioned in chapter 2. Then chapter 3 highlights the details about proposed algorithms in this thesis and the theory behind them. Chapter 4 shows the experimental results and discusses those results. Finally, chapter 5 includes the conclusion and the future work.

14 2.1. Traffic signs in Vietnam CHAPTER II LITERATURE REVIEW The traffic sign system in Vietnam Traffic signs are signs erected at the side of roads to provide information to road users. There are many kinds of traffic sign in Vietnam, they have been designed to be principally distinguishable from the natural and man-made backgrounds. The traffic signs are designed, manufactured and installed according to tight regulations. They are designed in fixed 2-D shapes like triangles, circles, rectangles, squares or octagons [1, 2]. The colors of the signs are chosen to be far away from the environment, which make them easily recognizable by the drivers [3]. The information on the sign has one color and the rest of the sign has another color. The tint of the paint that covers the sign should correspond to a specific wavelength in the visible spectrum [4, 5]. The signs may contain a pictogram, a string of characters or both. They are characterized by using fixed character heights [5]. In Vietnam, [6] the traffic signs are categorized as follows: obligation signs (blue circles) prohibition signs (red circles) danger signs (red triangles) indication signs (blue squares) work in progress signs (red and yellow triangles) priority signs (yellow rhombs) yield signs (red reverse triangles)

15 The speed limit sign in Vietnam The speed limit sign helps improve safety and reduce accidents on road where high speeds are a serious danger by encouraging motorists to drive within the speed limit. In traffic system of Vietnam: Urban: km/h Rural: km/h Expressway: over 60 km/h Figure 2.1 Some speed limit signs in Vietnam. The speed limit signs are designed in a circular shape with a thick red rim, and black number inside white background. All numbers inside speed limit sign have the same height [6] Color segmentation RGB color space In the RGB model, each color appears in its primary spectral components of red, green and blue. This model is based on a Cartesian coordinate system. The color subspace of interest is the cube shown in Figure 2.2, in which RGB values are at three corners; cyan, magenta, and yellow are at three other corners; black is at the origin; and white is at the corner farthest from the origin. In this model, the gray scale (point of equal RGB values) extends from the black to white along the line joining these two points. The different colors in this model are

16 points on or inside the cube, and are defined by vectors extending from the origin. For convenience, the assumption is that all color values have been normalized so that the cube shown in Figure 2.2 is the unit cube. That is all values of R, G, and B are assumed to be in the range [0, 1]. Figure 2.2 Schematic of the RGB color cube. The RGB color model is additive in the sense that the three light beams are added together, and their light spectra add, wavelength for wavelength, to make the final color's spectrum. Red + Green = Yellow Red + Blue = Purple Green + Blue = Cyan (blue-green) Red + Green + Blue = White Figure 2.3 RGB color model.

17 In color segmentation, the systems sets up an image filter with RGB color space to extract three colors: red, white and black from the given image. Their color information will be stored respectively. The idea of this algorithm is to test if a candidate region is a speed sign by checking if there is a white region in a red region and if there is a black region in that white region [7, 8]. Figure 2.4 Color gamut produced by RGB monitors. Another way using RGB color in color segmentation is to convert from RGB image to grayscale image. They threshold the grayscale image to generate the binary image and then find out most likely white region as the candidate for further processes [9]. RGB color space is high sensitivity to the light condition. The segmentation result could vary in different light conditions due to weather and/or shadows. An appropriate fixed threshold value is also impossible to specify with this color space.

18 HSV color space One of the most common color space in computer vision area is HSV, as well as in road sign detection and recognition field. This color space comprises three components: Hue, Saturation, and Value. Figure 2.5 Value of Hue in HSV color space. Hue is defined color information. 0 degrees is RED 120 degrees is GREEN 240 degrees is BLUE Saturation is defined as the range of grey in the color space. Values range from 0 to 1. Value is defined brightness of the color. It varies with color saturation.. Values range from 0 to 1 In order to segment color, the RGB original image is converted to HSV color space, then using a fixed threshold value extracts the color regions of interest. The threshold values are chosen by analyzing the hue, saturation and value information of the image [10, 11]. Figure 2.6 shows that a good result of red color detection in HSV color space.

19 Figure 2.6 a. RGB original image. b. HSV image. c. Binary image extracts red color from HSV image. The HSV color space has less variation to light condition than RGB. Hence the effect of light can be reduced to an acceptable level, the fixed threshold value can be applied for segmentation Binarization In order to analyze content inside the traffic sign, the traffic sign must be binarized. Hence, adaptive thresholding is applied into the image. Adaptive thresholding is designed to overcome limitations of conventional, global thresholding by using a difference threshold at each pixel location in the image. This local threshold is generally determined by the values of the pixels in the neighborhood of the pixel. Thus, adaptive thresholding works from the assumption that illumination may differ over the image but can be assumed to be roughly uniform in a sufficiently small, local neighborhood [12]. An approach for handing such a situation is to divide the original image into subimages and then utilize a different threshold to segment each subimage. The key issues in this approach are how to subdivide the image and how to estimate the threshold for each resulting subimage.

20 Figure 2.7 a. Gray scale image. b. Binarized image by adaptive thresholding. The following algorithm can be used to obtain threshold (T) automatically for each subimage [12]: 1. Select an initial estimate for T. 2. Segment the image using T. This will produce two groups of pixels: G 1 consisting of all pixels with gray level values > T and G 2 consisting of pixels with values T. 3. Compute the average gray level values μ 1 and μ 2 for the pixels in regions G 1 and G Compute a new threshold value: T = 1 2 (μ 1 + μ 2 ) 5. Repeat steps 2 through 4 until the difference in T in successive iterations is smaller than a predefined parameter T Circular sign detection In order to detect circle object, the system will calculate the circularity of the objects. Based on the properties of circle object, the circularity is computed by the formula: circularity = perimeter2 4 π area According to Thomas B.Moeslund [13], be researched as:

21 Items Circularity (C) Circle 0 < C 1.2 Square 1.2 < C 1.6 Isosceles triangle 1.6 < C 1.8 Others C > 1.8 Table 2.1 The circularity values Speed limit sign recognition Artificial Neural Networks An Artificial Neural Networks (ANN) is a mathematical or computational model which consists of an interconnected group of artificial neurons. It processes data using a connectionist approach, akin to the vast network of neurons in the human brain. Nowadays, modern ANNs are non-linear statistical data modeling tools and widely used to detect patterns in data or model complex relationships between inputs and outputs. The recognition stages in paper [14, 15, 8] are implemented by using a multilayer feed forward neural network. The basic feed forward network performs a non-linear transformation of input data in order to approximate the output data. This neural network contains three different layers: input layer, hidden layer and output layer [16]. Each layer consists of numbers of nodes. The input layer is passive which means it doesn t modify the data and the other two layers are active. For example, every node in input layer is the color value (usually 0 for black and 1 for white) of a single pixel in a binary image. Each value from the input layer is duplicated and sent to all of the nodes in hidden layer. This is called a fully interconnected structure. The

22 hidden layer is usually about 10% the size of the input layer [16]. The values entering the hidden node are multiplied by weights, which a set of predefined value for hidden nodes. After all input values are multiplied by the weight, they will be added to produce a single sum. Before leaving the node, this number is passed through a nonlinear mathematical function called a sigmoid. This is an S shaped curve that limits the node's output. That is, the input to the sigmoid is a value between - and +, while its output can only be between 0 and 1. Each of these outputs from the hidden layer are duplicated and applied to the output layer. The active nodes in this layer will combine the entering values together to generate the outputs, which are usually possibilities of output categories. The category with most likelihood is the final output of this feed forward ANN. Before using the ANNs to process data, it must be optimized through various training algorithms [16]. In paper [14], the ANN is trained using binary images of digits 0~9 with size 32 X 32 pixels, which means there are 1024 nodes in its input layer. The model contains 15 and 10 neurons in its hidden and output layer respectively. The 10 output neuron consists of 9 desired output neurons, which are representing 9 most common readings of speed limit sign (15, 30, 40, 50, 60, 70, 80, 90 and 110km/h), and one undesired output neuron which representing non-digit input. After all output layer neurons get their result, the final output will either be a speed limit sign reading (for example 50 km/h) or non-digit reading (i.e. not a speed limit sign) depends on the most likelihood possibility output. Papers [8, 15] also use the same ANN model to recognize the readings of speed limit sign. In paper [15], the input layer of ANN contains 400 neurons (20 X 20 pixel samples). The size of the hidden layer was set by empirical experimentation, which shows that the 30 neurons in hidden layer giving the best performance. There are 12 neurons in the output layer corresponding to the following signs types: 10, 20, 30, 40, 50, 60, 70, 80, 90, 100 km/h, national-speed limit (in UK/Poland) and

23 false positive (i.e. not a sign). Each output value is the classification likelihood between 0 and 1 for the corresponding class (i.e. type) of sign and the largest value indicates the output of ANN. Figure 2.8 ANN structure for speed limit sign reognition [14]. Researchers claim at the end of paper [8, 14, 15] that the system has achieved a very accurate performance with 90 to 92% recognition accuracy. The recognition speed of ANN models is also acceptable for real time processing. However, the cost of precise ANN recognition is large amount of training samples and building a model of complex multilayer structure Fuzzy template & local feature matching Contrary to ANN approaches, template matching has no need of collecting samples and training the model. However, the traditional template matching algorithms are not suit for recognizing the readings of speed limit sign due to image blur, weather or illumination condition [17, 18]. To overcome that, Paper [17] proposed a algorithm which combines template matching and local feature matching together. Firstly, the system applies fuzzy template matching method to coarse recognise the sign number character. Then, to avoid mismatching and achieve robust recognition performance, a fine recognition process based on

24 local feature vector is applied. The approach of first part of the system is template matching using Normalized Cross Correlation as the matching criterion. After this, the features of the sample image will be extracted to form the local feature vectors. The reason for applying the local feature vector is to identify the similar characters under blurs and different luminance. The features of character can be number / position of holes [18] and distribution of character pixel (different from background pixel) at the top / bottom of the image[17]. An example of distinguish character 4 and 6 on the speed limit sign is given in [17]. Because the numbers of hole for 4 and 6 are both one, the feature to identify the correct character is the difference between distributions of white pixel (background of the road sign) at the bottom. To form the local feature vector, the system search background pixel at the bottom of the image. The search of a column terminated with character pixel (nonbackground pixel) encountered. Then, system counts the number of background pixel passed and starts to search the column next. After all columns have been searched, counts of background pixel are normalized to size 20 according to the searching order (i.e. left to right) to consist local feature vector. Figure 2.9 Two local feature vectors for digit 6 and 4 [17].

25 The local feature vector of fuzzy template can be obtained in the same way. Then, the Euclidean Distance of local feature vectors between character image and fuzzy template can be calculated by using the following equation: 20 Distance i = (z j z i j ) 2 j =1, (i = 4, 6) Where Z i = {z i 1,, z i 20 } is the feature vector of the fuzzy template i. and Z = {z 1,.., z 20 } is the local vector of the character to be recognized. Hence, the reading of the speed limit sign is finally recognized as the best matching result which has the smaller Euclidean distance. Similarly, the system in paper [18] uses this algorithm in recognition stage with some other local features (i.e. position of top/ bottom row of character pixel, number of left / right column across of character pixel). This algorithm retains the simplicity and flexibility of the template matching. It enhances the reliability and improves performance of digit recognition Scan-line based digit recognition Scan-line based OCR technique is widely used in automated meter-reading (i.e. automated record readings from ammeter dial plate [20, 21]) and document acquisition because its simplicity and fast recognition speed. In fact, this algorithm is suit for various purposes including speed limit sign recognition [19, 21]. The idea of this algorithm is using multiple vertical and/or horizontal scan-lines to go through the input image and mark those nonbackground pixels encountered. These pixels and relevant patterns are critical features which identify the out digit character by matching with pre-defined digit character (0 ~ 9) feature matrixes. The digit character with the highest similarity with critical features is the output of system. If the similarity is below the threshold value (i.e. 95%), the input is treated as a nondigit character.

26 Paper [19] gives a detailed example of how this efficient algorithm works. Firstly, the system detects the left (X min), right (X max), top (Y max) and bottom (Y min) boundary of the input character image. Then, a vertical scan line is drawn through the middle point of image width (i.e. through point which X = X min + [X max X min]/2). All non-background pixels the scan line encountered are marked as 1, 2,, n, from top to bottom of the character. For example, there are 6 critical points for images contain digit 2 and 8. Figure 2.10 The distribution of critical points the vertical scan-line encountered, and two horizontal scan-lines for digit 8 and 2. After marking critical points, the algorithm to determine the character is as follows: 1) If there are only two critical points (minimum of possible number of critical points), the pattern in the input image is digit character 1 (i.e. the feature matrix for digit 1 is one critical point ). 2) If the number of critical points is more than six (i.e. seven critical points), the pattern is not a digit character (i.e. return None ). 3) Otherwise, if the number of critical points is more than three (i.e. four critical points), for all even critical point, except the last one (i.e. critical points No.6 for 8 ), do inner

27 edge tracing. Mark Feature1 = true if first tracing success and Feature2 = true if second tracing success. Otherwise, mark false. (i.e. for pattern 8, the first tracing starts at critical point No.2 and the second starts at No.4. Both Feature1 and Feature2 are marked as true. Similarly, for pattern 2, both are marked as false ). 4) Otherwise, if the number of critical points is six, Feature1 = true and Feature2 = true, the pattern is digit character 8. 5) Otherwise, if the number of critical points is six, Feature1 = true and Feature2 = false, the pattern is digit character 9. 6) Otherwise, if the number of critical points is six, Feature1 = false and Feature2 = true, the pattern is digit character 6. 7) Otherwise, if the number of critical points is four, critical point No.4 is not Y min and Feature1 = true, the pattern is digit character 4. 8) Otherwise, if the number of critical points is four, critical point No.4 is Y min and Feature1 = true, the pattern is digit character 0. 9) Otherwise, if the number of critical points is four, Feature1 = false and Feature2 = false, the pattern is digit character 7. 10) Otherwise, if the number of critical points is six, Feature1 = false and Feature2 = false, do two horizontal scan-lines (towards right) through the middle point of critical point No. 2 & No.3 and through the middle point of No.4 & No.5 respectively (See figure2.8 above). Mark Feature 3 = true if S1 encountered any non-background pixel and Feature 4 = true if S2 encountered non-background pixel. Otherwise, mark false. 11) If the number of critical points is six, Feature1 = false, Feature2 = false, Feature 3 = false and Feature 4 = true, the pattern is digit character 5

28 12) Otherwise, if the number of critical points is six, Feature1 = false, Feature2 = false, Feature 3 = true and Feature 4 = true, the pattern is digit character 3 13) Otherwise, if the number of critical points is six, Feature1 = false, Feature2 = false, Feature 3 = true and Feature 4 = false, the pattern is digit character 2 14) Otherwise, the pattern in the input image is a non-digit character, returns None. Through these 14 conditions above, the pattern of the input image can be identified as either a single digit character or non-digit. Since the reading of a speed limit sign could be vary in number of digit (i.e. 5 km/h one digit, 10 km/h two digits and 115 km/h three digits). However, if the number of digit is 3, the algorithm is applied at most two times to generate the final reading as the first digit is always 1. Some obvious features of this algorithm include fast process speed, stronger resistance to image blurs and variation of font. The concept of this algorithm is straightforward and easy to implement. The similar technique is used in systems proposed by paper [20, 21]. The difference is that in paper [21], an image thinning and burrs deleting process are applied to the input image to be recognized before drawing scan-lines in order to reduce the effect of noise. It is not really necessary because the width all three scan-lines is only 1 pixel.

29 CHAPTER III METHODOLOGY 3.1. Database In order to test and evaluate the performance of the result of speed limit sign detection and recognition algorithm, a sample database is set up. The database is collected in different times of a day, and different environments (urban area, countryside). Database consists of 300 static images covering 9 different speed limit signs (5, 10, 20, 30, 35, 40, 50, 60, 80 km/h) and other regulatory traffic signs with 640x480 image resolution. The sample image is captured with a resolution of 640x480 because a lower resolution will result in loss of detail, otherwise, higher resolution will affect to the time running of program. The table 3.1 shows the structure of database which uses in this thesis. The speed limit signs occupy approximate 31% of the database.

30 Database Number of images Total 5 km/h 1 10 km/h 2 20 km/h 3 30 km/h 14 Total of speed Speed limit sign 35 km/h 1 40 km/h 28 limit sign km/h km/h 4 80 km/h 1 Other traffic sign 207 Table 3.1 The structure of database Methods description In this thesis, a method for the speed limit sign detection and recognition is proposed. It consists of three main processing stages, which are color segmentation, speed limit sign detection, and recognition stage. The method is described as follows: Read an image (Input) Color segmentation Speed limit sign detection Speed limit sign recognition

31 In the first stage of system, the input image will be segmented into numbers of smaller regions based on the color information. This process is to focus the processing only on the important image areas, and reduce the effort wasted on search speed limit sign over the entire image. The project researches about speed limit sign, which has surrounded by red outer, hence this step only searches for red color regions. The speed limit sign detection stage is the process that identifies speed limit signs from those candidate regions. It identifies all circles from these regions and then, checks the content inside the circle. Only those circles with similar visual features with speed limit signs survive as candidates of the recognition stage, the others will be discarded. In the last processing stage, the optical digit character recognition is applied to identify digit inside the circle background. In general, there are two most important tasks for the proposed method, which are distinguishing between the speed limit sign and the other objects faked speed limit sign, and recognizing the specified speed limit on the sign correctly Methods specification Color segmentation Color segmentation Red and non-red color determination Enhancing binary image Locating non-red region in traffic sign

32 The color segmentation is used to extract regions in interested color from input image. The regions surrounded by red color are considered as candidate regions which may contain speed limit sign. In order to achieve this, the input image will be converted into HSV color space. The reason why this color space was chosen is because it is immune to lighting changes. Each image element is classified according to its hue, saturation, and value. In this thesis, the RGB original image is converted to HSV color space by the formula: H = H, if B G 360 H, otherwise S = Max R, G, B Min(R, G, B) Max(R, G, B) V = Max(R, G, B) 255 Where: H = cos R G + R B R G 2 + R B G B Figure 3.1 a. The input image. b. The HSV image.

33 Red color Hue (H) 0 < H < 0.08 ( ) 0.92 < H < 1 ( ) Saturation (S) 0.25 < S < 1 Value (V) 0.25 < V < 1 Table 3.2 The threshold values used in color segmentation stage to extract the red color. After that, the value pixels satisfy above table will be marked as white color, and all rest pixels are marked as black. The purpose of this step is to separate red color regions from input image. Figure 3.2 a. The HSV image. b. The binarized image.

34 To eliminate small noise, a 3x3 median filter is applied to binary outputs. Application of a 3x3 median filter to binary outputs greatly eliminates these unwanted pixels to reduce the speed processing of system. After the application of the median filter, in order to improve the binary image, dilation with a 3x3 rectangular structuring elements is applied. Figure 3.3 Image enhancements by median filter and then dilation. When a red traffic sign appears on the red background (in front of a red building, or in front of the other red objects), this stage can be fail because the red rim of the traffic sign is melted into the background. In order to overcome this problem, the image is inverted, the system will process on none red region inside of traffic sign. Then based on the area of traffic sign with input image, the system will locate that none red.

35 According to database of this project, the traffic signs usually occupy an area about 0.3% - 5% of input image. Figure 3.4 a. Inverting image. b. Locating the candidate region. The first stage of process is done with result is candidate regions and/ or fake regions go to detection stage Speed Limit Sign Detection Speed Limit Sign Detection Circular detection Content extraction/ segmentation Speed limit sign Identification The main purpose of this stage is determining speed limit signs within those candidate regions got from color segmentation stage. There are 3 processes in this stage; those are circular detection, content extraction/ segmentation and speed limit sign identification. In first process, the candidate regions will be checked whether circle is or not. Only circle objects can go through to further processes and all non-circular regions are eliminated.

36 To detect circle object, the system will calculate the circularity of the objects. Based on the properties of circle object, the circularity is computed by the formula: circularity = perimeter2 4 π area According to research of Thomas B.Moeslund and database, the circularity value is about for the circle traffic sign. Thus, non-red circle inside red round traffic sign have been identified, some of regulatory traffic signs can be eliminated. However, the system cannot be sure that traffic sign is speed limit sign or not because some other traffic signs which have similar features with speed limit sign have not been eliminated yet. Figure 3.5 The speed limit sign and some traffic signs have similar features with speed limit sign. In order to make sure that only speed limit sign can be detected by the proposed system, the content of identified non-red circle is analyzed. The candidate circles inside the traffic sign are extracted from original image, and converted to gray-scale image. Then, adaptive thresholding method is applied to obtain the binary image. Adaptive thresholding method was chosen is because surface of traffic signs are usually uneven illumination, the adaptive thresholding method solves this problem very well.

37 Figure 3.6 a. Candidate circles in gray-scale. b. Binarized image by adaptive thresholding. Figure 3.7 a. Candidate circles in gray-scale b. Binarized image by adaptive thresholding. After that the result binary image will be labeled objects to identify speed limit sign or not. In the system, all of objects will be computed the height and width. If there are 2 3 labeled objects found with roughly the same height ( km/h), the corresponding circle is considered as a speed limit sign. Figure 3.8 Speed Limit Sign ( km/h)

38 In case only 1 object is found inside of traffic sign (5 km/h), the system will based on the height and width of object to determine the speed limit sign or not. According to database, the height of character is about pixels, and width is about 5 15 pixels, the candidate is speed limit sign. Figure 3.9 Speed Limit Sign (5km/h) By this method, most of traffic signs and other objects with high similarity of a speed limit sign are eliminated. The results when combining the segmentation and speed limit sign methods will be sent to recognition stage Speed Limit Sign Recognition In last stage, the digits inside speed limit sign are recognized. As mentioned in previous chapter, Artificial Neural Network and Template Matching are two traditional approaches to recognize digit in this field. However, there are still several problems with these techniques. The Artificial Neural Network requires large amount of training to operate before applying. It also requires high processing time for a well-trained robust neural network. For Template Matching, the performance is varying in different fonts, damage and rotation of speed performance is also poor. Furthermore, it also requires high time cost. Therefore, scan-line algorithm is used in this thesis. The main advantages of this method are:

39 It is able to run in real time. Good resistance to variation of font. Do not require any pre-processing or training. Speed Limit Sign Recognition Scanning character Visual features identification Digit recognition The scan-line algorithm scans character with three vertical scan lines and three horizontal scan lines. The three vertical scan lines are: middle line, left line (between column of left boundary and middle line) and right line (between column of middle line and right boundary). The three horizontal scan lines are: middle line, upper line (between row of top boundary and middle line), and lower line (between row of middle line and bottom boundary). The result will more precisely when using more lines, but 3 lines in vertical and 3 lines in horizontal are able to achieve exactly result. If the number of scan-lines is increased, the speed processing will be affected.

40 Figure 3.10 Three pattern speed limit sign after being scanned by scan-lines. Pixels at the border between object and background which encountered by scan lines will be marked. The number of marked points combine with their location is considered as features of each. The system based on this feature to classify the digit. Digits #CP #LLP #RLP #MP #ULP #LLP Table 3.3 The features for each digit. # CP: No. of points in middle line. (Vertical axis) # LLP: No. of points in left line. (Vertical axis) # RLP: No. of points in right line. (Vertical axis) # MP: No. of points in middle line. (Horizontal axis) # ULP: No. of points in upper line. (Horizontal axis) # LLP: No. of points in lower line. (Horizontal axis)

41 CHAPTER IV RESULT AND DISCUSSION This chapter presents results for tests conducted on a signs database, containing most of the possible Vietnam signs, as well for real signs that appear on the photos taken along Vietnam roads. Since this thesis was divided in three different chapters: color segmentation, speed limit sign detection, speed limit sign recognition, each one will be tested for result analysis Color segmentation results Applying the color segmentation method on the database signs has proved that the segmentation works well for correctly colored signs. Note that this stage only extracts red traffic sign because this thesis focuses on speed limit sign which has surrounded by red outer. Total Segmented False segmented % of segmented Database % Table 4.1 The result of color segmentation. The error occurs in case of the traffic signs are much darker under poor lighting conditions. Figure 4.1 The error in color segmentation stage.

42 The segmentation proved to be efficient, detecting most of the presented red signs. In this stage, there are exist 3 traffic signs (none speed limit sign) cannot be segmented successful. The performance is 98.9% Speed limit sign detection results The speed limit sign detection on database signs has also proved to be successful. Total Detected False detected % of detected Database % Table 4.2 The result of speed limit detection. However, there is still 1 none speed limit sign detected. The mistake happened in case of weight limit signs which do not obey the standard form (the character T is too small). Figure 4.2 The mistake in speed limit detection stage Speed limit sign recognition results Finally, the recognition of traffic signs is tested. All the database signs are fully segmented and well detected. These results are shown in the following table:

43 Table 4.3 The result of recognition stage. The average speed processing by using Matlab simulation is: 0.97 seconds. From previous chapter, the average distance from camera to traffic signs is: 10 meters. Then, the maximum velocity of vehicle is: Speed limit sign Total Recognized False recognized % of recognized 5 km/h % 10 km/h % 20 km/h % 30 km/h % 35 km/h % 40 km/h % 50 km/h % 60 km/h % 80 km/h % Velocity = distance time = 10 meters meters = seconds seconds 37 km/h

44 4.1. Conclusion Chapter V CONCLUSION & FUTURE WORKS In this thesis, the algorithm for speed limit sign detection and recognition has been studied and implemented. A scan-line based numerical digit recognition algorithm has been introduced. Experimental results indicate that the proposed system of this research achieved speed limit sign recognition with high overall accuracy. The proposed algorithm was divided into three main process stages, which are color segmentation, speed limit sign detection and digit recognition. Color segmentation discards all uninterested regions in HSV color space to smooth speed limit sign detection. Possible speed limit signs in the scene are detected and extracted from regions of interest. The circularity method was used to identify circular region of interest and speed up the process. In the last stage, candidate signs are classified via proposed scan-line based digit recognition algorithm, which provide reliable performance with less complexity factor. The algorithm attained 100% recognition accuracy and an average processing speed of 0.97 seconds, which mean the main research goal of this thesis has been achieved Future works Due to shortage of the time, some limitations have not researched, although some of them can be solved. First of all, in order to overcome speed limit sign misclassification caused by sloped signs, geometric transformation algorithms could be introduced. These algorithms are frequently used to introduce distortion into a scene. In this case, for any sloped sign, the system can calculate the angle between the bottom of the digit character and the lower border of the image first, and rotate every pixel according to the rotation operators.

45 Another possible improvement is unclosed red regions fixing scheme to speed limit sign detection stage. Sometimes, part of the outer red ring is covered by other object which causing unclosed red regions and misdetection. If the system can fix those unclosed red regions by evaluating the shape and size of the gap (whether the gap is small enough; whether the gap is a curve), the detection accuracy will be increased. Last but not least, the computation cost must be decreased.

46 LIST OF REFERENCES [1] P. Parodi and G. Piccioli, A feature-based recognition scheme for traffic scenes, present at Intelligent Vehicles 95 Symposium, Detroit, USA, [2] J. Plane, Traffic Engineering Handbook: Prentice-Hall, [3] G. Jiang and T. Choi, Robust detection of landmarks in color image based on fuzzy set theory, presented at Fourth Inter. Conf. on Signal Processing, Beijing, China, [4] S. Vitalbile and F. Sorbello, Pictogram road signs detection and understanding in outdoor scenes, presented at Conf. Enhanced and Synthetic Vision, Orlando, Florida, [5] S. Vitabile, A. Gentile, and F. Sorbello, A neutral network based automatic road sign recognizer, presented at The 2002 Inter. Joint Conf. on Neural Networks, Honolulu, HI, USA, [6] The traffic system in Vietnam, [7] A. Broggi, et al., Real Time Road Signs Recognition, in Intelligent Vehicles Symposium, 2007 IEEE, 2007, pp [8] J. Torrensen, et al., Efficient recognition of speed limit signs, in Intelligent Transportation Systems, Proceedings. The 7 th International IEEE Conference on, 2004, pp [9] H. Yea-Shua and L.Yun-Shin, Detection and recognition of speed limit signs, in Computer Symposium (ICS), 2010 International, 2010, pp

47 [10] C. Hsin-Han, et al., Road speed sign recognition using edge-voting principle and learning vector quantization network, in Computer Symposium (ICS), 2010 International, 2010, pp [11] N. Barnes and A. Zelinsky, Real-time radial symmetry for speed sign detection, in Intelligent Vehicles Symposium, 2004 IEEE, 2004, pp [12] R. C. Gonzalez and R. E. Woods, Digital Image Processing, Prentice Hall, 2nd Edition, 2002, pp [13] Thomas B. Moeslund, Image and Video Processing, 2 nd Edition. [14] K. A. Ishak, et al., A Speed Limit Sign Recognition System Using Artificial Neural Network, in Research and Development, SCORED th Student Conference on, 2006, pp [15] M. L. Eichner and T. P. Breckon, Integrated speed limit detection and recognition from real-time video, in Intelligent Vehicles Symposium, 2008 IEEE, 2008, pp [16] C. M. Bishop, Neural Networks for Pattern Recognition, Oxford University Press, [17] L. Wei, et al., Real-Time Speed Limit Sign Detection and Recognition from Image Sequences, in Artificial Intelligence and Computational Intelligence (AICI), 2010 International Conference on, 2010, pp [18] W. Yongping, et al., A Method of Fast and Robust for Traffic Sign Recognition, in Image and Graphics, ICIG 09. Fifth International Conference on, 2009, pp

48 [19] W. C. Xu Han Wei, A New Algorithm for Numeral Recognition, Surveying and Mapping of Geology and Mineral Resources, vol. 2, p.31, [20] D. Castells-Rufas and J. Carrabina, Camera-based Digit Recognition System, in Electronics, Circuits and Sytems, ICECS th IEEE International Conference on, 2006, pp [21] L. Yibo and Q. Hongjuan, Automatic Recognition System for Numeric Characters on Ammeter Dial Plate, in Young Computer Scientists, ICYCS The 9 th IEEE International Conference on, 2008, pp