Car Parking Vacancy Detection and Its Application in 24-Hour Statistical Analysis

Transcription

1 th International Conference on Frontiers of Information Technology Car Parking Vacancy Detection and Its Application in 24-Hour Statistical Analysis Jermsak Jermsurawong, Mian Umair Ahsan New York University Abu Dhabi Abu Dhabi, UAE {jj1192; Abdulhamid Haidar Massachusetts Institute of Technology Cambridge MA, USA Haiwei Dong, Nikolaos Mavridis New York University Abu Dhabi Abu Dhabi, UAE {haiwei.dong; Abstract Finding empty parking spaces is a common problem in densely populated areas. Drivers spend an unnecessarily large amount of time searching for the empty spots, because they do not have perfect knowledge about the available vacant spots. An effective vacancy detection system would significantly reduce search time and increase the efficiency of utilizing the scarce parking spaces. The proposed solution uses trained neural networks to determine occupancy states based on visual features extracted from parking spots. This method addresses three technical problems. First, it responds to changing light intensity and non-uniformity by having adaptive reference pavement pixel value calculate the color distance between the parking spots in question and the pavement. Second, it approximates images with limited lighting to have similar feature values to images with sufficient illumination, merging the two patterns. Third, the solution separately considers nighttime vacancy detection, choosing appropriate regions to obtain reference color value. The accuracy was 99.9% for occupied spots and 97.9% for empty spots for this 24-hour video. Besides giving an accurate depiction of the car park s utilization rate, this study also reveals the patterns of parking events at different times of the day and insights to the activities that car drivers engage with. Index Terms Empty slot detection, single camera, multi-car parking monitoring. I. INTRODUCTION Finding empty parking spaces is a common problem in densely populated areas. Drivers individually find empty parking spaces, searching without information of the parking status. These drivers take an unnecessarily long time to find the parking spot. This task should be specialized by one automatic system to reduce the search time. Another problem is that drivers often end up finding the same parking spot at the same time. While one would get the empty spot and reap the benefits from his search time, the other drivers search time is completely wasted. This is inefficient because priority is not given to those who need the parking spot the most. The information on the locations of available spots can be used to efficiently allocate this scarce resource. Therefore, an effective vacancy detection system is needed. Real-world 24-hour statistics to assess the current utilization of car park is helpful to drivers searching for parking slots. In addition, different parking patterns at different time periods can also be analyzed to give useful information about the activities that car drivers engage with. This paper seeks to identify vacant parking slots in an outdoor car park for a continuous 24-hour period and obtain statistics of the parking activities. There have been various approaches to detect empty parking spaces. While wirelessbased [1] or wired-based sensor [2] methods may prove to be effective, they are less efficient than one-to-many sensor in image-based detection system. In this vision-based field, much research has focused on object detection. Bong used pixel detection with threshold to differentiate between cars and empty slots. Although this threshold value is non-adaptive to changing light conditions, he compensated this with edge detection [3]. Still, the method does not work well when the car-to-camera distance is high and the car images have few pixels. Fabian based his method on the homogeneity of the car pixel values, counting the number blocks containing pixels of similar homogeneity values [4]. Again, this requires a highly detailed image with limited applications in large-scale singlecamera detection system. Besides, his method does not consider when the parking place is under limited illumination. Another method, object tracking, is not always practical, as many existing surveillance cameras process videos at large intervals and sufficiently continuous image stream cannot be assumed [5]. There have also been attempts using machine learning to classify the parking states [6] [7], but they did not consider the changing light conditions. Huang developed a /12 $ IEEE DOI /FIT

2 Bayesian framework detection method that is robust to changes in light conditions. This detailed study even modeled the shadow-from-sunlight direction, based on U.S. Naval Observatory to anticipate the false detection of black-colored vehicles [8]. However, this method is too complicated, because although the shadow could be recognized as the car, it has a uniform appearance. Measurement of randomness or spread of pixel values can account for this shadow problem. Little research has attempted to build a comprehensive system for nighttime detection. Macdonell and Lobo looked at nighttime detection but the whole picture frame consisted of only five parking spots [9]. As a result, little is known about 24-hour statistics of car parking activities. The proposed solution uses trained neural networks to determine occupancy states based on visual features extracted from parking spots. This method addresses three technical problems. First, it responds to changing light intensity and nonuniformity by having adaptive reference pavement pixel value calculate color distance between the parking spots in question and the pavement. Second, it approximates images with limited lighting to have similar feature values to images with sufficient illumination, merging the two patterns. Third, the solution separately considers nighttime vacancy detection, choosing appropriate regions to obtain reference color value. The accuracy was 99.9% for occupied spots and 97.9% for empty spots for this 24-hour video. This method therefore reports a reliably large data set of parking states over space and time, leading to accurate indication of car park utilization rate and meaningful statistical analysis. Overall, this method has two major advantages. First, it relies on only a few pixels compared with other methods, being able to cover more than 150 parking spots with in a single camera frame. Second, the approach is robust to changing light conditions and non-uniformity due to shadows from the surrounding buildings. II. TECHNICAL PROBLEMS A. Problem 1: Changing Light Intensity and Non-uniformity in Daytime In an open space car park, light intensity not only changes with time, but also may be non-uniform (Fig.1). Partial shadow from the surroundings extend an across the car park. This problem is especially significant in urban context, because buildings often surround car parks. Therefore, image analysis gives varying features of parking spots at different times, although the parking statuses remain the same. (c) Fig. 1. Different light intensity and non-uniformity in the car park. Morning. Noon. (c) Afternoon. Table 1 shows the decrease in percentage of average statistical measurements of non-shaded parking spots when they are in a shaded region. The proposed solution should dynamically respond to the light conditions both intensity and non-uniformity. TABLE I. PERCENTAGE DECREASE OF AVERAGE STATISTICAL MEASURES WHEN NON-SHADED SPOTS BECOME SHADED Statistical Measures Occupied Empty Standard Deviation -29.8% -32.7% Interquartile Range -28.9% -38.4% Entropy -6.45% -20.7% B. Problem 2: Limited Light in Early Morning and Evening Time Even though changing illumination level throughout daytime can be solved, severely limited light intensity compromises the details of images to be analyzed. Such low light intensity happens during the two transitions between daytime and nighttime before the light posts are turned on in the early morning, and after they are turned off in the late evening (Fig. 2). 85

3 with a sample of video frames. It was then used to analyze the whole 24-hour video recording. Fig. 2. Limited light intensity. In the early morning just after the light posts are turned off. In the late evening just before light posts are turned on. Statistical measurements of pixel values from such images give a different range of results compared to ones with sufficient lighting. For example, the standard deviation and range of the pixel values at evening are significantly lower than ones in the afternoon. As a high level of car activity takes place during this late evening, it is important to accurately detect the vacant parking spaces in such a critical period. C. Problem 3: Different Light Pattern in Nighttime Although constant lighting from light posts at night does not complicate the feature values over time, a single light source at a short distance from the light posts significantly changes patterns of lighting compared to daytime (Fig. 3). Because illumination from streetlights is concentrated only on parking spots, the color of an empty parking space is much brighter than that of the nearby road pavement. Directly comparing the two regions gives a false indication of car presence, as the color difference between the empty spot and the pavement is now significant. Fig. 3. Nighttime image of the car park. Concentrated light source from the light posts. III. METHOD The 24-hour video recording of an outdoor car park in Downtown Abu Dhabi took place from 09:00 July 3th to 09:00 July 4th. The camera was placed at 29 th floor of the adjacent tall building to avoid occlusions. The recording spanned 9 lanes of car parking spaces each with 14 parking spots. In a frame, a total of 126 parking spots were considered. This large scope reduced the size of each parking spot analyzed to only 30x14 pixels. A neural network was constructed, and trained A. Neural Network Neural network classification was used to determine the parking states. The two networks, for daytime and nighttime, were two-layered feed-forward networks, with sigmoid hidden and output neurons. They took in the various features extracted from each parking spot as the input vector to give the predicted output as the parking states. In training, each network took a sample consisting of input features and the target output, which is the actual status of the corresponding parking spaces, manually collected using a developed annotator. This 24-hour video recording has 25 frames per second. The training data set for daytime used only 326 frames. However, one problem was that there were significantly less empty parking spots than the occupied ones. Taking sample for ground truth at an equal interval would result in proportionally fewer vacant spaces. These 326 frames thus span equally throughout the day, except in the evening, when with a doubled frequency was used to capture the period of higher parking activity with more empty spots. Nonetheless, the ground truth sample is not biased because the evening period is shorter than the daytime. In fact, doubling the evening sample makes the overall ground truth sample more representative of different lighting conditions. For nighttime, 112 frames were taken, spanning equally 4 hours after the light posts were turned on. This time interval has significantly more empty parking spots than the late night interval. Still, the sample is representative of the whole night period because at night the light intensity was constant. As each frame contained 126 observed parking spaces, the daytime training set had in total parking spots consisting of occupied ones and 1329 empty ones. The nighttime training set had parking slots consisting of occupied spots and 381 vacant ones. For both training data samples, each of the two networks daytime and nighttime further separated its sample into three groups. Firstly, it randomly selected 70% of the data for training to adjust according to its error using scaled conjugate gradient back propagation method. This compares the predicted output from the network to ground truth occupied or vacant and put back the error in the network to find optimal weight and bias that give the output closer to the ground truth. Secondly, the network used another 15% of the data for validation to measure the network performance. The network continued to train until the performance in this validation sample stops improving. The last 15% of the sample data provided an independent assessment of the network. The selected daytime network contained 11 hidden layers with 99.9% of occupied spots and 98.5% of empty spots from the training data identified correctly. The selected nighttime network contained 10 hidden layers with 99.9% of occupied spots and 98.4% of empty spots from the training sample classified accurately. The accuracy for the vacant spots was lower than the occupied ones in both time intervals, because of the much smaller size of empty spots in the training set. Another observation is that the misclassification in nighttime was equally spread throughout the time, while most of the 86

4 errors in daytime were concentrated in the early morning and in the evening. It confirmed the problem of limited sunlight in those two periods outlined in Problem 2 and suggested the need to approximate them to the other time periods that worked well with the selected daytime network. B. Features Features extracted from the parking spots are categorized into five categories: light-related features, pixel value statistical features, edge features, color-related features, and time-related features. Light-related Features Features in this category tackle the problem of light nonuniformity and varying light intensity. First, the color distance feature has an adaptive reference point. It must be noted that because the size of the each parking frame in this study is the same, the color distance is just the sum of the color distance of each pixel in the parking spot. The sum of the color distance per unit area would serve the same purpose. Second, the shadow status feature associated with each parking spot determines if it is in a non-shaded region, a shaded region, or in the image with no significant shadow. The values assigned are zero, one, and two respectively. Pixel-related Features This feature group considers dispersion and spatial arrangement of the image pixels of the parking spots. Cars are more likely to have greater spread and more non-uniform spatial arrangement of pixel values than the empty spots. Nevertheless, certain cars in limited light had a similar range of these statistical measures compared to the empty pavement. These measures therefore have two levels of strictness so that through training of the neural network, the ideal weightage can be found for each level of strictness to find the optimal contribution from these statistical measures. Less strict features are the standard deviation and entropy of the whole observed frame and the range of pixels. When cars are present, these standard deviation and entropy measures inevitably include the pavement pixels in the calculations, because cars often do not occupy the total frame of the parking slots. Although including the whole frame further increases the dispersion and randomness of an occupied spot, strengthening the indication of car presence, it may also include the shadows and overlapping of the adjacent cars falsely increasing the spread and randomness. The other feature is range, which measures the maximum difference of the pixel values in the parking spot. It is also less strict because it may include outliers of the pixel values. Stricter features are the standard deviation of an inner frame of the parking spot and interquartile range. This standard deviation reduces the noise from adjacent cars by taking in pixel values only from the inner fame, which is 20% the size of the actual observed frame. The interquartile range is robust to outliers, because it measured the difference between the twenty-fifth and seventy-fifth percentile of the pixel values, excluding the extreme values. Edge Features Again, edge features have two levels of strictness. The less strict one uses Canny edge detection method. Canny method is sensitive to weak edges, because it identifies edges from the local maxima gradient of the grayscale image. In addition, two different thresholds for weak and strong edges are used. It outputs not only strong edges and also weak edges that connect to the strong ones. Although this less strict edge detection can identify cars with similar colors to the empty pavement, it included unnecessary weak edges such as stains on the empty parking spot. The stricter edge feature uses Sobel method, identifying edges only at the maximum gradient of the grayscale image. Details of a car have to be much stronger for Sobel method to recognize. Color-related Features The color-related features are good indicators of colored cars. Maximum and mean values of the red, green, and blue filter of colored cars are likely to be higher than that of the empty parking space. The other feature, maximum difference in RGB filters, does a similar job in identifying colored cars from empty pavement. Grey tone colors have very close values in the red, green, and blue filter. A vacant spot is therefore more likely to have a smaller maximum difference between any two of the color filters. Time-related Features This feature group considers the information from the previous frame. When parking status changes from one frame to another, the features extracted show a significant change. If the spread, edge, and randomness values decrease from the previous frame, this suggests that the parking spot of interest in the current frame has now become empty. On the contrary, when the values increased, the parking spot is more likely to be occupied. This relative change at each new frame negates the effect of changing light conditions throughout the day, as it is robust to constant or slowly changing noise. Although these features are not useful when these parking states remain constant, they can be key features in determining the parking states during high traffic flow when parking states are frequently changing. C. Solution for Problem 1: Adaptive Pixel Value Reference The light-related features address the changing light intensity and non-uniformity. The color distance measures the sum of each distance between a pixel in the parking spot and the most prevalent pixel of the road pavement. This most prevalent pixel serves as an adaptive reference point, changing with different light intensity throughout the day. When light is not uniform, two most prevalent pixel values are returned each corresponding to a shaded and non-shaded region. The shadow status then determines if the parking spot of interest is in a shaded region. After that, the distance color is calculated, comparing each pixel in the parking spot to the corresponding reference pixel in the shaded or non-shaded region. A high color distance extracted from a parking spot suggests the presence of cars, while a low distance suggests a high degree of similarity to the empty pavement of empty space. 87

5 To find the most prevalent pixel value as the reference point, the images were first converted to YCbCr color space. The regions of empty pavement were marked out by hands along the region indicated in yellow strips in Fig. 4. A histogram of pixel values from all those regions was made. When the light is uniform, the distribution has one peak, shown in Fig. 4(c). However, when shadow moves in the image, the histogram shows bimodal distribution, seen in Fig. 4(d). These two situations would be now considered separately. lighting and non-uniform lighting due to the shadow. (c) Histogram of pavement pixels in uniform lighting corresponding to. (d) Histogram of pavement pixels in non-uniform lighting corresponding to (d). If the histogram of all the regions has only one peak, each pavement region is now considered individually to find the most frequent pixel values. These most prevalent pixel values are averaged to give the final reference pixel value for a uniform lighting condition. In the event that there are cars along the road pavement that could potentially distort the prevalent pixel value in a region, the value from that region could be discarded as an outlier. When the histogram shows two distinct peaks, one peak corresponds to the most frequent pixel value in the non-shaded region and the other corresponds to the shaded region. The mean of these two values is used as a threshold to distinguish between shaded and non-shaded region. Depending if each pixel exceeds the threshold value, each pixel is assigned one or zero. The biggest connected region in now a binary image is then identified, as seen in Fig. 5. The image is filtered to remove noises before being morphologically eroded to fill in the wholes to get the final separation of shaded and non-shaded region, as seen in Fig. 5(c). Each parking spot can then be identified if it is in the shade and the reference pixel value is one of the two peaks corresponding to the parking spot s current shade status. (c) (d) Fig. 4. Uniform and Non-uniform lighting. Yellow strips indicate the region of pavement used. and Comparison between uniform (c) Fig. 5. Shade Detection. Image with non-uniform lighting. Detect region with shade. (c) Final separation of the two region. 88

6 D. Solution to Problem 2: Approximating Sufficient Lighting As outlined in problem 2.2, the dispersion features and pixel values in color-related features are significantly lower in limited light intensity. In daytime, the image has high contrast due to the sufficient sunlight. Such images have a well spreadout histogram in gray scale or intensity layer. Therefore, with the right amount of histogram stretching, the poorly-contrasted image with limited light can be approximated to the time of sufficient lighting that the selected neural network works well. A measure of contrast can be extracted from a gray-level co-occurrence matrix constructed from an image of interest. Each element in this co-occurrence matrix, with index values i and j, indicates how often the pixel values i and j from the image of interest occurred in a specified spatial relationship. In other words, each entry in the matrix indicates the number of co-occurences between two pixel values, and that pixel values are the two index values of the entry. In this study, spatial relationship was specified to be adjacent to the left. From this constructed matrix, the contrast sums the product of the squared difference between the two index values and the entry value at the two index values. When the image is constant, the contrast from corresponding co-occurrence matrix is zero. It is found that the contrast value in the late evening declines rapidly (Fig. 6). Histogram stretching of poorlycontrasted image could compensates for the contrast. This was done by mapping pixel values that exceed a certain threshold to higher values. The threshold value decreases until the new contrast value reaches that of a well-contrasted image. In other words, more and more pixels are pushed to the right in the brighter region. The target contrast value was selected from the graph, shown in Fig. 6 to be 0.28, just before the contrast starts to fall. Stretching of histogram to this desired contrast value is illustrated in Fig. 7. Fig. 6. Contrast value over time. Contrast value declines rapidly in insufficient sunlight. Fig. 7. Comparison of image histogram before and after the stretching. The contrast value increases from in to in after the adjustment. E. Solution to Problem 3: Selecting Appropriate Regions for Reference Fortunately, because the lighting condition in nighttime remained constant, it did not need pre-processing. Still, the minor problem was finding appropriate regions to find the reference point for calculating color distance. At night, the light source was concentrated only along the parking spots, and not on the pavement. Color of the empty space was therefore significantly different from that of the pavement. The semicircle regions at the top of each parking lanes were chosen, because these pavement sections also received constant and similar lighting from the light posts. IV. RESULTS A. Accuracy Analysis The 24-hour video was analyzed using the selected networks for nighttime and daytime and with pre-adjustment to the images in limited sunlight. Throughout the video recording, frames were extracted for every 10 seconds, a duration that safely captured changes in parking states even in the high traffic period. From these frames, features of the parking slots were extracted and given to the neural network to determine the parking states. After that, every one frame for 60 analyzed frames was taken as a sample set to evaluate accuracy. This means that the predicted parking states for every ten minutes of the recording were verified. In total, parking spots were checked. Only 9 out of occupied spots and 11 out of 537 empty spots were misclassified. The accuracy was 99.9% for occupied spots and 97.9% for empty spots for this 24-hour video. B. 24-hour Statistical Analysis It is noted that, due to technical limitations in video recording, the 24-hour statistics has missing data for 40 minutes from 21:20 to 22:00 and 15 minutes from 08:45 to 09:00. These two time intervals are indicated by grey regions in the graphs to be discussed in this section. The car dynamics can be seen from a macro-view in the Free Space Plot over 24 hours in Fig. 8. It shows the number of 89

7 empty parking spots over time. The highest number is 33 out of 126 available spots or 26% of the total parking slots. The parking spaces start to free up from 6:00 to 9:00. People who park the cars overnight go for their daily work. After 9:00, the peak declined as people from other areas arrive in this area for work and park their cars. The second highest number of vacant spots is 18 or 14%, almost half of the highest number. It is between 13:00 to 14:00 suggesting that many people leaving in lunchtime. This peak in the afternoon has lower spread than that in the morning. This is probably because while people have different starting times for their morning work activities, people who leave for eating places generally go for lunch at the same time. Generally, the number of empty parking spots is below 10. This means that for most of the day, 92% of the parking slots remained occupied. Fig. 8. Free space plot over 24 hours. The number of free parking spots over time, out of 126 total spots. The rate of change in the number of free parking spots (Fig. 9) illustrates many positive and negative values alternating throughout the time, suggesting high rates of activity. When cars leave, there are new cars taking up the spots. The periods of consecutive positive values correspond to the two major peaks discussed. The highest increase in empty spots occurred at around 21:00 to 22:00, when people who finish their activities in the early night are likely to leave. V. CONCLUSION The proposed solution uses trained neural networks to determine occupancy states based on features extracted from parking spots. This method addresses three technical problems. First, it responds to changing light intensity and non-uniformity by having adaptive reference pavement pixel value to calculate the color distance between the parking spots in question and the pavement. Second, it approximates images with limited lighting to have similar feature values to images with sufficient illumination, merging the two patterns. Third, the solution separately considers nighttime vacancy detection, choosing appropriate regions to get reference color value pixels. The presented method relies only few pixels compared with other methods, being able to cover large number of parking spots with a single camera. Moreover, the system is robust to changing light conditions and light non-uniformity due to shadow from the surrounding. The accuracy for this 24-hour period is 97.9% for empty spots. Besides giving accurate depiction of the car park s utilization rate, this study also revealed the patterns of parking events at different time of the day and insights to the activities that car drivers engaged with. REFERENCES [1] G. Yan, W. Yan, D. B. Rawat, and S. Olariu, "Smart parking: a secure and intelligent parking system," IEEE Intelligent Transportation Systems Magazine, vol. 3, pp , [2] J. Wolff, T. Heuer, H. Gao, M. Weinmann, S. Voit, and U. Hartmann, "Parking monitor system based on magnetic field sensors," presented at the IEEE Intelligent Transportation Systems Conference, [3] D. B. L. Bong, K. C. Ting, and K. C. Lai, "Integrated approach in the design of car park cccupancy information system (COINS)," IAENG International Journal of Computer Science, vol. 35, pp. 7-14, [4] T. Fabian, "An algorithm for parking lot occupation detection," in 7th Computer Information Systems and Industrial Management Applications, 2008, pp [5] S. Funck, N. Mohler, and W. Oertel, "Determining car-park occupancy from single images," presented at the IEEE Intelligent Vehicles Symposium, [6] N. True, "Vacant parking space detection in static images," University of California, San Diego, [7] Q. Wu, C. C. Huang, S. Y. Wang, W. C. Chiu, and T. H. Chen, "Robust parking space detection considering inter-space correlation," presented at the IEEE International Conference Multimedia Expo, [8] C. Huang and S. Wang, "A hierarchical bayesian generation framework for vacant parking space detection," IEEE Transactions on Circuits and Systems for Video Technology vol. 20, pp , [9] A. Macdonell and J. Lobo, "Visual sensor for smart parking," Boston University, Fig. 9. Rate of change in number of free parking spots over 24 hours. Each net change is over 10 minutes. 90