Application of the Automatic Incident Detection (AID) to the open section on highway in Japan Takehiko Kato 1, Hidenori Yamamoto 1, Nobuaki Miura 2, Kosuke Setoyama 1, Hiroshi Nagano 1, Masayuki Doi 2 1 Systems & Electronics Division, Sumitomo Electric Industries, Ltd., Japan 2 System Development Division, Sumitomo-Electric System Solutions, Ltd., Japan 1. Abstract On highway, to ensure safety of the road, changes of weather such as rainfall, snowfall, strong wind and so on as well as ever-changing circumstances including traffic congestion and accidents are monitored, and such information is immediately provided for drivers and necessary regulations are quickly taken. Visual information provided by CCTVs is as important as traffic flow measurements and weather information including wind velocity and temperature to understand road and traffic conditions. Therefore, CCTV cameras are installed at curves, in tunnels, at exits, entrances, and junctions of the IC (interchange), and around-the-clock surveillance is conducted at the surveillance center with multiple screens. Specifically, if a traffic accident occurs in a tunnel and many vehicles enter the tunnel without noticing the accident, multiple accidents including chain collision and fire (secondary disaster) may occur. In a long tunnel, to support surveillance staff s work, sudden incidents such as stopped or low-speed vehicles are detected by processing all camera images in real time. Then, drivers who are about to enter the tunnel are alerted by means of information boards (: Variable message signs) and signals, ventilation devices are halted to prevent expansion of fire, lighting devices in the tunnel are controlled, and the occurrence of the incident is notified to the police and patrol cars. Also in the open section on highway, the environment and circumstances greatly change due to rainfall, snowfall, sunlight, wind, and other variations in weather, curves, elevation differences, and incoming and outgoing traffic at interchanges. Thus, road administrators needs for the automatic incident detection have basically been high, and it has been already applied in some of open sections. Recently, popularization of CCTV cameras and progress of video compression transmission technology including IP (Internet Protocol) networks have increased the coverage of surveillance areas. Accordingly, detecting sudden incidents in the open section is more expected to be utilized to increase safety. SEI (Sumitomo Electric Industries) has supplied the AID s of more than 1,000 cameras for tunnels and open sections in Japan. Currently, SEI is further developing the image processing technology for the open section that has been established during the development of vehicle sensors for the traffic flow measurement and for both domestic and overseas traffic signal control.
SEI is working on an automatic incident detection for the open section that has a high affinity with a CCTV by combining the image processing technology with the image compression transmission and storage technologies. In this report, we will introduce SEI s AID and its efforts to cope with technical issues that arise when the AID for the tunnel section is applied to the open section. 2. Overview and configuration of the AID 2-1. Overview of the Obviously, when surveillance is conducted manually, it is not possible to watch more videos than the actual number of monitors (video display devices), which may lead to delay of the initial motion. Also, for the surveillance staff to constantly maintain concentration to detect abnormalities is a great burden on them. Since the AID constantly processes image data, there is no restriction of the number of monitors, and an incident can be detected in real time. Surveillance staff can closely check the incident location or the circumstances before and after the occurrence of the incident based on the information provided by the and by the use of the zooming and turning functions of the CCTV cameras. This makes it possible to effectively provide information for drivers via and also quickly notify the police and patrol cars of the incident. For the AID for the open section, while dedicated cameras are sometimes installed, it is sometimes required to be combined with the CCTV in terms of efficiency in the road administrator s equipment investment. A CCTV camera usually has zooming and turning functions, and detection of a sudden incident is possible when the camera is located at a specified preset position. Since the CCTV transmits video data, a large-capacity, high-speed IP network has been established, and the way of directly receiving IP video packets and processing the video data is becoming mainstream. SEI s can not only cope with it but also make it possible to store images including before and after the occurrence of incidents by use of its detection as a trigger. By collecting all of the image data in the central server, arbitrary images can be browsed. Analysis of those images can contribute to the investigation of the cause of the accident as well as the road improvement. As for image processing, basic operations are as follows: a vehicle is detected by the image processing methods including the background difference and the time difference known as popular ones, based on the trajectory and speed of the vehicle, whether the vehicle is stopping, slowly driving, or swerving and so forth is detected. SEI has been applying a variety of measures to reduce the effects of sunlight and so on to the image processing algorithm, which will be described in detail in Section 4.
2-2. System configuration As stated above, there are two cases in the AID configuration: a case where an image processing device is added to an existing CCTV, and a case where dedicated cameras for automatic incident detection are additionally installed. The first case has an advantage of using existing surveillance cameras; however, since those cameras are for surveillance, they may not be suitable for image processing. For example, the effects of vehicle s headlight reflection from the road surface on the rainy day and so on may cause a decrease in detection accuracy. On the other hand, the second case requires the installation of dedicated cameras; however, images with the best brightness or angle of view suitable for the image processing can be obtained, which enables the maintaining of higher detection accuracy than that of the existing camera. Images taken by the dedicated camera can also be used for the traffic flow measurement including the number of vehicles and speed as well as used for surveillance although the angle of view is fixed. Figure 1 shows a configuration example that uses images taken by an existing CCTV. Images captured by the CCTV are entered into an image processing device, where a sudden incident is detected by processing those images. When an incident is detected, it is possible to output information including the number of the camera that has captured the incident to the CCTV. This information is expected to be used so that the CCTV can fix the display images on the surveillance monitor to the relevant camera when an incident occurs. Also, when a sudden incident is detected, it is also possible to output information including the relevant camera number to the to alert drivers. Figure 2 shows a configuration example in which dedicated cameras for AID are additionally installed. An MPEG stream video from the dedicated camera is entered into the image processing device, where the video data is decoded to process the images. It is possible to convert the decoded video images into analog signals and output to the CCTV as surveillance images. Other collaborative operations with a CCTV or a are the same as those in the example that uses images taken by the existing CCTV. Camera Camera Camera Video CCTV control Video Preset signal Incident camera number etc. Image processing device Control Surveillance monitor control Incident camera number etc. Management center Figure 1: System configuration example using video images captured by existing CCTV
Camera device Camera unit Video compression unit Image processing device Video CCTV control Video (MPEG2) Transmission unit Measurement target point Fiber-optic network Video (MPEG2) Transmission unit Incident camera number etc. Surveillance monitor Control control Incident camera number etc. Management center Figure 2: System configuration example using dedicated cameras for AID 2-3. Advantages of introduction Advantages of introducing the AID are summarized in Table 1. Table 1: Advantages of introduction of AID No Item Contents 1 Improved safety Improved surveillance Since automatic incident detection reduces the work load on the surveillance staff, the staff can have remaining capacity to more carefully monitor the road conditions. 2 Quicker initial motion Incidents are always watched for without depending on visual inspections, which will lead to quicker initial motion. Also, video images that detected an incident can be displayed in a pop-up window on other surveillance screens. 3 Information provision Calling for attention Based on the detected incident, information can be provided via and signals to alert drivers. 4 Analysis of incident Improvement of road construction, etc. Statistical analysis of the recorded incident type and occurrence location will contribute to the improvement of road and facility conditions. Storage of images of both before and after the incident will enable the analysis of the process of the incident. 5 Expansion to TFM The AID can be used in combination with the traffic flow measurement (TFM) device that measures the road occupancy, average speed, and the number of vehicles, and determines the type of vehicles. This is helpful for making plans of road construction.
3. Image processing device SEI develops its original image processing device and dedicated camera device which are core devices of the AID. Configuration and features of the image processing device will be described below. 3-1. Configuration of the image processing device Figure 3 shows the configuration example and appearance of the image processing device. Image processing device CCTV control Analog video input (up to 16 videos) Analog video output (up to 16 videos) Image processing unit Up to 16 ch SW- HUB Video (MPEG2) up to 32 LAN (1000Base-T) Only when MPEG2 video is entered Analog video input (up to 16 videos) Analog video output (up to 16 videos) Image processing unit Up to 16 ch LAN (100Base-Tx) AID management terminal 2 m or less KVM switch AID control unit TFM control unit *1 KVM Video storage server *2 System console unit LAN (100Base-Tx) LAN (100Base-Tx) Preset signal Incident camera number, lane, incident type *3 control Incident camera number, lane, incident type *3 LAN (100Base-Tx) *1 Mounted when traffic flow measurement is provided *2 Mounted when the image storage function is provided *3 Only when the function is provided Figure 3: Configuration example and appearance of the image processing device
Each component will be overviewed below. (1) Image Processing Unit The image processing unit is compatible with both analog video and MPEG2 stream video. It is expected that analog video is entered from an existing CCTV and MPEG2 stream video is entered from a dedicated camera for the AID. In the latter case, images captured by the dedicated camera can also be sent as analog video signals to the existing CCTV. The image processing unit detects a sudden incident by processing entered images. (2) AID Control Unit The AID control unit stores the history of occurrence and resolution of sudden incidents. (3) AID Management Terminal In addition to the display of real-time situation of a sudden incident and the history of occurrence and resolution of sudden incidents, the AID management terminal can make settings of the automatic incident detection function. (4) TFM Control Unit The TFM control unit is mounted when the traffic flow measurement function is additionally provided. It stores the traffic flow measurement results measured by the image processing unit. (5) Video Storage Server The video storage server is mounted when the video storage function is additionally provided. It stores video images before and after the occurrence of sudden incidents. (6) System Console Unit The console unit consists of a KVM (a display, mouse, and a keyboard), and is used to make settings for the AID control unit, TFM control unit, and the video storage server. 3-2. Features of the image processing device Features of the image processing device are described below. 3-2-1. Main functions (1) Automatic incident detection Table 2 shows main functions of the automatic incident detection. Table 2: Main functions of the automatic incident detection Item Contents Type of incident Stopped vehicles, slow driving, traffic congestion, swerving, overspeeding, opposite-direction traveling, shoulder road traveling Detection range Up to 4 lanes, a section up to 140 meters Target vehicle Vehicle traveling at a speed of 160 km or less per hour Vehicle traveling crossing over the lane can also be detected. (2) Processing of multiple camera videos
One image processing device can process up to 32 videos, which contributes to space saving and cost reduction. (3) Combined use with the CCTV When an incident is detected, the image processing device can output information including the camera number that captured the incident and the type of the incident to the CCTV. The camera number is expected to be used for the CCTV control to automatically display the incident image on the surveillance monitor, and the type of incident is expected to be used for the CCTV control to overlay the contents of the incident on the screen. (4) Combined use with the When the needs to display speed warning message at the time an overspeeding vehicle is detected, if the surveillance staff operates the after the detection, the target vehicle may have passed the, which may result in missing appropriate timing. If the image processing device outputs the target camera number information to the s control when detecting an overspeeding vehicle, it is possible to automatically display a speed warning message on the at an appropriate timing. (5) Video storage before and after the incident and primary storage to semiconductor memory When an incident is automatically detected, videos captured for 60 seconds or more (at least 30 seconds before the incident detection plus at least 30 seconds after the incident detection) can be stored in the HDD of the video storage server. Those videos can be used for post confirmation of the incident and analysis of the incident occurrence mechanism. Since it is unknown when an incident occurs, to store video before the incident detection, images must be constantly written into a storage medium. However, if images are constantly written to the HDD of the video storage server, the service life of the HDD will be reduced. On the other hand, as a storage medium, the use of an SSD (Solid State Drive) that does not have a mechanical portion can be considered; however, the capacity of the SSD is comparatively small, resulting in high costs. In the light of the above, MPEG2 video is constantly written to a semiconductor memory on the image processing board (primary storage), and when an incident is detected, images stored in the semiconductor memory are transferred to the video storage server. Thus, the service life of the HDD of the video storage server is made longer. 3-2-2. Flexibility and extensibility of the architecture (1) Additional function In addition to the automatic incident detection, the traffic flow measurement function can be additionally provided. The traffic flow measurement function recognizes the type of vehicles (large, small), counts the number of passing vehicles, and measures the occupancy and average speed for each lane per one or five minutes by processing images similar to the automatic incident detection, and outputs the data. This makes it possible to detect traffic
flow in real time, and the data can be utilized for restricting the entrance of vehicles during traffic congestion. Also, conducting statistical analysis will contribute to road management including making plans of road construction and so on. Furthermore, the traffic flow measurement function, video storage function, combined use with the CCTV, and the combined use with the indication board can be freely selected and added according to the needs. 3-2-3. Reliability (1) Original image processing board SEI has developed a highly reliable, original image processing board that uses highly reliable parts and so on, for example, using no any aluminum electrolytic capacitors which deteriorates with passing age, since the circuit board is subject to 24-hour, 365-days continuous operation. (2) Adoption of SSD A highly reliable industrial PC is used for the AID control unit and the traffic flow measurement control unit that do not require a large-capacity storage medium as the video storage server. Also, an SSD is used as a storage medium instead of using an HDD that has a moving portion. Thus, the factor of mechanical failures is eliminated, and reliability of the device is increased. 3-2-4. Easy installation and setup (1) Setup by the original GUI software Setup procedures at the installation of the equipment can be conducted easily by the original GUI software installed in the PC. 3-2-5. Maintenance (1) Easy maintenance by centralized image processing Because centralized image processing is conducted at the management center instead of doing so at a local measurement point, usual maintenance does not require lane closure on site and maintenance can be carried out at one location. (2) Remote communication of the camera device When the dedicated camera device is used, remote maintenance can be conducted by the image processing device located at the management center via the IP communication. (3) Analog video output port and dedicated LAN port for maintenance Each image processing board is equipped with an analog video output port and a dedicated PC-connecting LAN port for maintenance. Therefore, it is possible to conduct maintenance work without affecting the normal operation and other image processing boards. (4) Power switch for each image processing board
Power switches are provided for each image processing board. Therefore, when an image processing board needs to be replaced, replacement work can be conducted without affecting other image processing boards operation. 4. Issues about automatic incident detection in the open section and our efforts A general problem of the image sensor is the decrease in detection accuracy under certain conditions. Unlike the inside of the tunnel where environment is stable, there are many factors that may cause the decrease in detection accuracy in the open section. Typically, sudden changes of sunlight, vehicle s headlight reflection from the road surface on the rainy day and so on are those factors. To cope with the above problem, representative software solution will be described. 4-1 Characteristics extraction Typical methods to extract characteristics from images include the difference (background difference) between the image (background image) in which there are no vehicles and the input image, and the difference (time difference) between the past image and the input image. With regard to the background difference, it is difficult to regularly update correct and stable background images. With regard to the time difference, there is a problem in that stopped vehicles cannot be detected in a stable manner. Accordingly, SEI adopts a characteristic, known as increment sign [1], which focuses only increase and decrease in the brightness value between adjacent pixels. Since the increment sign does not indicate the brightness value of individual pixels, but describes the magnitude relation between adjacent pixels, the sign is not essentially affected by the increase and decrease in the brightness of the entire image due to changes of sunlight. Furthermore, even in the scene where there are shades, the sign of the shaded portion does not change much in comparison with the scene where there are no shades. Therefore, possibility of mistakenly recognizing shadow of vehicles as a vehicle can be reduced. By combining the increment sign which is not subject to changes of environment with background difference and time difference, SEI makes it possible to extract characteristics more stably. 4-2 Recognition of vehicles As a vehicle recognition method, there is a method in which feature patterns of many vehicles have been stored beforehand and the type of vehicle is recognized by determining whether each input feature pattern matches a vehicle feature pattern that has been stored. However, it is usually difficult to prepare beforehand teaching data which is commonly effective in all fields. Accordingly, SEI adopts a method that uses perspective projection transformation. Based on the actual measurement data at several points on the road obtained at the installation of cameras, a relational expression (perspective projection transformation matrix) between the image captured
by camera and the actual road plane is obtained. As is known, perspective projection transformation makes it possible to uniquely project the each point on the coordinate (world coordinate ) as the origin is a certain point on the road plane onto the coordinate on the image plane (image coordinate ). Generally, perspective projection transformation is essentially irreversible transformation because it expresses dimensionality reduction from three dimensions to two dimensions. However, with limitation to a plane expressed by the world coordinate, it is possible to reversely project a point on the image onto the plane uniquely. Therefore, by choosing the road plane as the plane, it is possible to reversely project a point on the image onto the road plane uniquely. By the use of the above, the characteristic point group obtained in the previous process is labeled while attention is paid to noise components, and all of the characteristic points that are considered at the front (rear end) of a vehicle are reversely projected onto the road plane. Then, areas in which there is likely to be a vehicle are set for all of the reversely projected points, and a sign of vehicle is obtained for each area. Herein, the sign of vehicle means the score that is comprehensively calculated from the number of characteristic points and restrictions imposed when a vehicle is regarded as a rigid body. When the score is equal to or greater than a certain level of threshold, the object is determined to be a vehicle, and the size of the vehicle can be also determined by the score. Eventually, it is possible to determine vehicle s location and vehicle type on the road. 5. Conclusion We have developed the automatic incident detection with improved environment resistant properties that can be applied to the open section as well as the conventional tunnel section. Next aim is to cope with the problem in that when image processing is conducted by the use of conventional camera images, because of limitation of the number of pixels, as a target moves away from a camera, the vehicle speed measurement accuracy decreases and the range in which occurrence of incident can be determined is limited. We will continue to study to enable the automatic incident detection in a wider range by updating the function to be compatible with high-definition images of the HDTV camera to increase accuracy and measurement range. References [1] S. Kaneko, I. Murase, and S. Igarashi, Robust image registration by increment sign correlation, Pattern Recognition 35 (2002) 2223-2234