Automatic Maritime Surveillance with Visual Target Detection Domenico Bloisi, PhD bloisi@dis.uniroma1.it
Maritime Scenario Maritime environment represents a challenging scenario for automatic video surveillance due to gradual and sudden illumination changes (e.g., clouds) motion changes (e.g., camera jitter) high frequency background objects (e.g., waves, raindrops) reflections Iniziativa Software 2 Page 2
System Architecture OBJECTIVE 3 DL KB SITUATION ASSESMENT + DL detection OBJECTIVE 1 IMAGE ANALYSIS tracking multiple camera data fusion feature extraction event understanding image classification validation radar + situation awareness AIS IR OBJECTIVE 2 DATA FUSION user OBJECTIVE 4 SITUATION AWARENESS
Objectives 1/4 1) Image Analysis Automatic analysis of the images coming from the cameras - Detection of the objects of interest - Tracking determines correspondences between a set of observations separated over time. Iniziativa Software 2 Page 4
Objectives 2/4 2) Data Fusion data coming from different sources - Video Day-Light - Video Infrared - RADAR - AIS are used to reduce the false positive detections Iniziativa Software 2 Page 5
Objectives 3/4 3) Situation Assessment a symbolic description of the observed scene is built - Feature based Image Classification Iniziativa Software 2 Page 6
Objectives 4/4 4) Situation Awareness An automatic reasoning system is able to respond appropriately as the situation evolves. - Track Split - Track Merge Iniziativa Software 2 Page 7
Detection Module OBJECTIVE 3 DL KB SITUATION ASSESMENT + DL detection OBJECTIVE 1 IMAGE ANALYSIS tracking multiple camera data fusion feature extraction event understanding image classification validation radar + situation awareness AIS IR OBJECTIVE 2 DATA FUSION user OBJECTIVE 4 SITUATION AWARENESS
Port Scenario A series of criticisms related to the port scenario have been individuated from the analysis of the literature: the use of Pan-Tilt-Zoom (PTZ) cameras, the presence of targets having very different size the presence of apparently motionless boats anchored off the coast. Iniziativa Software 2 Page 9
Background Subtraction VS Classification-based Detection Two different approaches Background Subtraction can be used on video sequences Classification-based Detection can be used with still images Iniziativa Software 2 Page 10
Foreground Extraction (Background Subtraction Technique) THRESHOLD T (based on illumination conditions) T foreground image current frame > background image blobs Iniziativa Software 2 Page 11
Background Subtraction Examples Waving Trees Water Surface Jug Page 12 12
Background Subtraction limitations Background subtraction is a common approach to detect moving objects in video sequences taken with a static camera. It is not straightforward to extend the approach to moving cameras. Cannot detect non-moving objects. Iniziativa Software 2 Page 13
Classification-based Detection current frame detection Iniziativa Software 2 Page 14
Pedestrian Detection Pedestrian detection in still images is considered among the hardest examples of object detection problems. Feature Extraction most informative object descriptors regarding the detection process are obtained from the visual content Detection the obtained object descriptors are utilized in a classification framework to detect the objects of interest. Iniziativa Software 2 Page 15
People Detection on Riemannian Manifolds Ensemble-of-feature approach (EOFA) (Tuzel et. al in PAMI 2008) A human is described as an ensemble of covariance matrices. Learning and feature selection is done simultaneously using LogitBoost of Riemannian Manifolds Iniziativa Software 2 Page 16
People detection: EOFA Results on Heathrow Zone A - Lobby Iniziativa Software 2 Page 17 17
Haar-like features POSITIVES NEGATIVES HAAR-LIKE FEATURES BASED CLASSIFICATION [Viola and Jones 2001, OpenCV] XML File Iniziativa Software 2 Page 18
Cascade of weak-classifiers Iniziativa Software 2 Page 19
Detection Scheme Input: Camera Frame, Haar Classifier Output: List of Observations (x,y) DETECTION XML file observations Workshop Iniziativa Software Page 20
Examples A LIVE test of the detector performance has been carried out at the VTS control center in Civitavecchia Iniziativa Software 2 Page 21
Sea-sky line vs Sea-coast line Since in presence of the coast the probability of finding false positives increases, erroneous detections laying above the seacoast line should be filtered out Sea-sky line Sea-coast line
Detection + Camera Heading Input: Camera Frame, Haar Classifier, Camera Heading Output: List of Observations, sea-sky line / sea-coast line Camera Heading DETECTION XML file sea-sky line observations Iniziativa Software 2 Page 23
Reflections and wakes Reflections and wakes on the water surface, can increase false positive detections Iniziativa Software 2 Page 24
Solution: Adding levels Iniziativa Software 2 Page 25
Detection Results Detection Rate False Alarm Rate Domenico Bloisi, Luca Iocchi, Michele Fiorini, Giovanni Graziano AUTOMATIC MARITIME SURVEILLANCE WITH VISUAL TARGET DETECTION Iniziativa Software 2 Page 26
Tracking Module DL KB OBJECTIVE 3 SITUATION ASSESMENT + DL detection OBJECTIVE 1 IMAGE ANALYSIS tracking multiple camera data fusion feature extraction event understanding image classification validation radar + situation awareness AIS IR OBJECTIVE 2 DATA FUSION user OBJECTIVE 4 SITUATION AWARENESS
Input: List of observations Visual Tracking Output: List of tracks (observation with an ID, filtered over the time) Tracking is crucial in filtering out false positives Iniziativa Software 2 Page 28
Validation Module OBJECTIVE 3 DL KB SITUATION ASSESMENT + DL detection OBJECTIVE 1 IMAGE ANALYSIS tracking multiple camera data fusion feature extraction event understanding image classification validation radar + situation awareness AIS IR OBJECTIVE 2 DATA FUSION user OBJECTIVE 4 SITUATION AWARENESS
VTS data VTS data are decoded in order to read boat information (name, System Track Number, latitude, longitude) and then projected on a Google Earth map Iniziativa Software 2 Page 30
Data Fusion First step: synchronization Time synchronization between VTS data and Video data Second Step: localization Localization of the VTS tracks navigating in the field of view of the camera Third Step: rotation VTS track rotation with respect to the camera heading Fourth Step: projection VTS tracks and Visual tracks are projected in a COMMON space Fifth Step: association Probabilistic association between VTS tracks and Visual Tracks Iniziativa Software 2 Page 31
VTS Track Transformation COMMON SPACE Iniziativa Software 2 Page 32
Visual Track Projection COMMON SPACE Iniziativa Software 2 Page 33
Data Association VISUAL TRACK VTS TRACK Iniziativa Software 2 Page 34
A Complete View DETECTION VISUAL TRACKING + VTS TRACKING SEA-SKY LINE EXTRACTION VTS DATA Iniziativa Software 2 Page 35
Multiple Radar Detections Visual analysis can help in correcting erroneous detection made by the radar. Detection results (a) are filtered over the time to obtain visual tracks (b) that are projected over a common space with radar and AIS tracks (c) Iniziativa Software 2 Page 36
Future Directions Iniziativa Software 2 Page 37
References Viola and Jones, Rapid object detection using a boosted cascade of simple features, CVPR 2001 Florian Adolf, How-to build a cascade of boosted classifiers based on Haar-like features Vadim Pisarevsky, OpenCV Object Detection: Theory and Practice Y. Freund, R.E. Schapire. A Decision-theoretic Generalization of On-line Learningand an Application to Boosting. Journal of Computer and System Sciences. 1997 Iniziativa Software 2 Page 38
Automatic Maritime Surveillance with Visual Target Detection Domenico Bloisi, PhD bloisi@dis.uniroma1.it