HIMA - project update: Human detection. Sensor Technologies for Autonomous Work Machines Jukka Laitinen VTT Technical Research Centre of Finland

Transcription

1 HIMA - project update: Human detection Sensor Technologies for Autonomous Work Machines Jukka Laitinen VTT Technical Research Centre of Finland

2 2 Topics HIMA - review Camera systems Radars (Juhana Ahtiainen) Laserscanners Algorithms Proximity detection (Jari Saarinen)

3 3 HIMA Objectives were: 1. Detection of humans in workmachine (working)area 2. Environment perception so that workmachine can safely move and carry out a work cycle Detection was done with sensor combination of multiple (different wavelengths) cameras and (automotive) radar Requirements for conditions of use were: Max speed for vehicle 60 km/h, temperature >100 C, detection area (stationary vehicle) <70 m, all-kinds of clothing possible

4 4 HIMA Four different use cases: Detection of humans with moving vehicle Detection of humans with stationary vehicle Detection of obstacles with moving vehicle Detection of obstacles with stationary vehicle Alue, jossa ihmisen tai esteen läsnäolo pystytään havaitsemaan Työkone Alue, jossa ihminen tai este pystytään paikantamaan. Työkone Liikesuunta. Alue, jossa ihmisen tai esteen läsnäolo pystytään havaitsemaan.

5 5 HIMA Used sensors: HDR camera (Gevilux Cam 3L) FIR camera (FLIR Photon 320) Stereo camera (Point Grey Bumblebee2) Wideangle stereo (Sony cameras with Fujinon optics) Radar (Bosch LRR2-LGU) Doppler radars (MDU1100T and HR-12)

6 6 HIMA Data fusion was done in 3D Distance information was needed With (single) cameras distance information was modelled with ambiquity Overall applicability to fusion: Stereocameras worked well FIR camera worked well when it was combined with other sensors HDR camera had some difficulties Radar gave accurate distance information but was unable to classify objects Doppler radar results were poor in this case

7 7 HIMA Test sets were collected from two different environments Vuosaari port and Otaniemi testcave In Vuosaari the best sensor combination was radar, stereocamera and FIR camera Detection rate was 95,0% with false detections of 0,1 / second Both the distance information and the classification were duplex With addition of HRD camera the detection rate was same Combination of FIR and radar the detection rate was 84,0% but with lower false detection rate (0,04 / second) In Otaniemi testcave the best sensor combination was FIR with stereocamera Detection rate was 96,6% with false detetions of 0,28 / second Radar got too many reflections which decreased it s performance

8 8 HIMA Data fusion improved the detection rate It was also possible to predict the position of earlier detected human (if it disappeared) Conclusions of data fusion were: Fusion module needed accurate distance information Own movement affected to position measurements and visibility time of objects Synchronization of sensors (time labels) and calibration of coordinate systems were very important

9 9 HIMA Sensors were divided to two groups 1. Early (detection/warning) sensors Radar, fast detection, lots of targets, no classification 2. Classifiying sensors Cameras, slow detection, object classification Multiple sensors complement each other System duplication (i.e. radar + stereo + FIR) Preprocessing (images are processed based on radar information) Sensor independent data fusion Difficulties: Cameras: high contrast situations, characters, symbols, images/posters, unexpected thermal differences (FIR) Radar: large obstacles near human

10 10 HIMA It was suggested that first pedestrian recognition systems should be driver assistant systems (not fully autonomous) Restricted area of operation would ease the job First target should be safety level of SIL1 and after that SIL2

11 11 Cameras on the market Automotive industry NIR/FIR cameras on some luxury models (with pedestrian detection), both active and passive Daimler, Toyota, Audi, BMW (Autoliv) -> driver assistance systems TOF cameras Blaxtair Mobileye C2-270

12 12 TOF cameras Typically sensor pixel array is 160x120 Typical range up to 10 m Connection USB2, Ethernet Many suppliers: Panasonic - D-Imager (indoor) SoftKinectic DepthSense (depth + colour image, indoor) Fotonic - many models (outdoor, Laser) MESA Imaging SwissRanger PMD Technologies CamCube3

13 13 Blaxtair Stereo camera system for pedestrian detection which is aimed for mobile industrial machinery Detects pedestrians within a range of 0 to 14 metres around the machines depending on the settings. Min/max (m) detection: 5/15, 1.5/6, 0.8/4 Alerts the driver and pedestrian of any risk of collision. Effective day and night, it is built to withstand harsh conditions (construction sites, mines, etc.). More information from the website:

14 14 Mobileye C2-270 One camera system which has different on-board applications: Daylight Pedestrian Collision Warning, including Bicycle Detection Forward Collision Warning, both in Highway and Urban areas, including Motorcycle Detection Lane Departure Warning Headway Monitoring and Warning Intelligent High-Beam Control Mobileye system is used in Volvo S60 and V60 for pedestrian detection (combined with radar)

15 15 HDR cameras Modern HDR cameras cover the VS+NIR spectrum Promising technology at the moment is stereo cameras with HDR technology OmniVision Many different models, i.e. sensor OV10630 up to 1280x800 pixels, dynamic range up to 115 db, 30 fps

16 16 Radars

17 17 Ultra-wide band (UWB) radars Radars Not a new technique but new applications Adverse weather Object detection in vegetation Through-the-wall irobot DareDevil Xaver 800

18 18 Laserscanners Sick LMS5xx series Detection up to 65 meters, response time 10 ms 5-echo technology

19 19 Laserscanners Sick LD-MRS (same as IBEO Lux) Multi-layer, multi-echo technology Velodyne HDL-32E / HDL-64E 40 vertical field of view 32 lasers are aligned from +10 to -30 vertical field of view Rotating head design delivers a 360 horizontal field of view Generates a point cloud of 700,000 points per second with a range of 100 meters and typical accuracy of +/- 2cm Prize range $ Velodyne HDL-32E

20 20 Sensor fusion Radar/LIDAR detections define Region of interest in camera images Automotive applications Estimation / Filtering Bayesian inference Multi-target tracking Social behavior

21 21 Sensor fusion Publications (or information) for different sensor combinations: Laser + FIR (Kalman filter) 1D laser scanner + VIS Radar + velocity + steering sensor + VIS/FIR Radar + thermopile, steering angle, ambient temperature sensor VIS + FIR stereo Radar + VIS + FIR

22 22 Algorithms Preprocessing Camera calibration Foreground segmentation the promising algorithms in foreground segmentation are the road based ones the presence of the stereo information for this task is mandatory for any future system Classification (simplified) HOG + SVM most promising one mostly focused on gradient-based features and several typical learning algorithms multiclass/multipart approaches are also gaining importance

23 23 Algorithms Tracking Kalman filter is the most used one More work is needed here Data fusion Ideal combination of sensors must be clarified Each sensor has its own difficulties More research is needed No public datasets for FIR images of pedestrians

24 24 Algorithms Best detectors combine multiple features, best accuracy is achieved by combining multiple features incl. motion Nighttime (or harsh condition) detection has been barely researched a system that works only at daytime, under good weather conditions (no heavy rain/snow/fog), over a range of distances up to 50 m is, from our viewpoint, the first intermediate challenge for the community - David Gerónimo on his PhD Thesis 2009

25 25 Conclusion? IEEE Transactions on pattern analysis and machine intelligence, vol. 32, no. 7, July 2010 Survey of Pedestrian Detection for Advanced Driver Assistance Systems, David Gerónimo, Antonio M. López, Angel D. Sappa, and Thorsten Graf After reviewing 108 published papers on pedestrian detection ( ) the conclusion was: However, the feeling is that we are still far from developing an ideal system.

26 26 Proximity detection Three candidates considered by MSHA 2009 Two were approved, while third was not yet tested All are based on active generation of electro-magnetic fields Criteria: Detect and alarm when in warning zone Power off when in danger zone Basics: Active EM field generated around the machine Magnitude corresponds to distance Requires calibration Predefined thresholds used

27 27 Proximity detection Nautilus international Coal-Buddy Omnidirectional antenna as transmitter in tue machine Placed roughly into the center of the machine Belt device as receiver Strata mining Hazard Avert Number of field generators placed on the machine Personal alarm device on a user Safety functions based on communication? Matrix design group The Matrix M3 The user has transmitter, machine has 4-6 receivers Tracking of tags? No use case data available?

28 28 Proximity detection

29 29 Proximity detection General: Proximity systems can be retrofitted Has to be calibrated Tradeoff between usability and safety The two first ones are based on a belt unit that detects the signal emitted by machine. Precalibrated fields. Based on initial analysis the Matrix system would be the most interesting, althought this requires additional investigation Multiple tracking units added easily Actual tracking

30 30 VTT creates business from technology