The Use of Camera Information in Formulating and Solving Sensor Fusion Problems

Transcription

1 The Use of Camera Information in Formulating and Solving Sensor Fusion Problems Thomas Schön Division of Automatic Control Linköping University Sweden Oc c

2 The Problem Inertial sensors Inertial sensors Inertial sensors Inertial sensors Camera Camera Radar Camera Barometer (altitude) Radar Wheel speed sensors Barometer (altitude) Terrain elevation DB Steering wheel angle sensors Might all seem very different at first sight. However, as we will see there is a general underlying sensor fusion framework that applies everywhere.

3 Sensor Fusion Def. Sensor Fusion Sensor fusion is the process of using information from several different sensors to compute an estimate of the state of a dynamic system. Surrounding infrastructure Sensors s Sensor Fusion Applications State estimation.. Dynamic model Sensor model.. State Estimation Dynamic models Sensors models

4 Dynamic Systems We are dealing with dynamic systems! Mathematics Application examples State variable Input signal Measurement Stochastic noise

5 Sensors It is important to realize that sensor fusion is a multi-disciplinary activity! It is not uncommon that each sensor is associated with its own discipline, Cameras Radar Laser Maps (GIS). Conclusion, you should not be afraid to enter and learn new fields!

6 Ego-Motion Estimation Using Night Vision Make use of sensor fusion to make better use of information already present in modern premium cars. Aim: Show how images from an infrared (IR) camera can be used to obtain better estimates of the ego-vehicle motion and the road geometry in 3D. Sensors Sensor Fusion Inertial sensors IR camera Wheel speed Steering angle State estimation Dynamic model Sensor model Estimates Nov vember 23, 2009

7 Dynamic Model Coordinate frames: World: Inertial fram. Body: Attached to the moving vehicle. Camera: Attached to the camera. Vehicle state vector: 3D position Longitudinal velocity Yaw angle Front wheel angle Pitch angle of the road Pitch angle of the vehicle Nov vember 23, 2009

8 Using the Measurements from the Infrared (IR) Camera Inverse depth parameterization of the landmark position relative to the camera. Landmark position Camera position at time t Landmark state vector: Camera position when the landmark is first observed. Mathematically we can treat the IR camera as if it was a standard camera. Nov vember 23, 2009

9 Sensor Fusion Algorithm 1. Initialize the vehicle states and the landmark states (first image). 2. If there are new proprioceptive measurements, do an EKF measurement update. 3. Predict the landmark positions in the new image. 4. Perform data association and detect and remove outliers. 5. Perform an EKF measurement update using the new camera measurements. 6. Perform an EKF time update. 7. Search for new landmarks in areas of the image where landmarks are missing. 8. Repeat from 2. Nov vember 23, 2009

10 Results Measurements recorded d during night-time ti driving i on rural roads in Sweden. Using CAN data and IR camera Only CAN data Showing the ego-motion estimates reprojected onto the images. Nov vember 23, 2009

11 Example 2 Fusing Inertial Sensors and Camera Objective: Produce high quality estimates of the position and orientation (pose) of a camera in real-time using measurements from inertial sensors and a camera. Schematic overview of the solution: Camera Computer vision 3-D Scene model IMU Sensor Fusion Position and orientation estimates The work was conducted within a 6 th framework EU project (MATRIS).

12 Example 2 Fusing Inertial Sensors and Camera Sensors Sensor Fusion Accelerometers Gyroscopes Magnetometer Camera State estimation Dynamic model Sensor model State t estimate Sensor unit developed within the project Position Velocity Acceleration Unit quaternion describing the orientation ti Angular velocity Bias term (gyroscopes) Bias term (accelerometer)

13 Example 3 Automotive Sensor Fusion Sensors Inertial sensors Camera Radar Wheel speed Steering angle Sensor Fusion State estimation Dynamic model Sensor model Estimates 1. Host vehicle motion 2. Road geometry 3. Leading vehicle motion

14 Example 3 Automotive Sensor Fusion Results in a nonlinear state estimation problem, which is solved using an extended Kalman filter. Vision only Reference Sensor fusion

15 Take Home Message Many different applications, all handled using the same underlying framework: 1. Model the dynamics 2. Model the sensors 3. Estimate the states using these models together with a suitable estimator and, do not underestimate the surrounding infrastructure Much interesting research remains to be done!