Principles of inertial sensing technology and its applications in IHCI Intelligent Human Computer Interaction SS 2011 Gabriele Bleser Gabriele.Bleser@dfki.de
Motivation I bet you all got in touch with inertial sensors already What can be done with inertial sensors? Think of common devices that you use Smartphones, digital picture frames, mobile PCs: Align image content Fall detection Sports & fitness products: Step count/length, velocity, travelled distance Activity classification, sleep analysis, etc. Game controller: Gesture recognition Motion sensing (control by physical interaction) Pointing 22.06.2011 Lecture IHCI SS 2011 2
Motivation What do you know about inertial sensors? Accelerometers: Measure linear acceleration (body acceleration + gravitational acceleration) When non accelerating: indicate up down direction Gyroscopes: Measure angular velocity around an instantaneous axis (turning rate) 22.06.2011 Lecture IHCI SS 2011 3
Motivation: trends in the game industry Yesterday: Classic controller (gamepad, joystick) Button presses, stick control Today: Controller with motion sensing capability (mostly inertial, magnetic and optical sensors) Gestures, motion physical interaction more intuitive Games involving physical interaction (fitness & health) 22.06.2011 Lecture IHCI SS 2011 4 http://www.wikipedia.org
Motivation: game controllers Wii MotionPlus + Sensor Bar 3D accelerometer: Detect rapid motions Roll and pitch angles 3D gyroscope: 3D turning rate 3D orientation Distinguish body acceleration and acceleration due to gravity Distinguish position and orientation Optical sensor: Detect LED clusters of Sensor Bar pointer, roll angle, distance 3D position (if 3D rotation known) Reminder: insideout tracking 22.06.2011 Lecture IHCI SS 2011 5
Wii MotionPlus + Sensor Bar Derive roll angle from angle of detected LED clusters Can pitch and yaw be derived from horizontal and vertical shift of the detected LED clusters with respect to principal point alone? Requires information from inertial sensors! Image plane Roll angle 22.06.2011 Lecture IHCI SS 2011 6
Motivation: game controllers PlayStation Move + Eye Camera 3D accelerometer, 3D gyroscope, 3D compass in the controller: Drift free 3D orientation Compass: drift correction for yaw angle Temporary dead reckoning for position (during occlusion) LED orb + external camera: Detect orb 3D controller position (distance by size of orb in image) Z Reminder: outside in tracking 22.06.2011 Lecture IHCI SS 2011 7
Outline Until here motivation and a lot of (new) terms: Gyroscope, turning rate Accelerometer, body acceleration vs. gravity Compass, heading direction 3D orientation: yaw, pitch, roll angles Drift (correction) Dead reckoning Now the technologies and principles behind 22.06.2011 Lecture IHCI SS 2011 8
Outline 1. Inertial sensor basics 2. Inertial measurement units (IMUs) 3. Orientation estimation principles 4. Orientation and position estimation principles 5. Outlook: advanced applications 22.06.2011 Lecture IHCI SS 2011 9
Inertial sensor basics Why do we call accelerometers and gyroscopes inertial sensors? Their functionalities are based on the principle of inertia, stating the resistance of an object to a change in its state of motion or rest/to be accelerated. Many different types and categories of inertial sensors available Here: micro machined electromechanical systems (MEMS) technology Small size, low weight, low power consumption, etc. But also reduced accuracy and bias stability 22.06.2011 Lecture IHCI SS 2011 10
Accelerometers Principle (of mechanical type): A spring suspended mass in a housing will be displaced when subjected to a force The displacement is proportional to the specific force and can be measured The output is an electrical signal that by calibration can be related to the physical quantity 22.06.2011 Lecture IHCI SS 2011 11
Accelerometers Measurement in 1D: Specific force, f, in direction of sensitive axis, n: Sensitive axis Accelerometers measure the difference between body acceleration and gravity acceleration compared to free fall Gravity Acceleration Assuming perfect calibration: What does an accelerometer measure when lying still with the sensitive axis leveled? 9.81 m/s 2 (assuming positive axis points up) What is measured in free fall with sensitive axis leveled? 0 m/s 2 22.06.2011 Lecture IHCI SS 2011 12
Linear velocity: Reminder: translational motion Linear acceleration: Position: Holds in 3D with vectors Initial position 22.06.2011 Lecture IHCI SS 2011 13
Gyroscopes Principle (vibrating mass type): A mass is actuated to vibrate in direction r act and a displacement is measured in perpendicular direction r cor An angular velocity, ω, perpendicular to the plane induces a Coriolis force, which results in a proportional displacement along r cor From this ω can be calculated Measurement in 1D: Angular velocity, ω, around the sensitive axis 22.06.2011 Lecture IHCI SS 2011 14
Reminder: circular motion Coriolis acceleration: A person moving northward towards the outer edge of a rotating platform must increase the westward speed component (blue arrows) to maintain a northbound course. The acceleration required is the Coriolis acceleration. 22.06.2011 Lecture IHCI SS 2011 15
Angular velocity: Reminder: rotational motion (1D) Rotation: In 3D, a bit more involved (later) Initial orientation 22.06.2011 Lecture IHCI SS 2011 16
Inertial measurement units (IMUs) Triads of gyroscopes and accelerometers to obtain 3D measurements + compass triad to obtain 3D earth magnetic field Calibrated to provide all measurements in one orthogonal righthanded body coordinate system (typically aligned to housing) in physical units, typically at 100 Hz Commercially available IMUs: Magnetometer Wireless Trivisio Colibri Wireless InertiaCube3 Xsens MTi 22.06.2011 Lecture IHCI SS 2011 17
Inertial measurement units (IMUs) Typical coordinate system definitions: IMU coordinate system (s) Global reference system (g) Taken from Xsens MTi/MTx User Manual 22.06.2011 Lecture IHCI SS 2011 18
Inertial measurement units (IMUs) Measurement models in 3D (idealized!): 3D gyroscope [rad/s]: 3D accelerometer [m/s 2 ]: IMU orientation with respect to global frame 3D compass [tesla or gauss]: under no magnetic disturbances, measures magnetic north 22.06.2011 Lecture IHCI SS 2011 19
Earth magnetic field http://www.magnetic declination.com Magnetic north Inclination angle Field strength 22.06.2011 Lecture IHCI SS 2011 20
Accelerometers: acceleration/gravity ambiguity z y y z Ambiguity! Once we know the IMU s rotation, we can separate body acceleration and acceleration due to gravity. 22.06.2011 Lecture IHCI SS 2011 21
Orientation estimation principles Assume perfect measurements and negligible body acceleration: the measurement tells us, where down is: How do we know, whether acceleration is present? The measurement provides the last column of the IMU s rotation matrix roll and pitch angles. The yaw angle (rotation around global z axis) can t be determined. 22.06.2011 Lecture IHCI SS 2011 22
Orientation estimation principles Naive solution under negligible body acceleration: Accelerometer provides negative z axis of global frame in IMU frame yields last column of required IMU rotation matrix Magnetic north How can we use the magnetometer information? Yields y axis of global frame in IMU frame 22.06.2011 Lecture IHCI SS 2011 23
Orientation estimation principles What about the gyroscopes? Naive solution using gyroscopes: integrate angular velocity measurements to obtain absolute rotation Easy in 1D: integration based on rectangular rule yields: In 3D: Angular velocity vector describing turning rate around instantaneous rotation axis 22.06.2011 Lecture IHCI SS 2011 24
3D rotational kinematics In 3D: Integration of angular velocity based on rectangular rule: For the derivation of the differential equation and the matrix exponential as required for integration see, e.g., [Woodman 2007, Shuster 1993, ] Relative rotation in axis angle representation resulting from constant angular velocity, ω, over time, δt Rodrigues rotation formula 22.06.2011 Lecture IHCI SS 2011 25
Orientation estimation principles Bad news: IMU measurements are not perfect! More realistic models including bias and noise terms: Zero mean white noise (typically modelled as Gaussian) Even worse are magnetic disturbances! What does this mean for naive solution based on accelerometers and magnetometers? Jitter and systematic error What does this mean for naive solution based on gyroscopes? Error accumulates over time (drift) 22.06.2011 Lecture IHCI SS 2011 26
Orientation estimation principles Solution? Sensor fusion! Gyroscopes provide short term indication of rotation (depends on, e.g., bias stability and noise scale, independent of acceleration) Accelerometers provide drift correction for roll and pitch angle during periods of negligible body acceleration Magnetometers provide drift correction for heading direction during periods of no magnetic disturbance Typically, a statistical filter (e.g. extended Kalman filter) is used for fusion [Rehbinder and Hu 2001, Harada et al 2007, ] Bias terms can also be estimated Applications, e.g.: Head tracking for VR (HMD) 3D pointing devices Reminder: improved motion sensing of Wii MotionPlus and PlayStation Move 22.06.2011 Lecture IHCI SS 2011 27
Orientation and position estimation principles Dead reckoning: Reminder: PlayStation Move What problems do you expect here? Additional references required, e.g. visual information Applications, e.g.: 6 DOF camera tracking for AR Inertial navigation systems (aircrafts, submarines, spacecrafts much better sensors!!!) 22.06.2011 Lecture IHCI SS 2011 28
Outlook: advanced applications Body motion tracking Pedestrian tracking (NavShoe) X X IMU integrated in shoe to estimate travelled distance Body worn IMU network to capture human motions 22.06.2011 Lecture IHCI SS 2011 29
References Inertial sensors: O. J. Woodman: An introduction to inertial navigation. Technical Report UCAM CLTR 696, University of Cambridge, Computer Laboratory, Aug. 2007 D. Titterton and J. Weston: Strapdown Inertial Navigation Technology, American Institute of Aeronautics and Astronautics, 2004 Orientation estimation: T. Harada, T. Mori and T. Sato: Development of a Tiny Orientation Estimation Device to Operate under Motion and Magnetic Disturbance, The International Journal of Robotics Research, 2007, 26, 547 559 H. Rehbinder and X. Hu: Drift free attitude estimation for accelerated rigid bodies, IEEE International Conference on Robotics and Automation (ICRA), 2001 Rotation representations and rotational kinematics Shuster, M. D.: A Survey of Attitude Representations, The Journal of the Astronautical Sciences, 1993, 41, 439 517 22.06.2011 Lecture IHCI SS 2011 30
We are searching for students in this area! Contact: Gabriele Bleser (Dr. Ing.), Senior Researcher German Research Center for Artificial Intelligence (DFKI) Department Augmented Vision Trippstadter Straße 122, 67663 Kaiserslautern E Mail: Gabriele.Bleser@dfki.de 22.06.2011 Lecture IHCI SS 2011 31