Understanding and Modeling of WiFi Signal Based Human Activity Recognition Wei Wang, Alex X. Liu, Muhammad Shahzad,Kang Ling, Sanglu Lu Nanjing University, Michigan State University September 8, 2015 1/24
Motivation Modeling Design Experiments Conclusions Motivation WiFi signals are available almost everywhere and they are able to monitor surrounding activities. 2/24
Motivation Modeling Design Experiments Conclusions Motivation WiFi signals are available almost everywhere and they are able to monitor surrounding activities. 2/24
Problem Statment WiFi based Activity Recognition Using commercial WiFi devices to recognize human activities. Wireless router Wireless signal reflection Laptop Advantages Work in dark Better coverage Less intrusive to user privacy No need to wear sensors 3/24
Problem Statment WiFi based Activity Recognition Using commercial WiFi devices to recognize human activities. Wireless router Wireless signal reflection Laptop Advantages Work in dark Better coverage Less intrusive to user privacy No need to wear sensors 3/24
Problem Statment WiFi based Activity Recognition Using commercial WiFi devices to recognize human activities. Wireless router Wireless signal reflection Laptop Advantages Work in dark Better coverage Less intrusive to user privacy No need to wear sensors 3/24
Problem Statment WiFi based Activity Recognition Using commercial WiFi devices to recognize human activities. Wireless router Wireless signal reflection Laptop Advantages Work in dark Better coverage Less intrusive to user privacy No need to wear sensors 3/24
Problem Statment WiFi based Activity Recognition Using commercial WiFi devices to recognize human activities. Wireless router Wireless signal reflection Laptop Advantages Work in dark Better coverage Less intrusive to user privacy No need to wear sensors 3/24
Challenges Measurement from commercial devices are noisy and have unpredictable carrier frequency offsets Needs robust and accurate models to extract useful information from measurements 75 70 65 60 11 11.5 12 12.5 4/24
Understanding Multipath Key observations Multipaths contain both static component and dynamic component Each path has different phase Phases determine the amplitude of the combined signal Wall LoS path Reflected by wall Sender d k (0) Receiver Reflected by body d k (0) 5/24
Understanding Multipath Sender Q LoS path Reflected by wall d k (0) Reflected by body d k (0) Static component Combined Dynamic Component I Wall Receiver 6/24
Understanding Multipath LoS path Reflected by wall Sender LoS path d k (t) Reflected by body Q Static component Combined Dynamic Component I Wall Receiver 6/24
Understanding Multipath LoS path Reflected by wall Sender d k (t) Reflected by body Q Combined Static component Dynamic Component I Wall Receiver 6/24
Understanding Multipath Interpreting amplitude Phases of paths are determined by path length Path length change of one wavelength gives phase change of 2π Frequency of amplitude change can be converted to movement speed Q Static component Combined Dynamic Component I 7/24
-Speed Model How accurate is it? Wave length 5 6cm in 5 GHz band 200 power 150 100 50 2.5 3 3.5 4 4.5 5 5.5 6 6.5 7 Waveform with regular moving speed amplitude changes are close to sinusoids 8/24
-Speed Model How accurate is it? Wave length 5 6cm in 5 GHz band 200 1 power 150 100 CDF 0.8 0.6 0.4 0.2 50 2.5 3 3.5 4 4.5 5 5.5 6 6.5 7 Waveform with regular moving speed 0 0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 Measurement error (meters) Moving distance measurement error amplitude changes are close to sinusoids Average distance measurement error of 2.86 cm 8/24
-Speed Model How robust is it? Robust over different multipath conditions and movement directions Linear combination of multipath do not change frequency Probability 0.25 0.2 0.15 0.1 0.05 running walking sitting down 0 0.2 0.4 0.6 0.8 1 1.2 1.4 Estimated speed (m/s) Speed distribution of different activities in different environments 9/24
-Activity Model Activities are characterized by Movement speeds Change in movement speeds Speeds of different body components 15 10 40 20 5 0 5 10 20 0 20 0 20 15 2 2.5 3 3.5 4 Walking 40 0.5 1 1.5 2 Falling 40 0 0.5 1 1.5 2 Sitting down 10/24
-Activity Model Use time-frequency analysis to extract features Use HMM to characterize the state transitions of movements Walking Falling Sitting down 11/24
-Activity Model Build one HMM model for each activity Determine states based on observations in waveform patterns State durations and relationships are captured by transition probabilities State 1 State 2 State 3 State 4 12/24
System Architecture measurement collection Activity data collection Noise reduction Online monitoring Activity detection and segmenting Feature extraction HMM based activity recognition HMM training Model generation HMM Model Monitoring records 13/24
Data Collection 30 subcarriers N M 30 streams 75 75 70 70 65 65 60 60 11 11.5 12 12.5 11 11.5 12 12.5 75 75 70 70 65 65 M 60 75 60 11 11.5 12 12.5 75 11 11.5 12 12.5 N 70 65 70 65 60 60 11 11.5 12 12.5 11 11.5 12 12.5 75 75 70 70 65 65 60 60 11 11.5 12 12.5 11 11.5 12 12.5 75 75 70 70 65 65 60 11 11.5 12 12.5 60 11 11.5 12 12.5 14/24
Noise Reduction Correlation of on different subcarriers Subcarriers only differ slightly in wavelength Subcarriers have the same set of paths, with different phases 312.5kHz Wave length = 5.150214 cm Wave length = 5.149662 cm Frequency 15/24
Correlation in Streams 16/24
Noise Reduction Combines N M 30 subcarriers using PCA to detect timevarying correlations in signal 75 70 65 75 70 65 60 11 11.5 12 12.5 Original 10 11 11.5 12 12.5 Low-pass filter 5 0 5 10 11 11.5 12 12.5 PCA 17/24
Real-time Recognition Activity detection - Use both the signal variance and correlation to detect presence of activities Feature extraction - Time-frequency analysis (DWT) HMM model building - Eight activities Walking, running, falling, brushing teeth, sitting down, opening refrigerator, pushing, boxing - More than 1,400 samples from 25 persons as the training set 18/24
Evaluation Setup Commercial hardware with no modification - Transmitter: NetGEAR JR6100 Wireless Router - Receiver: Thinkpad X200 with Intel 5300 NIC A single communicating pair is enough to monitor 450 m 2 open area Measurement on UDP packets sent between the pair Sampling rate 2,500 samples per second 19/24
Evaluation Results True activity Activity recognized R W S O F B P T E Running 1.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 Walking 0.000 1.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 Sitting 0.000 0.000 0.947 0.030 0.011 0.000 0.012 0.000 0.000 Opening 0.000 0.005 0.150 0.803 0.042 0.000 0.000 0.000 0.000 Falling 0.000 0.010 0.041 0.010 0.939 0.000 0.000 0.000 0.000 Boxing 0.000 0.000 0.000 0.000 0.000 1.000 0.000 0.000 0.000 Pushing 0.000 0.000 0.000 0.000 0.000 0.000 1.000 0.000 0.000 Brushing 0.000 0.000 0.000 0.000 0.000 0.000 0.000 1.000 0.000 Empty 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 1.000 Ten-fold validation accuracy: 96.5% Detects human movements at 14 meters Real-time recognition on laptops Packet sending rate can be as low as 800 frames per second 20/24
Evaluation on Robustness Models are robust to environment changes Train once, apply to different scenarios Training use database collected in lab with different users Test in with users not in the training set Open lobby Apartment (NLOS) Small office Fridge Table Training location Walking/running route 7.7 m Lab Testing location Kitchen Table Bath room Tx Fridge Rx 1.6 m Tx Rx Table 6.5 m Appartment 21/24
Evaluation on Robustness Consistent performance in unknown environments, with more than 80% average accuracy 1 Accuracy 0.5 0 R W S O F B P T lab lobby apartment office Environments 22/24
Conclusions measurements contains fine-grained movement informations -Speed model quantifies the correlation between value dynamics and human movement speeds -Activity model quantifies the correlation between the movement speeds of different human body parts and a specific human activity Our models are robust to environment changes 23/24
Q & A Thank you! Questions? 24/24