System and Methodology for Usng Moble Phones n Lve Remote Montorng of Physcal Actvtes Hamed Ketabdar and Matt Lyra Qualty and Usablty Lab, Deutsche Telekom Laboratores, TU Berln hamed.ketabdar@telekom.de, matt.lyra@gmal.com Abstract I n t h s p a p e r, w e p r o p o s e a s y s t e m a n d m e t h o d o l o g y f o r u s n g m o b l e p h o n e s f o r m o n t o r n g p h y s c a l a c t v t e s of a user, and ts applcatons n assstng elderly or p e o p l e w t h n e e d f o r s p e c a l c a r e a n d m o n t o r n g. T h e m e t h o d s b a s e d o n p r o c e s s n g a c c e l e r a t o n d a t a provded by accelerometers ntegrated n new moble p h o n e s. A s t h e m o b l e p h o n e s c a r r e d r e g u l a r l y b y t h e user, the acceleraton pattern can delver nformaton r e l a t e d t o p a t t e r n o f p h y s c a l actvtes the user s e n g a g e d n. T h s n f o r m a t o n c a n b e s e n t t o a m o n t o r n g s e r v e r, a n a l y z e d a n d p r e s e n t e d a s d f f e r e n t h e a l t h r e l a t e d factors for assstance, montorng and healthcare p u r p o s e s. 1. Introducton Today, moble phones are becomng an essental devce n our daly lfe, and carred comfortably and regularly by a huge percentage of populaton. In ths paper, we propose a system and methodology whch can turn moble phones to devces for constantly montorng physcal actvtes of a user. Such an applcaton can send nformaton related to physcal actvty pattern of the user and related health factors to a server for montorng purposes. Ths can be especally useful for people who are n constant need for assstance and montorng, such as elderly or people wth specal movement or psychologcal dsorders. In addton to ths group of users, healthy people can also beneft form such an applcaton by constantly recevng reports on detals of ther actvty level and energy consumpton over dfferent desred perods of hours, days, etc. We propose to analyse user actvty based on data collected by accelerometers whch are ntegrated n new moble phones. Over the past few years, many moble phones are equpped wth sensors that can captur e nformaton related to physcal varables such as acceleraton. As the moble phone s carred by the user, these physcal varables, e.g. acceleraton recorded by phone s accelerometer can provde some nformaton related to physcal actvtes of user. Accelerometer sensors have been successfully used n several other applcatons, especally for sensng devce orentaton. Rekmoto [1,2] dscussed the potental of ths technque for tasks such as navgatng menus and scrollng. Hnckley e t a l. [ 3 ] d e monstrate how accelerometers could be useful for an automatc screen orentaton devce and scrollng applcaton. Oakley and O Modhran [4] descrbe a tlt based system wth tactle augmentaton for menu navgaton. Regardng the ssue of heath care and actvty analyss, there are a few moble phone based commercal applcatons especally for Phone platform [5,6]. These applcatons use data provded by GPS poston sgnal to analyse amount of actvty durng sports. However, these a p p l c a t o n s a r e d e p endent on avalablty of GPS sgnal for actvty level estmaton. In many places (such as home), GPS sgnal s not avalable due to coverage problems. In addton, lmted and mostly expensve moble phones are equpped wth GPS recevers, whle a c c e l e r a ton sensor ntegraton n moble phones s more wdespread and much less expensve. After all, user poston nformaton s not strongly correlated wth amount or pattern of user actvty. In contrast, acceleraton nformaton are strongly correlated wth amount and pattern of force exerted by user, and thus more drectly related to amount and pattern of user actvty. We propose to use data provded by moble phone s accelerometer for examnng actvty pattern of ts user. The moble phone equpped wth these sensors should be carred normally by the user n hs pocket. The results of examnaton can be presented to the user or a montorng agent n dfferent ways as ndcatons of dfferent health related factors. In addton to presentng these data to the user, the moble phone can then optonally analyse these data, or send t to a server for further analyss. The moble phone or server can analyse physcal actvty pattern of the user and compare t aganst normally accepted pattern for the same user, or normally accepted actvty pattern for the users of the same age. In the followng, we present our approach for analyzng actvty pattern of a user usng acceleraton data captured by a moble phone, and ts applcatons n montorng and asssted lf e. In ths work, we are mostly nterested n estmatng and montorng amount (level) of user actvty, as well as classfyng user actvtes nto certan groups. We also present some ntal experments and results. In
Secton 5, we talk about the setup and functonaltes of a system we developed based on the dea presented n ths p a p e r. 2. Analyss of Acceleraton Data Acceleraton sensors ntegrated n a moble phone provde lnear acceleraton nformaton along x, y, and z axs. In ths work, we assume that the moble phone s carred by a user n hs pant pocket. Most of usual daly actvtes nvolve movement of legs, therefore the best place to poston the moble phone (and acceleraton sensors) s pant pocket. Dfferent physcal actvtes results n dfferent patterns (sgnatures) n data provded by acceleraton sensors, and thus can be classfed accordngly. Fgure 1 s showng an example of acceleraton sgnals (along x, y, and z axs) captured by acceleraton sensors over tme (samples). T he data s captured over a consecutve sequence of walkng and restng scenaros. Dfferent actvtes (walkng or restng) has been marked n the fgure. As can be seen n the fgure, there s a sgnfcant dfference n pattern of acceleraton for dfferent actvtes. In ths work, acceleraton sgnal samples are sent to a server for further analyss, however the analyss can potentally be done on the moble devce as well. 2.1. Pre-processng Acceleraton pattern captured by a moble phone (when carred by a user) can be caused by dfferent sources. The acceleraton pattern s manly correlated wth physcal actvty of the user, however other factors such as beng n a vehcle can affect acceleraton pattern. For nstance, f the user s n a vehcle, acceleraton of the vehcle n a certan drecton can affect sensory data. As we are nterested n analyzng user physcal actvtes, other acceleraton sources such as those due to vehcle should be fltered out. Accordng to our studes, accelerat on pattern caused by physcal actvtes has hgher frequency content, whle other sources such as vehcle and gravty force result n lower frequency contrbuton. F o r p r e -processng step, we have used a hgh pass flter to remove low frequency components and preserve hgh frequency components whch are more representatve of actual user actvty. Ths also removes the constant component whch s due to gravty force. The hgh pass flter s appled on x, y, and z acceleraton sgnals. 2.2. Feature Extracton In ths work, we are nterested n estmatng actual level (amount) of user physcal actvty, as well as classfyng actvtes nto basc classes of walkng, runnng, restng, and no actvty (e.g. moble phone s left on a table). In order to acheve ths goal, we extract certan features from acceleraton sgnals whch represent dfferent actvtes n a dscrmnatve way. We have used absolute magntude of acceleraton, as well as the rate of change n absolute magntude acceleraton as featur es. Absolute magntude of acceleraton at a sample s defned as: 2 x 2 y a = a + a + a 2 z where a x, a y, and a z are acceleraton sample values along x, y, and z axs respectvely. Rate of change n acceleraton s defned as the dfference between absolute magntude acceleraton for current sample and prevous one. 2.3. Estmatng Actvty Level One of nterestng factors for montorng a user s hs level (amount) of physcal actvty. Acceleraton magntude ( a ) s correlated wth actvty level, however due to movements of legs, acceleraton pattern comes wth hgh frequency oscllatons. In order to extract actvty level, we measure the absolute dfference between a pck n acceleraton magntude and subsequent valley. Actvty level estmates can be presented to a montorng person (agent) at a remote server sde. A hstory of actvty level estmates can be also stored for browsng and analyss by a medcal doctor. Montorng user actvtes can be further facltated f the system on the server sde s able to assst the montorng person (agent) by classfcaton of actvtes. In ths way, a sem -automatc montorng scheme can be appled. Ths means that montorng by a person can be appled only n case of rsky user actvtes, or f the actvty level extends below or above a threshold (over certan perod of tme). In the next secton, we descrbe our approach for automatc classfcaton of user actvtes. 3. Actvty Classfcaton As mentoned earler, n addton to montorng user physcal actvty level, we are also nterested n classfcaton of user actvtes. Ths facltates analyss and browsng of user actvtes by the montorng agent at the server sde, and also helps the agent to detect rsky and emergency scenaros related to a certan user. The agent can choose to browse actvtes belongng to a certan class, search for certan events or actvtes, or get alarm n case of certan actvtes. Addtonally, the agent can montor several users when he s asssted by an actvty
classfcaton system. Snce actvty classfcaton can send alerts, montorng can be lmted only to cases whch a user s engaged n a rsky actvty. Therefore, the agent needs less concentraton for montorng and would be able to montor several users smultaneously. In ths work, we classfy user actvtes n 4 man classes: walkng, runnng, restng, and no actvty. We buld reference statstcal models fo r these classes durng a tranng phase. The statstcal models are bult usng features extracted from acceleraton sgnals. As statstcal model, we have used Gaussan mxture models (GMMs) [7]. For each class, a GMM s traned to maxmze lkelhood of nstances for that class: θ ˆ = arg maxθ p ( θ ) X Where θ s the set of parameters of GMM for class, whch s adjusted durng tranng to maxmze the lkelhood and obtanθ. Maxmzaton of lkelhood s done usng Expectaton-Maxmzaton (EM) algorthm [8]. Durng the test of the system, the traned GMM models are matched aganst actual nstances of acceleraton based features. The actvty class whch maxmzes lkelhood s selected as ongong actvty class of user: ˆ = arg max p ( X θ ) Where ) s the selected actvty class (result of classfcaton). The actvty classfcaton results can be presented to the agent along wth the estmaton of actvty leve l for a user by markng dfferent actvty dagrams wth dfferent colors. The actvty class label can be also stored along wth actvty level, n order to allow the agent to browse/search actvty data later. 4. Experments and Results We set up ntal experments for estmatng user physcal actvty level and actvty classfcaton. We have used Phone 3G [9] as the moble phone for the experments. Lnear acceleraton sgnals are provded along x, y, and z axs by the Phone accelerometer at 5 H z rate. We recorded a database of 320 actvty nstances wth 4 subject users. The database s portoned nto 208 actvty nstances for tranng and 112 actvty nstances for testng the system. There are four actvty classes: walkng, runnng, rest ng 1, and no actvty (e.g. moble phone on a table). Each actvty class has equal number of nstances n the database. Each actvty nstance lasts 10 1 Durng restng actvty class, the user s sttng on a char, and workng wth a laptop or wa t c h n g T V. seconds. The subject users carry the moble phone n ther pocket n a regular manner durng dfferent actvtes. The acceleraton sgnal s preprocessed as explaned n Secton 2. Features vectors are extracted for every sample of acceleraton sgnal. The actvty level estmaton step s done by measurng absolute dfference between consecutve pcks and valleys n acceleraton magntude. For actvty classfcaton, values of features are averaged over actvty nstance nterval (10 seconds n ths case), resultng n an average feature vector for every actvty nstance. Extracted features are used to t ran GMMs for each class. As mentoned before, we are nterested n classfyng user actvtes nto 4 classes of walkng, runnng, restng and no actvty. We have used 2 Gaussan mxtures for each class. The parameters of Gaussans are traned usng EM a lgorthm to maxmze lkelhood for each class. Durng testng of the system, extracted features are matched aganst models for dfferent classes. Each class model provdes a lkelhood score ndcatng the match between the actual actvty nstance and the model. Therefore, we obtan 4 lkelhood scores for each actvty nstance. The class havng hghest lkelhood score s selected as the outcome of actvty classfcaton. We have evaluated the actvty classfcaton system n terms of the accuracy n detecton of actvtes. The overall accuracy s 92.9%. Table 1 shows a confuson matrx for the errors. Ths table ndcates whch classes are mostly confusable. For nstance, we can see that a walk actvty nstance s detected 26 tmes as walkng, 2 tmes as runnng, and never as restng or no actvty class. Accordng to the table, confuson between walkng and runnng classes, also between restng and no actvty classes s hgher. Table 1. Confuson matrx for dfferent actvtes. Actvty Walk Run Rest No Act. W a l k 2 6 2 0 0 Run 2 2 6 0 0 Rest 0 0 2 5 3 No Act. 0 0 1 2 7 5. ActvtyMontor: The Developed System We have developed a system called as ActvtyMontor based on the dea presented n ths paper for lve remote montorng of several users physcal actvtes. In ths secton, we explan the setup and functonaltes of ths system.
In order to setup and operate the system, two applcatons are requred. The frst applcaton s nstalled and executed on an Phone. Ths applcaton sends acceleraton data through avalable data servce (W-F, GPRS, etc.) to a desgnated server. The user can set certan confguraton parameters such as a name for ongong experment and samplng frequency. In addton, the appl caton allows capturng data n snapshots n order to reduce traffc of the server. The second applcaton s a Java applcaton (ActvtyMontor) whch can be nstalled on any ordnary computer. Ths applcaton connects to the desgnated server and reads acceleraton data. The data s then analyzed and presented (to an agent) n real tme as physcal actvty nformaton, and dfferent statstcs and health related factors (Fgure 2). The applcaton s able to classfy actvtes n certan categores, and ssue warnngs n case of rregular actvty patterns. It can addtonally store the data, browse t, or search for certan actvty category. The ActvtyMontor screen has dfferent sectons. The man part of the screen s allocated by plots of actvty related data. In these plots, actvty level and category of the user, warnngs, and energy (calore) consumpton can be vsualzed (Fgure 3.a). The ActvtyMontor screen also comes wth a log feld on the left sde, whch provdes textual nformat on about the ongong actvty and dfferent statstcs (Fgure 3.b). There s also a settngs tab (Fgure 3.c) whch allows confgurng server connecton, managng data download, and formattng the data. The montorng agent can also choose to observe s t a tstcal data n a separate wndow (Fgure 4). The desktop applcaton can be used to montor several users smultaneously. It can automatcally analyze actvty data and detect unexpected patterns such as shocks or long perods of hgh or low actvty (Fgure 5). Upon detecton of an unexpected event for a certan user, the agent s nformed by a vsual or audo alert, and the montorng screen related to that user pops up. Ths allows montorng multple users n an automatc or sem-automatc manner. 6. Socal Implcatons of ActvtyMontor As mentoned n the prevous sectons, ActvtyMontor system turns regular moble devces to devces for constantly montorng physcal actvtes of users. Such a system can be provded to publc a s a software fo r dfferent moble devces, as well as a server and montorng centr e. Users who may wsh to be montored, can regster themselves n ths servce. The regstraton can be somethng smlar to regstraton for advanced servces such as MMS, Vop, etc. By r egsterng n ths servce and actvatng the respectve software on the moble phone, the user allows hs actvty nformaton to b e transferred to the server and b e montored by an agent. As mentoned before, the montorng process can be done n a sem-automatc manner due to the fact that the desktop montorng applcaton can automatcally check for unexpected patterns. T h e r e f o r e, montorng by a human agent can be necessary only f somethng unexpected happens. Ths allows possblty of montorng s e veral users by an agent smultaneously. As an alternatve, the system can b e p r o v d e d n a prvate manner. Ths means that the desktop montorng applcaton can be also sold as a software for personal use. In ths way, two p e o p l e (montorng agent and the user), or a group of people can establsh ther own mon torng process based on a local server. For nstance, one can personally take care of hs/her elderly parents usng such a system. In ths case, a software can be downloaded and nstalled on moble phone, and a second software (desktop montorng applcaton) p lus some space on a server should b purchased. All these steps can be done onlne very effcently whch means a prvate montorng system can be establshed wthn a few mnutes. Although such a system can be useful for healthcare purposes, as studed n [ 1 0 ], t can come wth some prvacy ssues. Smply, p eople may not feel comfortable wth havng ther actvtes always beng montored. Although ths can be an mportant ssue to be studed d e eply before commercalzng such systems, there are already some potental solutons. For nstance, the user may smply swtch of the montorng applcaton on hs moble phone, or leave t unattended when he d o e s no t want to be montored. Another soluton could be desgnng the montorng software n way that only very general nformaton about ongong actvty of the user such as actvty level can be transferred. Ths allows montorng the user and helpng hm n case of unexpected events, and n the same tme those not expose detaled nformaton to the agent. 7. Conclusons and Future Work In ths paper, we have presented a system and methodology based on processng data provded by moble phone accelerometer sensor for montorng physcal actvte s of users. As a moble phone can be convenently and constantly carred by a user, and t does not mpose burden of wearng extra sensors, such an applcaton can enable the moble phone to become a user frendly, precse and constant health and actvty montorng devce. In addton to montorng actvty level and actvty classfcaton whch s already dscussed n ths paper, such a system can be used for more detaled analyss of certan actvtes. For nstance, walkng pattern of a user
c a n b e a nalysed to see f there s any devaton form the normal pattern of walkng for the same user or users of the same age. Many dseases show ther early symptoms n changes of daly actvty pattern. Our system can be used for advanced analyss of certan actvtes and early detecton of certan problems, as well as montorng progress of user after a surgery or medcal treatment. Although such a system can provde a lot of advantages n terms of health care and montorng, dfferent socal mplcatons relate d t o t h e system such as prvacy ssue s should be consdered and studed before the publc use. 8. References [ 1 ] Rekmoto, J. 2001. Gesturewrst and gesturepad:unobtrusve wearable nteracton devces. n Proceedngs of Ffth Internatonal Symposum on Wearable Computers. [ 2 ] Rekmoto, J., Tltng Operatons for Small Screen Interfaces, U I S T ' 9 6, 1 6 7-1 6 8. [ 3 ] Hnckley, K., Perce, J. and Horvtz, E. 2000. Sensng Technques for Moble Interacton. n Proceedngs of ACMUIST, 91-1 0 0. [ 4 ] Oakley, I. and O'Modhran, M. 2 0 0 5. T l t t o s c r o l l : evaluatng a moton based vbrotactle moble nterface. In Proceedngs of World Haptcs: IEEE, 4 0-4 9. [ 5 ] http://www.runkeeper.com [ 6 ] http://www.mapmyrde.com [ 7 ] McLachlan, G.J. and Basford, K.E. 1988. Mxture Models: Inference and Applcatons to Clusterng", Marcel Dekker. [ 8 ] Blmes, J. 1998. A Gentle Tutoral of the EM algorthm and ts Applcatons to Parameter Estmaton for Gaussan Mxture and Hdden Markov M o d e l s. Internatonal Computer Scence Insttute, Berkeley CA. TR-9 7-0 2 1. [ 9 ] http://www.apple.com/phone/ [ 1 0 ] Frank Kargl, Elane Lawrence, Martn Fscher, Yen Yang Lm, "Securty, Prvacy and Legal Issues n Pervasve ehealth Montorng Systems," Moble Busness, Internatonal Conference on, pp. 296-3 0 4, 2008 7th Internatonal Conference on Moble Busness, 2008. Rest Rest Rest Walkng Walkng Fgure 1. An example of acceleraton sgnals captured by acceleraton sensors durng walkng and restng actvtes. It can be observed that pattern of acceleraton durng dfferent actvtes shows a sgnfcant dfference.
Fgure 2. ActvtyMontor desktop and montorng agent a b c Fgure 3. Dfferent parts of ActvtyMontor screen: a) man part whch vsualzes actual actvty of user, b) log vewer whch can show the ongong actvty class, warnngs, etc. as text, and c) control and settngs part whch can be used for adjustng dfferent parameters n relaton wth the connecton to server, vsualzaton, etc. Fgure 4. Vsualzng dfferent statstcs on health related factors Fgure 5. Warnngs n case on unexpected events