Media Adaptation Framework in Biofeedback System for Stroke Patient Rehabilitation

Transcription

1 Media Adaptation Fraework in Biofeedback Syste for Stroke Patient Rehabilitation Yinpeng Chen, Weiwei Xu, Hari Sundara, Thanassis Rikakis, Sheng-Min Liu Arts, Media and Engineering Progra Arizona State University, Tepe, AZ, 8528, USA E-ail: {yinpeng.chen, wwxu, hari.sundara, thanassis.rikakis, ABSTRACT In this paper, we present a edia adaptation fraework for an iersive biofeedback syste for stroke patient rehabilitation. In our biofeedback syste, edia adaptation refers to changes in audio/visual feedback as well as changes in physical environent. Effective edia adaptation fraeworks help patients recover generative plans for ar oveent with potential for significantly shortened therapeutic tie. The edia adaptation proble has significant challenges (a) high diensionality of adaptation paraeter space (b) variability in the patient perforance across and within sessions (c) the actual rehabilitation plan is typically a non first-order Markov process, aking the learning task hard. Our key insight is to understand edia adaptation as a real-tie feedback control proble. We use a ixture-of-experts based Dynaic Decision Network (DDN) for online edia adaptation. We train DDN ixtures per patient, per session. The ixture odels address two basic questions (a) given a specific adaptation suggested by the doain expert, predict patient perforance and (b) given an expected perforance, deterine optial adaptation decision. The questions are answered through an optiality criterion based search on DDN odels trained in previous sessions. We have also developed new validation etrics and have very good results for both questions on actual stroke rehabilitation data. Categories and Subject Descriptors J.3 [Life and Medical Sciences]: Health; H.5.3 [Group and Organization Interfaces]: Coputer-supported cooperative work; I.6.4 [Siulation and Modeling]: Model Validation and Analysis General Ters Algorighs, Experientation, Huan Factors, Keywords Biofeedback, Media adaptation, Dynaic decision network, Mixture of experts Perission to ake digital or hard copies of all or part of this work for personal or classroo use is granted without fee provided that copies are not ade or distributed for profit or coercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific perission and/or a fee. MM 07, Septeber 23 28, 2007, Augsburg, Bavaria, Gerany. Copyright 2007 ACM /07/ $ INTRODUCTION The goal of this paper is to develop a Dynaic Decision Network (DDN) based edia adaptation fraework for use in a ultiodal biofeedback syste [] for stroke patient rehabilitation. Stroke rehabilitation is an iportant proble every 45sec. soeone in the United States suffers a stroke, often leading to physiological ipairent. Effective edia adaptation fraeworks can potentially lead to significantly shortened therapeutic procedures (typically taking years). In this paper, edia adaptation refers to changes in audio/visual feedback as well as changes in physical environent. There has been prior work on biofeedback therapy. Virtual Reality (VR) is an eerging and proising technology for task-oriented biofeedback therapy to iprove otor function in individuals with stroke [5]. VR can provide an effective huan coputer interface that allows users to interact with a coplex, highly detailed and engaging ultiodal feedback in response to physical action. This has significant potential in augenting traditional task-oriented therapy training. It has been shown that task learning in a VR can be transferred into real world task perforance [7]. Holden et al. [5] utilized VR to train reaching and hand orientation of stroke patients by developing a virtual ailbox environent with different slot height and orientation. There has been extensive prior work on using dynaic odels in ultiedia (e.g. [2]) as well as in coputer vision (e.g. [9]). However since our research is analogous to a real-tie feedback control proble, the ost relevant work appears in robotic control / planning probles using Partially Observable Decision Markov Processes [8,]. There, Theocharous et al. [] use hierarchical POMDPs to represent ulti-resolution spatial aps for indoor robot navigation. This paper is otivated by these control probles and applies the to ultiedia research. The edia adaptation proble has four significant challenges. (a) the joint subject oveent and edia adaptation is high diensional, aking odel learning difficult (b) the patient perforance across sessions is highly variable due to external factors such as edication, lack of proper sleep (c) the perforance within session is affected by physiological factors such as being tired and (d) the physical therapist / doctors and artists plan two to three steps ahead this is a non first-order Markov process, aking the learning task hard. The ain contribution of this paper is in the developent of a ixture-of-experts based DDN fraework for online edia adaptation. The subject functional task is to reach for a target. The decision network contains three types of nodes adaptation decision (A t, observable), state describing the oveent plan (S t, hidden) and subject oveent observations (O t, observable). The

2 decision node A t represents the edia adaptation decision after subject oveent observations at tie t. We ake decision on changes rather than on values to ake the DDN paraeter learning proble tractable. The state node S t represents the subject hidden state in reaching/grasping plan acquisition. The seantics of the hidden state represent the goodness of the body plan (in oveent sequence) to reach the target. The observation node O t represents the subject s reaching/grasping perforance in the set t. We train DDN ixtures per patient, per session. In online adaptation, we dynaically search for a proper DDN odel fro all DDN odels trained in previous sessions and use the selected odel to give rehabilitation tea adaptation recoendation. Our fraework addresses two issues (a) given a specific adaptation suggested by the doain expert, predict patient perforance and (b) given an expected perforance, deterine optial adaptation decision. We have very good results for both questions on actual stroke rehabilitation data. The rest of the paper is organized as follows. In the next section we overview the biofeedback syste. In Section 3 discuss the edia adaptation proble in biofeedback systes and present the key challenges. In sections 4, we present the adaptation odel using DDN. We propose the perforance prediction algorith and adaptation recoendation algorith in the section 5. In section 6, we show how to use DDN based adaptation in biofeedback syste. We show the experiental results in section 7 and conclude this paper in section BIOFEEDBACK SYSTEM In this section we suarize our current biofeedback environent for stroke patient rehabilitation [] and discuss liitations for therapist decision aking. Our syste situates participants in a ulti-sensory engaging environent, where physical actions of the right ar are closely coupled with audio/video feedback. Participants are guided by our syste to explore the novel environent. Through exploration, the participants begin to discover rules ebedded in the environent. Those rules have been designed to couple action to feedback, consistent with the functional task. 2. Functional Task The functional task for our patients is reaching out with the right ar and grasping the target. This task was chosen because it is the basic ar oveent for stroke patient rehabilitation. This function task includes three sub-goals: (a) reach reach for and grasp the target successfully, (b) open open the ar to reach for the target without shoulder and torso copensation, and (c) flow reach for the target soothly. Therefore, our biofeedback environent encourages the subject to achieve these three subgoals through engaging and purposeful audio/visual feedback. 2.2 Syste Overview We now suarize our current biofeedback syste []. The syste integrates seven coputational subsystes (ref. Figure ): (a) Motion capture; (b) Task control; (c) Motion analysis; (d) Visual feedback; (e) Audio feedback; (f) Media archival and (g) Visualization. All seven subsystes are synchronized with respect to a universal tie clock. The otion capture subsyste we are using is produced by Motion Analysis Corporation. We use six near-infrared caeras running at 00 fraes per second to track the three-diensional position of reflective arkers that are placed on the subject. The task control subsyste provides a paraeter panel where we can adjust the paraeters related with the reaching and grasping task. The real-tie otion analysis subsyste soothes the raw sensing data, and derives an expanded set of task specific quantitative features. It ulticasts the analyzed data to the audio, visual and archival subsystes at the sae frae rate. The audio and visual subsystes adapt their auditory and visual response dynaically to selected otion features under different feedback environents. The archival subsyste continuously stores the otion analysis as well as the feedback data for the purpose of annotation and off-line analysis. Visualization subsyste visualizes the analysis results of subject s perforance [3]. Subject Motion Capture adaptation Task Control Artist Audio Feedback Doctor Rehabilitation Tea Motion Analysis Therapist Engineer Visual Feedback Visualization Media archival Proposed autoultiedia adaptation recoendation engine Figure. The flowchart of the biofeedback syste. The rehabilitation tea includes therapists, edical doctors, artists and engineers. The visualization subsyste helps the rehabilitation tea to ake the decisions about how to change the environent for achieving a successful rehabilitation. The syste can be fine-tuned through adjusting the paraeters in task control, audio feedback engine and visual feedback engine. In this paper, we are proposing an autoatic edia adaptation recoendation engine (ref. Figure ) that provides edia adaptation recoendation to be used by the rehabilitation tea. 2.3 Coupling Moveent to Feedback The structure of the feedback environent and its relationship to the achieveent of the goals are based on well established principles regarding the role and function of art [4]. At the highest level of its structure the environent ust counicate to the patient the essages that can encourage the accoplishent of the oveent goals. These essages are: reach, open, flow. The overall idea driving the appings is that spatial and target inforation is better counicated through visuals and coplex tie series data is better counicated through audio. The oveent paraeters allowing successful anipulation of the environent are the key paraeters of an everyday reaching and grasping oveent. Thus, the environent can be easily connected in ters of action to its goal and does not require specialized oveent training. Figure 2 shows visual feedback under four cases. The appings and content follow a siilar structural hierarchy as the oveent paraeters and goals with sub-essage levels supporting the counication of each larger essage. As is the

3 case of oveent paraeters, there are feedback paraeters that control the feedback generation. The subject can quickly understand the apping fro ar oveent to feedback. Through practice, the subject can control the feedback by oving the right ar to achieve the goals. The detail of oveentfeedback coupling is discussed in []. 3. MEDIA ADAPTATION In this section, we shall discuss the edia adaptation proble in our biofeedback syste, present the key challenges of adaptation proble and finally propose our solution. 3. The Proble The edia adaptation proble is how to adapt the biofeedback environent to help subject acquire a generative plan for reaching and grasping oveent. Matheatically, the proble is stated as follows: how to find optial change in feedback space f which results in an expected change in the observation space (patient oveent) O (ref. Figure 3). Patient perforance is variable both across subjects and within the sae subject. The use of edia adaptation allows the subjects under different conditions to successfully regain the generative plan for reaching / grasping the target. Figure 2. Visual feedback. Left-top: copletion of iage when subject grasps target successfully. Right-top: particles begin to for the iage as the hand approaches the target. Left-botto: Iage pulled to the right when subject is off target. Right-botto: Vertical bands appear when the subject has wrong target height. 2.4 Rehabilitation Procedure We now introduce the rehabilitation procedure by using our biofeedback syste. Let us denote every subject visit as session. For each session, there are several sets. In each set, the environental condition (physical state, audio and visual paraeters) reains fixed. Each set includes ten reaching trials. In each trial, the subject reaches out the right ar toward the virtual target, grasps the target and return back to the rest position. The rehabilitation tea (includes the therapists, doctors, artists and engineers) adapt the syste during the short break (typically two inutes) between two consecutive sets. The tea discusses the subject s oveent perforance through the visualization subsyste which shows the subject s perforance for the previous sets. Then the group decides how to fine-tune the syste (e.g. change usical instruent) to help the patient achieving a generative reaching/grasping plan. 2.5 Liitations of the current syste The ajor liitation of the current syste [] is that the edia adaptation is totally done by the huan expert (rehabilitation tea) without accurate data driven analysis. The huan expert ay iss the inforation about the relationships between decisions on edia adaptation and patient s oveent. This is because the diensionality of edia paraeters and oveent paraeters is very high (~00 paraeters) and each rehabilitation takes long tie (about 2 hours). Therefore, we are proposing an autoatic edia adaptation recoendation engine (ref. Figure ) that suggests optial edia adaptation to be used by the tea. Our proposed adaptation engine is data driven which analyzes the relationships between subject oveent and edia adaptation in previous rehabilitations. This adaptation recoendation subsyste will help the rehabilitation tea to ake correct adaptation decision. Figure 3. Diagra of edia adaptation. The environent adaptation includes five parts:. reaching/grasping task paraeters (e.g. target position) 2. audio feedback paraeters (e.g. usical instruent) 3. visual feedback paraeters (e.g. the speed of iage particles coing together) 4. physical environent paraeters (e.g. table height) 5. therapist instruction (e.g. focusing on elbow extension) We found that in patient rehabilitation, a cobination of these five adaptations works well for subject to achieve good reaching/grasping plan. In this paper, we address the autoatic data-driven recoendation for adaptation -3. We plan to address 4-5 in future work. 3.2 Why is it difficult? The edia adaptation is challenging because of four reasons:. High diensionality the diensionality of the space for both subject s ar oveent and environent paraeters are very high. There are ore than one hundred paraeters related with ulti-joint coordinated ar oveent training and they are correlated. And there are five groups of environent paraeters (ref. Section 3.). The diensionality of cobination of environent change is very high and the relationship between the iproveent of oveent perforance and environent change is coplicated and needs to be learnt fro the data. 2. Consistency we have had over 30 sessions with the patients and a key observation of stroke rehabilitation is that the subject physiological condition is different for each session. This can happen due to any reasons including, taking of edication, not sleeping correctly (i.e. posture) the night before therapy. Hence the rehabilitation tea can not ake perforance continuity assuptions (i.e. patient will ove at

4 0c/s etc); or ake assuption on what specific environent condition is optial. 3. Engageent it is a challenge for the subjects to reain attentive and otivated during a long and tedious session and they easily becoe physically and entally tired. Therefore, the edia adaptation should also consider how to hold attention through engageent of subjects. 4. Not st order Markov The artists and therapists do not only ake changes for the next set, but ay ake plans for set groups (say next three sets). However, learning higher order Markov processes is severely constrained due to liited data. In this paper, we still use the first Markov process to approxiate the real-world decision aking process. 3.3 Our Solution 4. DDN BASED ADAPTATION MODEL In this section, we present the adaptation odel based on Dynaic Decision Network (DDN) [0]. In Section 4. we show how a single DDN can be learned and used for edia adaptation. In Section 4.2 we shall generalize the idea to a ixture of experts. 4. Adaptation Model We use DDN to odel the relationship between adaptation decision and subject s oveent. We shall discuss three key coponents in adaptation odel: DDN structure, observation prediction and DDN learning in this subsection. 4.. DDN structure Figure 4. Diagra of autoatic adaptation recoendation for queries of rehabilitation tea. We present a huan-coputer joint adaptation fraework in our biofeedback syste. Our idea is to copute the relationship between subject oveent perforance and environent change and provide useful suggestions for the rehabilitation tea. Based on the autoated suggestions and therapist doain knowledge, the rehabilitation tea akes the decision for edia adaptation. Figure 4 shows the interaction between the rehabilitation tea and adaptation recoendation engine. The reason why a fully autoated adaptation is not the attepted is that it is very difficult to account for the extensive experience of the therapist. For exaple, this experience is very valuable in assessing input at the beginning of the session when subject state is only available qualitatively e.g. I do not feel too good today I slept on y couch last night, I took sleep edication last night. We propose to provide adaptation suggestions for the following: Q. Perforance prediction: given the target environent adaptation, the algorith provides expectation of subject oveent perforance for the next set. (e.g. what will happen if the instruent is changed to guitar?) Q2. Adaptation suggestion: given the expected subject oveent perforance, coputer adaptation provides the optial change of the environent. (e.g. what needs to be changed for straighter hand trajectory?) In this paper, coputer adaptation only suggests which paraeter should change, without considering the quantity of paraeter change. The ain reason is because of the high diensionality of the edia paraeter space and not enough evidence to ake accurate predictions. This suggestion by our syste is valuable as it draatically reduces the paraeters of the therapist to consider. The rehabilitation tea can then decide the exact quantity. In the following three sections, we discuss coputer adaptation odels. Figure 5 (a) Dynaic Decision Network structure for adaptation. A t is the decision (i.e. environent change) at tie slice t, S t is the state of subject oveent, O t is the observation of subject oveent. Shaded nodes are observable and blank nodes are hidden. Rectangular nodes are discrete and elliptical nodes are continuous. (b) Transition diagra for hidden node S t. The hidden node has three values:, 2 and 3. The transition probability P(S t =3 S t+ =2, A t ) represents the probability of transition fro state 2 to state 3 under the decision A t. We now propose the general adaptation odel in biofeedback syste. Figure 5 shows the DDN structure for adaptation. The tie slice in the DDN corresponds to the set (ref. Section 2.4). This is because we only change the environent (task, audio, visual, physical, and therapist instruction) between the sets and the environent is fixed for all trials within the set. At each tie slice t, there are three nodes: (a) adaptation decision A t, (b) state S t, and (c) observation O t. For the sake of conciseness, we use the ter decision to refer adaptation decision in rest of this paper. The decision and state nodes are discrete (rectangular nodes in Figure 5 (a)) and the observation nodes are continuous (elliptical nodes in Figure 5 (a)). The decision and observation nodes are observable (shaded nodes in Figure 5 (a)) and the state nodes are hidden (blank nodes in Figure 5 (a)). For exaple, let us assue the DDN odel is used for hand velocity adaptation in biofeedback. The adaptation decision A t could be the change of usical instruent. The hidden state S t represents the goodness of the subject oveent plan to achieve sooth velocity. The observation O t is the average hand jerkiness over ten trials in set t. Hand jerkiness is a well defined easureent in bioengineering. We shall discuss hand jerkiness later in this paper (eq. <3> in Section 6.3). We now discuss the nodes in the DDN (ref. Figure 5(a)) in detail: The decision node A t represents the edia adaptation between set t and set t+. The change of edia paraeter is the decision. For exaple, assue that we consider the usical instruent and tepo as the edia adaptation decision option.

5 Each paraeter (instruent or tepo) is a binary variable: (no change/change). Therefore we have four possible decision values by cobining these two usical paraeters. We ake decision on changes rather than on values to ake the DDN paraeter learning proble tractable. The state node S t represents the subject hidden state in reaching/grasping plan acquisition. The seantics of the hidden state represent the goodness of the body plan (in oveent sequence) to reach the target. We set three possible values for hidden state which indicate three different states in reaching/grasping plan acquisition. Figure 5 (b) shows the transition diagra between these three values. The transition probability P(S t+ S t, A t ) is represented as a conditional probability table (CPT): R={r i,j,k }={P(S t+ =k S t =i,a t =j)}. The observation node O t represents the subject s reaching/grasping perforance in the set t. It is the observable output of hidden state node S t. Since observation node is continuous, we use probability density p rather than probability P. We use Gaussian distribution for sensing probability density p(o t S t =i)=n(µ i, Σ i ), where µ i and Σ i are ean vector and covariance atrix when the value of hidden nodes S t equals to i Observation Prediction In this section, we address the first question posed in Section 3.3. We show how to predict the subject oveent perforance in the set t+ (i.e. Õ t+ ) given the decision and observation sequence for the pervious t sets using DDN odel. Let us denote the decision and observation sequence fro the first set to the set t as A :t and O :t respectively. The prediction includes three parts: (a) predict the probability of hidden nodes S t+ : P(S t+ O :t, A :t ), (b) predict the probability density of observation nodes O t+ : p(o t+ O :t, A :t ), and (c) predict the expected value of observation nodes: Õ t+. It is shown in [0] that P(S t+ O :t, A :t ) can be coputed as follows: P( S O, A ) = P( S S, A ) P( S O, A ), <> t+ : t : t t+ t t t : t : t S t where P(S t O :t, A :t- ) can be coputed iteratively as follows: P( S O, A ) = α p( O S ) P( S S, A ) P( S O, A ) t : t : t t t t t t t : t : t 2 S t <2> where p(o t S t ) is sensing probability density (single Gaussian pdf), P(S t S t-, A t- ) is transition probability, α is a noralization constant. Using eq<>, we can predict the probability density p(o t+ O :t, A :t ) as follows: p( O O, A ) = p( O S ) P( S O, A ). <3> t+ : t : t t+ t+ t+ : t : t S t+ Therefore, we can copute the conditional ean of O t+ as the prediction result: Oɶ ( O, A ) = E( O O, A ), <4> t+ : t : t t+ : t : t where E( ) is the expectation operator DDN Learning There are four kinds of paraeters that need to be learned: (a) Probability of decision: (b) Initial probability of state: η i =P(A t =i). π i = P(S =i). (c) Sensing probability density: b i (o t )=p(o t =o t S t =i). (d) Transition probability: r i,j,k =P(S t+ =k S t =i,a t =j). We use the EM algorith [2] to learn these paraeters. We assue that we have L training data, {y l }, l=,,l. Each training data is a teporal sequence of decision and observation (y l ={a l :Tl,o l :Tl}) where T l is the length of the sequence for the l th training data. For the sake of siplicity, we do not discuss the details of EM learning in this paper. The details can be found in [2]. Note that we assue a unifor prior for transition probability (i.e. P(S t+ = S t,a t )=P(S t+ =2 S t,a t )=P(S t+ =3 S t,a t )=/3) if the decision A t is not observed in the training data. 4.2 Mixture of Experts In this section, we extend the basic DDN described in section 4. to a fraework using ixture of experts. There are two key ideas: () In offline training, each session is considered separately. For each session, we use bagging to train K DDN odels. (2) In online adaptation, we dynaically search for a proper DDN odel fro all DDN odels trained in previous sessions and use the selected odel to give rehabilitation tea adaptation recoendation. Figure 6 shows the diagra of offline training and online suggestions. We discuss the offline training in this section and present the online adaptation algorith in the next section. Figure 6. Diagra for offline training and online adaptation suggestion. We train adaptation odels for session to session i- separately and select a proper odel for each set in session i to do online suggestion. D i,j is the j th DDN odel for the i th session Why do we use ixtures? We use ixture of experts (considering each session separately) because we find that the subject s perforance is weakly correlated. The subject s perforance is correlated across different diensions over different sessions. This prevents a single odel to give accurate prediction. We also find that there is no strong carry over effect even for two consecutive sessions conducted over consecutive days. By carry over we iply that the perforance values of the previous session can not be assued to continue to the current session. This is because subjects ay have different aount of use of their ipaired ar which results in different physical conditions of their ar ability. Therefore we train different sessions separately. We use bagging [3] to train K DDN odels for each session as we have liited training data Offline Training We now present the offline DDN training process using the data fro previous sessions. The offline training includes two parts: (a)

6 bootstrap training dataset and (b) DDN bagging. For the sake of definiteness, we show the offline training process for one session. The process generalizes for all sessions. We now show how we generate bootstrap training saples for DDN learning. Let us assue that we have T sets for current session and that each set has ten trials. We copute the observation easureent (e.g. hand jerkiness) for every trial. For each set, we use Huber s ethod [6] to copute robust ean and variance and detect outliers. Then we randoly select a trial fro each set to construct an observation sequence o :T. We replace this selected trial back to the set and repeat this rando selection until we have N observation sequences (o k :T, k=,..,n). For each set, we use unifor distribution on non-outlier trials and zero probability on outlier trials prior to rando selection. Figure 7 shows the diagra for generating N observation sequences. The decision sequence a :T- is the decision taken by the rehabilitation tea in the session, where a i represent the decision between set i and set i+. Therefore, we cobine the decision sequence and observation sequence to construct the N training saples y k ={a :T-, o k :T}. Using EM algorith (ref. Section 4..3), we can train an DDN odel on these N training saples for the current session. Figure 7. Construct N bootstrap training observation sequences. R i,j is the observation value for the j th trial in the i th set. We repeat the process of generating N bootstrap training saples K ties and hence create K training datasets. Each of these bootstrap datasets is used to train a different DDN adaptation odel for the current session. Therefore, we train K DDN odels for each session. The final online adaptation recoendation is done by selecting an optial DDN odel fro K DDNs and using its adaptation suggestions. The DDN odel is optial in the sense that it has the closest relationship between adaptation decision and oveent perforance with the current session. 5. ADAPTATION RECOMMENDATION We now present our online adaptation recoendation algorith based on ixture of DDN experts to address online adaptation recoendation. We shall address both two questions presented in Section 3.3: (a) perforance prediction and (b) decision suggestion. 5. Perforance Prediction The key idea of our perforance prediction algorith is that we first select an optial DDN fro all previous sessions dynaically, and use it to predict the perforance for the next set in the current session. The DDN is optial in the sense that it has the best prediction results prior to the current set. Let us assue that we have Q previous sessions. Each session has K trained DDN adaptation odels. Let us denote the j th DDN trained for the i th session as D i,j (i=,,q, j =,,K). For the current session, we copute the observation for each trial and copute robust ean [6] of observation for each set. Let us assue the current session has T sets. We denote the ean of observation of the t th set as o t and the actual decision between the t th set and the t+ th set as a t. Therefore the perforance prediction is forulized as predicting the observation for the set t+ (i.e. o t+ ) given Q*K DDN adaptation odels for previous sessions and the decision and ean observation fro set to set t (i.e. a :t, o :t ). We use the robust ean of observation over ten trials as the observation of the set because it is an iportant easureent for therapists to evaluate subjects. In this paper, we also refer the actual decision taken by the rehabilitation tea and the ean observation of set (i.e. a :t, o :t ) as ground truth decision and ground truth observation respectively. The initial environental paraeters are set by the rehabilitation tea based on observations of physical target reaching at the beginning of the session. At the end of the first set, therapist gives a decision query for the second set (i.e. a ). Since this is the first prediction for the current session, we do not know which adaptation odel can predict well. Hence, we use the edian of prediction results of all DDN odels as the prediction for the second set: oɶ ( a ) 2 edian[ o(2 D, a )] if Ω( a ) ɶ Di, j Ω ( a ) = Di, j i, j edian[ oɶ (2 D, a )] otherwise i, j, <5> where õ 2 (a ) is the expectation over the observation for the second set for decision a, õ(2 D i,j,a ) is the prediction result for the second set for decision a by using the DDN odel D i,j, Ω(a ) is the set of DDN odels D i,j that have decision a in the training data. The idea is that when we predict the results for the decision a, we should use the previous sessions in which the sae decision was taken. If this decision has not been used before (i.e. Ω(a ) is epty), we assue unifor prior transition probability and to predict the observation. Since we consider all DDN odels equivalently, the prediction results ay be noisy. Hence, we infor the therapist that this result ay not be very reliable. We can use the previous prediction error to help us find a good DDN adaptation odel starting fro the third set. We select the DDN odel with iniu prediction error before the current set as the adaptation odel to predict the next set. The prediction error until the current set (t) using odel D i,j (i.e. ε(t D i,j, a :t-, o :t )) is coputed as follows: t ε t Di, j a: t o: t = o k Di, j ak ok k= 2 (,, ) ɶ (, ), <6> where õ(k D i,j,a k- ) is the prediction result for the k th set for decision a k- by using the DDN odel D i,j using eq.<4>, o k is the actual observation for the k th set. This equation coputes the overall prediction error fro the beginning of the session to the current set (set t) by using DDN D i,j. Hence the optial DDN odel for the prediction of the t+ th set for the decision a t D * (t+ a t ) is represented as follows:

7 arg in[ ε ( t Di, j, a: t, o: t )] if Ω( at ) * Di, j Ω( at ) D ( t+ at ) =. <7> arg in[ ε ( t D, a, o )] otherwise Di, j i, j : t : t Therefore, using eq.<4>, we can copute the prediction results for the t+ th set by using DDN odel D * (t+ a t ): oɶ ( a ) oɶ ( t D ( t a ), a ), <8> * t+ t = + + t t where õ t+ (a t ) is the prediction result for the t+ th set for decision a t, õ(t+ D * (t+ a t ),a t ) is the prediction result for the t+ th set for decision a t by using the DDN odel D * (t+ a t ). 5.2 Decision Suggestion We now propose the decision suggestion algorith. The idea is to find the decision whose prediction result is the closest to therapist s expectation. For exaple, a therapist ight say if I want to iprove hand trajectory straightness by 0%, what decision should I take? In this section, we first discuss a general decision suggestion fraework by using utility function. Then, we apply a siple utility function to provide online decision suggestion for rehabilitation tea. Let us assue that the observation is scale variable in this section. It can be easily extended for vector variable. Let us assue that there are P possible decisions in the adaptation odel which constructs an decision set Φ={α,, α P }. For the case of adaptation odel for hand trajectory, we have 27 possible decisions (P=27). Let us denote Φ h as a subset of Φ in which all decisions are taken in the previous sessions. We only suggest the decisions taken in the previous sessions, because the DDN adaptation odels do not learn the cases of decisions that have not been observed and the suggestion ay have large error. The optial decision is the one that iniizes prediction error. This can be achieved by defining an appropriate utility function. In [0], it is shown that the optial decision at tie slice t is the decision that axiizes the expected utility: aɶ = arg ax[ u ( a )] t a t a Φ t h,<9> ua( at ) = U ( St+ ) P( St+ St, at ) P( St o: t, a: t ) St+ St where u a (a t ) is the utility of decision a t, P(S t+ S t, a t ) is transition probability, P(S t o :t, a :t- ) can be coputed using eq.<2>, U(S t+ ) is the utility function on the state nodes at tie slice t+. The utility function can be defined differently for different purposes by rehabilitation tea. In this paper, we define the utility as a function of prediction error: e e t+ t+ t = λt+ t+ t+ t+ U ( S o, a ) ( E( O S ) o ) e if oɶ t+ ( at ) < o <0> t+ λt+ = otherwise where E(O t+ S t+ ) is the expectation of observation at t+ th set given S t+, o e t+ is the rehabilitation tea s expected observation for the t+ th set, λ t+ is the sign indicator, õ t+ (a t ) is the prediction results for the observation at the t+ th set for decision a t using eq.<8>. The utility function is not only related to the state S t+ but also related to the rehabilitation tea s expected observation o e t+, and the decision a t. The utility is negative. The absolute value of the utility is the distance fro the observation prediction õ t+ to the rehabilitation tea s expected observation o e t+. The larger the utility (closer to zero), the saller the prediction error. λ t+ equals + when the predicted observation is less than the rehabilitation tea s expected observation and equals - otherwise. Maxiizing the utility is equivalent to iniizing the prediction error. In practice, for each possible decision, we search for the DDN adaptation odel to axiize the utility (or iniize the prediction error). Then we use the decision which axiizes the utility over all possible decisions as the recoended decision. We represent the decision recoendation in ters of iniizing the prediction error as follows: aɶ arg in( oɶ ( a ) o ), <> e t = t+ t t+ a Φ t h where ã t is the suggested decision between set t and t+, õ t+ (a t ) is the prediction result for the t+ th set for decision a t (ref. eq. <8>), o e t+ is the therapist s expected observation for the t+ th set. 6. ADAPTATION IN BIOFEEDBACK We now show how to use DDN based adaptation odel in biofeedback syste. We apply DDN odel on three key aspects of subject oveent perforance: (a) spatial accuracy, (b) straightness of hand trajectory and (c) jerkiness of hand velocity. These three aspects are iportant for stroke patient rehabilitation as they are connected to the reach, open and flow sub-goals respectively (ref. Section 2.). In this paper, we consider these three adaptations separately. Each adaptation has a DDN adaptation odel. For these three adaptations, we use the change of edia paraeter as the decision and we only consider the direction of the change (i.e. increasing/decreasing/no change). The exact quantity of paraeter change is decided by the therapist. The doain experts (therapists and artists) suggested that the adaptation should be over three different edia spaces: (a) reaching/grasping task control, (b) audio feedback and (c) visual feedback. For each adaptation, therapists deterine the goal for stroke patient rehabilitation. 6. Adaptation for Spatial Accuracy Figure 8. (a) Diagra for spatial accuracy, (b) diagra of adaptation decision for spatial accuracy. We now propose the edia adaptation for spatial accuracy. We describe the adaptation goal, the easure of the observation, the eaning of the state and the decision as follows: Goal: the goal is to help subject to reach close to the target. Observation: we use the distance fro the subject hand to the virtual target during grasping as the easureent of spatial accuracy (Figure 8 (a)). The subject achieves grasping when his/her hand stay in the grasping zone (Figure 8 (a)) for continuous 250s. In this paper, we only focus on the grasping zone on the table plane.

8 State: the state node has three values ( 3) that indicate three different body coputational plans to reach for the target. Decision: we have nine possible decisions in the adaptation odel for spatial accuracy. These are the cobination of the change of grasping zone radius and virtual target position Figure 8 (b). The change of grasping zone has three values (i.e. -: shrink, 0: no change, +: enlarge). The change of target position also has three values (i.e. -: ove closer, 0: no change, +: ove further). 6.2 Adaptation for Hand Trajectory We now present the goal, easure of the observation, the eaning of the state and the adaptation decision for hand trajectory. Goal: the hand trajectory adaptation is geared towards aking the hand trajectory straight. Observation: The straightness of hand trajectory G is defined as follows: ax( z) = ( ), <2> 0 G x z dz where x and z are x and z coordinates of subject s hand. The x and z axis is shown in Figure 9 (a). The straightness equals the area of trajectory region in Figure 9 (a). The saller the area, the straighter the hand trajectory. State: In the siilar to spatial accuracy adaptation, the state node in trajectory adaptation DDN has three values ( 3). Figure 9. (a) Trajectory curve and trajectory region along table plane. Four pictures are visual feedback corresponding to four positions along the trajectory curve. (b) Adaptation decision for hand trajectory (change x-z hull, change coalescing point and ove target position). Decision: We have 27 possible decisions hand trajectory adaptation which are the cobination of change of the x-z hull paraeter (size), coalescing point and target position (Figure 9 (b)). Each change has three values. The x-z hull is a forgiving region for subject hand oveent related to the sensitivity of the iage pulling in visual feedback (Figure 9 (a)). The coalescing point is a point on the z axis between the rest position and virtual target (Figure 9 (b)). It indicates the position where the iage particles coe together (coalescing) and controls the coalescing speed. The position of virtual target also has effect on trajectory. 6.3 Adaptation for Hand Velocity We now discuss the goal, observation, state and adaptation decision for hand velocity adaptation. Goal: the hand velocity adaptation attepts to ake the hand velocity sooth. Observation: We use a well understood easure (in Bioengineering) for velocity soothness (Jerkiness) as the observation of DDN odel for hand velocity. The jerkiness J is defined as follows: T d x d y d z J = dt <3> dt dt dt where T is the tie when grasping state is achieved, x, y and z are 3-D coordinates of subject s hand trajectory. The saller the value, the soother the hand velocity. State: In the siilar anner, the state node in velocity adaptation DDN has three values ( 3). Decision: We have 2 possible decisions in velocity adaptation. They are related to the change of usical instruent, tepo, and task difficulty. Musical instruent has two values (i.e. : change, 0: no change). Tepo has three values (i.e. -: decreasing, 0: no change, : increasing). Task difficulty has three values (-: aking task easier, : aking task ore challenge, 0: the difficulties of the consecutive sets are in the sae level). The task difficulty is annotated by the therapist. It relates to ultiple edia paraeters such as x-z hull, grasping zone, target position, etc. 7. EXPERIMENTAL RESULTS We now discuss the experiental results. Our biofeedback syste is in a state of continuous iproveent. Three stroke patients and twelve non-ipaired subjects are recruited to test the syste (30 sessions, 60 hours). Their data is not used in this paper since we keep debugging and iproving the syste in these 30 sessions. We recruited one stroke patient to use our biofeedback syste for rehabilitation after the syste is finalized. The recruited patient is iddle aged feale who suffered ild stroke in the right ar. The recruited patient was unfailiar with the syste prior to the rehabilitation. The patient did eight sessions in two consecutive weeks in January The eight sessions include one physical pre-test, one physical post-test and six sessions of rehabilitations using our biofeedback syste. Each session lasted approxiately two hours. The rehabilitations are lead by physical therapist that has one year experience of using our syste. We use the six sessions of rehabilitation as the experiental dataset. These six sessions have 5, 7, 4, 7, 9 and 9 sets respectively. 7. Validation Measure We now propose the validation easure for: (a) perforance prediction and (b) decision suggestion. Let us denote M as nuber of sessions (M=6), T as nuber of sets for the th session, a t as the ground truth decision (rehabilitation tea s decision) after the t th set in the th session, o t as the ground truth observation of the t th set in the th session. 7.. Validation of Perforance Prediction We use both ean of relative prediction error (e ) and edian of relative prediction error (e ed ) to evaluate perforance prediction algorith. The e and e ed is coputed as follows:

9 M T M ot oɶ t ( a ) t e = ( ) T 2 t 2 ot = = = 2, <4> M M ot oɶ t ( at ) eed = ( T ) Med ( T ) 2 2 t T = ot = 2 where õ t (a t- ) is the predicted observation of the t th set in the th session given the decision a t- (ref. eq. <8>), Med( ) is edian operator. e represents the average relative prediction error over all sets in all sessions. e ed averages the edian of relative prediction error over all sessions using the weight (T ). (T ) indicates the nuber of sets for which we predict the perforance Validation of Decision Suggestion We use three validation easures to evaluate the decision suggestion: (a) average rank, (b) relative utility ratio and (c) difference of prediction error. As we discussed in Section 5.2, we need to know the rehabilitation tea s expected observation before the decision suggestion. We use the actual observation o t+ (ref. Section 5.) as the expected observation and use the actual decision a t as the ground truth decision. This is because the observation o t+ is actually caused by the actual decision a t. The average rank is the average rank of ground truth decision (ade by rehabilitation tea) over all sets in all sessions. As we discussed in Section 5.2, we sort all possible decisions by the utility (ref. eq. <9>) in the descending order and select the first rank decision. We use average rank of ground truth decision to indicate the accuracy of our decision suggestion algorith. If the ground truth decision has low rank, it has the large utility to achieve the rehabilitation tea s expectation for the next set. The lower the average rank, the ore accurate the decision suggestion. The ideal case is that the average rank equals to one. This eans that all ground truth decisions are recoended. The average rank of ground truth decision is coputed as follows: r rank ( a o ) M ave = t t+ M = 2 T t= 2 T <5> where rank(a t o t+) is the rank of ground truth decision a t over all possible decisions given the rehabilitation tea s expected observation o t+. However, the average rank does not consider the utility difference between the ground truth decision and our recoendation. For exaple, if ground truth decision has higher rank, its utility ay be very close to the first rank decision. We should consider it differently with other high rank decision with uch saller utility. Therefore, we use the relative utility ratio (RUR) r u to evaluate online suggestion: [ ( ˆ ())] [ ( ))], <6> Et ua at Et ua at u ({ at }) = uax Et[ ua( aˆ t ()] r where {a t} is the set of ground truth decision, u a (â t()) is the utility (eq.<9>) of the first rank decision after the t th set in the th session, u a (a t) is the utility of the ground truth decision a t, u ax is the axiu of possible utility and E t is expectation operator on set. In this paper, the axiu utility u ax is zero which eans the predicted observation is exactly the sae as the rehabilitation tea s expected observation. For the ideal case in which all ground truth decisions are selected as the first rank decision, the relative utility ratio equals to zero. The saller the relative utility ratio, the ore accurate the online suggestion. The difference of prediction error easures the difference between the ground truth decision and our suggested decision (first rank decision) in ters of prediction error. The difference of prediction error (DPE) is coputed as follows: ot oɶ t ( at ) ot oɶ t ( aˆ t ()) d({ at }) = Et ot, <7> where õ t (a t- ) is the predicted observation of the t th set in the th session given the ground truth decision a t- (ref. eq. <8>), õ t (â t()) is the predicted observation of the t th set in the th session given the recoended decision (i.e. first rank decision â t()) and E t is expectation operator on set. 7.2 Results of Perforance Prediction We now discuss the experiental results of perforance prediction. In the training phase, we train three kinds of DDN adaptation odels for each session: (a) adaptation for spatial accuracy, (b) adaptation for hand trajectory and (c) adaptation for hand velocity. For each odel, we train K=0 (ref. Section 4.2.2) different DDN odels for each session. For each single DDN odel training, we generate N=50 observation training sequences using bootstrap technique (ref. Section 4.2.2). In the prediction phase, the observation for each set is the robust ean observation over ten trials in the set. The adaptation decision is ade by the rehabilitation tea. Note that not all possible decisions are taken in the rehabilitation (ref. Section 6). We observe 5, 4 and 5 adaptation decisions taken for spatial accuracy, hand trajectory and hand velocity respectively. We present two kinds of prediction result in this section: (a) online prediction and (b) offline prediction. In online prediction, we predict the observation for the session k using the DDN odels for the session to session k-, and we do not predict the observation for the first session. In the offline prediction, we predict the observation for the session k using the DDN odels for all sessions except the session k. While in practice, only online predictions are possible. The offline results are indication of the possible iproveent of the results with ore data. Figure 0 (a) (b) (c) shows the prediction results for spatial accuracy, hand trajectory and hand velocity respectively Table shows the average prediction error over six sessions using eq.<4>. Table. Online/offline prediction error (eq.<4>) for session 2 6 Adaptation Model Online prediction Offline prediction ean edian ean edian Spatial accuracy 8.58% 7.32% 6.02% 3.56% Hand trajectory 7.85% 4.02% 2.64% 9.27% Hand velocity 3.76% 0.60% 8.00% 6.40% We have two observations fro the experiental results:. Online prediction: the online prediction error (both ean and edian error eq.<4>) for three adaptations (spatial accuracy, hand trajectory and hand velocity) is low (less than 8%). This indicates that our DDN odel works well to odel the relationship between patient oveent and edia adaptation.

10 This also indicates that the algorith of ixture of experts works well for prediction. Note that copared to the single odel, ixture of experts significantly reduces the prediction error. The results for single odel prediction have been left out for the sake of brevity. We also observe the prediction error at the beginning of session is larger (especially for the second set) and the error decreases as the set nuber increases (Figure 0). This is because our expert selection is based on the prediction error of previous sets. The probability to find the proper DDN expert increases as ore sets coe. We also notice that predictions are not very good for soe sessions (such as session 2, 4 in spatial accuracy, session 3, 5 in hand trajectory, session 4 in hand velocity). This is because the oveent-adaptation relationship in previous sessions does not repeat in the current session. 2. Coparison between online prediction and offline prediction: The offline prediction has saller prediction results for sessions 2-5 (the online prediction is equivalent to the offline prediction for session 6). This is because that we can only use early session to predict the late session in online prediction. While in offline prediction, the siilarity between the early session and late session in ters of oveent-adaptation relationship results in good prediction for both early session and later session. We can see the prediction results of session 2 and 4 for spatial accuracy, session 2 for hand trajectory and session 4 for hand velocity are iproved significantly by using offline prediction. We understand that in practice, we can only do online prediction. We use the offline prediction to show that the online prediction results for soe early sessions are not good because we do not have enough data. As we get ore data, our algorith provides ore accurate prediction. 7.3 Results of Decision Suggestion We now discuss the experiental results for online decision suggestion. Table 2 shows average rank (ref. eq.<5>), relative utility ratio (ref. eq.<6>) and difference of prediction error (ref. eq.<7>) for online decision suggestions for three adaptation odels (spatial accuracy/hand trajectory/hand velocity). We can see that the average rank is close to one. This indicates that the ground truth decision has larger utility copared with other decision options. There are two reasons why soe ground truth decisions are not chosen as the first rank decision. First, the perforance prediction (related to the utility) for the ground truth decision is not accurate for soe sets. Second, other decisions ay result in siilar perforance to the ground truth decision. We observe that the relative utility ratio (RUR) and difference of prediction error (DPE) of the ground truth decision are very close to zero for hand trajectory adaptation which eans the ground truth decision and the first rank decision have very close utility and prediction error. The RUR for spatial accuracy adaptation and hand velocity adaptation are a little larger, but the DPE are very sall. The sall DPE eans that although soe ground truth decision are not rank first, their prediction error are very close to first rank suggestions. The RUR is a little larger because the prediction error of the first rank decision is close to zero which results in a sall denoinator in eq. <6>. We copare the Figure 0. Experiental results for perforance prediction. (a) Prediction results of spatial accuracy adaptation. The online prediction and offline prediction results overlap for session 3, 5 and 6. (b) Prediction results of hand trajectory adaptation. The online prediction and offline prediction results overlap for session 6. (c) Prediction results of hand velocity adaptation. The online prediction and offline prediction results overlap for session 5 and 6.

11 Table 2: Online suggestion results. The average rank is coputed by using eq.<5>. The average rank for spatial accuracy.65/5 eans the average rank is.65 and the nuber of observed decisions in rehabilitation is 5. In the last three coluns, the first nuber (out of bracket) is the relative utility ratio (RUR) (ref. eq. <6>), the second nuber (in bracket) is the difference of prediction error (DPE) (ref. eq.<7>). We copare the RUR and DPE of ground truth decision and the second rank decision and the last rank decision. Adaptation Model Average Rank Relative Utility Ratio (RUR) and Difference of Prediction Error (DPE) Ground truth decision The second rank decision The last rank decision Spatial accuracy.65/ (.78%).3 (7.02%) 6.44 (35.63%) Hand trajectory.48/4 0.2 (.54%) 0.99 (6.83%) 2.76 (44.59%) Hand velocity.77/ (5.79%).85 (.9%) 6.86 (38.35%) relative utility ratio (RUR) and difference of prediction error (DPE) for ground truth decision with the second rank decision and last rank decision. The RUR and DPE for the second rank decision can be coputed by using the second rank decision as the input eq.<6> and eq.<7> (i.e. r u ({â t(2)}) and d u ({â t(2)}) ). In the siilar anner, we can copute the RUR and DPE for the last rank decision. We can see that both RUR and DPE for ground truth decision are significantly less than the second rank decision and the last rank decision. This indicates that our online suggestion algorith works well. 8. CONCLUSION This paper presented a new fraework for edia adaptation, for biofeedback rehabilitation. The proble was challenging for several reasons (a) high diensionality of paraeter space, (b) within session and across session subject perforance variation and (c) doain expert decision aking was a non first order Markov process. Our key insight is to understand edia adaptation as a real-tie feedback control proble. We used a ixture-of-experts based Dynaic Decision Network (DDN) for online edia adaptation. The ixture of experts odel was adopted after we realized that patients exhibit significant variability across sessions. The expert ixture was trained using the failiar EM algorith. The odels were used to answer two basic questions (a) given a specific adaptation suggested by the doain expert, predict expected patient perforance and (b) given an expected perforance, deterine optial edia adaptation decision. The questions are answered through an optiality criterion based search on DDN odels trained in previous sessions. Our experients on the real stroke patient data show excellent results on both perforance prediction and adaptation decision recoendation. We plan to extend our research in several ways - (a) extend decision aking to changes in the physical set-up, and suggestions on when the therapist should discuss with the patient. (b) joint decision aking by incorporating the relationship between different aspects of patient's oveents (e.g. cobine spatial accuracy, hand trajectory and hand velocity), (c) prediction and decision suggestion for higher order Markov process rather than the first order Markov process. We also plan to continue our patient rehabilitation trials over the next few onths. 9. REFERENCES [] Y. CHEN, H. HUANG, W. XU, et al. (2006). The Design Of A Real-Tie, Multiodal Biofeedback Syste For Stroke Patient Rehabilitation, SIG ACM Multiedia, Santa Barbara, CA, Oct [2] A. DEMPSTER, N. LAIRD and D. RUBIN (977). Maxiu likelihood fro incoplete data via the EM algorith. Journal of the Royal Statistical Society 39(): -38. [3] R. O. DUDA, P. E. HART and D. G. STORK (2000). Pattern Classification, Wiley. [4] D. J. GROUT and C. V. PALISCA (200). A history of western usic. New York, Norton: xvi, 843, 86 of plates. [5] M. HOLDEN, E. TODOROV, J. CALLAHAN, et al. (999). Virtual environent training iporves otor perforance in two patients with stroke: case report. Neurology Report. 23: [6] P. J. HUBER (98). Robust Statistics, Wiley. [7] D. JACK, R. BOIAN, A. S. MERIANS, et al. (200). Virtual reality-enhanced stroke rehabilitation. IEEE Trans. Neural Syst Rehabil Eng 9: [8] L. P. KAELBLING, M. L. LITTMAN and A. R. CASSANDRA (998). Planning and acting in partiallly observable stochastic doains. Artificial Intelligence 0: [9] K. MURPHY, A. TORRALBA and W. FREEMAN (2003). Using the Forest to See the Trees:A Graphical Model Relating Features, Objects and Scenes. Proc. NIPS'03 (Neural Info. Processing Systes). Whistler, Canada. [0] S. RUSSELL and P. NORVIG (2003). Artificial Intelligence, A Modern Approach, Pearson Education, Inc. [] G. THEOCHAROUS, K. MURPHY and L. P. KAELBLING (2004). Representing hierarchical POMDPs as DBNs for ulti-scale robot localization, International Conference on Robotics and Autoation, [2] L. XIE, S.-F. CHANG, A. DIVAKARAN, et al. (2003). Unsupervised Discovery of Multilevel Statistical Video Structures Using Hierarchical Hidden Markov Models. Proc. IEEE Conference on Multiedia and Expo Baltoore MD, USA. [3] W. XU, Y. CHEN, H. SUNDARAM, et al. (2006). Multiodal archiving, real-tie collaborative annotation and inforation visualization in a biofeedback syste for stroke patient rehabilitation, 3rd Workshop on Capture Archival, Retrieval of Personal Experiences, in Conjunction with ACM MM 06, Santa Barbara, CA, Oct