gaming Motion tracking technologies for rehabilitation

Chapter 1. gaming Motion tracking technologies for rehabilitation Chapter 1 gaming Motion tracking technologies for rehabilitation Albert de Graaf University of Twente 19-7-2010 Abstract Recently, there have been several motion tracking devices introduced by the gaming industry, like the Nintendo Wii with the Wiimote and the PlayStation3 Eye. This in-depth paper research investigated if these techniques can be used for home based upper extremity rehabilitation. The results are promising: Although current applications are not ready to hit the market yet, test results show that the techniques meet the requirements and already proved to help in rehabilitation. 1.1 Introduction Stroke is the second leading cause of death and disability worldwide 1, with about 795,000 Americans experiencing a stroke yearly 2. After six to nine months physical therapy is generally stopped, despite rehabilitation science evidence of the potential for improving motor function years after stroke [Holden and Todorov, 2002][Merians et al., 2002]. Acute rehabilitation currently concentrates on the lower extremity. Theferore, 30 to 66% of stroke survivors will not regain use of their affected arm [Truelsen and Bonita, 2003]. The literature is clear that intensive, repetitive practice of functional movement helps people recover [Butefisch et al., 1995][Feys et al., 1998][Dickstein et al., 1997]. Importantly, self-improvement does not stem directly from the therapist coaching and guiding, but rather from information processing by the subject with consolidation of everything inherent to the task leading to a refined motor output[winstein et al., 1994]. Thus, subjects should be given opportunities to train themselves to improve their (control of) upper extremity motions, especially arm and hand motion. The easiest place to do that would be at home, where people can train whenever they want. Recent developments in the gaming industry might be enabling the use of gaming consoles for home-based rehabilitation. With the introduction of the Nintendo Wii with the Wiimote 3, PlayStation 2 s (PS2) EyeToy 4, PlayStation 3 s (PS3) Eye 5 and the anouncement of Microsoft Kinect 6 and the PS3 Move 78 there is a whole new range of motion tracking devices available for gaming consoles. 1.2 Research question The research question, which follows from the introduction: 1 http://www.theuniversityhospital.com/stroke/stats.htm, last visited on juli 17, 2010 2 http://www.strokecenter.org/patients/stats.htm, last visited on juli 17, 2010 3 http://nl.wii.com/wii/nl NL/index.html, last visited on july 17, 2010 4 http://www.eyetoy.com/index.asp, last visited on july 17, 2010 5 http://us.playstation.com/ps3/accessories/scph-98047.html, last visited on july 17, 2010 6 http://www.xbox.com/en-us/kinect/default.htm, last visited july 17, 2010 7 http://us.playstation.com/ps3/accessories/scph-98060.html, last visited july 17, 2010 8 http://us.playstation.com/ps3/accessories/scph-98061.html, last visited july 17, 2010 Gaming motion tracking technologies for rehabilitation - Albert de Graaf 1

P XEL Simulations & Games Which existing motion tracking technologies used in the game industry are usable for implementing upper extremity home based rehabilitation? In order to answer the research question, the following aspects need to be reviewed: ˆ information on and requirements for home based upper extremity rehabilitation ˆ The available motion tracking technologies and the usability of these technologies (with examples) 1.3 information on and requirements for home based upper extremity rehabilitation To take advantage of home-based rehabilitation several tools are necessary. The first of them being a device showing an exercise. During the exercise, the movements of the user need to be recorded and processed. Considering that we will use gaming console s, the exercises will be shown as games. As already proven by several studies, games are an excellent medium for these exercises. They provide a way of measuring performance via scores and they add competitiveness to the exercises which motivates users to go on just that little bit longer[burke et al., 2009]. Custom games could also save more information so a physician could use the results too. Some of the consoles (the PS3 and the XBOX 360) have an internet connection, for the Wii it is necessary to buy addition equipment. This enables the option to upload results to a physician, so he/she can provide feedback to the user. The most important aspect is recording the movements, because this is the only tool console s them self cannot provide. The first requirement for an input device is that it works on a gaming console (the PS2, PS3, Xbox, Xbox360 or the Wii) without big modifications. The most recent console s are preferred because they are faster and provide better graphics. The second requirement is that the device should be easy to configure, not everyone is very technical. The last requirement is that it works at home. 1.4 The available motion tracking technologies and the usability of these technologies Currently there are two device categories, one which is using inertial sensors to detect movement (Wiimote and gloves) and the other which is using camera s (PS2 EyeToy and PS3 Eye). All 1.4.1 Inertial based the Wiimote When Nintendo released the Wii with the Wiimote a new idea of game control started. Players no longer used their fingers for game control, but rather used their whole arm 1. When you see films of players they move their arms a lot. But is this a good platform for rehabilitation? [Deutsch et al., 2008] uses a Wii for rehabilitation with a patient who can already walk himself. The user plays Wii Sports for several sessions and he improved in visual-perceptual processing, postural control, and functional mobility. There are a few drawbacks though, it is necessary for one extra person to be there, in this case even two and the requirements are not fully met, because it is not used in a home environment, but on a school. Several others also use off-the-shelf games in combination with the Wii and also get positive results on using the wii for rehabilitation [Tanner, 2008][Fung et al., 2010][Saposnik et al., 2010][Anderson et al., 2010][Lange et al., 2009]. There are some other studies in which the Wiimote is used, but they don t use a gaming console (i.e. the Wii) but rather use some custom hardware and a pc[tam, 2010][Decker et al., 2009][Alankus et al., 2010]. Although this is slightly out of our scope, it is still very interesting to see the research done here. In one study, the motion of the wrist joint was picked as an ideal candidate for their rehabilitation system. It consists of a wiimote in a casing with several LED s and reflective markers on the wrist, hand and arm. See 1.1 for a picture of the setup. They 1 http://money.cnn.com/2006/11/15/commentary/wii review/index.htm, last visited on July 17, 2010 2 Study Tour Pixel 2010 - University of Twente

Chapter 1. gaming Motion tracking technologies for rehabilitation also confirm the rehabilitative working of the setup and the possibility to distribute it around the world for approximately $450 [Decker et al., 2009]. resources/usableforv2/specificresearchdevices/wii/wiihabilitati Figure 1.1: The setup for the wiihabilitation study resources/usableforv2/specificresearchdevices/webcam/webcamebas Figure 1.2: example of webcam based motion tracking using a coloured sock The Wii has so far only be used with stock games, this limits the use of results for physicians considering only game scores are provided. Wiimotes are used in combination with a pc for rehabilitation, some special hardware is required to use Wiimotes on a pc, it might even be necessary to create custom hardware. Because this, and because pc s are no gaming consoles, this won t be considered an option. gloves Both the Playstation 3 and the Xbox360 have USB connections for extra hardware, and the first Xbox has the option of using USB via an adapter. This enables the use of gloves with sensors like the 5DT glove 1. There have been several studies for the usability of such a glove in hand rehabilitation. One of them is using an Xbox together with a P5 glove. Before the Xbox could be used, several hardware modifications had to be made, including a Xodus Xenium Mod chip 2 and an additional CD-Rom drive. It was also necessary to install a customized operating system to be able to use the glove. Unfortunately, there are not much results known for this implementation [Morrow et al., 2006]. Another study is using a PS3 together with the before mentioned 5DT glove, the PS3 s software needed to be modified in order to use the glove, just like on the Xbox, although no hardware modification was required here. This modification could be prevented if the developer would buy the Sony Software Development Kit (SDK) for developing applications, which costs approximately $10,000. It would be an option if the cost could be spread over enough users. This system used the internet connection to upload results to an on-line database where a physician could have access to [Golomb et al., 2010]. The conclusion of the second study was that it is a good way to regain hand movement although there is a need for a better glove, which is more rugged and easier to put on and take off. More rehabilitation games to be developed and a more open platform than the PS3 is needed [Huber et al., 2010]. As already said, it could be solved by buying the SDK, another option would be to use another platform. In short, it seems like a good option to use for rehabilitation, but it still needs development and further research. 1 http://www.vrlogic.com/html/datagloves.html, last visited on July 18, 2010 2 http://www.modchipstore.com/xodus-xenium-ice-xbox-mod-chip-official-16182.html, last visited on July 18, 2010 Gaming motion tracking technologies for rehabilitation - Albert de Graaf 3

P XEL Simulations & Games 1.4.2 Camera based the PS2 EyeToy Some research has been done on using the EyeToy for rehabilitation. Both studies report that the EyeToy has potential where one study suggests it could best be used for home-based rehabilitation. This study consists of two groups of individuals, one using of the EyeToy for 4 weeks and the other one just watching the games. This study reveals that it is useful to use the EyeToy games for recovery [Yavuzer et al., 2008]. The other study reports some limitations of using the EyeToy. The most important being the requirement of therapists, therapists are needed to make games easier or to change the movement patterns. Another limitation is the lack of recording the performance. If this was the case, the first could be remotely done in evaluation sessions, enabling more frequent use. It also states that it has potential to promote exercise for high-functioning individuals with stroke[rand et al., 2008]. Considering the limitations found in the second study, it is not very likely that this system can be used directly, an alternative for the EyeToy should be searched[rand et al., 2008]. webcam Comparable to the EyeToy is the use of a webcam with a computer. Although it doesn t meet al the requirements, it sketches a good image of the possibilities. A very simple version of its use is found in a study done in [Alankus et al., 2010]. Here, they put a coloured sock over the hand of the user, which is being tracked by the camera. The pc to which the camera is connected translates this movement to movement in the game. See 1.2 for a picture. The only results reported is that 3 patients played the game called baseball catch for 25 minutes using the webcam. Another system also uses a webcam. In order to track the players movements, the player holds a single-coloured marker in each of his hands, this might be gloves worn, or objects held. After calibration, the system can track these items. There were a few games developed using these techniques. The first game, Rabbit Chase, was developed for single-arm rehabilitation. In this game, the user tries to catch a rabbit popping out one of four holes. The player can see the rabbit running from one hole to another, so he can anticipate. A second game, Arrow Attack, was developed for rehabilitation of both arms. 2 arrows, one corresponding to the left, and one to the right hand, move between four boxes on the screen. The player must try to touch the arrows at the moment they hit a box. Finally, two variants of a game called Bubble Trouble have been created: one for rehabilitation of the affected arm and another which requires the use of both arms. In this game the user needs to pop bubbles by touching them in the virtual environment. If the version is played where both arms are used, each bubble is colour-coded to match the marker colour. Although there are no results on improvement of motor function, the test results are positive. The able-bodied users and three people from a stroke group, all enjoyed playing the games. One person from the stroke group actually really wanted to have the games herself[burke et al., 2009]. Although there are no results on rehabilitation importance, the second article clearly states that people enjoyed playing these games, which makes it more likely to do repetitive movements. 1.5 Discussion and Conclusion Of the motion tracking techniques reviewed, the use of data gloves seems the most advanced. Although there is a need for more rugged equipment, it works. One of the reviewed systems even had the ability to upload results to an online database. The Wii has no custom games developed for rehabilitation, the only research done so far is using off-the-shelf games. These games were not primarily designed for rehabilitation, which renders the results almost unusable for physicians. Almost the same problem occurs for the EyeToy. Although it has potential, it also has the drawback that no games made for rehabilitation on this platform exist. One of the reasons for this could be the expensiveness of a SDK for these devices. In the near future, Microsoft will come with a new camera device called Kinect 1, which is supposedly able to track 3d movements using two camera s. This, and the availability of Microsoft XNA Game Studio Express[Burke et al., 2009] for only $99 a year could make sure the disadvantages of previous systems are being 1 http://www.xbox.com/en-us/kinect/default.htm, last visited july 17, 2010 4 Study Tour Pixel 2010 - University of Twente

Chapter 1. gaming Motion tracking technologies for rehabilitation solved, but further research has to be done after Kinect hits the market in November 2. Webcam technologies reveal some parts of the potential of Kinect and PS3 s Eye, the latter already on the market, but not used for rehabilitation purposes yet. Overall, there is high potential in using gaming techniques for rehabilitation. 1.6 References Alankus, G. et al. Towards customizable games for stroke rehabilitation. In Proceedings of the 28th international conference on Human factors in computing systems, pages 2113 2122. 2010. Anderson, F., Annett, M. and Bischof, W. Lean on wii: physical rehabilitation with virtua reality wii peripherals. Studies in Health Technology and informatics, 154:229 234, 2010. Burke, J.W. et al. Optimising engagement for stroke rehabilitation using serious games. The Visual Computer, 25(12):1085 1099, 2009. ISSN 0178-2789. Butefisch, C. et al. Repetitive training of isolated movement improves the outcome of motor rehabilitation of the centrally paretic hand. 130:59 68, 1995. Decker, J. et al. Wiihabilitation: Rehabilitation of Wrist Flexion and Extension Using a Wiimote- Based Game System. 2009 Governor s School of Engineering and Technology Research Journal, 2009. Deutsch, J.E. et al. Use of a low-cost, commercially available gaming console (Wii) for rehabilitation of an adolescent with cerebral palsy. Physical Therapy, 88(10):1196, 2008. Dickstein, R. et al. Repetitive practice of a single joint movement for enhancing elbow function in hemiparetic patients. 85:771 785, 1997. Feys, H. et al. Effect of a therapeutic intervention for the hemiplegic upper limb in the acute phase after stroke: A single-blind, randomized, controlled multicenter trial. 29:785 792, 1998. Fung, V. et al. The Utility of a Video Game System in Rehabilitation of Burn and Nonburn Patients: A Survey Among Occupational Therapy and Physiotherapy Practitioners. Journal of Burn Care and Research, 2010. Golomb, M.R. et al. In-Home Virtual Reality Videogame Telerehabilitation in Adolescents With Hemiplegic Cerebral Palsy. Archives of physical medicine and rehabilitation, 91(1):1 8, 2010. Holden, M. and Todorov, E. Use of virtual environments in motor learning and rehabilitation. Handbook of Virtual Environment Technology. Lawrence Erlbaum Assoc, Inc, page 9991026, 2002. Huber, M. et al. Feasibility of Modified Remotely Monitored In-Home Gaming Technology for Improving Hand Function in Adolescents With Cerebral Palsy. IEEE Transactions on Information Technology in Biomedicine, 14(2):526 534, 2010. ISSN 1089-7771. Lange, B., Flynn, S. and Rizzo, A. Initial usability assessment of off-the-shelf video game consoles for clinical game-based motor rehabilitation. Physical Therapy Reviews, 14(5):355 363, 2009. ISSN 10833196. Merians, A., Jack, D. and Boian, R. Virtual reality-augmented rehabilitation for patients following stroke. 82:898 915, 2002. Morrow, K. et al. Low-cost virtual rehabilitation of the hand for patients post-stroke. In 2006 International Workshop on Virtual Rehabilitation, pages 6 10. 2006. Rand, D., Kizony, R. and Tamar, L.W.P. The Sony PlayStation II EyeToy: Low-Cost Virtual Reality for Use in Rehabilitation. Journal of Neurologic Physical Therapy, 32:155 163, 2008. Saposnik, G. et al. Effectiveness of Virtual Reality Using Wii Gaming Technology in Stroke Rehabilitation. Stroke, 41:1477 1484, 2010. Tam, S. An Environment for Stroke Therapy Game Authoring. 2010. Tanner, L. Doctors use Wii games for rehab therapy. USA Today, page 200802, 2008. 2 http://www.computerandvideogames.com/article.php?id=250862, last visited on July 17, 2010 Gaming motion tracking technologies for rehabilitation - Albert de Graaf 5

P XEL Simulations & Games Truelsen, T. and Bonita, R. Stroke in developing countries: a continuing challenge. 7:61 66, 2003. Winstein, C.J., Pohl, P.S. and Lewthwaite, R. Effects of Physical Guidance and Knowledge of Results on Motor Learning: Support for the Guidance Hypothesis. 65:316 323, 1994. Yavuzer, G. et al. Playstation eyetoy games improve upper extremity-related motor functioning in subacute stroke: a randomized controlled clinical trial. European journal of physical and rehabilitation medicine, 44:237 244, 2008. 6 Study Tour Pixel 2010 - University of Twente