Motion tracking using Matlab, a Nintendo Wii Remote, and infrared LEDs.

Transcription

1 Motion tracking using Matlab, a Nintendo Wii Remote, and infrared LEDs. Dr W. Owen Brimijoin MRC Institute of Hearing Research (Scottish Section) Glasgow Royal Infirmary 16 Alexandra Parade Glasgow G31 2ER United Kingdom owen@ihr.gla.ac.uk Introduction This document describes how to use a Wii remote to do simple single-axis motion tracking. The Wii remote is an inexpensive peripheral for a video game system built by Nintendo. It has buttons, a 3-axis accelerometer, and an infrared camera mounted on the front. The Wii remote is a useful tool for researchers: the buttons on the device can be used for recording subject responses, the accelerometers can be used for detecting movement, and the camera can be used to do reasonably accurate motion-tracking of objects such as the head. Combined with Matlab and digital signal processing, it is possible to use this system to (e.g.) adjust signals presented over headphones in real-time so that they appear to remain fixed in space, rather than moving with the head. Note: this code does not do stereoscopic 3D tracking: it can only measure rotations that are in the plane of the camera. This will allow you to track yaw, rotations of the head in the vertical axis (shaking your head no ), but not pitch or roll. If you find this code useful for your science, please cite Brimijoin et al (in press 2013, PLoS One). Requirements 1) 1 or more Nintendo Wii remotes 2) 1 Microphone stand with clip or other mounting hardware for the remote 3) 1 Bluetooth Chip and associated software driver stack 4) 1 array of infrared LEDs (battery powered) 5) 1 copy of Matlab 6) 1 copy of the WiiLAB dynamic link library 7) 7 Matlab files (included here) Detailed Requirements 1) Wii remotes. The most recent iteration of the Nintendo Wii remote has been known to have compatibility problems. We recommend either an original (pre-2010) Wii remote (the ones without the words Motion-Plus printed on the remote), or a first generation Motion-Plus remote. 2) Stand for wii remote. The wii remote should be mounted above the head, pointing down at the participant (see figure 1). Microphone stands can

2 2 be used for this purpose, provided they have a clip style microphone mount with jaws large enough to fit around the body of the remote. Given the limited field of view of the Wii remote s infrared camera, it is recommended that the remote be mounted at least a meter above the head. The height you use will be a tradeoff between tracking accuracy (closer is better), and the area over which you can capture motion (farther is better). Given the size of the infrared LED array you are using (see point 4 below) and typical subject movements, an ideal distance that allows accurate tracking but little loss of the subject s head is about 5 times as far away as the length of the LED array. 3) Bluetooth chip. It is recommended that you use a third-party Bluetooth USB dongle such as those made by Belkin (F8T065) and install the Bluetooth drivers either from the included compact disc or downloaded from the manufacturer. The reasons for this are that the built-in Windows Bluetooth chips and stack have been known to raise compatibility issues. 4) Infrared LED array. (Figure 2). The array must be of 3 infrared LEDs arranged in a single row. The exact dimensions of the array are not critical (the one we use is 0.2 meters in length) but the important thing is that the rear two LEDs must be closer together than they are from the front LED. The reason for this is that this asymmetry allows Matlab to determine the front and back of the array, so making it possible to record 360 of motion. A circuit diagram of a simple version of a suitable LED array is shown in Figure 2. The array can be built by anyone with any soldering experience this example is powered by a single 9 volt battery, has a single on-off switch, one 47 Ohm resistor, and three 1.5 volt infrared LEDs wired in series. The LEDs, resister, battery, and switch can be mounted on anything, provided that this object can in turn be mounted on the head. We recommend using the inside of a construction hard hat, as the adjustable plastic bands inside a hard hat can be made to fit any head and are easily disinfected. 5) A licensed copy of Matlab (r13 or later). 6) WiiLAB library. You will need to download and install the WiiLAB library from Brindza and Szweda at Notre Dame. The installer can also be obtained from its original source here: Download the.zip file containing the installer and source code (currently this link is called Get the zip file (full install). ). Unzip the directory to a spot on the matlab path. Run the installer batch file: InstallWiiLab. The resulting directories will contain a number of files, the critical ones being WiiLAB.dll, WiimoteLib.dll, and WiiLAB.tbl. Either copy these and place them into

3 3 a directory with the matlab functions listed below, or move the Matlab functions to the directory containing them. Compatibility note: For older versions of Matlab (or for instances where installation or connection from Matlab proves to be a problem), an alternative version of the WiiLAB library has been provided by Ian Stevenson and may be found here: Select the link: WiiLab_2007aCompat.zip 7) Matlab functions. The functions included in the zip file are detailed below: Core functions: connect_n_wiimotes(n); %this connects n wiimotes (one for tracking, one for response, etc). disconnect_ wiimotes(); %this disconnects all connected wii remotes. buttons = get_wiimote_buttons(remote_num) %tell it which remote you want and it will return what button(s) are being pressed. accel = get_wiimote_acceleration(remote_num) %tell it which remote you want and it will return the output of the 3 accelerometers [yaw, x, y] = get_wiimote_angle(remote_num) %tell it which remote you want and it will return the angle and xy position of the infrared LED array. Example Functions: test_wiimote(remote_num) %this simply plots all data that is received from the wiimote (buttons, acceleration, LED locations, and angle). It calls the other functions to demonstrate how they are used. [yaws, xvals, yvals] = record_head_angle(duration,remote_num) %this is another example function showing how to use get_wiimote_angle.m to record head motion over a specified time. Motion Tracking Methods: 1) Pairing the Wii Remote to Windows: The Wii remote must first be connected to the computer*. This is accomplished through the Show Bluetooth devices dialog box (typically found in the tray at the bottom right of the screen). Click Add a device then immediately press down buttons (1) and (2) on the wii remote. The LEDs on the remote will begin flashing and the remote will now show up in the list of devices (as something like Nintendo RVL-CNT-01 ). Select this, click next, and then Pair without using a code. These mouse clicks must be performed within about

4 4 ten seconds or else the wii remote will stop flashing and the pairing will fail. If it works, a dialogue box should now read This device has been successfully added to this computer. The blue LEDs on the wii remote will continue to blink. Click dialogbox at this point. Once you are finished motion tracking, it is recommended that you remove the device (by right clicking and selecting remove device ). *Note that these instructions are based on pairing the device in Windows 7. 2) Connecting to the Wii remote in Matlab: To connect a single Wii remote, open matlab and type the following at the Matlab command line: >> connect_n_wiimotes(1) If the connection is successful, the blue LEDs on the Wii remote will now stop flashing and a single one will light steadily. If this does not happen, remove the device from Windows Bluetooth devices and start again at step 1). To test whether you have a good connection, run test_wiimote(1). The active buttons and the output of the accelerometers should be displayed in the figure window. Once you are finished motion tracking, it is recommended that you disconnect the device by typing >> disconnect_wiimotes at the Matlab command prompt and then remove the device from the Windows Bluetooth devices. Warning! If you shut off the wii remote, disconnnect it from Windows, remove the battery, or let it run down on charge all the way, this will cause Matlab to crash unceremoniously without saving your workspace. 3) Motion tracking: Turn on the LED array and ensure it is in the field of view of the wii remote. You can run >> test_wiimote(1) in Matlab to visualize the output of the wii remote camera. If the remote can see all three LEDs of the array, they will be displayed in real-time on the screen as red green and blue dots. The subtended angle of the array will also be displayed on the figure. The core functions detailed above can be used as components in any Matlab script or function. An example for how to record head movements over time and report them as a vector of head angles is given with the function: >> record_head_angle(duration,remote_num). This example function takes as its input duration (in seconds), and remote number. For example, to record 2 seconds of angle and array position, call the function as follows: >> [yaw,x,y] = record_head_angle(2,1); >> plot(yaw) Other possible uses for the motion tracking system would be to alter in real-time some aspect of a signal being presented, for example the ITD or ILD or to choose a binaural room impulse response. For these applications, one should call

5 5 >> yaw = get_wiimote_angle(1) ; in a loop to query the wii remote for head angle. You may then use this angle to adjust audio playback. We have successfully implemented this system using Windows XP with Matlab r13 and Windows 7 with Matlab versions 2011a and 2013b. We have not tested it on any other system, and we cannot guarantee that it will work on your system.