Web Camera Mouse with Smart Base: An Electrical Engineering Design Project

Web Camera Mouse with Smart Base: An Electrical Engineering Design Project 1. Introduction Richard Muysenberg a, Lisa Zyonse b, Olaoluwa Fadiran c, Wenjun Li d, Qin Hu e Central Michigan University a-e Mt Pleasant, MI 48858 email: muyse1rl@cmich.edu a, zyonse1lm@cmich.edu b, fadir2oa@cmich.edu c, li1w@cmich.edu d, hu1q@cmich.edu e The primary purpose of this project is to design a system that will act as an assisting device for people with severe cases of mobility impairment diseases. Over the years, research has focused on developing assistive technology using various techniques, such as nasal pressure [1], brain activity [2], and eye movement [3]. However, these techniques are extremely complex, difficult to implement, and very expensive. The Web Camera Mouse with Smart Base (WCMSB) provides a simpler system in which a user can control the computer cursor and screen alignment by the movement of a facial feature like the nose. The overall system comprises of real time video image processing using MATLAB. Facial recognition will be followed by feature recognition which provides the coordinates of a marker. This marker will be constantly monitored in real time and used to decide when the servo and cursor will be controlled. The computer cursor will be controlled by a facial feature (marker), such as the nose. Cursor control will be implemented by the use of a microcontroller housed inside the base. The programming for this will be very similar to that of a regular mouse. The microcontroller will integrate two systems: cursor control and base rotation as shown in Fig.1. The base rotation process will activate when the marker is outside of the cursor limits described later in the paper. This will allow the computer/monitor to center itself in the user s view. The cursor control process will then reactivate and the user will again control the computer cursor with the marker. 2. Contemporary Issues People who are diagnosed with severe mobility impairment diseases, such as Cerebral Palsy (CP) [4], Muscular Dystrophy (MD) [5], and Multiple Sclerosis (MS) [6], are facing severe limitations in their everyday life. CP is a disease caused by irregular development or damage of the brain and it affects body movements, balance and posture, and limits overall motion of the body. Similarly, MD, which also affects thousands of people annually, limits movement capability by damaging muscle fibers. Victims of diseases like CP and MD usually find it difficult to use computers, ipods, cell phones, and other portable devices because of their mobility limitations. In severe cases, individuals with CP or MD need others to help them with computer processes like opening files, web-surfing, and video conferencing. The WCMSB is designed specifically to assist people with these life-challenging diseases. 1

Webcam Turn Table and Base (connect with servo) Microprocessor Servo Fig. 1 WCMSB design in which microcontroller controls all components (mouse and servo). 3. Customer Needs There are two groups of customers. Those with severe mobility impairment diseases, such as severe cases of CP, MD, and MS, fall into the Physically Disabled Group. Those who are highly active and generally multitask, such as a stay-at-home parent, business people, and nurses, fall into the General Population Group. The customer needs for each group is addressed accordingly: a. Physically Disabled Group: Flexible working/living environment; ease of access to a computer; ease of access to communication devices; to increase quality of life; compact and portable. b. General Population Group: Ease of multi-tasking; more convenient tasks; compact and portable. The prototype design will focus on harnessing the needs of the Physically Disabled Group. The design will ensure that these patients can use their computer more independently. 2

4. Research 4.1 Image Processing The web camera mouse will recognize the face of the user, track the marker, and save the marker s Cartesian coordinate in a text file. This process is accomplished using the MATLAB Image Acquisition toolbox and Image Processing toolbox. The web camera used in the prototype is limited to a field of view of 640 x 480 pixels, which represents a Cartesian coordinate system with the origin located in the upper left hand corner of the field of view. The marker coordinate is used in further processing by the microcontroller. Microcontroller processing will be discussed in greater detail in section 4.4. Two methods are proposed to be used in facial recognition: Adaptive skin-color model and eigenface method [7,8]. For face detection using the adaptive skin-color model; the image will be represented by the chromatic color system; separate skin areas from other parts based on the different chromatic color of the skin; separate the face area from skin area by feature of the face; and match a created face template into the suspected region to recognize the face area. Eigenface method provides a similar approach for facial recognition. In this case, a training set that is made of many faces is made and an average face matrix is developed from the mean of the training set. The eigenvectors (eigenfaces) are calculated from the average face matrix. The facial recognition system works by comparing the test image s eigenfaces with the training set. 4.2 Cursor Limit and Average Position The WCMSB will operate in one of two processes. The first process controls the computer cursor while the second rotates the smart base. In order to distinguish from the two processes, operation regions are defined within the web camera input window (field of view). The mouse operation region is a range within the cursor limits, 64 to 576 pixels. Fig. 2 gives a visual representation of the regions and cursor limits. The cursor limits are defined by a vertical line that is located at 1/10 of the horizontal pixel dimension from each end of the web camera input window. The base trigger region is outside the mouse operation region and within the web camera input window. When the marker is within the base trigger region, the direction of rotation is determined by the horizontal location of the marker. If the value of the marker s x- coordinate is less than 64, the base will rotate clockwise. If the value of the marker s x- coordinate is greater than 576, the base will rotate counter clockwise. The degree of rotation is approximately 20 which takes the marker back to the average position located near the center of the web camera input window. The prototype uses the SABRENT USB Color Web Camera which has a 42 field of view. The degree of rotation is 21 which is half the field of view. This will allow the marker to be re-centered in the web camera input window and cursor control can be re-established. 3

Web Camera Input Cursor Limit Base rotates clockwise Base rotates counter-clockwise Mouse Operation Region Base Trigger Region Fig. 2 Web camera input window. 4.3 Servo Control Servo control is usually implemented by a microcontroller and a servo motor. A servo is an electromechanical device that uses a feedback system to correct errors for precise movement. RC servos are used by most hobbyists because of their simplistic universal control since they can be controlled with a simple Pulse Width Modulation (PWM) circuit. The width of the electric pulse going to the RC servo controls the position of the servo horn/arm. Similarly, speed control can be obtained by repeatedly sending the same signal to a servo modified for continuous rotation. The HS-311 Standard Servo [9] is chosen because it meets the specifications required by the demands of this project. This servo has a maximum torque of 49 oz-in and maximum speed of 0.15 sec/60, which exceeds the minimal requirements. This servo is also modified for 180 rotation by the factory. Additionally, the HS-311 is the cheapest of its kind and was recommended by other colleagues. 4.4 Microcontroller The microcontroller is the brain of the system. It runs the different processes that allow the WCMSB to function appropriately. The WCMSB requires a microcontroller board that is able to interact with a computer via Universal Serial Bus (USB) and a RC servo motor via servo headers. The Cerebot 32MX4 Microcontroller Board from Digilent Inc. [10] meets all of the above requirements. It also has the ability to add pitch control, in addition to rotation, in the future. It has a PIC32MX460F512L 32-bit MIPS microprocessor embedded on the board. This microcontroller board has plenty of space for an efficient program to run effectively without errors. 4

The microcontroller program will start off by initializing variables. The present point, data point, and cursor point variables will all hold Cartesian coordinates. There are also two cursor limit variables which hold x-axis (horizontal) coordinates that mark the separation of the Mouse Operation Region (MOR) and Base Trigger Region (BTR) defined in 4.2. The microcontroller program will call the marker coordinates from the text file and store it in the cursor point variable. This initializes the cursor in a position on the screen relative to the marker. Then, the same data will be stored in the data point. Next, the data point is checked to see what region it is in. If it is equal to or within the cursor limits, it is in the MOR and the program enters the cursor control subroutine (Fig. 4). Otherwise, it is in the BTR and the program will enter the base control subroutine (Fig. 5). After the subroutine is processed, the program will continuously loop to fetch new data points as shown in Fig. 3. Microcontroller Initialize Variables: Present Point = (0,0) Data Point = (0,0) Cursor Point = (0,0) Cursor Limit1 = (64) Cursor Limit2 = (576) Fetch data from file --> Cursor Point Fetch data from file --> Data Point Cursor Limit1 <= Data Point (x) <= Cursor Limit2? Cursor Control Base Control Fig. 3 Microcontroller flowchart. 5

In the cursor control subroutine, the data point will be tested for differences from the previous point. The series of loops shown in Fig. 4 allows for the cursor to follow the movements of the marker. After differences are tested between both Cartesian coordinates, the data point is stored in the previous point. This will allow for the new data point to be compared with the previously tested data point. The subsystem will then end and the microcontroller process will continue. Cursor Control Data Point (x) > Previous Point (x)? Inc Cursor Point (x) Data Point (x) < Previous Point (x)? Dec Cursor Point (x) Data Point (y) > Previous Point (y)? Inc Cursor Point (y) Data Point (y) < Previous Point (y)? Dec Cursor Point (y) Data Point --> Previous Point End Fig. 4 Cursor control subsystem. 6

In the base control subroutine, the data point is checked to see if it is greater than cursor limit2 (Fig. 5). This will determine which direction the base needs to rotate to re-center the computer/monitor in the user s view. If it is greater than cursor limit2, then the base will rotate 21 counterclockwise. Otherwise, the data point must be less than cursor limit1 and the base will rotate 21 clockwise. After this subroutine has ended, the microcontroller process continues. Base Control Data Point (x) > Cursor Limit2? Rotate 21 degrees counterclockwise Rotate 21 degrees clockwise End Fig. 5 Base control subsystem. 5. WCMSB Design The final design of the smart base can be easily described by five parts which are the top base, turntable, servo, insert and bottom panel. The whole weight of top base will be supported by a turntable, which means that the servo will not obtain any weight from the top base. This allows the servo to rotate the base more efficiently. The servo will be set up on the top of the bottom panel and an insert piece will be used to cover the rest of the base (Fig. 6). The circuit and microcontroller is set up in the inside space of the insert. Fig. 7 shows a technical drawing of the design from the side and top of the base. Fig. 6 The 3-D Section View Image of the Smart Base. 7

Fig. 7 The side view and top view of the smart base with dimensions. In order to minimize the load on the servo motor, a turntable was incorporated into the design (Fig. 8). The load will rest on the turntable, which is placed on the bottom half of the smart base. The servo motor will mount so that it will make horizontal contact with the top of the base, thus preventing it from handling any vertical force. The method used for preventing the servo from handling any weight is to partially hollow out the area on the bottom of the top base where the servo is sitting. The partially hollowed out area is the exact shape of the servo horn, so when the servo turns the base there will not be any slipping, thus allowing the servo to turn the base efficiently. The materials used to create the final design of the smart base were chosen for many reasons. One reason was that the materials of the smart base, plywood, were environmentally safe and will not emit any harmful toxins. The plywood was also a great, durable material that will ultimately result in a more economical product. 8

Fig. 8 Turntable analysis. 6. Summary The WCMSB is a device will serve as an aid to people with severe mobility impairment diseases, such as CP, MD, and MS. The purpose of the WCMSB is to improve their quality of life and increase their independence. This project that integrates real time image processing with servo and cursor control. The system is implemented by facial feature recognition using adaptive skin color model and eigenface techniques. The final base design is made of the top base, turntable, servo, insert and bottom panel. The circuit and microcontroller will be mounted inside the insert. This will provide a compact and portable system for implementation. Prototype development is in progress and a working prototype is feasible in the near future. Bibliography [1] A. Plotkin, L. Sela, A. Weissbrod, R. Kahana, L. Haviv, Y. Yeshurun, N. Soroker, and N. Sobel, Sniffing enables communication and environmental control for the severely disabled in Proceedings of the National Academy of Sciences of the United States of America, vol. 107, no. 32, pp. 14413-14418, August 2010. [Online] Available: www.pnas.org/cgi/doi/10.1073/pnas.1006746107 [2] D. Murph, UCLA/Caltech researcher help patients move mouse cursor with their brains, engadget, 2 Nov 2010. [Online] Available: http://www.engadget.com/2010/11/02/ucla-caltech-researchers-help-patients-movemouse-cursors-with/ [3] P. DiMattia, F.X. Curran, J. Gips. An Eye Control Teaching Device for Students without Language Expressive Capacity: EagleEyes. Edwin Mellen Press, 2001. [4] emedicinehealth. Cerebral Palsy [Online]. Available :http://www.emedicinehealth.com/cerebral_palsy/article_em.htm 9

[5] Mayo Clinic. Muscular Dystrophy [Online]. Available: http://www.mayoclinic.com/health/musculardystrophy/ds00200 [6] National Institute of Neurological Disorders and Strokes. Multiple Sclerosis: Hope Through Research [Online]. Available: http://www.ninds.nih.gov/disorders/multiple_sclerosis/detail_multiple_sclerosis.htm#158953215 [7] L. Xuan, S. Nitsuwat, Face Recognition in Vide, a Combination of Eigenface and Adaptive Skin-color Model, International Conference on Intelligent and Advanced Systems, 2007. [8] S. Zhou, V. Krueger, R. Chellapa, Probabilistic Recognition of Human Faces from Video, Computer Vision and Image Understanding, 2003, pp. 214-215. [9] ServoCity. HS-311 Standard [Online]. Available: http://www.servocity.com/html/hs-311_standard.html [10] Digilent. Cerebot 32MX4 [Online]. Available: http://www.digilentinc.com/products/detail.cfm?navpath=3,719,755&prod=cerebot32mx4 10