USING EYE TRACKING TECHNOLOGY FOR WEB SITE USABILITY ANALYSIS: THE APPLICATION OF ERICA TO GEFANUC.COM

Transcription

1 2002 IEEE Systems and Information Design Symposium University of Virginia USING EYE TRACKING TECHNOLOGY FOR WEB SITE USABILITY ANALYSIS: THE APPLICATION OF ERICA TO GEFANUC.COM Student Team: Alyssa Boaz, James Cuneo, David Kreps, Marsden Watson Faculty Advisor: Thomas E. Hutchinson Department of Systems and Information Engineering Client Advisors: Wende Hope and Emily Turner GE Fanuc Automation Route 606 & Seminole Trail Charlottesville, VA Phone: (434) KEYWORDS: ERICA, GazeTracker, usability, graphical user interface (GUI). ABSTRACT Given its role as a tool for business and information sharing, efficient navigation of the Web has become imperative. In order to evaluate and improve a Web site, usability has emerged as an important field of study. In the business world, an intuitive Web interface mitigates the frustration of customers and arguably relates directly to increased sales. Despite this fact, many contemporary usability techniques remain subjective. Our team conducted an innovative analysis of the GE Fanuc Web Site. Using ERICA (Eye-Gaze Response Interface Computer Aid), we examined the paths test subjects used to accomplish tasks outlined in a testing protocol. GazeTracker, ERICA s associated software, allowed the team to record and quantify user behavior patterns. Analysis of this data enabled us to make recommendations for design improvements to the content owners of gefanuc.com. GE Fanuc s relies on the Six Sigma design process, which requires quantitative data to ensure the quality of their products and services. Traditional usability techniques could not provide GE Fanuc with adequate quantitative data. Through this project, we used ERICA to collect this previously unavailable data. Our study found that the addition of ERICA to a usability study makes testing more precise and quantitative. We found it to be a valuable complement to traditional, qualitative usability analysis. INTRODUCTION This project demonstrates the benefits of using ERICA for usability studies by analyzing and making design recommendations for the GE Fanuc Web site. Usability is the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use [Ferre, Xavier and Juristo 2001]. Using ERICA, we were able to track and analyze the path of a person s gaze on a series of Web pages. This information provides businesses with insights into precisely what their customers are seeing. Thus, a company can potentially improve efficiency, production, and customer satisfaction by using this eye gaze analysis to improve its graphical user interface (GUI) design. Client Background GE Fanuc Automation is a subsidiary of General Electric Corporation, headquartered approximately ten miles north of Charlottesville, VA. It is a global corporation devoted to providing solutions for factory and process automation. Like many competitive companies today, GE Fanuc is strongly committed to e- commerce and customer service, citing both issues as principle business initiatives. Since the launch of the gefanuc.com website, the marketing department has received feedback citing customer frustration in attempts to locate information on the site. Figure 1 illustrates the original design of gefanuc.com. GE Fanuc feels very strongly that their Web site is a critical piece of its business, serving as both an advertisement to prospective customers and a reference 157

2 Using Eye Tracking Technology For Web Site Usability Analysis for current customers. Furthermore, they view frustrated customers as a serious threat of forfeited revenue. GE Fanuc estimates the cost of soliciting each new customer lead to be five hundred dollars [Graney]. The company expressed an interest in increasing the rate of online lead generation form submissions for their Ethernet products and Supply Chain Execution services. EXPERIMENT DESIGN We conducted a series of tests to determine and validate design recommendations for gefanuc.com. Our methodology incorporated three phases. Phase One: Protocol Development Phase One involved creating and piloting a testing protocol. Aimed at determining proper instructions for the tasks, Phase One consisted of nine subjects. We completed three rounds of testing, with three subjects in each round. Observations of user behavior given the protocol instructions allowed the team to alter the wording to make instructions more intuitive without leading the subjects. Our goal was to make the protocol clear so that the user could accomplish the desired task without asking questions to the tester. The final protocol was then used for all other phases of testing. Phase Two: Subject Testing of the Current Site Figure 1: Original GE Fanuc Homepage ERICA ERICA is a non-invasive eye tracking system that determines where a user is looking on a computer display [ 2001]. The system consists of a camera and an infrared light emitting diode (LED) and two mirrors that direct the camera s line of sight to a user s eye. The LED illuminates the eye and generates a reflection off the surface of the eye (the glint) and causes the pupil to appear as a bright disk (the bright eye) in contrast to the surrounding iris. The system calculates where the user is looking on the computer screen based on the vector relationship between the pupil center and the glint. ERICA relies on GazeTracker software for analysis [Lankford 2000]. This application records a subject s eye response to still images, videos, or GUIs. The evaluator can then analyze the eye-gaze patterns. One feature of GazeTracker that was central to our analysis the LookZone feature. A LookZone is a user defined area of interest on the screen, created to all eye gaze data in this area can be combined. Phase Two testing involved three GE Fanuc employees and nine UVA students as test subjects. These subjects performed the tasks indicated on the protocol and GazeTracker recorded their eye movements and mouse clicks across the pages they visited. We designed Phase Two so we could analyze the data and create two alternate designs of the site to complete Phase Three testing. Phase Three: Subject Testing of Alternative Designs Using the same testing protocol, we conducted our final phase of testing. In this Phase, we again tested four GE Fanuc employees and fourteen UVA students. As in Phase Two, GazeTracker recorded all eye movements and mouse clicks during the tasks. The goal of Phase Three was to validate changes on Versions 1 and 2 and to determine final design recommendations. PROTOCOL TASKS Task 1: Free Scan The first protocol task allowed each subject to look at the Homepage for thirty seconds. Subjects were permitted to scroll up and down the page, but instructed not to navigate to any other page. Data collected from the free scan offered insights as to which features on the page naturally attract a user s attention when they first reach the site. 158

3 2002 IEEE Systems and Information Design Symposium University of Virginia Tasks 2 and 3: Information Finding: Ethernet and Supply Chain Execution Tasks After a subject became mildly acclimated to the interface on the Homepage through the free scan, we instructed them to assume to role of a business owner and told them to find and request as much information as possible concerning purchasing an Ethernet System for their company, or learning about GE Fanuc s Supply Chain Execution process. We analyzed the eye gaze patterns and navigational paths chosen by the subjects as they attempted to complete these tasks. Also, we recorded whether a subject found and/or completed the associated lead generation form. We conducted a similar evaluation of the Ethernet and Supply Chain pages and associated lead generation forms. Given that the interfaces of these two pages are very similar, the changes we made to the Ethernet page were very similar to the changes we made to the Supply Chain page. Figure 3 illustrates the breakdown of the percentage of time the subjects spent examining various LookZones on the Ethernet page. Since users spent a significant amount of time looking at the middle of the screen (Description and CenterPic) and the Right Navigation bar, we reasoned that changes to these sections of the page would be effective. DATA ANALYSIS Initial Data Analysis The purpose of this step is to determine areas of the website where changes will have the most significant effect. To do so, we examined the LookZone data generated by user gaze patterns on each page. LookZones are defined areas of interest on the interface that can be drawn using GazeTracker tools. Figure 2 revealed that the users were spending the majority of their time looking at the text in the middle of the page as well as the What s New section. Given that this area of the Homepage naturally attracted user attention, we reasoned that alterations to these sections could help to create more intuitive paths to the target pages. Figure 3: Total Time in LookZones Ethernet Page Alternative Design Drafting Next, we analyzed the data gathered to make two alternative designs of the homepage, Ethernet page, and Supply Chain page. Version 1 contained less obtrusive changes, while Version 2 contained more substantial changes. Homepage Figure 2: Total Time in LookZones Homepage One of the metrics we examined when drafting alternative designs was the time to target metric. This metric examines the amount of time that a user spent before looking at a specific region of the screen. The data gathered by GazeTracker provides us with a means to determine how long a user spends between arriving at a page and glancing at a particular region of the site. The time to target data on the original version of the site indicated that users were confused by the wording of the link to the Ethernet page, which was located in the What s New? box. This link read, Plug into Productivity. When completing the Ethernet task, all 159

4 Using Eye Tracking Technology For Web Site Usability Analysis users glanced at this link, but not all users arrived at the Ethernet page. Furthermore, the average user spent seconds on the homepage before first viewing this link leading to the Ethernet page, but the user took seconds before actually arriving at the Ethernet page. This indicates that, while the user was viewing the link to the Ethernet page, the wording was not intuitive enough for him or her to actually click on it to arrive at the Ethernet page. We therefore changed the wording of this link to read, Ethernet Solutions, in both alternate designs of the site. Figure 4: Original and New What s New Box Version 2 incorporated additional links to the Ethernet and Supply Chain pages, located in the center of the homepage. On the original site, users spent as long as 242 seconds finding the Ethernet page, with 27% of users not finding it at all. During the Ethernet task, users spent an average of 25% of their time on the homepage looking at the center of the page. In addition, 91% of users on the Free Scan task looked at the middle section of the homepage within 10 seconds. Thus, we reasoned that adding a link to this section of the page would allow users to find the target pages faster. In the original version of the site, the link to the lead form was located in the left navigation bar LookZone on the Ethernet and Supply Chain pages. Every user looked at this section of the page and eventually found the lead form, but it took users an average of seconds to find it. We felt that adding additional links to the lead form and changing the wording to the existing link would allow users to find the form in less time. For Version 1 and 2, we added a link to the lead form on the right hand side of the Ethernet or Supply Chain pages. The design and placement of these links were similar to the What s New box on the homepage. As described above, this was one of the most looked at sections of the homepage. Thus, we inferred that these links on the Ethernet and Supply Chain pages would also be effective. We also changed the wording of the link in the left navigation bar from Tell Us About Yourself to Request Additional Information. Users spent 14% of the time looking at the left navigation bar, and every user looked at this LookZone. Therefore, by making the wording of the link more intuitive, we expected to decrease the amount of time it took find the form. Figure 6: Version 2 Ethernet page. Note the additional link on the right side of the page (Versions 1 and 2). Also note the link in the middle of the page (Version 2 only). Figure 5: Version 2 Homepage. Note the additional links to the Ethernet and Supply Chain pages located in the middle of the screen. Ethernet and Supply Chain pages In the more obtrusive Version 2, we also added a link to the middle of the page. Users spent 22% of the time looking at this section of the page, which was more than any other section. As a result, we believed this would be a very visible location for the link, and it would allow users to find the lead forms in less time. 160

5 2002 IEEE Systems and Information Design Symposium University of Virginia These findings specifically highlight the critical role that eye-gaze technologies can play in usability analysis, as this sort of data could not be captured through any other method. Alternative Design Comparison / Recommendation Drafting We then tested a new group of subjects on the two alternative designs using the protocol established in Phase One. Again, using quantifiable data gathered with GazeTracker TM, we were able to compare the subjects performance on the alternative designs. As shown below in Figures 6 and 7, Version 2 performed the best on our metrics. Figure 6: Percentage to complete each task. Figure 6 shows the completion percentages of the various tasks. Subjects performed equally well or better on the alternate versions of the site as compared to the Original design. On the original, 73% of users arrived at both the Ethernet and Supply Chain forms. On Version 1, this metric improved to 88.9% and 100% for the Ethernet and Supply Chain pages, respectively. On Version 2, 100% of users reached both the Ethernet and Supply Chain forms. This metric indicates that Version 2 helps the user navigate the site and arrive at the desired page. Beyond achieving the primary goal of increasing the percentage of users to reach the target pages, we also wanted to decrease the amount of time it took subjects to find each page. Figure 7 compares the average amount of time it took to complete the various task on each of the designs. On the original version of the site, users spent and seconds to find the Ethernet and Supply Chain pages, respectively. On Version 2, these metrics fell to and 7.46 seconds. While Version 1 also marked an improvement in time to task metrics, Version 2 showed a more substantial improvement. Figure 7: Average time to complete each task. As GE Fanuc indicated, they were most concerned with having the user arrive at the lead form. As Figures 6 and 7 illustrate, more subjects who used the alternate versions found these forms, and it took these users less time to find the forms. Through analyzing the eyetracking data, we were able to determine why the alternate versions outperformed the original. In both alternate versions, a link to the lead form was added to the right hand side of the screen, and in Version 2 was also added a link to the center of the screen. In the original version of the site, the average user took over 70 seconds to view the link to the lead form. With Version 1, the average user first viewed either of the two links to the lead form in 43 seconds, and in Version 2, the average user first saw any of the three links to the lead form in 17 seconds. These data suggested users of Version 2 found the lead form faster because it took them less time to first see the link to the form. In addition to the forms, we also wanted to make it easier for the user to find the Ethernet and Supply Chain pages from the homepage. As described earlier, we changed the wording of the links to these pages in the What s New box to make the link more intuitive. The time to target data for Version 2 indicated that these changes in wording were effective. Rather than spending seconds between first viewing this link and arriving at the Ethernet page, the average user spent only seconds between first seeing the link and arriving at the page. A similar result was obtained by changing the wording of the link to the Supply Chain page. In the original version, a user first viewed the Supply Chain link after15.16 seconds on the homepage, but did not arrive at the Supply Chain page until seconds later. In Version 2, this differential was reduced to 5.12 seconds. We concluded from these data that it took users less time to find the Ethernet and Supply Chain pages on the alternate designs because the wording of the links were more intuitive. 161

6 Using Eye Tracking Technology For Web Site Usability Analysis DISCUSSION This project was unique in that it was the first ERICA study in which eye tracking technology was used for multiple pages of a Web site. Rather than focusing on one page, this study examined and aimed at optimizing the navigational path of a user across multiple pages within the GE Fanuc site. As described previously, the results of the project indicate that eye tracking was successful at improving user efficiency on the site and increasing the number of lead form submittals in the testing environment. The differential analysis highlights the advantage of using eye tracking for a usability study. The data showed how users would see a certain part of the page, but neglect to click on the link associated with this part. This indicates that this part of the page is not intuitive. For example, we noticed that users viewed the links to the Ethernet and Supply Chain pages in the What s New? box, but chose not to click on them. This granted valuable insight and led us to change the wording of the links within this box. Similarly, users on the Ethernet and Supply Chain pages spent a lot of time on the page before seeing a link to the lead form. Adding additional links to these forms on the product pages significantly reduced the time to task data on these parts of the site. Because eye tracking is a relatively new technology, its place for usability studies has not been validated. This project indicates that eye tracking technology and its associated quantitative data are effective for a usability study. The recommendations we arrived at are similar to those that would be made in a more traditional usability study and have quantitative data to back them up. CONCLUSIONS Traditional usability studies rely largely on qualitative data for making Web site design recommendations. ERICA is unique in its ability to gather quantitative data for usability analysis. GE Fanuc s reliance on its Six Sigma methodology requires gathering this kind of quantitative data for all business projects. Providing GE Fanuc with quantitative data allowed them to do additional analysis for the Six Sigma methodology to validate all design changes. The data gathered allowed us to explore the use of quantitative data to determine the potential ERICA has in usability studies. Analysis allowed us to create and test alternative design recommendations and form a final design recommendation to GE Fanuc. The quantitative data suggested areas of the site to change to improve efficiency on the site. The analysis proved successful in that the final design of the site allowed the user to navigate the site more efficiently, finding critical information more quickly than in the original site, view more key information, and arrive at the lead generation form. Again, GE Fanuc approached us with the goal of increasing the number of users filling out lead generation forms. The use of eye gaze data led to a final design recommendation in which 100% of test subjects filled out the form. This arguably translates into a larger customer base for GE Fanuc and, hence, more profit. REFERENCES ERICA, Incs Online. Internet. 16 April Available: Ferre, Xavier and Juristo, Natalia and Windl, Helmut and Constantine, Larry Usability Basics for Software Developers. IEEE Software (Vol. 18, No. 1) Graney, Glenn. Personal Interview conducted October Lankford, C GazeTracker: Software designed to facilitate eye movement analysis. In proceedings of the 2000 Eye-tracking Research and Application Symposium, sponsored by ACM s SIGCHI and SIGGRAPH (West Palm Beach Florida, November). BIOGRAPHIES Alyssa Boaz is a fourth year Systems Engineering student from Stuart, Florida. James Dennis Cuneo is a fourth year Systems Engineering student from Orinda, California. Next year, Mr. Cuneo will be an associate with The Boston Consulting Group in D.C. David W. Kreps is a fourth year Systems Engineering student from Bethlehem, Pennsylvania. Mr. Kreps looks forward to a lucrative career as a business analyst for Applied Predictive Technologies in the D.C. area. Marsden Watson is a fourth year Systems Engineering student from Rochestor, New York. 162