Conceptual and Practical Issues in the Statistical Design and Analysis of Usability Tests

Transcription

1 Conceptual and Practcal Issues n the Statstcal Desgn and Analyss of Usablty Tests John J. Bosley (Bosley_J@bls.gov), BLS, John L. Eltnge (Eltnge_J@bls.gov), BLS, Jean E. Fox (Fox_J@bls.gov), BLS, Scott S. Frcker (Frcker_S@bls.gov), BLS Presenter and Contact Author: John J. Bosley, Offce of Survey Methods Research, PSB 1950, Bureau of Labor Statstcs, 2 Massachusetts Avenue NE, Washngton, DC 2021 Key Words: Computer-asssted ntervewng (CAI); Data dssemnaton; Mean and dsperson effects; Predctve dstrbuton. 1. Introducton In recent years, the Federal statstcal communty has focused a consderable amount of effort on usablty testng for data collecton nstruments and for data dssemnaton facltes. Usablty testng can help ensure that data are collected easly, effcently, and accurately. It can also help ensure that those who retreve Federal statstcs, partcularly from webstes, fnd and understand what they are lookng for. At the Bureau of Labor Statstcs (BLS), we have ncorporated usablty testng nto the development process for many data collecton nstruments, ncludng the Commodtes and Servces (C&S) survey for the Consumer Prce Index, the Consumer Expendtures Survey, and the Amercan Tme Use Survey. We have also conducted usablty testng for data dssemnaton facltes such as BLS own webste, LabStat. The goal of usablty testng s to fnd problems wth an applcaton and dentfy solutons before the applcaton s dstrbuted to end users. In a usablty test, typcal users perform typcal tasks wth a prototype of the applcaton. The usablty specalsts collect objectve data, such as number of users who completed a task correctly and the number of screens accessed to complete the task. Usablty specalsts also collect subjectve data such as ther own observatons and partcpant comments and ratngs, and then evaluate the data to reveal problems and solutons. The evaluaton usually ncludes an analyss of the quanttatve measures collected, but the analyss tends to be very smple. Usablty specalsts look for cases where the values seem napproprately hgh or low. For example, they may look for unusually hgh task tmes or error rates or unusually low satsfacton ratngs. One reason for the smple analyses s that usablty tests often do not meet the assumptons necessary for more rgorous statstcal analyses. Dumas and Redsh (1993) lst a varety of reasons that usablty test results may not warrant more detaled statstcal analyss. Accordng to these authors, sample szes generally are small (5-10 partcpants per user populaton), and sample selecton and assgnment of treatments to expermental unts may not nclude randomzaton. Further, many of the measures are nomnal or ordnal, but not nterval varables, so they volate the assumptons of many statstcal tests typcally used n psychologcal research. In addton, to the extent that statstcal methods are used n the analyss of usablty data, prevous authors have tended to focus on formal hypothess tests (e.g., t, ch-square or F tests) for equalty of treatment means. As nterface development becomes more refned, one encounters mportant questons that are not readly addressed by the smple analyses descrbed above. For example, non-usablty specalsts, e.g., managers of software development projects, may not readly accept expert conclusons based on a casual analytc approach. Smlarly, knowledgeable clents may rase nformed questons regardng trade-offs n performance as measured n multple dmensons. Consequently, ths paper notes some ways n whch statstcal methods n expermental desgn, formal nference and exploratory analyss may shed addtonal lght on the abovementoned usablty testng ssues. Secton 2 outlnes four ssues that may be of specal nterest n the statstcal analyss of usablty data. Secton 3 ntroduces an example nvolvng testng of

2 alternatve approaches to a topcal-lnk functon for federal statstcs; and presents some related exploratory data analyses. Secton 4 suggests some areas for addtonal work. 2. Four Statstcal Issues n Usablty Testng 2.1. Exploratory Analyses Lnked wth the Practcal Implcatons of Extreme Outcomes In usablty testng, the prmary practcal goal generally s to dentfy nterfaces that may ental unreasonably hgh rsks of unusually hgh or low values of one or more outcomes Y. Wthn ths framework, the concepts of unreasonably hgh rsks and unusually hgh or low values often are very context dependent. For example, n some applcatons there may be an expectaton that almost all users should be able to complete a task n less than a specfed number of mnutes, and any nterface that dd not meet ths crteron would be deemed unacceptable. On the other hand, n other applcatons there may be a greater tolerance for the fact that some users may take a much longer amount of tme to complete a task wth a gven nterface, or may not be able to complete the task at all. Also, even wthn a gven context, the terms unreasonably hgh rsk and unusually hgh or low values often are not precsely defned a pror; and the relatve mportance of dfferent outcome measures (e.g., tme to completon or reported frustraton) are not prespecfed. Consequently, agences may obtan better nsghts nto the usablty of an nterface through exploratory analyss of the outcome data, rather than through formal hypothess testng procedures. For these reasons, t may be advsable to structure analyses of usablty data n forms that allow relatvely drect examnaton of the full predctve dstrbuton of outcome measures Y that are expected to arse wth the partcular user group(s) of nterest. 2.2 Dsperson Effects In some cases, one may address the ssues of Secton 2.1 through statstcal analyss of mean and dsperson effects. For example, f for each nterface, the vector of outcomes ( Y 1,..., ) follows a normal dstrbuton wth mean Y depends entrely on the parameters Y r µ and varance-covarance matrx Σ, then the prevalence rate of extreme values µ and Σ. Thus, for ths case, t s approprate to focus prncpal analytc effort on dentfcaton of nterfaces that have exceptonally large means or varances. To date, mean effects have receved prncpal attenton n the statstcal analyss of usablty data, and tend to follow relatvely standard approaches to formal hypothess testng for the equalty of means, as covered, e.g., n Johnson and Wchern (1998). However, n some cases dfferent treatments may have relatvely smlar means, but have marked dfferences n ther varances. For these cases, a treatment that leads to a hgher varance may n turn lead to an unacceptably large proporton of users havng exceptonally hgh (or low) outcomes Y. In such cases, t s worthwhle to consder detaled modelng of the underlyng varances as functons of the treatments and relevant covarates. A large body of lterature on varance functon modelng has been developed n the lterature for bostatstcs and engneerng statstcs (e.g., Carroll and Ruppert, 1988, Chapter 3; Box and Meyer, 1986; and Davdan, 1990), and could be appled to analyss of dsperson effects n usablty data Nonparametrc Comparson of Predctve Dstrbutons In cases for whch we have relatvely large sample szes and for whch the outcome vectors do not follow an approxmate normal dstrbuton, t s of nterest to carry out a nonparametrc comparson of the predctve dstrbutons of the outcomes Y across dfferent nterfaces. For some outcome varables (e.g., user preferences recorded on a fve-pont Lkert scale) ths nonparametrc comparson s relatvely smple. For other outcome varables (e.g., the tme or number of mouse clcks requred to complete a gven task) the nonparametrc comparsons may be somewhat more complex. One opton nvolves dsplay of predcton ntervals for new observatons Y, or assocated tolerance ntervals. Another opton nvolves dsplay of sde-by-sde quantle plots or boxplots of observatons Y.

3 2.4 Dstnctons Among User Subpopulatons In prncple, a gven nterface may be used by members of dfferent subpopulatons defned by dfferences n such characterstcs as levels of formal tranng, experence, self-perceved ablty to use computers, or motvaton. For example, n work wth usablty testng n computer-aded ntervewng, the prncpal subpopulatons of nterest may be defned by levels of tranng and experence wthn the ntervewer corps. In ths example, three subpopulatons mght be feld supervsors, experenced ntervewers and relatvely new ntervewers. In the data dssemnaton context, Marchonn and Hert have attempted to defne varous qualtatvely dstnct user types, wth a vew toward desgnng and provdng dfferently-confgured nterfaces for each dfferent type. (Hert and Marchonn, 1997; Marchonn, 2000.) In general, these researchers sought to make role-wde dstnctons, e.g., teacher, student, economc research professonal, busness owner, and so on. They dd focus on attemptng to defne a task-based classfcaton scheme on occason, but ths was not the prmary approach. Ths has two practcal mplcatons for statstcal work wth usablty data. Frst, t s mportant for the desgn of the usablty study to select persons approprately from the user subpopulaton(s) of nterest; and for results to be nterpreted accordngly. Second, f the usablty study nvolves more than one subpopulaton, then there may be specal nterest n evaluaton of nteractons between nterface effects and subpopulaton effects. 3. An Example: Usablty Testng of Alternatve Approaches to a Topcal-Lnk Search Functon for Federal Statstcs One commonly-used technque to enable users to fnd nformaton on a large, nformaton-rch webste s to create a content ndex to the ste and then use that ndex as a lst of hypertext lnks that wll take the user to the topc that the lnk label descrbes. The cross-agency portal ste, Fedstats ( s one of several major federal statstcal webstes for data dssemnaton that uses a topcal ndex to support searches. Ths ste contans lnks to data that resde at more than 100 dfferent U.S. federal statstcal agences wth a wde range of mssons. The Fedstats home page presents a varety of structured search functons as alternatves to the search engne. Among these s a Topcs A-Z ndex lst of hyperlnks, along wth a search by agency and a search by geography ( Mapstats ). Pror usablty tests (Marchonn and Hert, 1997) of the Fedstats nterface had rased concerns about the effectveness of the Topcs A-Z ndex for certan tasks and for certan types of users. To address these concerns, Ceaparu and Shnederman (2003) conducted three emprcal studes of alternate organzatonal schemes for FedStats Topcs A-Z. After each tes t, the FedStats ste was modfed to ncorporate the results. Each study had 15 partcpants, all of them graduate students from a wde varety of dscplnes. The gender composton of the three samples was reasonably equvalent. In each test, all partc pants performed three scrpted search tasks ( scenaros ) per verson. These three tasks were always presented n the same order. The frst scenaro requred constructon of an answer to a complex queston nvolvng urban development n formerly rural areas. The second scenaro had a much more specfc goal--locatng data about sales trends for organc products n a metropoltan area. The thrd scenaro asks the partcpant to fnd data that would allow comparson of two U.S. ctes for lvablty by user-defned crtera. Of the three, the last appears to be the least well-defned or structured, wth the second the most clearly-defned task and the frst somewhere between them. Ceaparu and Shnederman (2003) recorded task completon, the total elapsed tme for a gven task, an overall measure of reported user frustraton (on a scale of 1-10), and seven measures of specfc types of user frustraton (each also measured on a scale of 1-10). For the current paper, we have carred out addtonal analyses of these data; some relatvely smple results are dsplayed n Fgures 1 through 7. More detaled analyses wll be reported n a longer verson of ths paper. Fgure 1 presents boxplots of the Overall Frustraton scores for subjects who receved treatment 1, wth separate plots for each scenaro. For a gven scenaro, the top and bottom of the grey box represent the seventy-ffth and twenty-ffth percentles of the observed values; the mddle lne represents the medan; and the sold dot represents the mean. In addton, the upper and lower whskers represent the extreme upper and lower values of the

4 observed scores. Fgures 2 and 3 present related boxplots for treatments 2 and 3, respectvely. Comparsons of Fgures 1 through 3 ndcate that the degree of varablty n the overall frustraton scores (as reflected n the nterquartle range, equal to the dfference between the seventy-ffth and twentyffth percentles) vares consderably across scenaros and across treatments. For example, the nterquartle range for Scenaro 3 s substantally larger for Treatment 2 than t s for Treatments 1 or 3. In other analyses, not reported n detal here, we observed smlar (and n some cases, more pronounced) patterns of heterogenety of varance n task completon tme and other scores. Fgures 4 through 6 present a correspondng set of boxplots for the Confusng Search component of the frustraton score. These plots agan dsplay some ndcaton of heterogenety of varance (e.g., for Scenaro 1 n Treatments 1 and 2). In addton, note that for Treatment 3, the dstrbuton of the Confusng Search scores was also nfluenced by a floor effect snce most of the scores were relatvely close to zero. Fnally, Fgure 7 presents a scatterplot of frustraton scores aganst completon tmes, wth dfferent plottng symbols used for each treatment. The frustraton scores and completon tmes are jttered n ths plot to avod problems wth overstrkes. The plot ndcates a moderate degree of postve assocaton between the (subjectve) frustraton scores and the (objectve) completon tmes; a related smple lnear regresson of 2 frustraton score on completon tme had an R value equal to Subjects correspondng to ponts n the upper left and lower rght corners of the scatterplot may be of specal nterest for follow-up analyses. For example, subjects n the upper left corner completed the assgned task relatvely quckly, but nonetheless reported relatvely hgh levels of frustraton. It would be of nterest (but beyond the scope of the current work) to explore the extent to whch these phenomena may be assocated wth subpopulaton membershp (whch n turn may be lnked wth expectatons regardng comp leton tme); or assocated wth specfc components of the frustraton score that are relatvely ndependent of completon tme. 4. Dscusson Ths paper has noted some areas n whch statstcal analyss methods may offer some nsghts nto usablty data. A more detaled examnaton of ths area would need to consder several addtonal topcs, e.g., sample sze selecton and power analyss; nonresponse; measurement error; Hawthorne and other nterventon effects; and use of formal factoral or fractonal factoral expermental desgns to dentfy mportant factors n mean and dsperson effect models, and to construct correspondngly mproved nterfaces. In consderng these ssues, t would be mportant to dentfy specfc cases n whch the costs of ncreased sample szes, complexty of study desgn, and analytc burden can reasonably be justfed by the resultng addtonal nsghts nto human-computer nteracton that would not have been obtaned through smpler approaches. (See also, for example, Nelsen, 1993). Acknowledgements The authors wsh to thank Ms. Irna Ceaparu and Prof. Ben Shnederman of the Unversty of Maryland s Human-Computer Interacton Laboratory for kndly permttng use of ther data n ths paper. We also thank Dr. Cornell Julano of the Xerox Corporaton for a conversaton that helped motvate wrtng ths paper, and for obtanng corporate permsson to use some of hs data n our work. References Box, G.E.P. and Meyer, R.D. (1986). Dsperson Effects from Fractonal Desgns Technometrcs, 28, Carroll, R.J. and Ruppert, D. (1988). Transformaton and Weghtng n Regresson. New York: Chapman and Hall. Ceaparu, I., and Shnederman.B. (2003). Fndng Government Statstcal Data on the Web: Three Emprcal Studes of the FedStats Topcs Page. Unversty of Maryland HCIL Report HCIL , May 28, Onlne: ftp://ftp.cs.umd.edu/pub/hcl/reports-abstracts -Bblography/ html/ htm,

5 Fgure 1: Dstrbuton of Overall Frustraton Fgure 2: Dstrbuton of Overall Frustraton Ratngs by Scenaro for Treatment 1 Ratngs by Scenaro for Treatment 2 Fgure 3: Dstrbuton of Overall Frustraton Fgure 4: Dstrbuton of Confusng Search Ratngs by Scenaro for Treatment 3 Ratngs by Scenaro for Treatment 1 Fgure 5: Dstrbuton of Confusng Search Fgure 6: Dstrbuton of Confusng Search Ratngs by Scenaro for Treatment 2 Ratngs by Scenaro for Treatment 3

6 Fgure 7: Scatterplot of Overall Frustraton Score Aganst Elapsed Tme, Wth Separate Plottng Symbols for Treatments 1 Th rough 3 (Frustraton Scores and Tmes are Jttered) Davdan, M. (1990). Estmaton of Varance Functons n Assays wth Possbly Unequal Replcaton and Nonnormal Data. Bometrka, 77, Dumas, J.S. and Redsh, J.C. (1993). A Practcal Gude to Usablty Testng. Norwood, NJ: Ablex Publshng. Hert, C.H. & Marchonn. (1997). Seekng Statstcal Informaton on Federal Webstes: Users, Tasks, Strateges, and Desgn Recommendatons. Fnal Report to the Bureau of Labor Statstcs, July 18, Onlne: Johnson, R.A. and Wchern, D.W. (1998). Appled Multvarate Statstcal Analyss. Englewood Clffs, NJ: Prentce-Hall Inc Lazar, J., Bessere, K., Ceaparu, I., Robnson, J., and Shnederman, B. (2003). Help! I m Lost: User Frustraton n Web Navgaton, IT&Socety, 1(3), Wnter 2003, pp Onlne: Marchonn, G. (2000). Interfaces to Support Customzed Vews and Manpulaton of Statstcal Data, Paper presented at the Second Internatonal Conferences on Establshment Surveys, Buffalo, NY, Onlne: Nelsen, J. (1993). Usablty Engneerng. New York: AP Professonal. Tourangeau, R., Couper, M.P. and Conrad, F. (2003). The Impact of the Vsble: Images, Spacng, and Other Vsual Cues n Web Surveys. Paper presented at the WSS/FCSM Semnar on the Fundng Opportunty n Survey Methodology, May 22, 2003.