Usability Tst of UCRS -Larning DVD Dcmbr 2006 Writtn by Thrsa Wilkinson
Excutiv Summary Usability Tst of th UCRS -Larning DVD, Dcmbr 2006 This rport documnts th findings of a usability tst of th UCRS -Larning DVD. Th main purpos of th tst was to assss th usability of th DVD for UCRS CSRs. Background Novmbr 24, 1993, Prsidnt Clinton signd law rquiring stats to stablish and utiliz a systm of profiling all nw claimants for rgular UC bnfits. Th purpos of th law is to idntify claimants who ar likly to xhaust UC bnfits and nd job sarch assistanc to mak a succssful transition to nw mploymnt. Th REA program is an xpansion of th currnt Unmploymnt Compnsation Rmploymnt Srvics (UCRS) program. It rquirs a fac-to-fac, on-on-on assssmnt to b prformd in On Stops with UC claimants and is intndd to idntify ligibility issus and possibl barrirs to mploymnt, as wll as to provid th ncssary tools to claimants to assist thm in dvloping a work sarch plan to obtain mploymnt. UCRS has ngagd in a projct to crat an -Larning DVD that will b distributd to ach of th CSRs in th On Stops throughout th stat. Th DVD will b an orintation to rmploymnt srvics. It is fully narratd and contains vidos, animations and simulations of OJI and SCOTI. It has bn dsignd to b ducational and has incorporatd many aspcts dsignd to rtain th attntion of th studnt. Positiv aspcts about -Larning DVD: Ability to us buttons. Most tst participants had no problms using th buttons to mov from pag to pag or paus and stop th scrns. Quizzs. Tst participants considrd th sction quizzs to b vry asy to us. Animations. Most tst participants commntd positivly on th animatd charactrs. Chaptr 2. All tst participants found Chaptr 2 layout, graphics, and scrn colors to b plasing and asy to us. Improvmnt aras for -Larning DVD: Audio and txt not matching in th Timlin and in Chaptr 1. All participants notd that th audio and txt did not match. Thy found this vry problmatic: I want to rad along but I can t. Misspllings. Most tst participants found th misspllings to b distracting. Ability to rad txt. Dark background and drop shadows on txt mad th txt vry hard to rad on many scrns in Chaptr 1. Pag 2
Usability Tst of th UCRS -Larning DVD, Dcmbr 2006 Basd on th usability problms idntifid in this tst, rcommndations ar as follows: Ensur txt and audio match up on most scrns in th DVD. Chang background colors pr standards. Eliminat drop shadows on txt. Fix misspllings. Hav somthing chang or add mor graphics if a graphic is on th scrn for ovr a minut. For mor spcific changs, s Usability tst rsults.doc Thrsa Wilkinson Dcmbr, 2006 Pag 3
Tabl of Contnts Usability Tst of th UCRS -Larning DVD, Dcmbr 2006 Excutiv Summary 1 1. Approach 5 1.1. Purpos 5 1.2. Mthod 5 1.3. Discussion of Mthod 5 1.4. Equipmnt 5 1.5. Tst Participant Profils 6 1.6. About th Rport 6 2. Findings 7 2.1. Navigation bar 7 2.2. Timlin 8 2.3. C1_1_12 - Clinton 9 2.4. C1_1_13 Profiling Intro 9 2.5. C1_1_14 Statistical modl scrn 9 2.6. C1_2_1 B&W movi 10 2.7. C1_2_3 10 2.8. C1_2_4 Color movi 10 2.9. C1_2_6 US Map 11 2.10. C1_2_7 Piggy bank 11 2.11. C1_2_9 11 2.12. C2_1_1 11 2.13. C2_1_6 Showing profiling pool by county rport 11 2.14. C2_2_2 - Guys 12 2.15. Quiz 12 2.16. Gnral commnts 12 Appndix A. Usability Tst Script 14 Appndix B. Tst Task Rsults 19 Pag 4
1. Approach Usability Tst of th UCRS -Larning DVD, Dcmbr 2006 Th usability tst was conductd for th -Larning DVD in Dcmbr, 2006. Th tst was carrid out on th portions of Chaptrs 1 and 2. 1.1. Purpos Th primary purpos of th tst was to assss th usability of DVD for CSRs. 1.2. Mthod Th tst has bn carrid out with svn usrs who d to participat in th tst. Th profils of th tst participants appar in sction 1.5. Tst participants wr tstd on by on. A usability profssional actd as tst facilitator. All tsts wr conductd in cubicl on sit. Each tst took about on hour. Th usability tst consists of two phass: Solving tasks and Dbrifing. In this tst, th phass containd th following stps: Solving tasks: Dbrifing: Tst participants wr askd to carry out tasks using th DVD. Ths tasks ar includd as appndix C. Tst participants wr askd to think aloud and to commnt on th DVD whil thy wr carrying out thir tasks. Tst participants wr askd qustions and thn askd to fill in a qustionnair whil thinking aloud. W wr particularly intrstd in th vrbal commnts thy mad whil filling in th qustionnair. In addition, tst participants wr dbrifd about thir gnral imprssion of th DVD. Th qustionnair and th dbrifing qustions ar includd in th usability tst script in appndix A. 1.3. Discussion of Mthod Th basis for this usability tst is th rcognizd think aloud mthod. This mthod is dscribd in svral gnrally rcognizd books within th usability fild, for xampl: - Usability Enginring by Jakob Nilsn (Acadmic Prss 1993, s also www.usit.com). 1.4. Equipmnt Th quipmnt usd for this tst was a dsktop Compaq Evo with a 15 HP 1940 monitor. Microsoft Intrnt Explorr 7 was usd. Th computr was st up to run th DVD from fils on th dsktop. Pag 5
1.5. Tst Participant Profils Usability Tst of th UCRS -Larning DVD, Dcmbr 2006 Th tsts wr carrid out with tst participants who fulfilld th following rquirmnts: Btwn 30 and 55 yars old. Somwhat xprincd to xprincd Intrnt usrs. Th numbr of mn and womn in th group wr not qual. Participant Gndr Intrnt xprinc* 1 Mal Somwhat Exprincd 2 Fmal Somwhat Exprincd 3 Fmal Somwhat xprincd 4 Fmal Somwhat xprincd 5 Fmal Exprincd 6 Mal Exprincd 7 Fmal Exprincd * Intrnt xprinc was classifid by th tst participant according to ths groupings: 1. Non (.g. no Intrnt xprinc) 2. Bystandr (.g. has watchd othr prsons us th intrnt) 3. Bginnr (.g. has usd it onc or twic) 4. Somwhat xprincd (uss it rgularly) 5. Exprincd (uss sarch facilitis without problms) 6. Vry xprincd (has dvlopd wb sits) 1.6. About th Rport Quots: Th rport contains quots from tst participants. Appndix A contains th full usability tst script including th tasks givn to th tst participants. Appndix B contains a tabl that shows tst tasks and th tst participants succss in solving thm. Pag 6
Usability Tst of th UCRS -Larning DVD, Dcmbr 2006 2. Findings Findings ar catgorizd by th facilitator using th following catgoris: Good. This approach is rcommndabl. Good ida. A suggstion from a tst participant that could lad to a significant improvmnt of th usr xprinc. Minor problm. Causd tst participants to hsitat for a fw sconds. Srious problm. Dlayd tst participants in thir us of th DVD, but vntually thy wr abl to continu. Critical problm. Causd frqunt catastrophs. A catastroph is a situation whr th DVD wins ovr th tst participant, i.., a situation whr th tst participant cannot solv a rasonabl task or which causs th tst participant grat irritation. 2.1. Navigation bar Not: P1 - Is th navigation bar at th top and bottom ncssary to b up all th tim? It maks th scrn look too busy. Tak up ntir scrn lik th quiz. P1, P3, P4, P5, P7 - Usd scroll bar and Nxt button without any problms. P6 - Trid to us th back button on C1_1_14 and bcam frustratd. Participant thought it would work similar to a "rwind" button whr it would rwind within C1_1_14 and instad it rturnd th participant to scrn C1_1_13. Commnts: P2 - Wantd th systm to tll him/hr whn to click Nxt. H/sh did not us th scrollbar. P4 Can you mak th Nxt button mor obvious? P5 Can you mak it advanc for you? H/sh was using all buttons (Paus, Play) with no problms. P6, P7 Had to rmind him/hr to us th Nxt button. Pag 7
Usability Tst of th UCRS -Larning DVD, Dcmbr 2006 2.2. Timlin P1, P2, P3, P4, P7 Txt and audio dos not match. It is vry distracting. P1, P2, P3, P4, P7 - Noticd whr audio lft out th txt for 1998. P4 Can you shortn th JFK vido? Commnts: P2 - Sinc txt and audio do not match had to listn intntly and had no tim to rad txt. I want to rad along but I can t. P2, P5 Too much history. Tim is prcious. P2 Had problms rading whit txt on th dark background scrns. Sh had to mov closr to th scrn to s th txt (bifocals). P2 On 1973, how many mor dats to I hav to go though? P4 Can you shortn th JFK vido? P3 What is th rlvanc of th historical facts with UCRS, spcifically man on moon and Brlin wall? Th 1993 information did mak sns. P5 Why would I want to know about th Batls, Monopoly, Match Gam, or th othr stuff in thr? Just gt right to th point. Tim is a big considration. Suggstd that th history sction b shortnd. P6 At first, this participant mad commnts lik Sms lik a lot of fillr hr and this is th boring part but at th nd of th timlin, h/sh said th rfrnc to all th history and what is going on that yar is prtty cool. H/sh askd for mor dtaild information on REA in 2004. P7 H/sh would rathr rad or listn, not both sinc h/sh could not rad along with th txt. H/sh thought th youngr staff may not b abl to rlat to th information on th timlin (Batls, Match Gam). History was too long. Pag 8
Usability Tst of th UCRS -Larning DVD, Dcmbr 2006 2.3. C1_1_12 - Clinton P2 Can you bullt th txt to mphasiz th points bing mad? No txt on th scrn. Nd somthing to happn. P3, P5 What is th purpos of th magnifying glass? It maks it hard to rad th scrn txt. Commnts: P4 How dos Bill Clinton rlat to txt about CSRs? Likd th magnifying glass but txt had no corrlation with audio. P6 You ar losing m hr. I am not focusing or listning. Mntiond that txt and audio did not match up in this sction. P7 Why is thr a photo of Bill Clinton whn you ar talking about CSRs? Clickd Nxt button to advanc. 2.4. C1_1_13 Profiling Intro P1, P5, P6, P7 Misspllings ar distracting. Commnts: P5 H/sh bcam bord in this sction and startd to daydram. Just gt to th point. This chaptr sms vry long. P6 Th fading popl graphic is cool. P7 Th group of popl graphic is blurry it is starting to bothr m sinc I hav bn looking at it for so long. 2.5. C1_1_14 Statistical modl scrn P1, P2, P6, P7 found it vry distracting with th animation going all th tim. Too busy. P2, P7 Hard to rad txt with drop shadow. Pag 9
Usability Tst of th UCRS -Larning DVD, Dcmbr 2006 Commnts: P6, P7 Why would I want to pay attntion to th statistical information sinc OH did not choos this modl? Vry bord. Did not listn vry wll. P7 Rcivd on th scrn is missplld. 2.6. C1_2_1 B&W movi Commnts: P1, P7 Lins on th B&W movi ar distracting. P7 Audio is rally loud. 2.7. C1_2_3 P2 - Bst practics boxs sm to fly in from no whr. Can you hav thm look lik thy ar coming from th stats? P6 Bst practics boxs look blurry. P2 Can you mak th txt box biggr? Hard to rad th txt. 2.8. C1_2_4 Color movi P1 Can you us an arrow to point to whr you ar rfrring on th scrn? P4 Not obvious on th map what is bing rfrncd. Can you point to what you ar talking about? Can you pan slowr? Commnts: P7 H/sh was not sur th languag was corrct about th On Stops as local offics mad it sound lik no partnrs wr involvd. Pag 10
2.9. C1_2_6 US Map Usability Tst of th UCRS -Larning DVD, Dcmbr 2006 P2, P4 Dark background and whit txt is gtting to m. P6 also movd in closr to look at txt on th darkr scrns vn though h/sh did not mntion this as a problm. Commnts: P4 Local Oprations is not cntrd. 2.10. C1_2_7 Piggy bank P6, P7 H/sh likd th pig. Laughd out loud. 2.11. C1_2_9 P2 - Whit boxs and lin that gos up to govrnor. Can you mak th boxs largr? Hard to rad. P2 Dark background and whit txt is gtting to m. P4 Too much dark blu. Txt is hard to rad. 2.12. C2_1_1 P2 Can you spll out what OJI stand for as h says it? 2.13. C2_1_6 Showing profiling pool by county rport P4 Can you add an arrow to point to whr th txt is rfrring to on th rport? Circl th county or put an arrow at th top of th rport? This would mak it asir to follow along. P7 Can th audio and txt match up bttr? Pag 11
2.14. C2_2_2 - Guys Usability Tst of th UCRS -Larning DVD, Dcmbr 2006 P4, P5 Rally likd chaptr 2 ovrall. Likd th photos and bulltd txt. P6 Aftr this sction finishd, I rally likd that sction. Commnts: P7 This is th third tim w hav hard about th six charactristics. P7 For th first photo, you should show a group of popl instad of on guy. Likd photo of guy ripping up th lttr. 2.15. Quiz P1 Likd how quiz took up th ntir scrn. Vry asy to us. Likd th instructional txt. Commnts: P4 Can you mak th txt that says Ys, that is corrct in a diffrnt color? P5 No qustions about th Batls? P7 H/sh wantd to click th Nxt button in th cntr of th scrn, not on th sid. 2.16. Gnral commnts P1 and P2 Likd th txt boxs. P2, P4 A lot of audio. Rally had to listn. Could b ovrwhlming to som popl. Can you do it at a slowr pac? Too much information to absorb. P4 Rally likd th txt font and siz. All noticd that th Paus button did not always work. Pag 12
Usability Tst of th UCRS -Larning DVD, Dcmbr 2006 P4 Could not unclick th subtitls whn sh had th full txt scrn up. P5 H/sh wantd to click th Nxt button on all scrns. Sh clickd out of 1961 and 1962. Sh clickd through all animations. P5 Flt Chaptr 1 is a good ovrviw but it is too long. Kp things at a minimum. Us mor visual xampls. Too many photos. Us mor bullts (likd Chaptr 2). P5 Rmmbr thr ar oldr popl at th On Stops. On som scrns, th fonts ar too small. P6 Likd th animations spcially th pig and th profssor. P7 Likd whn th audio and th txt matchd up. H/sh did not xpct th bullts to match up xactly to th txt knows th information is condnsd. Whn th txt matchs th audio, I don t hav to listn as intntly. Pag 13
Usability Tst of th UCRS -Larning DVD, Dcmbr 2006 Appndix A. Usability Tst Script Thank you for coming today. W r working on th on th UCRS -larning DVD, and as part of th procss w r asking a varity of popl to attmpt to us th DVD to s what lmnts nd to b changd. I d lik to strss that w r tsting th DVD, and not your computr abilitis. If you find parts of th DVD that ar difficult to us or undrstand, so will othr popl, and it will b our job to mak sur w mak th appropriat changs to improv it. You cannot do anything wrong hr. W want to improv th DVD so w nd to know what you honstly think about it. As w go along, I am going to ask you to think out loud, to tll m what s going through your mind. This will b hlpful to us. If you hav any qustions, just ask. I may not b abl to answr thm right away, sinc w ar intrstd in what popl do whn thy don t hav somon sitting nxt to thm, but I will try to answr any qustions whn w ar don. If you nd to tak a brak, plas lt m know. I will also ask you som qustions whil you ar going through th DVD. Do you hav any qustions bfor w start? Pag 14
Usability Tst of th UCRS -Larning DVD, Dcmbr 2006 Tst tasks 1. Go through Ch 1 sctions 1 and 2 2. Complt 2 quiz qustions. 3. Go through Ch 2 sction 1 and 2 4. Tll m whr ar you in th DVD What chaptr? What pag numbr? 5. Can you locat th onlin hlp 6. Can you viw subtitls for th audio? 7. Can you viw th full txt for this pag? Aftr 60 minuts, th tst facilitator trminats th tst rgardlss of th numbr of tst tasks compltd. Individual tasks can also b intrruptd if th tst participant cannot advanc with th task or if th tst facilitator finds that no nw information would com of continuing with th task. Intrviw aftr tst A. Do you lik th scrn colors? Is anything to light? Is anything too dark? B. Do you lik th font? Font siz? Font color? C. Ar th animations distracting? D. Ar thr any unfamiliar trms? E. Dos th information mak sns? Pag 15
Qustionnair and rsults Usability Tst of th UCRS -Larning DVD, Dcmbr 2006 I hav brought a brif qustionnair that I would lik you to fill out. Plas continu to think aloud as you mak your slctions so I undrstand why you tick off what you do. 1 2 3 4 5 6 7 NA 1 Th DVD was asy to install. O NA 2 Ovrall, I am satisfid with how asy it is to us this DVD. O 4 usrs ratd 7 3 usrs ratd 6 3 It was simpl to us this DVD. O 2 usrs ratd 7 5 usrs ratd 6 4 I fl comfortabl using this DVD O 1 usr ratd 7 4 usrs ratd 6 1 usr ratd 5 5 Larning to oprat th DVD would b asy for m. O 2 usrs ratd 7 5 usrs ratd 6 6 Using th DVD would nhanc my ffctivnss on th job. O 1 usr ratd 7 4 usrs ratd 6 1 usr ratd 5 1 usr ratd N/A 7 Using th DVD would mak it asir to do my job. O 1 usr ratd 7 5 usrs ratd 6 1 usr ratd N/A 8 Th information providd for th DVD is asy to undrstand. O 4 usrs ratd 6 2 usrs ratd N/A 1 usr ratd 4 9 It was asy for m to dtrmin my location in th DVD. O 5 usrs ratd 6 2 usrs ratd 5 10 Th organization of information on th DVD scrns is clar. O 1 usr ratd 7 2 usrs ratd 6 2 usrs ratd 5 1 usr ratd 4 Pag 16
Usability Tst of th UCRS -Larning DVD, Dcmbr 2006 1 usr ratd 2 11 Th intrfac of this DVD is plasant. O 1 usr ratd 7 5 usrs ratd 6 1 usr ratd 5 12 I lik using th intrfac of this DVD. O 4 usrs ratd 6 2 usrs ratd 5 1 usr ratd 4 13 This DVD has all th functions and capabilitis I xpct it to hav. O 1 usr ratd 7 3 usrs ratd 6 3 usrs ratd 5 14 Ovrall, I am satisfid with this DVD. O 6 usrs ratd 6 1 usr ratd 4 15. How would you rat th ovrall as of us or difficulty of th DVD? 1 2 3 4 5 Vry Easy Nithr Difficult Vry Easy Easy Nor Difficult Difficult Commnt: 2 usrs ratd 1 Vry Nic. I nd somthing lik this for our troops in SCOTI. 5 usrs ratd 2 16. How asy or difficult was it to find information in th systm? 1 2 3 4 5 Vry Easy Nithr Difficult Vry Easy Easy Nor Difficult Difficult Commnt: Pag 17
Usability Tst of th UCRS -Larning DVD, Dcmbr 2006 6 usrs ratd 2 1 usr ratd N/A 17. What do you lik last about th DVD? Sction 1 History was too involvd. Could condns and gt to UCRS/REA soonr. Som of th bginning fillrs. Th narration at th bginning. Historical timlin, too much. Purpos? Talking pac too fast. Too much blu. Trivia rlvanc Rduc Chaptr 1 too long (ovrviw is ky hr) Scrn crowddnss Txt/audio dosn t match somwhat distracting 18. What do you lik bst about th DVD? Eas of navigating from on scrn to th nxt. Th ntir procss was vry asy not a problm to oprat. Intractiv Choic subtitls, listning Th profssor Corrlation of OLO, On Stop, UI Th quizzs Drop down info was hlpful Attmpt at humor and varity 19. What on thing would you tll th dsignrs to chang? Match scrn txt to audio txt whnvr possibl Th bginning Blu!! Automatically advanc to nxt pag. Rduc top and bottom navigation top and bottom bar. Mak th cntr mor visibl. Pag 18
Usability Tst of th UCRS -Larning DVD, Dcmbr 2006 Appndix B. Tst Task Rsults Th chart blow lists all tst tasks and all tst participants. Each ntry indicats how long it took th tst participant to complt that particular task. Usability tst rsults: 1 2 3 4 5 6 7 Task 1 35 min 25 min 34 min 35 min 32 min 41 min 41 min Task 2 09 min N/A 10 min 8 min 9 min 12 min 10 min Task 3 5 sc 5 sc 4 sc 1 sc 3 sc 5 sc 6 sc Task 4 10 sc 10 sc 3 sc 3 sc 3 sc 10 sc 4 sc Task 5 5 sc 3 sc 10 sc 2 sc 5 sc 5 sc 3 sc Task 6 15 sc 3 sc 5 sc 1 sc 12 sc 3 sc 3 sc Pag 19