User Survey on Q-interactive Examinee Behavior

Transcription

1 User Survey n Q-interactive Examinee Behavir Usability Findings Mark H. Daniel, PhD Senir Scientist fr Research Innvatin Nvember 2013 Cpyright 2013 NCS Pearsn, Inc. All rights reserved. Page 1

2 Intrductin Pearsn recently cnducted the first f a series f surveys f Q-interactive users t gather their bservatins abut hw the assessment system is wrking. User feedback can have several imprtant benefits. Reprts such as this can prvide the Q-interactive users with timely infrmatin abut patterns in the behavirs and respnses f examinees f different ages, demgraphic characteristics, r reasns fr referral. Learning abut the experiences f a large number f users can help individual practitiners interpret their wn clinical bservatins and be cnfident in making decisins abut using Q-interactive with particular clients. Regular feedback gives the Q-interactive develpers insights abut areas fr pssible enhancement f the system. This initial survey fcused n practitiners perceptins f whether the Q-interactive frmat affects the test-taking behavir f children and adlescents, with a special emphasis n examinees being evaluated fr particular clinical cnditins. WISC IV was the first child/adlescent assessment t becme available n Q-interactive, in August 2012, 14 mnths befre this survey was cnducted. Other instruments fr children and adlescents have since been added (D-KEFS, NEPSY,CMS, and WIAT-III). Althugh this survey was directed t WISC-IV users, the questins were nt limited t that instrument. Equivalence f scres btained frm Q-interactive and paper versins f the same tests has been largely demnstrated in a series f empirical studies (see Q-interactive Technical Reprts 1 thrugh 6, available at The feedback frm users presented here shuld be cnsidered in light f the general finding f scre equivalence between mdes f administratin. Prcedure In Octber 2013 an invitatin t take part in an nline survey was sent t 247 current Q-interactive users wh have administered WISC IV n system. Ninety-five f the practitiners respnded, fr a respnse rate f 38%. The survey included the fllwing questins: 1. Hw lng have yu been using Q-interactive? 2. What age grups d yu use Q-interactive with? (5 t 9, 10 t 18, 19 t 50, 51+) 3. Fr respndents wh use Q-interactive at ages 5 9: a. Have yu nticed a change in hw engaged yur clients, aged 9 and yunger, are during the assessment sessin when using Q-interactive? If yes, please explain. b. D yu find that yur clients aged 9 and yunger respnd differently t stimuli shwn n the ipad versus paper than lder clients (if applicable)? If yes, please explain. 4. Fr respndents wh d nt use Q-interactive at ages 5 9, but use it at ages 10 18: a. Have yu nticed a change in hw engaged yur clients, aged 10 and lder, are during the assessment sessin when using Q-interactive? If yes, please explain. Cpyright 2013 NCS Pearsn, Inc. All rights reserved. Page 2

3 5. The fllwing questins were asked fr each f five diagnstic grups: ADHD, autism, develpmental delay, intellectual disability, and learning disability: a. D yu use Q-interactive with clients diagnsed with [cnditin]? b. If yes: i. D yu find that clients diagnsed with [cnditin] respnd differently t stimuli shwn n the ipad versus the paper and pencil tests? If yes, please explain. ii. Please describe ther nuances yu have bserved when using Q-interactive with clients diagnsed with [cnditin]. 6. Have there been any clients with whm yu have had difficulty using Q-interactive? If yes, please explain. Results Abut tw-thirds f the respndents (68%) had been using Q-interactive fr three mnths r less, whereas 11% had used it fr nine mnths r mre. Almst all f the practitiners used Q-interactive with examinees ages 5 9 (82%) and (94%). Abut half (45%) used Q-interactive with ages 19 50, and 19% used it with examinees ages 51 r lder. Results by Age Level Ages 5 9 (59 respndents) Engagement: Of the 79 respndents wh use Q-interactive at ages 5 9, 74 answered the questin abut whether they had nticed an effect f Q-interactive n the children s level f engagement. Of thse, 77% reprted bserving an effect. The fllwing is the breakdwn f the effects reprted by these individuals: The great majrity (91%) described the effect as psitive (e.g., mre engaged, mre attentive, increased interest, mre willing t respnd, mre fcused ). Tw respndents (4%) reprted a negative effect ( mre distracted and want t play with the cmputer, sme becme preccupied trying t get the items t light up ). Three respndents (5%) made neutral cmments ( they like t push the buttns that light up smetimes this is gd, but smetimes distracting, ne lw-functining child said we were playing a game, it seems they want smething t happen when they [tuch the screen] ). Cpyright 2013 NCS Pearsn, Inc. All rights reserved. Page 3

4 Ages 5 9: Effect f Q-interactive n engagement (N = 74) 3% negative effect 3% n effect 23% psitive effect 71% Respnse t ipad stimuli: Of the 69 wh expressed an pinin, 38% said that children respnded differently t stimuli when presented n Q-interactive, rather than n paper. There were diverse descriptins f the nature f the difference, mst f which were repetitins f the greater engagement reprted in respnse t the previus questin. Hwever, sme respnses related specifically t hw children respnded t the presentatin f stimuli, such as: less distracted by turning f pages, etc. they find it easier t d they try t pick multiple ptins mre frequently smetimes they try t play with it by checking ut what happens if they push multiple respnses less impulsive the delivery is smther and mre familiar t them Ages (15 respndents) Engagement: Abut half (7 f 15, r 47%) reprted that Q-interactive affected the examinee s level f engagement, and 100% f these respndents described the effects as psitive: much mre engaged, mre interested, mre cmfrtable, very mtivated. There were n reprts at this age level f examinees being distracted r wanting t treat the ipad as a ty r game. Cpyright 2013 NCS Pearsn, Inc. All rights reserved. Page 4

5 Results by Clinical Grup Althugh this survey was intended t fcus n the use f Q-interactive with children and adlescents, the questins abut users experiences with individuals diagnsed with particular clinical cnditins were nt age-limited. The fllwing infrmatin is based, in part, n users experience with adult examinees. ADHD Of the 77 respndents wh had used Q-interactive with individuals with an ADHD diagnsis, 57% said that such examinees respnded differently t stimuli presented with Q-interactive than with paper. Out f thse wh thught Q-interactive had an effect, 82% reprted that examinees were mre attentive, 9% said they were less attentive, and 9% said that the effect was neutral r incnsistent. ADHD: Effect f Q-interactive n attentitiveness (N = 77) n effect 43% mre attentive 47% 5% less attentive 5% Cpyright 2013 NCS Pearsn, Inc. All rights reserved. Page 5

6 Autism Of the 34 respndents wh had used Q-interactive with individuals with an autism diagnsis, 59% reprted that presenting stimuli using Q-interactive had an effect n examinee behavir. Of these, 85% said that examinees were mre engaged and the ther 15% said that the effect was neutral r incnsistent. Autism: Effect f Q-interactive n engagement (N = 34) n effect 41% mre engaged 50% 9% less engaged 0% Cpyright 2013 NCS Pearsn, Inc. All rights reserved. Page 6

7 Develpmental Delay Of the 39 respndents wh had used Q-interactive with individuals diagnsed as develpmentally delayed, 41% reprted that presenting stimuli using Q-interactive had an effect n examinee behavir. Of these 16 respndents, 15 (94%) said that examinees were mre engaged, and the remaining respndent said that the effect was neutral r variable. Develpmental Delay: Effect f Q-interactive n engagement (N = 39) n effect 59% mre engaged 38% less engaged 0% 3% Cpyright 2013 NCS Pearsn, Inc. All rights reserved. Page 7

8 Intellectual Disability Results fr this grup were nearly identical t thse fr develpmental delay. Of the 45 respndents wh had used Q-interactive with individuals diagnsed as intellectually disabled, 42% reprted that presenting stimuli using Q-interactive had an effect n examinee behavir. Of these 19 respndents, 17 (89%) said that examinees were mre engaged, and the ther tw respndents said that the effect was neutral r variable. Intellectual Disability: Effect f Q-interactive n engagement (N = 45) n effect 58% mre engaged 38% less engaged 0% 4% Cpyright 2013 NCS Pearsn, Inc. All rights reserved. Page 8

9 Learning Disability Of the 79 respndents wh had used Q-interactive with individuals with a diagnsis f learning disability, 39% reprted that presenting stimuli using Q-interactive had an effect n examinee behavir. Of thse wh reprted an effect, 94% said that examinees were mre engaged, and the ther tw respndents said that the effect was neutral r variable. Learning Disability: Effect f Q-interactive n engagement (N = 79) n effect 61% mre engaged 37% less engaged 0% 2% Difficulty Using Q-interactive With Particular Types f Examinees Of the 95 respndents, seven (7%) reprted that they experienced difficulty using Q-interactive with specific types f examinees in the child/adlescent age range. These reprts were distributed acrss several diagnstic categries: ADHD (3) Develpmental Delay (1) Gifted (1) TBI (1 age nt indicated) Obsessive (1 age nt indicated) The larger number f reprts fr examinees with ADHD may partly reflect the fact that the ADHD sample was abut twice as large as the samples fr autism, develpmental delay, r intellectual disability. Hwever, the learning disability sample was abut as large as the ADHD sample, and there were n reprted difficulties in that grup. Cpyright 2013 NCS Pearsn, Inc. All rights reserved. Page 9

10 In additin, there were fur reprts f difficulty using Q-interactive with adults: Cllege age (1) Alzheimer s r dementia (1) Older (2), due t anxiety abut breaking the tablet r ding it wrng, r pssible malingering Additinal Feedback There were many cmments that examinees were mre engaged with the assessment prcess because Q-interactive seemed mre game-like. Mst f these stated a psitive effect, but there were a few reprts f children being distracted by their desire t use the tablet t play games. There were 3 such cmments (4%) regarding examinees ages 5 9 in general, 4 (5%) regarding children with ADHD, and ne each fr autism, develpmental delay, and intellectual disability. Summary In this survey f 95 practitiners wh have administered WISC IV using Q-interactive, there was a high incidence f reprts that Q-interactive either increased examinees engagement and attentin, r had n effect. Increased engagement appeared t be mre frequent fr yunger than lder examinees, with abut 75% f respndents reprting it at ages 5 9 and abut half reprting increased engagement at ages Amng the clinical grups f all ages, increased engagement r attentin was ften reprted, particularly fr ADHD and autism (abut half f respndents in each case). Abut 33% f respndents als reprted this fr thse with develpmental delay, intellectual disability, and learning disability. There were n reprts f decreased engagement r attentin in fur f the five clinical grups; hwever, decreased attentin was reprted by abut 5% f respndents fr the ADHD grup. Cpyright 2013 NCS Pearsn, Inc. All rights reserved. Page 10