1 5 METHODS OF LEARNING IN STATISTICAL EDUCATION: A RANDOMIZED TRIAL OF PUBLIC HEALTH GRADUATE STUDENTS 3 FELICITY BOYD ENDERS, PHD Mayo Clinic, Division of Biosaisics MARIE DIENER-WEST, PHD Johns Hopkins Universiy, Deparmen of Biosaisics ABSTRACT A randomized rial of 65 consening sudens was conduced wihin an inroducory biosaisics course: 69 received eigh small group cooperaive learning sessions; 97 accessed inerne learning sessions; 96 received no inervenion. Effec on examinaion score (95% CI) was assessed by inen-o-rea analysis and by incorporaing repored paricipaion. No difference was found by inen-o-rea analysis. Afer incorporaing repored paricipaion, adjused average improvemen was.7 poins (-.8, 5.) per cooperaive session and. poins (-.4, 5.5) per inerne session afer one examinaion. Afer four examinaions, adjused average improvemen for four sudy sessions was 5.3 poins (0.4, 0.3) per examinaion for cooperaive learning and 8. poins (3.0, 3.) for inerne learning. Consisen paricipaion in acive learning may improve undersanding beyond he radiional classroom. Keywords: Saisics educaion research; Acive learning; Cooperaive learning; Inerne learning; Randomized rial. INTRODUCTION The discipline of saisics provides criical quaniaive ools for public healh researchers and praciioners. Sudens pursuing graduae degrees in public healh mus become familiar wih key conceps in saisical reasoning and knowledge of he appropriae use and inerpreaion of classical biosaisical mehods such as esimaion, hypohesis esing, and mulivariable analysis. In paricular, he widespread availabiliy and accessibiliy of saisical compuing has increased he poenial for public healh professionals o confron saisical analyses in published repors, perform heir own daa analyses, or collaborae wih research eams. Because of heir quaniaive naure, courses covering saisical conceps and mehods may be challenging for sudens from oher fields of sudy. A variey of reasons have been proposed o explain why sudens migh have difficuly in developing inroducory saisical skills and compeencies. Such sudens frequenly harbor longheld anxiey regarding mahemaical courses and radiional didacic eaching mehods may no allow hem o sufficienly overcome such fears (Bradsree, 996). In addiion o hese barriers, sudens are ofen enrolled in muliple courses or concurrenly employed, Saisics Educaion Research Journal, 5(), 5-9, hp://www.sa.auckland.ac.nz/serj Inernaional Associaion for Saisical Educaion (IASE/ISI), May, 006
2 leading o a sressful background environmen (Simpson, 995). Finally, courses in inroducory saisics draw such a variey of sudens from diverse backgrounds and wih differen prior knowledge and innae skills ha i can be exceedingly challenging for insrucors o simulaneously ailor didacic course maerial o mee all suden needs (Simpson, 995). Recen advances in educaional psychology and compuer echnology sugges possible ways o improve sudens concepual undersanding of key saisical conceps. New insrucional mehods may enhance saisical educaion and sudens learning of saisical conceps. One way o ailor saisical educaion is o include acive learning mehodology. Acive learning refers o engaging a suden in an aciviy, as conrased wih a lecure forma or exbook which solely provides he suden wih informaion. A review of he lieraure in saisical educaion reveals ha sudens may learn more readily when maerial is presened hrough suden ineracion or aciviies, as compared o he radiional passive lecuring syle (Bradsree, 996; Garfield, 995a; Garfield, 995b; Love & Greenhouse, 000; Moore, 997). Ideally, his direc inerplay forces sudens o overurn misconcepions, fears, or learning difficulies ha hamper heir abiliy o develop correc saisical inuiion (Garfield, 995a; Garfield, 995b; Love & Greenhouse, 000). Including such mehodologies in he learning process migh help improve sudens undersanding of saisical conceps. By esablishing a hands-on environmen, acive learning may help alleviae difficulies heighened by anxiey relaed o mahemaical conceps. Acive learning can be faciliaed in a number of ways. Cooperaive learning is accomplished when sudens work ogeher on a srucured aciviy in small groups o gain concepual undersanding (Garfield, 993). This can be accomplished during, afer, or insead of a radiional lecure. One mehod is o reinforce conceps and echniques inroduced in a didacic lecure by subsequen small group aciviies faciliaed by a eaching assisan. By working ogeher, sudens no only engage in acive learning, bu derive benefis from heir combined knowledge base. Alhough he majoriy of previous aemps o implemen acive learning wihin saisical classrooms have used a cooperaive learning approach (Gnanadesikan, Scheaffer, Wakins, & Wimer, 997; Kvam, 000; Magel, 998), his migh be difficul o accomplish wih a large class size. Magel (998) used cooperaive learning in a class of 95 sudens and found i required significan advance preparaion o break sudens ino he small groups required and sill have a single insrucor serve as a faciliaor for all he groups. Creaing an inerface wih acive learning using currenly available inerne echnology provides an alernaive approach for improving suden undersanding in large classes wih a didacic course forma. JAVA apples (mini-applicaions) provide a venue for sudens o independenly examine saisical phenomena wihin a conrolled inernebased environmen. The ineracive naure of he apples allows acive learning o ake place on he compuer, i.e., inerne learning. Inerne learning is disinc from hybrid learning (Us, Sommer, Acredolo, Maher, & Mahews, 003; Ward, 004). In a hybrid course, he bulk of he course is online, and in person conac wih sudens is limied, ofen o approximaely an hour per week. By conras, inerne learning acs as an online componen added o a radiional didacic course. Previous sudies have described he use of cooperaive learning (Gnanadesikan e al., 997; Kvam, 000; Magel, 998; Shaughnessy, 977), bu very few sudies have compared cooperaive learning or echnologically-enhanced learning wih he more radiional didacic or lecure-based syle. This research sudy focuses on he implemenaion and evaluaion of he addiion of innovaive insrucional mehods o an exising didacic course sequence in inroducory biosaisics for non-saisicians. The 6
3 7 presen sudy was designed o evaluae cooperaive learning and inerne learning wihin a randomized seing, and o compare he relaive meris of cooperaive and inerne learning o each oher and o a conrol group... STUDY DESIGN. METHODS This sudy was conduced from Sepember hrough December 00 (6 weeks) in he conex of an inroducory biosaisics course ha was a requiremen for sudens in mos Masers and Docoral degree programs a a school of public healh. Sandard course insrucion included 3 hours of lecure and one -hour laboraory each week. The laboraory consised of a srucured review of examples peraining o lecure maerial bu in a smaller group seing ha permied more discussion. The firs half of he course reviewed inroducory conceps such as graphing, summary saisics, exploraory daa analysis, probabiliy conceps and disribuions, and esimaion and hypohesis esing. The second half of he course covered inference for one or wo groups, analysis of variance, and simple linear regression. Learning maerials consised of lecures, accompanying lecure noes, laboraory exercises, problem ses, online self-evaluaion problems, Saa TM (The Saa Corporaion, 00) compuing noes, quizzes, and examinaions. The sudy design was a randomizaion among consening sudens o one of hree groups: cooperaive learning (in person), inerne-based learning (online), and conrol (see Figure for a schema of he sudy design and paricipaion). During he firs week of classes, sudens were offered he opporuniy o paricipae in he sudy and asked o complee an online pre-sudy survey of heir mahemaical and saisical skill and apiude as well as demographic characerisics. All sudens were eligible, bu were enrolled in he sudy only afer providing wrien informed consen. In order o ensure represenaion of all degree programs, he randomizaion was sraified by degree program (Docoral, MPH, oher Masers degree, or Oher). Afer randomizaion, he inervenion phase was iniiaed. Each of he inervenion sessions began afer he inroducion of he relevan conceps in lecure, and followed he same basic framework. Sudens in he cooperaive learning group aended a one hour bi-monhly small group acive learning session faciliaed by a single experienced eaching assisan who did no paricipae in any oher course-relaed aciviies. Many of he acive learning sessions were moivaed by projecs described in Aciviy-Based Saisics by Scheaffer, Gnanadesikan, Wakins, and Wimer (996). A he same ime, sudens in he inerne learning group individually accessed a specially designed inerne learning websie and compleed an inerne-based aciviy ypically focused on saisical conceps illusraed by ineracive JAVA apples. The websie was comprised of apples publicly available on he inerne ha were designed o help sudens learn paricular saisical conceps. For each session, links o hese apples were embedded in a single compuer screen providing shor insrucions and quesions for he sudens. The apples and heir insrucions remained available o sudens hroughou he sudy.
4 8 376 sudens regisered for he course 65 (70%) consened o enroll in he sudy and were randomized (30%) did no consen o paricipae 56 (97%) compleed he pre-sudy survey 69 (6%) Cooperaive Learning Group 00 (38%) Inerne Learning Group 96 (36%) Conrol Group No adminisraive losses 3 adminisraive losses No adminisraive losses 69 sudens offered 8 cooperaive learning sessions 97 sudens offered 8 inerne learning sessions 96 sudens offered no inervenion All sudens compleed Examinaions I-IV 4 (6%) compleed possudy survey 57 (59%) compleed possudy survey 50 (5%) compleed possudy survey Figure. Sudy design and paricipaion The inervenion sessions covered eigh opics deemed inegral o he undersanding of course maerial: ) condiional probabiliy in a able; ) he Binomial and Poisson disribuions; 3) he sampling disribuion of he sample mean; 4) hypohesis esing; 5) confidence inervals; 6) he X disribuion; 7) analysis of variance; and 8) simple linear regression. Assessmens were based on suden performance as measured by four course examinaion scores. The firs course examinaion was adminisered afer he second sudy session; he second course examinaion was adminisered afer he fourh sudy session; he hird course examinaion was adminisered afer he sixh sudy session; and he fourh and final course examinaion was adminisered afer he eighh sudy session. Each
5 9 examinaion focused on maerial since he prior examinaion and included 0 five poin quesions so ha possible scores ranged from 0 o 00 poins... STATISTICAL ANALYSIS The primary goal of he analysis was o invesigae possible associaions beween inervenion and suden performance in he course as measured by course examinaion scores. Three separae linear modeling approaches were used o compare suden performance by sudy group (McCullagh & Nelder, 989). In he firs wo approaches, he four examinaion scores for each suden (0 o 00 poins) were used as a se of four longiudinal oucomes wih an exchangeable covariance srucure; in he hird approach, he oucome was he cumulaive examinaion score (0 o 400 poins). See Appendix A for equaions used in each of he hree approaches. Inen-o-rea models: The firs approach uilized he inen-o-rea principle; in Model, examinaion score was regressed on assigned sudy group. A random effec a he suden level was employed for he repeaed measures srucure resuling from he use of he four examinaion scores for each suden. Three indicaor variables were included o adjus for variabiliy in scores across he four course examinaions (he fourh examinaion served as he reference). Individual repored paricipaion models: The second approach incorporaed sudens repored paricipaion in he sessions. In Model a, paricipaion in he mos recen sudy session in eiher he cooperaive learning group or he inerne learning group or paricipaion in neiher session was used o predic he subsequen examinaion score. Model b used paricipaion in boh of he wo mos recen sessions. Similarly, Models c and d incorporaed paricipaion in he hree mos recen sessions (if available), or he four mos recen sessions (if available), respecively. Since he inervenion paricipaion effecs could vary by examinaion, wo-way ineracion erms beween inervenion paricipaion and examinaion were added o he models shown in Appendix A. A random effec a he suden level was employed for he repeaed measures srucure. Three indicaor variables were included o adjus for variabiliy across he four examinaions. Cumulaive repored paricipaion models: The hird approach accouned for he oal number of sudy sessions aended by each suden in he inervenion groups. Sudens in he conrol group were excluded from Model 3. Since cumulaive paricipaion in sudy sessions was no complee unil he end of he sudy, he oucome for his approach was he sum of he four examinaion scores (he cumulaive examinaion score). This approach esimaed he effec of inervenion on cumulaive examinaion score afer adjusing for he number of sudy sessions in which he suden repored paricipaion. Since only one observaion per suden was required for his analysis, no repeaed measures srucure was necessary. The session paricipaion used in he second and hird modeling approaches was based on self-repor eiher a he ime of compleion of he self-evaluaion problems afer individual sudy sessions or during he pos-sudy survey. Because self-repor was no requesed a he ime of he firs sudy session, he firs session was no included as a separae ime-poin. Each model was subsequenly adjused for baseline facors associaed wih performance which were idenified from he pre-sudy survey (daa no shown). Nonconsening sudens were no included in analyses of examinaion scores, according o he regulaions of our invesigaional review board. However, compleion of he pre-sudy survey was aken as aci consen for ha porion of he sudy among sudens who did no consen o join he whole sudy.
6 0 3. RESULTS 3.. STUDY PARTICIPATION A oal of 376 sudens regisered in he course; 65 (70%) of he sudens consened o enroll in he rial wih 69, 00, and 96 randomized o he cooperaive learning, inerne learning, and conrol groups, respecively. Three sudens randomized o he inerne learning group were excluded from he analysis due o early changes in suden course plans, reducing he oal number o 97. The disribuions of demographic and suden characerisics for boh randomized and non-enrolled sudens are shown in Table. As expeced by randomizaion, all hree groups were fairly comparable wih respec o pre-sudy characerisics, wih no saisically significan differences. In addiion, few differences were found beween sudens who consened o join he sudy and hose who did no enroll bu who sill compleed he pre-sudy survey. Approximaely 49% of he non-enrolled sudens volunarily compleed he pre-sudy survey. The primary difference beween hese wo groups was ha non-enrolled sudens repored greaer levels of concurren employmen. Individual access o he sudy inervenions was no racked in eiher inervenion group, alhough self-repor of inervenion paricipaion was colleced. In he cooperaive learning group, he number of sudens presen a each session was colleced. In he inerne learning group, he Supersas TM sofware was used o rack overall access o he inerne learning websie over ime (SieCaalys, 00). Figure compares he overall paricipaion by session from hese wo mehods. However, since he mehod of racking paricipaion differed by inervenion group, comparisons in Figure can only be made regarding overall paerns of paricipaion, raher han he paricipaion rae, because of differences in scale. Of he 69 sudens randomly assigned o he cooperaive learning group, 45 (65%) aended he firs session on Sepember 3, 00, wo days afer a naional ragedy in he US. Figure. Paricipaion in he cooperaive learning and number of imes he inerne learning websie was accessed, by sudy session
7 Gender Table. Disribuions of demographic and suden characerisics for randomized and non-enrolled sudens Cooperaive No. (%) Inerne No. (%) Conrol No. (%) p Male 0 (30.3) 7 (8.7) 30 (3.6) 9 (35.) 0.9 Female 46 (69.7) 67 (7.3) 65 (68.4) 35 (64.8) Age (58.0) 59 (60.8) 59 (6.5) 9 (53.7) (30.4) 3 (3.0) 9 (30.) 3 (4.6) (7.3) 3 (3.) 6 (6.3) (.9) (4.4) 4 (4.) (.) (.9) Degree MPH 5 (37.9) 36 (38.3) 3 (3.6) 4 (5.9) Oher Masers (33.3) 3 (34.0) 36 (37.9) 9 (35.) 0.99 Docoral (8.) 7 (8.) 7 (7.9) (.) Oher 7 (0.6) 9 (9.6) (.6) 9 (6.7) Credi Hours 5 3 (4.4) 7 (7.) 9 (9.4) 7 (3.0) 6- (.9) (.) 5 (5.) 4 (7.4) (59.4) 5 (53.6) 49 (5.0) 3 (4.6) 9+ 3 (33.3) 36 (37.) 33 (34.4) 0 (37.0) English Naive Language 4 (63.6) 5 (55.3) 60 (63.) 3 (59.3) 0.45 Second Language 4 (36.4) 4 (44.7) 35 (36.8) (40.7) Employmen 0+ hours/week 4 (36.4) 30 (3.9) 4 (43.) 44 (8.5) 0.8 <0 hours/week 4 (63.6) 64 (68.) 54 (56.8) 0 (8.5) Saisical Knowledge (Correc responses of 0) Mahemaical Skill (Correc responses of 5) Desire for a Tuor (Liker scale: 0=definiely no needed o 4=definiely needed) Mean (SD) Mean (SD) Mean (SD) p 4.8 (.84) 4.3 (.90) 3.7 (.3) (.33) 4.37 (.) 4.6 (.3) 4.44 (.0) (.9).5 (0.96).49 (0.89).53 (.) (.07) Toal sudens Saisical significance for he difference beween he randomized groups deermined by Chisquare es. Saisical significance for he difference beween he randomized groups deermined by Analysis of Variance es.
8 3.. STUDENT PREFORMANCE ON EXAMINATIONS The overall mean cumulaive examinaion score was poins (SD: 36.8 poins). There was variabiliy in mean score across he four course examinaions. The overall mean (SD) scores were 89.0 (.8) poins; 8.3 (.7) poins; 8.8 (0.6) poins; and 75.7 (4.) poins for he firs hrough fourh examinaions, respecively. In a previous analysis variables from he pre-sudy survey were used o model cumulaive examinaion score using forward sepwise regression incorporaing wo-way ineracion erms. Younger age, greaer mahemaical apiude (measured on a Liker scale from 0 o 5 based on a weighed scoring of he correc responses o quesions and 3 from Kemeny/Kurz Mah Series, 99, p. 6) and saisical knowledge (measured on a 0 poin scale adaped from Hoffrage, Lindsey, Herwig, & Gigerenzer, 000, and Wulff, Anderson, Brandenhoff, & Gule, 987), working less han 0 hours per week, and suden self-repor of no needing a uor (Liker scale of he repored need for a uor; 0=definiely no, =probably no, =no sure, 3=probably, 4=definiely) were idenified as pre-sudy facors associaed wih high performance. These five covariaes were added in all subsequen models of inervenion effec and performance. Evaluaing he Associaion of Inervenion wih Performance Based on Inen-o- Trea Models (s Approach) No saisically significan differences in performance by sudy group were observed in he unadjused inen-o-rea analyses. Afer adjusing for he five pre-sudy predicors of performance, esimaed mean scores for sudens randomized o cooperaive learning were 0.3 poins below hose of sudens randomized o conrol (95% CI: -3.4,.9); mean scores for sudens in he inerne learning group were 0.0 poins lower han sudens in he conrol group (95% CI: -.8,.8). Evaluaing he Associaion of Inervenion and Paricipaion wih Performance Based On Individual Repored Paricipaion Models (nd Approach) Table shows resuls of he models of suden performance on he four examinaion scores as a funcion of inervenion and repored paricipaion in sessions prior o he examinaions. The resuls sugges increased performance in boh inervenion groups; however, saisically significan increases in performance were only observed a he ime of he fourh examinaion. I should be noed ha models for hree or four consecuive sudy sessions could no be consruced for he firs examinaion because only wo sudy sessions had occurred by he ime of ha examinaion. All models in Table included ineracion erms of inervenion effecs and course examinaions idenified by Wald es resuls and were adjused for he five pre-sudy predicors associaed wih performance. Evaluaing he Associaion of Inervenion and Paricipaion wih Performance Based on Cumulaive Repored Paricipaion Models (3rd Approach) No saisically significan difference in performance (as measured by cumulaive examinaion score) beween he wo inervenion groups were observed afer adjusing for he number of sessions he suden repored aending (3 rd Approach, see Table 3). However, performance increased wih each addiional sudy session in which he suden paricipaed. Each session was associaed wih a. poin average increase (95% CI: 0., 3.9) in cumulaive examinaion score in he adjused model.
9 3 4. DISCUSSION 4.. CONCLUSIONS The goal of his sudy was o invesigae wheher he addiion of acive learning mehods o a didacic inroducory biosaisics course aided suden undersanding of key conceps, as measured by suden performance on course examinaions. The unadjused inen-o-rea analysis revealed no saisically significan differences in performance across he hree sudy groups (cooperaive learning, inerne learning, and conrol). This was likely aribuable o low paricipaion raes in he sudy inervenions; by he hird sudy session, 5% of he sudens in he wo inervenion groups had dropped ou. From commens on he pos-sudy survey, sudens in boh inervenion groups overwhelmingly cied lack of ime as he predominan reason for nonparicipaion. We also compared sudens who did no paricipae afer he second sudy session wih hose who did complee laer inervenion sessions. The only difference found beween paricipans and hose who dropped ou was ha hose compleing laer sudy sessions were enrolled in fewer credi hours. In he presence of significan noncompliance, inen-o-rea analyses may no adequaely reflec rue differences beween groups (Green, 00). Accordingly, alernaive analyic approaches were explored. The second modeling approach, using sudens repored paricipaion, suggesed improved performance for paricipans as compared o nonparicipans and conrols. The benefis of one sudy session were negligible. However, afer four consecuive sudy sessions a he ime of he fourh 00- poin examinaion, cooperaive learning paricipans scored an average of 5.3 poins higher (95% CI: 0.4, 0.3), and inerne learning paricipans scored an average of 8. poins higher (95% CI: 3.0, 3.), han nonparicipans or conrols, afer adjusing for he five pre-sudy facors associaed wih performance. The upper limi of he confidence inerval reflecs an improvemen in undersanding corresponding o perhaps wo addiional correc responses ou of 0 examinaion quesions. Under he 3 rd modeling approach, each addiional inervenion session in which he suden paricipaed was associaed wih a. poin increase in cumulaive examinaion score (on a 400 poin scale) (95% CI: 0., 3.9) in he adjused model. When his effec is muliplied by he number of available inervenion sessions, his increased performance may be subsanial. 4.. STRENGTHS AND LIMITATIONS A limiaion in he design of his sudy ha could inroduce bias was he requiremen of exra work beyond he regular course maerial for he wo inervenion groups. One effec was decreased paricipaion over ime, which is associaed wih wo poenial biases; possibly sudens who coninued o paricipae were more dedicaed and hus more likely o work hard, or sudens who coninued o paricipae did so because he inervenion was more helpful o hem han o hose who dropped ou. The effecs of hese poenial biases may be mos clearly observed in Table. By he ime of he fourh examinaion, hose who were sill paricipaing in he sudy inervenions had likely paricipaed in all four mos recen sudy sessions; consequenly, very lile variaion is observed in he increase in esimaed performance from he models reflecing a leas one sudy session versus he models reflecing a leas four sudy sessions. Conversely, i is possible ha sudens paricipaing in he wo inervenion groups simply spen more ime working wih saisical conceps, and ha addiional ime of any form would have led o he same improved performance.
10 4 Table. Linear models for sudens subsequen examinaion scores by he number of prior sudy sessions aended ( nd Approach, ineracion model) s Examinaion nd Examinaion 3 rd Examinaion 4 h Examinaion Inervenion Group Number of consecuive sessions aended Change in Examinaion Score (95% CI) Unadjused Esimae Adjused Esimae* Cooperaive session.8 (-.8, 7.3).7 (-.8, 5.) Group vs. No Inervenion sessions 4.8 (-0.3, 9.9).6 (-.6, 6.7) Inerne Group session 3.4 (-0.0, 6.9). (-.4, 5.5) vs. No Inervenion sessions 3. (-0.4, 6.6).7 (-.7, 5.) session 3. (-., 7.4) 3.0 (-0.6, 6.6) Cooperaive Group vs. No Inervenion Inerne Group vs. No Inervenion Cooperaive Group vs. No Inervenion Inerne Group vs. No Inervenion Cooperaive Group vs. No Inervenion Inerne Group vs. No Inervenion sessions 3.4 (-., 8.0) 3. (-0.8, 6.9) 3 sessions 3.5 (-.0, 8.0).9 (-.0, 6.8) 4 sessions 4. (-0.4, 8.6).9 (-., 7.) session -0.7 (-4.0,.7) -0.9 (-3.8,.0) sessions -0.4 (-4., 3.3) -0.9 (-4.3,.4) 3 sessions -0. (-4.0, 3.6) -0.8 (-4.,.6) 4 sessions -0.4 (-4.5, 3.7) -0.9 (-4.7,.8) session 0.8 (-3.4, 5.0). (-.6, 5.0) sessions.3 (-3., 5.9).7 (-.4, 5.9) 3 sessions.6 (-3., 6.).8 (-.3, 5.9) 4 sessions.6 (-3.0, 6.3).7 (-.4, 5.8) session.9 (-.7, 5.4).7 (-.7, 5.) sessions.3 (-.4, 6.0). (-.5, 5.7) 3 sessions.4 (-.3, 6.3). (-.4, 5.8) 4 sessions.6 (-.3, 6.4). (-.6, 5.7) session 3.6 (-.7, 0.0) 4.5 (-0.4, 9.4) sessions 3.6 (-.9, 0.0) 4.6 (-0.5, 9.6) 3 sessions 4. (-.3, 0.7) 5. (0., 0.3) 4 sessions 4.3 (-., 0.7) 5.3 (0.4, 0.3) session 8. (.9, 3.3) 7. (.3,.9) sessions 8.8 (3., 4.4) 7.7 (.6,.8) 3 sessions 8.9 (3.3, 4.5) 7.9 (.8,.9) 4 sessions 9.0 (3.4, 4.6) 8. (3.0, 3.) *Adjused models include pre-sudy facors associaed wih performance.
11 5 Table 3. Linear models for sudens cumulaive examinaion score by he number of prior sudy sessions aended (3 rd Approach) Comparison Cooperaive Group vs. Inerne Group Each addiional sudy session Change in Cumulaive Examinaion Score (95% CI) Unadjused Esimae Adjused Esimae* 0.4 (-.4,.) 4.7 (-5.3, 4.7).9 (-0.3, 4.0). (0., 3.9) *Adjused models include pre-sudy facors associaed wih performance. Anoher poenial bias is he Hawhorne effec. Individuals who are aware ha hey are being sudied may behave differenly han hey oherwise would (Franke & Kaul, 978). The average examinaion score was.5 poins higher among hose randomized o he conrol group (95% CI: -.0, 5.0), han among hose who did no consen o enroll in he sudy. However, i was no possible o adjus his difference for he oher performance predicors, since no all non-enrolled sudens compleed he pre-sudy survey. The key srengh of his sudy design was he randomizaion of sudens o differen sudy groups. Uilizing a longiudinal framework allowed comparison of he effecs of he inervenions on performance over ime. The large iniial sample size of he rial provided he power necessary for deecing differences in performance by inervenion and paricipaion in our analyses. The inclusion and comparison of wo differen ypes of acive learning (cooperaive and inerne) was anoher key componen of his sudy. Wih he adven of disance educaion, echnologically enhanced learning, such as ineracive online apples, affords a new way o offer acive learning wihin a disance-friendly forma. In he inerne learning group, no supervision was required and ye improved performance was observed ha was comparable o ha of he cooperaive learning group, wih far less inensive invesmen of insrucor ime. The websie for he inerne learning group required lile resources oher han providing an inroducory inerface and framework since publicly available apples were used. Developmen of new apples would have iniial coss bu require lile mainenance and resources over ime IMPLICATIONS FOR FUTURE INSTRUCTION AND RESEARCH Conducing a randomized rial wihin he framework of a large class such as his was exremely challenging. Despie he large number of sudens who iniially chose o join he sudy, overall paricipaion was low. Given sudens hecic schedules and demands on heir ime, paricipaion in any opional educaional research projec will be limied. Increased paricipaion is needed in fuure invesigaions. One opion is o incorporae inervenion maerials as a required course componen. A comparaive sudy could be made of consecuive offerings of a course in which he second offering inroduces required new maerial (such as acive learning sraegies) bu oherwise he course remains he same. This approach was used by Smih (998) o evaluae small group
12 6 cooperaive learning projecs. Such a sudy design assumes no differences in suden composiion and requires he same assessmen ools over ime. However, he inclusion of such a comparison group is a criical par of evaluaion of new saisical educaion mehods. Anoher consideraion in he fuure evaluaion of saisical educaion echniques is he specificaion of he amoun of course conen under evaluaion. In his sudy, he inervenion sessions covered a proporionaely small amoun of he course conen and ime relaive o he oher ime requiremens of he course. Consequenly, only small changes in overall performance could be expeced and heir deecion would require large sample sizes. Our choice of examinaion scores as he primary oucome variable resuled in high variabiliy. Alhough we inended o uilize specific self-evaluaion problems o evaluae he individual sudy sessions, hese problems were no mandaory and hus were no compleed by he majoriy of sudens. Fuure invesigaions warran he incorporaion of a required assessmen ool ha arges specific conceps emphasized hrough he inervenion. One way o address his concern is he use of a hybrid course for he online porion of he inervenion; however, ehical issues arise surrounding randomizing sudens o differen ypes of courses. These findings sugges ha sudens may be aided by learning inroducory biosaisics maerial via ineracive aciviies especially if such aciviies are a required course componen and offered hroughou he erm or semeser. Our findings of an associaion of coninued improvemen in performance wih compleion of addiional acive learning sessions in he hird approach is paricularly encouraging. Cooperaive learning aciviies and perinen echnological aids may boh be helpful addiions o onsie saisical educaion by eiher enhancing learning and/or reducing anxiey relaed o mahemaical conceps. Fuure research and evaluaion is needed o elucidae hese relaionships. In addiion, research on online acive learning mehodologies is also required in he area of disance educaion. Coninued developmen and evaluaion of saisical eaching mehodologies are criical and imely. Increasing numbers of public healh professionals are seeking skills in quaniaive mehods and are faced wih he challenge of masering knowledge of appropriae saisical echniques and applicaions. The widespread availabiliy of compuer echnology, boh wihin and ouside he classroom, provides an unparalleled environmen for innovaion in saisical educaion o maximize he poenial for learning. REFERENCES Bradsree, T. E. (996). Teaching inroducory saisics courses so ha nonsaisicians experience saisical reasoning. The American Saisician, 50(), Franke, R. H., & Kaul, J. D. (978). The Hawhorne experimens: Firs saisical inerpreaion. American Sociological Review, 43(5), Garfield, J. (993). Teaching saisics using small-group cooperaive learning. Journal of Saisics Educaion, (). [Online: hp://www.amsa.org/publicaions/jse] Garfield, J. B. (995a). Responden: How should we be eaching saisics? The American Saisician, 49(), 8-0. Garfield, J. (995b). How sudens learn saisics. Inernaional Saisical Review, 63(), Gnanandesikan, M., Scheaffer, R. L., Wakins, A.E., & Wimer, J.A. (997). An aciviybased saisics course. Journal of Saisics Educaion, 5(). [Online: hp://www.amsa.org/publicaions/jse]
13 7 Green, S. B. (00). Design of randomized rials. Epidemiologic Reviews, 4(), 4-. Hoffrage, U., Lindsey, S., Herwig, R., & Gigerenzer, G. (000). Communicaing saisical informaion. Science, 90(5500), 6-6. The Kemeny/Kurz Mah Series. (99). Algebra workbook. USA: True Basic, Inc. Kvam, P. H. (000). The effec of acive learning mehods on suden reenion in engineering saisics. The American Saisician, 54(), Love, M. C., & Greenhouse, J. B. (000). Applying cogniive heory o saisics insrucion. The American Saisician, 54(3), Magel, R. C. (998). Using cooperaive learning in a large inroducory saisics class. Journal of Saisics Educaion, 6(3). [Online: hp://www.amsa.org/publicaions/jse] McCullagh, P., & Nelder, J. A. (989). Generalized linear models. London: Chapman & Hall. Moore, D. S. (997). New pedagogy and new conen: The case of saisics. Inernaional Saisical Review, 65(), Scheaffer, R. L., Gnanadesikan, M., Wakins, A., & Wimer, J. A. (996). Aciviy-based saisics: Insrucor resources. New York: Springer-Verlag New York, Inc. Shaughnessy, J. M. (977). Misconcepions of probabiliy: An experimen wih a smallgroup, aciviy-based, model building approach o inroducory probabiliy a he college level. Educaional Sudies in Mahemaics, 8, Simpson, J. M. (995). Teaching saisics o non-specialiss. Saisics in Medicine, 4, SieCaalys (formerly SuperSa). (00). Nashville, TN: Omniure, Inc. [Online: hp://www.omniure.com/] Smih, G. (998). Learning saisics by doing saisics. Journal of Saisics Educaion, 6(3). [Online: The Saa Corporaion. (00). Inercooled Saa version 7, Houson Saion, TX. Us, J., Sommer, B., Acredolo, C., Maher, M. W., & Mahews, H. R. (003). A sudy comparing radiional and hybrid inerne-based insrucion in inroducory saisics classes. Journal of Saisics Educaion, (3). [Online: Ward, B. (004). The bes of boh worlds: A hybrid saisics course. Journal of Saisics Educaion, (3). [Online: Wulff, H. R., Anderson, B., Brandenhoff, P., & Gule, F. (987). Wha do docors know abou saisics? Saisics in Medicine, 6, 3-0. FELICITY BOYD ENDERS, PHD, MPH Division of Biosaisics Division of Healh Sciences Research The Mayo Clinic 00 Firs Sree, SW Rocheser, MN 55905, USA
14 8 APPENDIX A: VARIABLE DEFINITIONS AND MODELS USED FOR STATISTICAL ANALYSIS Variables used in he models: Y is he vecor of examinaion scores for he four course examinaions (Exam I) hrough (Exam III) are indicaor variables for he firs hree course examinaions (Coop) and (Inerne) are indicaor variables for randomizaion o he wo sudy inervenion groups (Number) is he number of sudy sessions for which he suden repored paricipaion Y cum is he sum of all four course examinaion scores (cumulaive examinaion score) The following ime-defined variables are each vecors of lengh seven, for which ime is defined as =, 3, 4, 8, represening sudy sessions wo hrough eigh. (Coop) and (Inerne) are he vecors of indicaor variables for repored paricipaion in he sudy inervenion groups across sudy session wo hrough eigh (conrol group is he reference group) (Coop) - and (Inerne) - are he vecors of indicaor variables for repored paricipaion in he wo sudy inervenion groups a he ime of he prior sudy session (conrol group is he reference group) (Coop) - and (Inerne) - are he vecors of indicaor variables for repored paricipaion in he wo sudy inervenion groups a he ime of he second prior sudy session (wih I=0 for ) (conrol group is he reference group) (Coop) -3 and ((Inerne) -3 are he vecors of indicaor variables for repored paricipaion in he wo sudy inervenion groups a he ime of he hird prior sudy session (wih I=0 for 3) (conrol group is he reference group) Y is he vecor of examinaion scores for he firs course examinaion afer he curren sudy session (Exam I) is he vecor of indicaor variables ha he course examinaion following he curren sudy session (a ime ) was he firs course examinaion [I(Exam = Exam I)] (Exam II) is he vecor of indicaor variables ha he course examinaion following he curren sudy session (a ime ) was he second course examinaion [I(Exam = Exam II)] (Exam III) is he vecor of indicaor variables ha he course examinaion following he curren sudy session (a ime ) was he hird course examinaion [I(Exam = Exam III)] : Inen-o-rea model Model. E Y = α + β Coop + β [ ] ( ) ( Inerne) + γ ( ExamI) + γ ( ExamII) + γ ( ExamIII) 0 inra suden iner suden. Individual repored paricipaion models Modela. E Y = α + β Coop + β Inerne [ ] ( ) ( ) + γ ( ExamI ) + γ ( ExamII ) + γ ( ExamIII ) 0 inra suden+ ε iner suden 3 3
15 9 Model b. [ ] = α 0 + β( Coop) + β ( Inerne) + β 3 ( Coop) + β 4 ( Inerne) + γ ( Exam I) + γ ( Exam II) + γ ( Exam III) E Y Modelc. E Y inra suden iner suden [ ] = α0 + β( Coop) + β ( Inerne) + β3( Coop) + β4 ( Inerne) + β ( Coop) + β ( Inerne) + γ ( ExamI) + γ ( ExamII) + γ ( ExamIII) 5 inra suden 6 iner suden 3 3 Model d. [ ] = α 0 + β( Coop) + β ( Inerne) + β3 ( Coop) + β 4 ( Inerne) + β 5 ( Coop) + β 6 ( Inerne) + β 7 ( Coop) 3 + β8 ( Inerne) 3 + γ ( Exam I) + γ ( Exam II) + γ ( Exam III) E Y inra suden iner suden 3 3. Cumulaive repored paricipaion models E Y = α + β Coop + β Number [ ] ( ) ( ) Cum 0 iner suden