How To Improve Your Online Learning

Transcription

1 Improving Course Completion and Student Retention Adapt Courseware Backgrounder and Results for the Academic Year Overview and Summary of Results Adapt Courseware undertook a series of academic validation and comparative outcome tests in the academic year, primarily in cooperation with several community colleges. These studies indicate that Adapt Courseware provides effective online learning, with gains in student course completion averaging 16% (at a 92% confidence level) and term-over-term retention averaging 18% (at a 97% confidence level). Students and instructors report satisfaction and benefits with using Adapt Courseware over conventional online methods. The results presented here provide, in the aggregate, substantial evidence that Adapt Courseware has successfully implemented the ambitious pedagogic and motivational design approaches selected, which themselves have substantial empirical support. A Better Online Learning Experience Online education is a great success story yet until very recently most online learning consisted of taking the conventional methods and materials of the classroom and simply placing them online resulting in the electronic version of correspondence courses. The best available multimedia, interactive, and analytic technologies have been underutilized. Online learning has been a great success but its future can be even brighter. In 2010, following an in-depth assessment of the state of the art in Key Design Principles online learning and a broad survey of empirically-supported theories of learning and learner motivation, Adapt Courseware adopted a set of Effective Multimedia five key design principles to guide development of our learning environment and curriculum materials. The goal was to develop the Mastery Learning best available online learning experience one that will produce Optimal Challenge measurable gains in learning outcomes, student satisfaction, course completion rates, and student retention. In addition to the best design Student Choice principles, the company committed to a large development budget for Social Learning each course, to support the best results. The courses that are the subject of this report each had a multimedia development budget of between $850,000 and $1,500,000, in addition to a multi-million dollar investment in a learning and analytics-reporting platform to deliver that multimedia content Adapt Courseware, LLC

2 The pre-existing research on the principles underlying the Adapt Courseware design is an independent source of validation for our approach. Reviewing this literature is not a substitute for direct, comparative validation testing, which is described later, but does supplement this research and compensate for the necessarily limited number of direct studies available to date. In that sense, the direct validation studies provide confirmation that the intended design approaches were effective and well implemented. EFFECTIVE MULTIMEDIA: In designing the Adapt Courseware multimedia strategy, we were guided by the outstanding work and analysis of Richard E. Mayer 1. Along with his colleagues, Mayer set out to understand the cognitive processes by which people construct meaningful learning outcomes from words and pictures. In his book Multimedia Learning, Mayer tells us people learn more deeply from words and pictures than from words alone. Mayer presents the results of meta-studies on effective multimedia instructional design and draws as his conclusions a set of principles of multimedia design. We selected the Mayer design principles most relevant to Adapt Courseware s overall design concept. These are listed in Statistical Method A commonly used, simple measure of learning effectiveness is effect size, which we will use in presenting various results. Effect size is defined as the difference in means between the experimental group and a control group, divided by the standard deviation of the control group: Effect Size =!!!!!!! That is, effect size is the difference between experimental and control groups in units of standard deviations. In terms of learning outcomes, a standard deviation is often comparable to about a letter grade difference. the table below. As with all the design principles presented here, there is a substantial body of preexisting research supporting their efficacy in learning and learner motivation Adapt Courseware, LLC 2

3 For each Mayer design principle selected the meta-analysis effect size and number of confirming research studies analyzed is shown below, along with the related Adapt Courseware design features. Selected Mayer Multimedia Effect Size Research Studies Adapt Courseware Design Principles Confirming Related Features Coherence Principle: People learn better when extraneous words, pictures, and sounds are excluded of 14 On-screen text limited to essential terms, carefully isolated. Speaker on-screen time limited. Redundancy Principle: People learn better from graphics and narration that from graphics, narration, and on-screen text of 5 On-screen text limited to essential terms. Segmenting Principle: People learn better when a multimedia lesson is presented in userpaced segments of 3 Short 3-5 minute video presentations and many short learning topics. Multimedia Principle: People learn better from words and pictures than from words alone of 11 Key ideas presented through rich narrated animations and graphics. Personalization Principle: People learn better from multimedia lessons when words are in conversational style of 11 Video scripting and hosts are informal and personable. Meyer s meta-analysis of existing research show strong support for his design principles, with good effect sizes and near-universal consistency across the research he reviewed. In Adapt Courseware these key ideas have been designed for maximum effect, as described Adapt Courseware, LLC 3

4 MASTERY LEARNING: Eminent University of Chicago learning theorist Benjamin Bloom, best known for his taxonomy of learning domains, set out in the early 1980s to answer the question, How effective can instruction be? Bloom published his findings in the seminal paper The Two Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring. 2 Bloom concluded that the most effective known approach to instruction consisted of two key elements: Mastery Learning: Each individual student stays on topic until that topic is mastered, and only then moves on. Contrast this with the traditional classroom and online approach of a fixed calendar of fixed assignments. Expert Tutoring: Learners received individual one-on-one tutoring from an expert teacher. The teacher would spot errors, answer questions, and clarify misconceptions as the student worked, one-to-one. Bloom found that these methods dramatically improved outcomes: Most striking were the differences in final achievement measures Using the standard deviation (sigma) of the control (conventional) class, it was typically found that the average student was about two standard deviations above the average of the control class The average students under mastery learning [only] was about one standard deviation above the average of the control class. Bloom found that attitudes as well as learning outcomes improved under a mastery learning approach: There were corresponding changes in students time on task and students attitudes and interests (least positive under conventional instruction and most positive under tutoring). Interestingly, Bloom additionally found that by using the best instructional methods, prior student performance was no longer highly predictive of current performance: There were great reductions in the relations between prior measures (aptitude or achievement) and the summative achievement measures. Typically, the aptitude-achievement correlations changed from under conventional [instruction] to under mastery learning and under tutoring. As Bloom notes, however, one-on-one tutoring is not a practical or affordable approach in most educational settings. The 2 sigma problem, then, is how to use the insights gained to improve practical, scalable, affordable methods of instruction: If the research on the 2 sigma problem yields practical methods (methods that the average teacher or school faculty can learn in a brief period of time and use with little more cost or time than conventional instruction), it would be an educational contribution of the greatest magnitude. It would change popular notions about human potential and would have significant effects on what the schools can and should do with the educational years each society requires of its young people Adapt Courseware, LLC 4

5 These insights are central to the design of Adapt Courseware. The problem posed in 1984 can be answered today, using multimedia networked computing devices with adaptive technology. Adapt Courseware curriculum is developed with hundreds of discrete, minute learning topics, each with an associated real-time mastery score, which provides immediate feedback to the student, instructor, and system as to student knowledge. Formative assessment is provided as long as needed for each student to reach mastery using adaptive stack technology that optimizes the learning and degree of difficulty for each student 3. An interactive hint engine and on-demand video instruction provides automated tutoring. Most importantly, instructor time is made available to respond one-on-one to students as needed, since repetitive instructional tasks are handled by the learning system. Selected Bloom Achievement Variables Effect Size Adapt Courseware Related Features Tutorial [mastery] instruction 2.0 Mastery based instruction, mastery meters, instructor role, hint engine Feedback-corrective 1.2 Adaptive stack, hint engine Student Time on Task 1.0 Mastery-based instruction Mastery Learning (Anania & Burke studies) 1.0 Mastery-based instruction Socio-economic status (for contrast) 0.25 It s remarkable how prescient Bloom s work was he clearly foresaw the opportunity for multimedia-based computer instruction to not only improve immediate academic performance, but student perceptions about learning and themselves as well: It is hoped that the more effective computer courses will also have positive effects on such affective characteristics as academic self-concept, interest in the subject, and desire to learn further As will be described below, we have seen evidence for these affective benefits Adapt Courseware, LLC 5

6 OPTIMAL CHALLENGE AND SOCIAL LEARNING: In addition to well-supported pedagogic features, Adapt Courseware design includes a set of motivational-support features. These keep students with varied prior learnings, metacognitive skills, and learning styles all engaged and moving forward. The first and most essential motivational design concept used is optimal challenge. The optimal challenge concept is grounded in the work of James Paul Gee on learning in video gaming contexts. In his essential text What Video Games Have to Teach Us About Learning and Literacy 4, Gee discusses the key insights to be gained from the evident effectiveness of video games in learning (that is, learning how to play the game). These insights have lead to a range of experiments in gamification placing core academic content in an explicit game environment. With Adapt Courseware, we chose not to do that instead, we focus on the underlying design principles that lead to game engagement, and use them directly. Selected principles and related Adapt Courseware features are listed here: Selected Gee Learning Principles Adapt Courseware Related Features Active, Critical Learning Principle: All aspects of the learning environment are set up to encourage active and critical, not passive, learning. Adaptive Learning Stack Psychosocial Moratorium Principle: Learners can take risks in a space where real-world consequences are lowered. Mastery meter allows only positive increments no penalties for incorrect answers Achievement Principle: For learners of all levels of skill there are intrinsic rewards from the beginning, customized to each learner s level, effort, and growing mastery, and signaling the learner s ongoing achievements. Adaptive stack provides challenge and mastery meter increments for all levels of learning Practice Principle: Learners get lots and lots of practice in a context where the practice is not boring. They spend lots of time on task. Students typically complete multimedia learning interactives per course 2013 Adapt Courseware, LLC 6

7 Regime of Competence Principle: The learner gets ample opportunity to operate within, but at the outer edge of, his or her resources, so that at those points things are felt as challenging but not undoable. Adaptive stack adjusts to each learner s demonstrated knowledge, supporting optimal challenge Multiple Routes Principle: There are multiple ways to make progress or move ahead. This allows learners to make choices, rely on their own strengths and styles of learning and problems solving, and explore alternative styles. Content is provided in three differentiated modes: instructional video, text, and learning interactives. Students can use varied learning strategies to engage with these modes. Affinity Group Principle: Learners constitute and affinity group ; that is, a group that is bonded primarily through shared endeavors, goals, and practices. Social media functions allow students to create profiles and form open and closed study groups. STUDENT CHOICE: People love to learn, but many students hate school. Why? Lack of choice and autonomy is a key reason. Edward Deci and Richard Ryan of the University of Rochester have done some of the best, and increasingly well-known, work on choice and motivation, under the rubric Self- Determination Theory (SDT) 5. They tell us Conditions supporting the individual s experience of autonomy, competence, and relatedness are argued to foster the most volitional and high quality forms of motivation and engagement for activities, including enhanced performance, persistence, and creativity. Specifically, through autonomy, or self-controlled behavior, students can play a more active role in their learning experience as decision makers and initiators the architects of their own educational paths. When they have the opportunity to make their own choices, they are more motivated and engaged. Deci s and Ryan s research concludes that, even in highly constrained situations, where choice is bounded, people still like choice where possible, and choice supports motivation. Using Adapt Courseware, students are free to determine the order, pace, schedule, and content modes (watch, read, practice) that best fit their individual learning style. Having the freedom to choose increases and supports motivation, allowing students to learn in a way that is more meaningful to them Adapt Courseware, LLC 7

8 Validation Testing, Learning Outcomes, and Success Rates Each Adapt Courseware course is benchmarked against nationally recognized and accepted standards for learning. The most common benchmarks are the College Board s College Level Examination Program (CLEP), the Advanced Placement exams (AP), the Defense Activities for Non-Traditional Education Support (DANTES) Subject Standardized Tests (DSST), and for precredit foundational work, the Common Core standards and the College Board s Accuplacer exam. Before use with colleges, Adapt Courseware tests each course by having a set of paid student testers (typically community college and high school students) complete courses and then take the relevant external exam. The student testing results are here, along with the publically available DANTES pass rate for all military personal as a comparison: Test Course Sample Mean Standard Pass Adapt DANTES Period Size (N) Score Deviation Score Courseware Pass Pass Rate Rate Summer 2012 Introduction to Psychology % 52% Fall 2012 Introduction to Psychology Low Prior GPA test group % 52% Summer 2013 Introduction to Sociology % 49% Summer 2013 Organizational Behavior % 42% Summer 2013 Financial Accounting (prior to revision) % 20% With the exception of Financial Accounting, 100% of student testers taking Adapt Courseware courses passed the relevant exam for credit, indicating highly reliable learning outcomes Adapt Courseware, LLC 8

9 In the case of Financial Accounting, on the first test, only 50% of students passed for credit. Although significantly better than the DANTES pass rate of only 20%, this does not meet our standards. The Financial Accounting course has been revised to add more case study and review activities and is in a second validation test now, to be completed in early January In addition to validation testing, a set of comparative use studies across several institutions was conducted in the academic year. The following table lists all studies conducted with not-for-profit institutions in this academic year: Southwest Virginia Community College (SWCC), Central Piedmont Community College (CPCC), Daytona State College (Daytona) and LeTourneau University (LeT). Results for for-profit institutions are generally not available for release. All grade percentages are on the denominator of students present in the section at the academic add-drop date. College Term Same Instructor / Exams? Sample Size (N) A-B Grade Rate A-C (Success) Grade Rate SWCC Adapt Fall N 16 56% 69% SWCC - Control 34 26% 53% SWCC - Adapt Spring N 22 55% 55% SWCC - Control 36 33% 53% CPCC - Adapt Spring Y 25 68% 80% CPCC - Control 21 76% 86% Daytona - Adapt Spring N 23 87% 87% Daytona - Control 37 62% 70% LeT - Adapt Spring Y 28 68% 75% LeT - Control 29 38% 69% Weighted Average Adapt % 74% Weighted Average - Control % 64% 2013 Adapt Courseware, LLC 9

10 The difference in sample success rates (74% as compared to 64%) was found to have a 92% statistical probability of indicating a real difference in population success rates. This is a gain of 10 percentage points, or 16%. In the studies there was also the possibility for systematic biases. In most cases the Adapt Courseware sections opened for registration much closer to the term start than the comparison section. Since late-registering students are well known to perform more poorly than early-registering students, this provided a bias against the Adapt Courseware results. On the other hand, use of Adapt Courseware requires access to broadband Internet access. This requirement could have provided a bias in favor of the Adapt Courseware results. These two systematic effects can be expected to have cancelled to some degree. In addition to grade comparisons, the two comparisons with the same instructor included common comparative exams. At CPCC, the Adapt Courseware section final exam average was 77.7, compared to 70.1 for the control section. LeTourneau conducted a pre- and post- test of learning outcomes and found a scaled improvement for Adapt Courseware students and a scaled improvement for students in a traditional textbook section. This difference was found by LeTourneau to be not statistically significant by t-test. A reasonable summary of the evidence to date on learning outcomes is that Adapt Courseware produces good, consistent learning outcomes with a 10-20% improvement in student success rate, defined as letter grades A-C. Statistical Method A full description of the statistical method used for confidence level would be too lengthy here to summarize we used a Welsh s t-test adapted to proportions between samples, with test statistic z! = p! p!!p (1 p )! 1 N! + 1 N! and p is a pooled estimate of common population proportion. This test is valid when the underlying distributions are normal, when N! p! (1 p! ) 10, and when the sample size is no more than 5% of the population size Adapt Courseware, LLC 10

11 Student Retention Rates In addition to learning outcomes, we collected data on student term-over-term retention. For the Fall of 2012, retained students were defined as any students registering for any course at the same college in the Spring of For Spring of 2013, retained students were any students registering for any course in the same college in the Summer of 2013 or the Fall of Retention data is not available for LeTourneau University. College Adapt Courseware Section Control Section SWCC Fall 12 75% 59% SWCC Spring 13 64% 64% CPCC Spring 13 68% 57% Daytona Spring 13 83% 62% Weighted Average 72% 61% Standard Deviation 8% 3% Effect Size 3.8 The difference in sample success rates (72% as compared to 61%) was found to have a 97% statistical probability of indicating a real difference in population success rates. This is a gain of 11 percentage points, or 18%. We can summarize these results as a consistent, meaningful increase in student retention when using Adapt Courseware. This effect is striking, considering that in each case Adapt Courseware comprised only one of the courses each student took, out of a typical load of 3-5 courses. Our hypothesis at this point is that a successful experience with a high quality masterylearning oriented course can increase student self-efficacy the student s perception of themselves as effective learners. A quote by Benjamin Bloom from above may bear repeating in this context: It is hoped that the more effective computer courses will also have positive effects on such affective characteristics as academic self-concept, interest in the subject, and desire to learn further We tentatively conclude that we are seeing this precise effect Adapt Courseware, LLC 11

12 Student Satisfaction and Feedback At community colleges using Adapt Courseware, we surveyed students on whether they would recommend the course to others, and solicited any comments they wished to make. Participation was voluntary and was incentivized with at gift card. Not all students participated. College Would recommend this course Adapt Would recommend this course - Control SWCC Fall 89% 87% SWCC - Spring 91% 86% CPCC 92% 100% Daytona 89% 89% Weighted Average 90% 89% This data seems to indicate general satisfaction with both Adapt Courseware and traditional online methods, among those students voluntarily participating in the survey. Some comments from Students taking Adapt Courseware are: I liked the different methods of answering the questions. The material was much easier to remember than when answering one basic multiple choice question after another. - SWCC student. I am an auditory learner and the audio that comes with the videos was of tremendous help. CPCC student. I think this type of teaching concept should be used in a lot of other classes. It was very user friendly and easy to understand. It made learning fun. The videos were helpful but I used them 50% of the course. I would recommend this type of learning to others. CPCC student. We can conclude that student acceptance of Adapt Courseware appears to be good, as it was for the control sections Adapt Courseware, LLC 12

13 Instructor and Administrator Feedback In a mastery-based instructional framework, the instructor role changes to more one-on-one concept clarification and enrichment, support, and coaching. We sometimes refer to this role as all office hours all the time. Not all instructors are open to the kind of change required to produce the improved results described here, but many are. Here are a few descriptive quotes from instructors and administrators using Adapt Courseware. They help convey the experiences and reactions of change-oriented faculty. I liked teaching online courses before this course. I like teaching online courses more after teaching this course. Senior Professor, Daytona State College Compared to the textbook I ve used, the information is very consistent. There s a lot of information within the sections. I still incorporate all of my own quizzes and all of my own assignments. I haven t changed any of that it goes along very much with the material presented, so I still use all of my stuff that I ve always used as far as my assignments go. Last semester I taught two classes I taught one with Adapt and one the way that I traditionally taught. I really liked using Adapt. I liked that it incorporates student engagement within it and that seemed to help the results that we got. My students taking the Adapt course tended to get better grades than students taking the course the traditional way, and I was using all the same quizzes and assignments and everything. When I had the two classes, the one using Adapt and the one traditional class my students using the Adapt program did better on the quizzes. More interestingly, I also have discussion board questions, and just qualitatively the quality of their responses was much better in the Adapt Courseware than in the traditional approach. Again, that s kind of a subjective measure but I felt like they really understood and grasped those concepts better after using the Adapt Courseware. I ve seen good results and had some good feedback from students as well. Personally, I prefer teaching in front of a classroom because you have that interaction. Using Adapt Courseware has seemed to help bring that online section kind of closer to that real classroom feel. Instructor - Psychology, CPCC. In comparison to courses that I have taught in the past, this format allowed me to have more one-onone personal interaction with the students. Instructor, SWCC When we first looked at the Adapt product I was so excited. I don t think anyone in education could look at the demo and not be excited. It really touches on, it really respects every modality that a learner may have. The good thing about that is it s not so cutting edge, it s not so high-tech that it alienates people that like a more traditional learning style. It respects every modality it gives the auditory learner the videos to watch and to listen to. For those who like to read the text that s also an option. The interactive activities give you the hands-on, and then with the social media platforms it just opens up every door and it really is such an amazing example of every best practice in distance learning in a very achievable and realistic way. Director of Distance & Distributed Learning and Blackboard Administrator, SWCC Adapt Courseware, LLC 13

14 Ongoing Research At this time, the primary research focus at Adapt Courseware is on validation studies at scale with groups of 500 or more student users. In addition to the general learning and retention benefits seen in studies to date, we believe that our learning environment may be well suited to demonstrations of excellent outcomes at large scale, a topic of interest due to the financial pressures colleges are experiencing. The Bill and Melinda Gates Foundation has taken an interest in Adaptive Learning as a path to improved access and outcomes in education. In March 2013 the Gates Foundation funded research report Learning to Adapt: A Case for Accelerating Adaptive Learning in Higher Education was released by Education Growth Advisors 6, in which Adapt Courseware is prominently described. Subsequently, North Carolina State University was awarded a $100,000 grant by the Gates Foundation to student adaptive learning using Adapt Courseware 7. This study, to be completed by mid-2015, is planned to include 500 or more students using Adapt Courseware, with another 500 students in one of two control groups. 1 See, for example, Richard E. Meyer, Multimedia Learning (Second Edition), Educational Researcher, Vol. 13, No. 6 (Jun-Jul 1984), pp An online copy is available here: 3 U.S. Patent Application No. 13/837,42 4 What Video Games Have to Teach Us About Learning and Literacy, Revised and Updated Edition (2007). 5 Deci, Edward L., and Richard M. Ryan, Intrinsic Motivation and Self-Determination in Human Beings (Perspectives in Social Psychology), Advisors_March-2013.pdf Adapt Courseware, LLC 14