Day 2: Analytics for Teaching, Learning, and Student Success

Transcription

1 BRIEF E D U C A U S E EDUCAUSE Analytics Sprint Day 2: Analytics for Teaching, Learning, and Student Success This brief summarizes the EDUCAUSE webinar Analytics for Teaching, Learning, and Student Success (part of the Analytics Sprint), held July 25, The speakers were Marsha Lovett, Director, Eberly Center for Teaching Excellence and Teaching Professor of Psychology, Carnegie Mellon University; and Ellen Wagner, Executive Director, WICHE Cooperative for Educational Technologies (WCET). P ublic expectations for accountability and transparency have increased in every sector, including education. Learning analytics can take advantage of large volumes of data and advanced technologies to empower students to manage their academic progress, to improve faculty visibility into student performance, and to match instructional resources to learner characteristics. The goals of learning analytics are to draw accurate conclusions from analytical insights and take targeted actions. Two projects clearly demonstrate the power of analytics in education. Carnegie Mellon University s Learning Dashboard has dramatically increased learning gains by using cognitive theory and statistical models to evaluate student progress. WCET s Predictive Analytics Reporting Framework has been used across institutions to identify factors related to student retention, advancement, and completion. Key Takeaways Predictive analytics might make education professionals uncomfortable, but public demand for measurable results can t be ignored. Thanks to big data and analytics, the public has heightened expectations for educause.edu accountability, transparency, and quality. The desire for measurable results extends to every sector, and education organizations cannot ignore these demands. As in the business sector, both big data and technological advances are driving adoption of predictive analytics in education. Students leave behind massive amounts of digital breadcrumbs when they engage in certain learning activities. Four technologies make it easier to mine this information to optimize teaching and learning: 1. Data warehouses and the cloud. With these technologies, massive numbers of student records can be collected and managed. 2. Sophisticated technology platforms. Powerful computer systems enable the analysis of huge data sets and the identification of meaningful patterns. 3. Data mining. Through descriptive and inferential statistics, actionable information can be found in large volumes of data. 4. Predictive techniques. Tools like neural networks and decision trees help anticipate behaviors and events. Many education professionals question whether mining student data can improve 2012 EDUCAUSE. Reproduction by permission only. 1

2 and personalize educational experiences. While analytics alone is not an answer, it has the potential to be a valuable component of larger efforts. The key is seeing how predictive analytics can lead to meaningful action. Marsha Lovett suggested that prediction plus understanding leads to targeted action. It is only through active engagement of educators that the best toolsets will be developed to support students and faculty. Education has to come up with ways of being accountable. If we can measure digital breadcrumbs in our consumer lives, we can measure them in our learning lives. Ellen Wagner Instructors and students can both benefit from quantitative, actionable learning data. Before instructors can improve learning, they must determine whether students are responding to instruction. Unfortunately, most instructors gauge the progress of a course based on gut feel. A recent study at Carnegie Mellon found that this qualitative approach often leads to incorrect conclusions and decisions and to suboptimal learning outcomes. Researchers studied a statistics course taught by an award-winning professor. Although students spent more than 100 hours per semester in the classroom, they demonstrated learning gains of only 3%, based on the difference between pre-test and post-test grades. Instructors and students can both benefit from more quantitative, actionable data about classroom progress. Today, most instructors only receive averages or distributions of student scores. This information is too high-level to be useful, and the results typically arrive only after units have been completed. Even if students find that they are falling short in a particular area, they may not know what to do. Many undergraduates aren t motivated to remediate because they quickly move on to the next topic. The Learning Dashboard uses cognitive theory and statistical models to improve learning through targeted actions. Carnegie Mellon s Learning Dashboard provides instructors and students with useful information about learning and instructional design choices. This tool uses two techniques to analyze clickstream data from students interaction with online learning objects: 1. Cognitive and learning theory. Learning is skill-specific. The Learning Dashboard s quantitative cognitive model of skill learning captures detailed information about students performance as they practice specific skills. 2. Statistical modeling. Since student learning states are not directly observable, inferences must be made about them using available data. Bayesian hierarchical modeling generates inferences about student learning, and these inferences become more accurate as data accrue across students, classes, and populations. As students interact with online learning modules, data are pushed to the cloud and the Learning Dashboard model estimates their learning states on several skills. The model tells students about patterns in their answers and about areas of strength and weakness, and it recommends areas for further study. The Learning Dashboard reveals how well students learned different skills, identifies patterns in learning behavior, and measures the effectiveness of instructional design choices. Marsha Lovett For faculty, the Learning Dashboard aggregates class data. Dot plots, bar charts, and green/yellow/red indicators show how classes are progressing on learning objectives. Instructors can drill down and identify specific students who may need additional assistance. educause.edu 2

3 Analytics can promote better learning in shorter amounts of time. Carnegie Mellon researchers studied three statistics classes to determine whether the Learning Dashboard improved learning. Faculty for the three experimental groups adapted their instruction based on Learning Dashboard feedback. In addition, the learning process was accelerated in two ways: 1) classes met twice a week rather than four times; and 2) one semester s work was covered in 8 weeks rather than 15. The research proved that with adaptive teaching and learning, students learned the same material in a shorter time than in a traditional course. They also showed equal or better performance on: Standardized tests. The experimental group demonstrated learning gains of 18% compared to 3% among the control group. Final exams. The adaptive/accelerated group had higher scores than the control group, though the results were not statistically significant. Follow-up evaluations of retention and transfer. After six months, the adaptive/ accelerated group scored higher on a retention standardized test than the control group. In addition, the experimental group scored considerably higher on open-ended data analysis tests for transfer of knowledge. When students were in a course with hybrid, online, and face-to-face learning with the Learning Dashboard, they learned more in less time. Marsha Lovett The Predictive Analytics Reporting Framework uses statistics to identify retention, progress, and completion factors. The Predictive Analytics Reporting (PAR) Framework was started by the WICHE Cooperative for Educational Technologies and funded by the Bill and Melinda Gates Foundation. This project analyzes patterns related to student success and educational approaches. Research questions examined included: What affects student disenrollment? What keeps students in school? What demographic variables affect student behaviors related to dropping out or changing institution? Ellen Wagner highlighted key aspects of the project s first phase: Six educational partners. The project included two for-profit institutions (American Public University, University of Phoenix), two four-year universities (University of Hawaii System, University of Illinois Springfield), and two community colleges (Colorado Community Colleges Online, Maricopa Community College). A large, multi-institutional data set. During the first phase, 33 common variables were identified across the six institutions. A multi-institutional data set aggregated 3.1 million course records and 640,000 student records. Descriptive, inferential, and predictive statistical methods. Analysis protocols focused on retention, progress, and completion and included regression analyses, group differences, decisions trees, CHAID (Chi-squared Automatic Interaction Detection) analyses, and more. The PAR Framework s preliminary findings suggest there is no apparent relationship between age, gender, or ethnicity as a function of students risk profiles. However, at-risk students are more likely to drop out if they take several courses concurrently. Taking more than one course in the early stages of college was highly correlated with an increased risk of disenrollment. The study also found that institution-specific factors predict success among students who aren t at risk of disenrollment. A second round of research is being planned that will include 10 institutions. The goal is to generate more confidence in the preliminary findings by aggregating one million student records and five million course records. The PAR Framework has identified opportunities for research into issues such as best practices for different types of institutions, enrollment educause.edu 3

4 management, and financial aid policies based on course load. A detailed description of the PAR Framework project methodology can be found in the Journal of Asynchronous Learning Networks. The education sector must recognize that learning can be advanced through analytics. Wagner s experience with the PAR Framework illustrates eight important lessons about learning analytics: 1. Analytics are here to stay. Education organizations must accept quantitative tracking of information. 2. What organizations do with analytical findings is what really matters. Institutions must interpret analytics findings based on their unique situations. Different institutions will find different applications for the insights. This is a shift for education organizations that are used to a single, top-down answer. I ve never seen a time where the evidence for efficacy is more important. Having valid, reliable research upon which we can depend to do our interpretations is absolutely key. Ellen Wagner 3. Doing research on analytics is different from applying results to help learners succeed. With the PAR Framework project, each of the institutional partners applied the findings in a different way. However, all shifted their policies to improve the student experience. 4. Since educational organizations have more data than they can handle, they need better ways to manage it. It is easier to manage this information, once important research questions have been articulated. 5. Even more interesting data-collecting opportunities await. For example, WCET has seen patterns of transactions in gamebased settings, and these can be used to anticipate different types of learning events. 6. Education must be prepared to live under the sword of data. In education, the prospect of freely sharing lots of data makes many uncomfortable because it isn t always clear how it will be used. Analytics will push higher education to think in new ways and work differently. 7. There is no such thing as sort of transparent. Once the data box is open, everyone can see it, and that can be uncomfortable. However, using data productively can improve the health of institutions. 8. Education is just starting to understand the true power that analytics brings to learning. Analytics has the potential to raise the bar and improve student success, but educators must take an active role in managing how these tools are used. Other Important Points Analytics and humanities. Learning in humanities courses such as literature or religion can be measured. With the Learning Dashboard, for example, instructors can grade essays according to a rubric and feed the information to the model. Benchmarking. Institutions might want to use common data frameworks that facilitate benchmarking against other organizations. EDUCAUSE is a nonprofit membership association created to support those who lead, manage, and use information technology to benefit higher education. A comprehensive range of resources and activities are available to all EDUCAUSE members. For more information about EDUCAUSE, including membership, please contact us at info@educause.edu or visit educause.edu. educause.edu 4

5