Demystifying Academic Analytics. Charlene Douglas, EdD Marketing Manager, Higher Education, North America. Introduction

Transcription

1 Demystifying Academic Analytics Charlene Douglas, EdD Marketing Manager, Higher Education, North America Introduction Accountability, stakeholders, dashboards this is the language of corporations, not academia, but more and more often these days, academic institutions have to prove their performance to external audiences in order to obtain funding and accreditation. In the United States alone, 19 states have some form of performance funding tying a percentage of budget allocations to an institution s achievement on specified performance criteria. Twenty-seven states have performance budgeting requirements that allow governors and legislators to consider institutional achievement on key indicators as one factor in determining allocations. Thirty-nine states have performance reporting programs mandating that colleges provide periodic updates regarding their performance on key indicators, but they are not directly tied to funding. And many states combine either performance funding or performance budgeting with reporting requirements. Furthermore, educational institutions need to comply with government mandates, compete globally for researchers and students, review programs and substantiate accreditation and make strategic decisions about whether to build on existing strengths or develop new areas. In order to do this, educational institutions need to better understand their own systemic strengths and weaknesses and create a model for accountability and continual improvement in education. Hard data are becoming the basis for all decisions in academia, including faculty hiring, compensation, and retirement as well as enabling individualized instruction and facilitating professional development. In order to obtain this indication of performance and campus achievement, academic analytics applications are often utilized. Academic analytics not only provides information to external audiences, it also helps drive decision making about which programs and initiatives are best suited to help the institution meet its goals. Academic analytics allow academic institutions to create an integrated, flexible process to steer performance and increase accountability. A good analytics tool can also clean up data, reduce inefficiencies and streamline the process of preparing and delivering necessary reports such as those often mandated by the state. Getting Started Even though there are a number of educational institutions employing academic analytics to better differentiate their institutions in the highly competitive higher education market, it is still a relatively new concept to education. Applying the principles of analytics to academia promises to improve student success, retention and graduation rates and demonstrate institutional accountability (Arnold, 2010). There is very little definition, however, of the categories that need to be measured and the metrics to quantify how well an institution is doing. Additionally, the data are often maintained in silos without appropriate policies governing standards and unique identifiers and there are no benchmarks for comparison. An essential component of continuous improvement is making decisions based on data. This will require fundamental changes in how we collect and use data and in the process we currently use for decision-making and for deriving meaningful interpretations relative to what we want to measure (NETP, 2010). Return on investment for higher education involves student retention, engagement, time to degree, and students post-graduation success, in other words, measuring people instead of dollars. Therefore, Copyright 2011 The Board of Regents of the University of Wisconsin System 1

2 parameters that need to be analyzed include a great deal of student information. As such, there are four main core systems of any educational institution human resources, finance, student information and academia. An enterprise academic analytics system has to provide student and academic reports; financial reports such as budget status of each department, dollar spent per student, dollar spent per class, payroll information, benefit information and so forth; recruitment and admission reports; registration reports; alumni reports; research information and the like. Educational institutions do not lack for data, but they tend to be unsure how to turn these data into meaningful information to drive change. Nor can they easily access, combine, and repurpose those data seamlessly to support analysis drive decision making and improve student success (Strategic Initiatives, Inc., 2009). Stakeholders Stakeholders typically thought of as needing to be able to run various reports and have access to others, include the Chief Information officer, Institutional Research personnel, Planning Directors, Presidents/ Chancellors, Provosts, Vice Presidents/Vice Chancellors, and Enrollment Management. However, it is important that faculty, staff and all administrators the end users understand how academic analytics can be leveraged to achieve institutional goals. Not including the campus community at large is to miss a golden opportunity to make sense of the current data and communicate an understandable analytics strategy and path for the future (Strategic Initiatives, Inc., 2009). Essentially, an institution cannot have a culture of constant improvement if they don t have an academic analytics solution available to the right people. In other words, all stakeholders need to have the appropriate data so that they can be successful. Capturing the Data The analytics process involves gathering and organizing information (often from different sources and in different forms), analyzing and manipulating data, and using the results to answer questions such as Why, What can we do about, or What happens if we do x ; analytics goes beyond traditional reporting systems by providing decision-support capabilities (Campbell & Oblinger, 2007). Additionally, these data are provided in a format whereby trends and issues can easily be identified. The why, what can we do about, and what happens if we do x analyses are moot if there is not an easy method to determine issues that need investigating. Goldstein (2005) provided a useful framework for categorizing key milestones in any implementation of academic analytics: Stage 1 Extraction and reporting of transaction-level data Stage 2 Analysis and monitoring of operational performance Stage 3 What-if decision support (scenario building) Stage 4 Predictive modeling and simulation Stage 5 Automatic triggers of business processes (alerts) Campbell and Oblinger (2007), simplified this model even more into Capture Report Predict Act Refine Copyright 2011 The Board of Regents of the University of Wisconsin System 2

3 But, even before an institution can capture, extract, and report the data, it needs to know what data to obtain. Most institutions are unclear as to what they should measure. Moving from just counting seats to business-focused impact metrics requires coordinated planning, meaningful measurement and effective communication (Kelly, 2010). It is important for an institution to identify their goals and expectations for using academic analytics in the first place. Is the institution ready to implement an academic analytics solution, what are the challenges, the risks? What are the problems they are trying to solve with analytics? What is the priority of these problems? Is it an institutional priority? It is also important to help stakeholders make the connection between those priorities and analytics. Analytics are most often used in education for administrative decisions but the use of analytics is growing in high stakes areas such as academic success and enrollment management (Campbell, DeBlois, & Oblinger, 2007). A few key indicators of academic success include Graduation rates Retention rates Program productivity Attendance Course/Learning Management System productivity Enrollment rates Transfer rates Passing scores on licensure exams Student satisfaction Job placement data Learning Management Systems With barely one in five Americans over 25 earning a bachelor s degree, retention of students who actually enter college is vitally important to our country s global competitiveness (U.S. Census Bureau, 2007). Yet, nationally, the six-year graduation rate for all colleges and universities is 63 percent (Berkner, 2002) and students in their second and third year of college can be among the least likely to persist (Lipka, 2006). Institutions are turning to academic analytics to help with this growing problem. The issue we are trying to solve is applying the insights of academic analytics to an intervention that current students will accept and not associate with academic profiling or big brother watching them (Campbell, DeBlois, & Oblinger, 2007; Fritz, 2009). Measuring performance and putting powerful information in instructors hands for just-in-time intervention is critical to student success. A relationship may exist between student performance as defined by grades and activity in the campus learning management system (LMS). For example, the University of Maryland, Baltimore County has determined that over two academic years, students earning a D or F in 72 courses used the LMS 35% less than students earning a grade of C or higher (Fritz, 2009). This kind of knowledge allows immediate intervention and support for those underperforming students who are at risk of leaving college due to low grades. Other higher education institutions such as Purdue University and the University of Georgia System have identified a clear relationship between LMS use and student achievement. Through the use of academic analytics, the University of Georgia System has shown that data from their LMS can be used to predict student success, show institutional effectiveness for accreditation, and show trends in student learning outcomes (Finnegan, 2009). One of the additional benefits LMS academic analytics provides is the ability to run day-to-day, end-ofterm or annual reports across multiple courses, sections and programs from one interface. Educators can Copyright 2011 The Board of Regents of the University of Wisconsin System 3

4 customize their own reports and combine data elements such as enrollment, student and faculty activity and grade reports into one big- picture report instead of seeing only one data element at a time. LMS academic analytics also provides a real-time element whereby the path a student takes towards their goals can differ depending on data that feeds workflows looking at such things as learning styles. Developing a Strategy Too many times you hear the terms data warehouse, ad hoc reporting tools, analytics, business intelligence and your mind starts racing on all the things you have always wanted to report on or questions you wanted to ask; therefore, the first step is to capture and categorize the questions (M. Cummings, personal communication, February 5, 2010). Ask the right people there are many people who capture and assemble information Review your current manually produced reports What is the cost of producing the manual reports? Review the reports available Review legislative and accreditation requirements What are your pain points, what is lacking? What would make faculty and staff more successful in their jobs? What reporting could faculty and staff use to help students be more successful? Prioritize these requirements How frequently does this information need to be created? How often is this information going to be used? Who uses this information? Why is it required? Review the requirements of an academic analytics solution considering: Difficulty to setup Difficulty to add information Standard reporting available Custom reporting available Ease of importing external information Ease of exporting information Money saved by implementing solution How the solution will affect the performance of the campus LMS Additional information provided that could not be provided in any other method Dashboard availability Three characteristics of successful academic analytics projects are worth highlighting (Goldstein Katz, 2005): 1. Leaders who are committed to evidence-based decision making 2. Administrative staff who are skilled at data analysis 3. A flexible technology platform that is available to collect, mine and analyze data In addition to the above, the educational institution also has to have a culture that is willing to invest in making the improvements identified. For all of their sophistication, academic analytics tools work best when they are kept simple. Resulting reports and dashboards are a call to action. There tends to be a disconnect between what people think they want and what they actually need (Ramaswami, 2010). Copyright 2011 The Board of Regents of the University of Wisconsin System 4

5 Conclusion Academic analytics will help shape the future of education just as evolving technology will enable new approaches to teaching and learning (Arnold, 2010). The need for unprecedented speed and agility, increasing compliance requirements and an urgent demand for visibility into real people-management are some of the challenges that will separate great institutions from those that fall behind. When academic analytics solutions are well designed and easy to use, they enable people to work smarter, learn faster and collaborate more closely. References and Resources Arnold, K. E. (2010). Signals: Applying academic analytics. EDUCAUSE Quarterly, 33(1). Campbell, J. P., DeBlois, P. B., & Oblinger, D. G.. (2007, July/August). Academic analytics: A new tool for a new era. EDUCAUSE Review. Campbell, J. P., & Oblinger, D. J. (2007). Academic analytics. EDUCAUSE. Finnegan, C. (2009). Academic analytics: Using the CMS as an early warning system. Retrieved from publications/impact2006/campbellfinnegancollinsgage_impact06.ppt Fritz, J. (2009). Using course activity data to raise awareness of underperforming college students. University of Maryland-Baltimore. Goldstein, P Academic analytics. Goldstein, P., & Katz, R. (2005). Academic analytics: The uses of management information and technology in higher education: Key findings. Retrieved from Kelly, T. (2010, February). Validation by metrics. Chief Learning Officer. National Educational Technology Plan. (2010). Transforming American education: Learning powered by technology. Ramaswami, R. (2010, February). Revving up performance. Campus Technology. Strategic Initiatives, Inc. (2009). Making analytics understandable and strategic: Strategic consulting services for colleges and universities. Retrieved from socdemo/education/cps2007.html About the Presenter Charlene Douglas has worked for Desire2Learn for almost seven years as an elearning Consultant, the Director of Small Client Initiatives, and now as the Marketing Manager for North America Higher Education. She has 14 years experience in the corporate environment and 19 years in higher education dealing mostly with the areas of elearning, data centers, hosting, and Help Desks. During her 5-year tenure as Director, dot.edu, an Application Service Provider of the University of Wisconsin System, she successfully proved the concept of centralized infrastructure. In 2002, dot.edu was selected by Sun Microsystems to become a Center of Excellence in elearning infrastructure. Charlene has a Bachelor's degree in Elementary Education, a Masters in Counseling and an EdD in Educational Psychology. Address: 151 Charles Street West Suite 400 Kitchener, Ontario N2G 1H6 CANADA Phone: charlene.douglas@desire2learn.com Copyright 2011 The Board of Regents of the University of Wisconsin System 5