How To Analyze Higher Education

Transcription

1 Dr. Craig Schoenecker Dr. Linda Baer MnSCU CAO/CSAO Meeting May 29, 2014

2 Why Analytics? Are you ready? Examples of outcomes Predictive Analytics Reporting (PAR) Overview Pilot participation in PAR

3

4 For much of the history of American Higher Education, the idea that many students would drop out was simply accepted as a given. And drop out they did. The Retention Agenda, ETS

5 Despite decades of reforms intended to improve student success, including hundreds of specific initiatives designed to facilitate student engagement during the first year of college, roughly 58% of firsttime full-time students who entered in 2004 completed in six years. EDUCAUSE 2014 At community colleges, fewer than a third complete an associate s degree within three years and only 17% earn a bachelor s degree within six years. EDUCAUSE 2014 One in four who begins college at a four-year college or university does not return the second year. (ACT 2010.) This has not changed for over 20 years!

6 Only 8 percent of CAOs strongly agree their faculty understand the financial challenges confronting their institutions. Few agree that MOOCs have great potential to make positive impact on higher education. Most are optimistic about the potential for adaptive testing and learning. Only 2 in 10 CAOs across sectors say their Institution is very effective in the area of identifying and assessing student outcomes. One in four CAOs are confident in their institution s effectiveness at using data to aid and inform campus decisions. A majority of chief academic officers say they will increase emphasis on cutting underperforming academic programs and collaborating with other institutions. Inside Higher Education 2013 Survey of College and University Chief Academic Officers

7 Definition: Analytics is the use of data, statistical analysis and explanatory and predictive models to gain insights and act on complex issues.

8 What are you doing with analytics? What would you envision as the use of analytics? What priorities?

9 If you use these analytical tools, you will know where you are, what you re doing, if what you are doing is working or not whether or not you need to be doing new things customized to fit your particular school or demographic infinitely more information to help students be successful Michael Crow, President Arizona State University

10 STRATEGIC QUESTION DATA ANALYSIS AND PREDICTION INSIGHT AND ACTION

11 1. On your campus, who is collecting what data, for whom, and how? 2. How is research about students being used to develop activities, interventions, and programs that promote student success?

12 What s the best that can happen? What will happen next? What if these trends continue? Why is this happening? What actions are needed? Where exactly is the problem? How many, how often, where?

13 Education Planning Counseling and coaching Risk Targeting and Intervention Transfer and Articulation Legacy ERP/SIS/LMS Vendor point solutions Homegrown point solutions Sinclair s MAP Valencia s LifeMap Austin Peay s Degree Compass Central Piedmont s Online Student Profile WICHE s Predictive Analytics Reporting Direct-tostudent 2010 Bill & Melinda Gates Foundation

14 What tools are you exploring or are you using? What are you learning about student success?

15 Have you inventoried your policies for student success impact? Are practices optional or mandatory?

16 First Year Seminars and Experiences Common Intellectual Experiences Learning Communities Writing-Intensive Courses Collaborative Assignments and Projects Undergraduate Research Diversity/Global Learning Service Learning, Community-based Learning Internships Capstone Courses and Projects

17 The LIPSS project is designed to allow analyses that test the links between specific policies and specific experiences for specific populations of students. Findings from the study will have the potential to shape college and university policies that will, in turn, contribute to student success and close gaps in student outcomes.

18 Technology that learns from experience (data) to predict the future behavior of individuals in order to drive better decisions. -Eric Seigel, Predictive Analytics: The power to predict who will click, buy, like or die (2013: John Wiley and Sons).

19 Consumer analysis Health care analysis Insurance analysis Amazon.com Crime prevention analysis role in education

20 Identifying students at risk Not being retained Not passing a course Not achieving satisfactory academic progress Identifying indicators that are important to monitor and may be actionable Targeting interventions efficiently The right students The right levels of interventions The right modalities The right timing

21 Culture and Process Data/Reporting Tools Investment Expertise Governance/Infrastructure

22

23 What is PAR and how are campuses using it?

24

25 Founding Partners (since 2011): American Public University System Colorado Community College System Rio Salado College University of Hawaii System University of Illinois Springfield University of Phoenix Implementation Partners (since 2012): Broward College Capella University Lone Star College System Penn State World Campus Sinclair Community College Troy University University of Central Florida University of Maryland University College Western Governors University New Members Northern Arizona University Kaplan University Excelsior College University of North Dakota Community Colleges of Spokane

26 Student Demographic s & Descriptive Gender Race Prior Credits Perm Res Zip Code HS Information Transfer GPA Student Type Course Catalog Subject Course Number Subject Long Course Title Course Description Credit Range Student Course Information Course Location Subject Course Number Section Start/End Dates Initial/Final Grade Delivery Mode Instructor Status Course Credit Lookup Tables Credential Types Offered Course Enrollment Periods Student Types Instructor Status Delivery Modes Grade Codes Institution Characteristics Student Financial Information FAFSA on File Date Pell Received/Awarded Date Student Academic Progress Curent Major/CIP Earned Credential/CIP Possible Additional ** Placement Tests NSC Information SES Information Satisfaction Surveys College Readiness Surveys Intervention Measures ** Future

27 IDENTIFY TARGET TREAT Show how institutions compare to their peers in student outcomes, by scaling a multiinstitutional database for benchmarking and research purposes. Identify which students need assistance, by using in-depth, institutional specific predictive models. Models are unique to the needs and priorities of our member institutions based on their specific data. Determine best ways to address weaknesses identified in benchmarks and models by scaling and leveraging a member, data and literature validated framework for examining interventions within and across institutions (SSMx)

28 Performance Benchmarks Measurable Results Common Data Definition s and Data Warehous e Intervention Benchmarks Action Predictive Models Scalable cross institutional improvements enabled by Collaboration via PAR

29 PAR Benchmarks Descriptive Analytics Cross Institutional Student/degree/major level insight into: 1. What did the retention look like for students entering in the same cohort? 2. How does your institution compare to peer institutions / institutions in other sectors? 3. How did performance vary by student attributes? PAR Models Predictive Analytics Institutional Specific insight into: 1. What students are being retained over time? 2. Which students are currently at risk for completing and why? 3. Which factors are directly correlated to student success? 4. What is the predicted course completion rate for a particular program?

30

31 Mechanism for inventorying & categorizing student success interventions/ supports In a common framework Based on why known predictors of risk and success In the context of timing in the academic life cycle Addresses Now What in a way that enable cross institutional benchmarking supports local and cross institutional cost/ benefit analyses. PAR Framework 2013

32 Literature-based tool for benchmarking student services and interventions

33

34 From Data to the Student Who needs additional support? Why? What should we do and when? How do we enable support staff to help? How do we continuously get better at this? Identify them with a suite of predictive models Cluster analysis to segment students combined with qualitative data Combine what we know about the why with what has historically been effective and design comprehensiv e intervention strategies Create dashboards and timely communications, provide recommended resources, and engage all roles in the development Create a mechanism for on-going data collection and a feedback loop to inform all parts of the process

35 A fundamental objective for developing common language and frameworks for reviewing student interventions is so that the most effective interventions can be applied at the points of greatest need to effectively remediate risk at the student level. PAR has paved the way for creating common understanding of student risk and common tools for diagnosing risk, but the road to developing consistent and applied measurement to student impact of intervention will take time and vigilance.

36 System leadership is considering a pilot participation in PAR System Office staff would coordinate participation and submit student records Colleges and universities could choose to use the services available through PAR Costs for implementation and two years of membership would be paid with grant funds

37 Gain experience and knowledge regarding use of predictive analytics Assess the value and potential of predictive analytics to improve student success Assess the effectiveness of intervention strategies Support achievement of Strategic Framework and Charting the Future goals

38 Consultation: ASA Advisory Council: March Academic Technology Council: April IR Directors: April CAO s, CSAO s and Deans: May CIO s: June Decision: Leadership Council

39 What do you need to know about PAR?

40

41 References AACU High Impact Practices Association of Community College Trustees Student Success Toolkit. Baer, Linda and John Campbell From Metrics to Analytics, Reporting to Action: Analytics Role in Changing the Learning Environment. In Game Changers, edited by Diana Oblinger. Bean, John P. and Barbara Metzner 1985 A Conceptual Model of Nontraditional Undergraduate Student Attrition in Educational Research Winter, 1985, Vol.55, No 4, Compete College America Crow, Michael. No More Excuses in EDUCAUSE Review, vol. 47, no. 4 July/August Davenport, Thomas Retrieved November 23, Gilbert, C., M. Eyring, and R. N. Foster. Two Routes to Resilience. Harvard Business Review, December, 2012, Graduate School Metrics. Association of American Universities Jones. Dennis Outcomes-Based Funding: The Wave of Implementation in September National Center for Higher Education Management Systems Kamenetz Anya Fast Company 2012 Most Innovative Companies 2012 Southern New Hampshire University University of Illinois Graduate Model. Kuh. George and Jillian Kinzie, John H. Schuh, Elizabeth J. Whitt and Associates Student Success in College: Creating Conditions that Matter National Commission on Higher Education Attainment An Open Letter to College and University Leaders: College Completion Must Be Our Priority. American Council on Education. Washington D.C. Norris, Donald and Robert Brodnick, Paul LeFrere, Joseph Gilmour, Linda Baer, Ann Hill Duin and Stephen Norris Transforming in an Age of Disruptive Change. Strategic Initiatives, Inc. and the Society for College and University Planning Predictive Analytics Reporting Framework. info@parframework.org Sowell, Bell and Kirby Ph.D. Completion and Attrition: Policies and Practices to Promote Student Success. Council of Graduate Schools Student Success Matix (SSM X ) A model classifying influences on student success within the PAR Project WCET Annual Meeting Presentation Mindy Sloan, Ashford University, Karen Swan, University of Illinois Springfield, Michelle Keim, Bridgepoint Education, Heidi Hiemstra, Predictive Analytics Reporting (PAR) Framework. November 15, 2013 Shugart, S. M The Challenge to Deep Change: A Brief Cultural History of Higher Education. Planning for Higher Education, December 28. Retrieved January 15, 2013, from the World Wide Web: Tinto, Vincent. 2012a. Leaving College: Rethinking the Causes and Cures of Student Attrition. University of Chicago Press. Tinto, Vincent. 2012b. Completing College: Rethinking Institutional Action. University of Chicago Press