Using Longitudinal Data to Measure Impact

Transcription

1 Using Longitudinal Data to Measure Impact ADP Network Leadership Team Meeting Alexandria, VA September 8, 2011

2 Increasing expectations Doing More With Less Improve efficiencies Improve student outcomes Effective use of data Improve system performance Decreasing resources Increase transparency

3 Changing the Culture Around Data Use

4 Critical Policy Questions: The Importance of Longitudinal Data to CCR Early Childhood K-12 Postsecondary Workforce What is the graduation rate by high school? Which early childhood programs best prepare students for kindergarten? To what degree are high school math grades predictors of readiness for college math? What industries are most employing high school and college graduates? How successful are college graduates in the workforce by major or credential?

5 States Have Made Unprecedented Progress Toward Building State Longitudinal Data Systems No state had all 10 Essential Elements 24 states report that they have all 10 Essential Elements

6 Lagging Essential Elements are those MOST Critical to College & Career Readiness Essential Element Data for Action 2010 (# of States) 1 A unique student identifier 52 2 Student-level enrollment, demographic, and program 52 participation information 3 The ability to match individual students test records from year to year to measure academic growth 52 4 Information on untested students 49 5 A teacher identifier system with the ability to match 35 teachers to students 6 Student-level transcript information, including information on courses completed and grades earned 37 7 Student-level college readiness test scores 46 8 Student-level graduation and dropout data 52 9 The ability to match student records between the P-12 and postsecondary systems A state audit system assessing data quality, validity, and reliability states cannot link teacher and student data 15 states do not collect coursetaking information 11 states can t connect K-12 and higher education

7 States Have Not Taken Action to Support Data Use No state has taken all 10 State Actions 2010

8 Data Are Not Linked and Accessible, and Stakeholders Do Not Have Capacity To Use Data Imperative Action # of states Link data systems across P-20 and the workforce to answer key questions Ensure that appropriate data can be accessed while protecting privacy Build capacity of all stakeholders to use longitudinal data 1. Link state K-12 data systems with early learning, postsecondary, workforce, and other 2. Create sustainable support for LDS Develop governance structures to guide LDS Build state data repositories Provide timely role-based access to data 8 6. Create progress reports with student-level data for educators, students, & parents to make individual decisions 7. Create reports with longitudinal statistics to guide change at system level 8. Develop a research agenda Implement policies to ensure educators know how to use data appropriately 10. Raise awareness to ensure all key stakeholders know how to access and use data states cannot link data 44 states don t provide timely access to data 51 states have not taken steps to build educator capacity

9 New Demands: Recalibrating the System for College and Career Readiness Longitudinal data are critical to conduct the research, analyses and evaluations necessary to understand system successes, challenges and opportunities for improvement. Longitudinal data are critical to help policymakers better understand which policies need to be created or changed Longitudinal data are critical to providing educators, parents and students timely and actionable information to inform their behavior and keep students on track to collegeand career-ready graduation

10 DATA REQUIRED CCR STRATEGY Establishing college & career readiness as the goal for every student (standards, assessments, graduation requirements, etc.) Offering students multiple pathways to & options for a college- and careerready diploma Ensuring students stay on track to graduate college and career ready REQUIRES DATA TO Provide evidence base re: skills and content knowledge necessary for postsecondary success Understand relationships among course labels, grades course completion, exam results and postsecondary success. Produce CCR indicators & high school feedback reports Understand & develop options for off-track students. Reveal industry/employment patterns to prioritize CTE, internship, and certificate opportunities. Measure impact of programs, strategies &policies on students postsecondary outcomes. Identify risk factors for dropping out and CCR benchmarks Produce &disseminate analyses to districts and schools. Notify students/parents/counselors re: merit-based aid eligibility Seamlessly provide students transcript information to college admission systems.

11 Ex: Kentucky Feedback Report

12 High School Feedback Reports View from DQC Survey Systems We have ability to link our K-12 and PS systems (44 states) Data Information We provide HS Feedback Reports (19 states) We have data about students transition, including remediation (27 states)

13 College and Career Readiness State of Illinois Achieve America Diploma Project Washington, D.C. September 8, 2011 Jason A. Tyszko Illinois Department of Commerce and Economic Opportunity (312)

14 Agenda 1) Defining the Challenge 2) Resources Available to Illinois 3) Example Categories of Use: Accountability, Personalization, and Planning 4) Challenges 5) Discussion 14

15 Defining the Challenge More than 70 percent of students who graduate high school start some kind of advanced training or education within two years. However, for many the journey ends long before graduation day. Just over half of student who start four year bachelor s degree programs finish within six years. Fewer than three out of ten students who start at community colleges graduate with an associate s degree in three years. In Illinois, of every th grade students enter college the fall after graduating from high school; 11 graduate with a bachelor s degree in four years; 4 graduate with an associate s degree in three years. Lack of college and career readiness has a significant economic impact on Illinois and the United States competitiveness. In addition, Illinois faces a talent gap where skill levels produced no longer align with market needs. 15

16 Defining the Challenge In Illinois, we have both a crisis and an opportunity with managing student transitions to college and employment......a narrowly defined college for all goal one that does not include a much stronger focus on career-oriented programs that lead to occupational credentialsseems doomed to fail Pathways to Prosperity Illinois needs a college and career readiness strategy where all residents have access to high quality degree and credential opportunities that prepare them for the jobs of the present and future. Illinois needs to increase the number of people with quality degrees and credentials with a particular emphasis on high need areas and those that align with the state s economic development objectives. 16

17 Defining the Challenge To address the challenge Illinois P-20 Council is implementing a Collaborative College & Career Readiness Framework: Define College & Career Readiness for Illinois Align P-20 Curriculum and Instruction to Career Pathways that include Common Core College & Career Readiness Standards Align Assessments and Certifications to College & Career Pathways Develop a New Public-Private Infrastructure for Employer Engagement and Partnership Expand Professional Development to Improve College & Career Readiness Target Educational Resources for and Access to Remediation Critical to the success of the Framework is to connect and maximize the use of data across local, state and national education and workforce systems as well as with other stakeholders, including industry and community organizations. 17

18 Resources Available to Illinois Illinois has secured a variety of diverse resources to build data systems that support college & career readiness policies and objectives... Public Act th General Assembly (SB1828) Enabling legislation for state longitudinal data system. Statewide Longitudinal Data System Initial and Expansion Grants Over $20 million. Workforce Data Quality Initiative Intent to apply for $1 million. Race to the Top Round 3 Estimated award between $30 million and $40 million. Gates Foundation and Carnegie Foundation Shared Learning Infrastructure Projected investment to be $100 million for five state consortium. Illinois Jobs Now Capital Bill Learning and Performance Management System $12 million. Industry Partnerships CompTIA data sharing agreement. 18

19 Example Categories of Use - Accountability Accountability By utilizing both education and workforce longitudinal data systems, state s have an increased ability to monitor accountability metrics for P-20 education and training systems. Increased accountability metrics can help identify and target resources and supports to underperforming institutions as well as individual students who are at-risk through early warning systems. Transparency can help inform consumer decisions. Project Examples 1) New K-12 School Report Cards 2) High School to College Success Report 3) Complete College America 4) Performance-Based Financing (Illinois Board of Higher Education) 19

20 Example Categories of Use Report Card Metrics Career Readiness Career Success Example of Metrics for Consideration 1) % of Students Achieving Proficiency on ACT 2) % of Students Completed Employability Assessment 3) # of Students Completing Student Learning Plans 4) % of Students Completing Pathway Skills Assessments 5) # of Early College Credits Earned 6) # of Students Participating in Work-Based Learning 7) # of Approved Programs of Study Offerings 1) % of Students who transition to Postsecondary, Military, or Employment 2) % of Students in Employment within 1 year of Graduation 3) % of Students who Enter and Complete Postsecondary Education within 6 years 4) % of Students Employed within Field of Study 5) Average Wages Earned of Student Cohorts by Degree/Level Attainment with % of Student who have Wages Increasing over Time. 20

21 Example Categories of Use Personalization/Guidance Personalization/Guidance Increasing calls for schools to become more personalized and student-centered in order to improve academic achievement and increase student engagement. Coupled with demands for more clearly defined college & career readiness skills and knowledge. Movement has traditionally emphasized more challenging graduation requirements and the adoption of the Common Core State Standards. Now focus is shifting to guidance systems that allow students to explore individual academic and career interests. Personalization empowers users to manage their data across systems to inform longterm goal setting and promotes student choice. Project Examples 1) Learning & Performance Management System (LPMS) 2) Shared Learning Infrastructure (SLI) 21

22 Example Categories of Use - LPMS Data Flow Source Systems District SIS Gradebook HR Finance Interim Assessment State Assessment Other Learning & Performance Management System Common Data Infrastructure: Source data provided continuously from district and state systems using common data model and rules Aligned to National Education Data Model, while enabling the capture of granular learning experiences Allows deep cross-district data mining Will feed aggregated systems such as the SLDS Common Functions and Portal Experience: Single sign-on, personalization provide access to the core functions of the LPMS, personalized by role Core functions include: Learning Communities, Data Vault, Portfolio, Education and Career Planning, and access to the broader Applications Exchange Standards and rules engine to address privacy, security, and reliability Applications Exchange: Programming interfaces allow for common apps across district Common catalogue system organizes applications by typology and links them to learning goals All apps are infused with appropriate data and provide usage and other information back to the LPMS environment Rating tool to tag apps with different measures of quality assurance Includes well-integrated versions of common district systems, (SIS, HR) as well as a marketplace of free and proprietary apps. Aggregation of demand and common platform supports rapid innovation State Systems State SIS SLDS Human Capital Data Early Childhood Health/ Social Services Workforce Post- Secondary 22

23 Example Categories of Use LPMS User Portal My Profile My Communities Education and Career Plan Integrated education and career planning tool (either state default, or district-selected) Representation of learning progression Financial planning and career/college applications Student Vault State/LEA performance history Competency view Data capture from LPMS applications Portfolio Capture of student work, organized by grade and learning goal Central High School Course Info and Apps (pushed by instructors) School Info and Apps (pushed by administrators) Science or Technology Student Association ABC Tutoring Assignment Wall (aggregated across all learning communities) My Applications Modular learning experiences Data capture from experiences Learning management meta-applications 23

24 Example Categories of Use - Planning Planning Access to longitudinal data across education and workforce systems provides an infrastructure to monitor the performance of talent pipelines across industry sectors. Can be used to anticipate critical skills shortages and better coordinate education and workforce resources, such as Perkins and WIA investments. Increases transparency around talent production to better inform industry stakeholders who are managing investments in education. Project Examples 1) STEM Learning Exchanges 2) CompTIA Certification Data Pilot 24

25 Illinois Pathways Initiative Preliminary Vision for a Governance Structure Illinois Pathways Interagency Committee - Established by IGA - Membership: DCEO, ISBE, ICCB, IBHE, ISAC, IDES - Functions: - Select Learning Exchanges - Data sharing across education and workforce systems - Establish Pathways Resource Center - Establish Illinois Pathways Advisory Council - Align programs and policies to support Pathways Resource Center and Learning Exchanges - Coordinate with LPMS Goal: To create a new, innovative publicprivate education infrastructure that can advance college and career readiness by coordinating statewide networks of P-20 education partners, business, labor, and other organizations based on career clusters. Illinois Pathways Advisory Council - Membership: IPIC, Learning Exchanges, P-20 Council, Business, and others - Responsibilities: - Advises Agencies, Governor, and General Assembly - Advises Pathways Resource Center on projects - Submits annual talent pipeline reports and Learning Exchange benchmark reports to Governor and General Assembly - Plans annual STEM/Learning Exchange project sharing event Pathways Resource Center - One or multiple government or non-governmental entities - Funded by Authority and submits reports to Adv Committee - Categories of service: - External Outreach - Funding Center - Technology Platform - Performance Management Learning Exchanges - Membership: Broad public-private partners, including P-20 education institutions, industry, labor, museums, and community based organizations - Selected by IPIC through submission of a 3 years strategic plan - Submits annual talent pipeline and benchmark reports to IPAC - Coordinates 9 Functions: 1. E-Learning Resources 6. Professional Development 2. Regional Resources & Assets 7. Career Development 3. Student Organization Supports 8. Education & Career Planning 4 Work-Based Learning Experiences 9. Review Performance 5. Sponsor Challenges

26 Example Categories of Use P-20 Sector Performance Labor Market Demand and Employment Transitions Program Completion and Credentialing Educational Engagement and Persistence Career Interest and Readiness Example of Metrics for Consideration 1) # of short-term projected job openings and number transitioning from P-20 pipeline into employment 2) # of long-term projected job openings relative to projected number transitioning from P-20 pipeline into employment 3) % making successful transitions to employment and achieving expected earnings 1) % of students completing programs with degrees and certifications, within expected time frames 2) % of students achieving recognized industry and professional certifications and licenses, within expected time frames 1) % of students persisting in programs without dropping out 2) % of students making progress toward program completion (e.g., competencies achieved, credits earned; within expected time frames) 1) # of students enrolling in programs of study within pathways and cluster 2) # of qualified students expressing career interests with pathways and cluster 26

27 Challenges While Illinois has taken important steps to advance college & career readiness priorities in the development of integrated longitudinal data systems, several important challenges remain... Common Identifier: State systems use a diverse set of identifiers ranging from social security numbers to student information system numbers. Given state and federal policy a centralized data warehouse is not a viable option. Instead, a federated approach to record matching needs to be designed and implemented. Transparency, Data Integration and Interoperability of Data: Broaden utilization beyond research to enable access to data by application developers; lifelong and life-wide learning partners; and the public to support increased personalization around college & career readiness. Ensuring a Diverse set of Stakeholders are Represented: As we continue to build longitudinal data systems and work towards data integration it is critical that a broad set of public-private partners are at the table to ensure that the data is meaningful, has utility and informs practice, policy, and planning. 27

28 Discussion 28

29 Achieve's ADP Network Leadership Team Meeting September 2011

30 Percent Working Age Adults in Hawaii P-20 Goal: 55% of working age adults will have a 2- or 4-year degree by % 8% 0% 20% 75% 50% 25% 30% 21% 40% 25% 55% 100% career and college ready 0% Current Educational Levels (2000) P- 20 Goal (2025) Associates or Higher Postsec ed & trng (no degree) Minimum HS Diploma Less than HS

31 U.S. & HI are losing ground on global competitiveness Differences in College Attainment (Associate and Higher) by Age Group U.S. and Leading OECD Countries, 2004 Source: OECD, Education at a Glance 2005

32 Of th graders 91 Hawaii s educational pipeline is leaky Graduate from HS on-time Enter college immediately Continue into sophomore year Graduate college on-time Source: The NCHEMS Information Center for State Higher Education Policymaking and Analysis, Best Performing State US Average Hawaii

33 Hawaii ranks 1 st in recent graduates denied military enlistment based on academic readiness Armed Forces Qualifying Test Ineligibility Rates of Recent High School Graduates, by State

34 Text Lexile Measure (L) 1600 For today s HS grads, Career Ready = College Ready academic skills NY Times Medical Insurance Benefit Plan Federal Tax Form (W-4) USA Today Installing Your Child Safety Seat GM Protection Plan DVD Player Instructions High School Literature College Literature High School Textbooks College Textbooks Military Personal Use Nat l Occupat. Entry-Level Occupations Competency Lexile Framework Tests for Reading Study Summary of Text Lexile Measures, Interquartile Ranges Shown (25% - 75%) SAT 1, ACT, AP* Harry Potter & the Sorcerer s Stone (bk 1) Adventures of Pinocchio Holes A Baby Sister for Frances

35 Employers find many graduates under-prepared

36 Using Data to Inform Practice Data Governance: Creation of a Statewide Longitudinal Data System Executive Committee Publication of the College and Career Readiness Indicators Report for Every Public High School University: Strategic Outcomes and Performance Measures Buildout of a College and Career Portal:

37 SLDS Executive Committee Data system requires inter-agency data which is outside the scope of institutional responsibilities and budgets Creation Impetus: Application for the 2009 Federal SLDS Recovery Act Grant Reinforcement: Passage of Legislation, Act 41, 2010 Quarterly Meetings: Superintendent, Dept. of Education Executive VP/Provost, University of HI VP for Community Colleges, University of HI Director of the Dept. of Labor & Industrial Relations, State of HI Governor s Education Policy Advisor Chief Information Officers, Dept. of Ed & Univ. of HI Director of Institutional Research, Univ. of HI Governor s Early Learning Coordinator Executive Director, Hawaii P-20 Partnerships for Education

38 College and Career Readiness Indicators Completed for every public high school Classes of 2009, 2010, 2011

39 College and Career Readiness 1. Key Issues: - Governance - Accountability - Data Literacy - Data Availability - Communication Indicators 2. Strategic Decisions - Class of - Available Data - Structure: Approaches, Meets, Exceeds (Achieve) - Low-Key Rollout

40 College and Career Readiness 3. Areas for Improvement - Career Data - True Longitudinal Data - Formal Accountability - Predictable Roll-Out - Ownership by Institutions - Better Format - Disaggregation by Special Indicators

41 College and Career Readiness 4. Impact: Indicators - Change in Emphasis in the HI Dept. of Education Strategic Plan - Incorporation into School Improvement Plans - Recruitment Purposes - Leverage for Outreach - Anecdotal

42 Telephone: (808) Website:

43 Other Data Initiatives UH Strategic Outcomes & Performance Measures Hawaii College & Career Portal

44 How Will We Know When We Are Successful? When all education stakeholders demand and use quality data to make decisions

45 As a result of states progress Conversations are changing but there s more work to be done to support effective data use Link data across systems Ensure appropriate access Build capacity for use and that requires tackling tough territory. Turf Trust Technical Time

46 Game-Changing Priorities for Fulfill the 50-state commitment to implement the 10 Essential Elements by September Link K 12 with early childhood, postsecondary and workforce data to answer critical policy questions. 3. Provide parents, students, and teachers with access to longitudinal student-level data. 4. Share data about teacher impact on student achievement with educator preparation institutions. 5. Enact statewide pre-service policies, including certification and licensure, and program approval, to build educator capacity to use data.