Performance Measures Primer WIOA Performance. What We (now) Know What We (still) Don t Know What s Keeping Us Up at Night

Transcription

1 Performance Measures Primer WIOA Performance What We (now) Know What We (still) Don t Know What s Keeping Us Up at Night

2 Introduction WIOA brings Massive Change Even for Texas Performance Measurement & Accountability Replace Existing Common Measures Changes in State & Local Negotiations Bigger Consequences for Failure Significant Reporting Requirements Now includes Eligible Training Provider (ETP) Reporting Opportunities Exist but Opportunities bring Complications 2

3 New Common Measures NonYouth Employed 2 nd Qtr Post-Exit Median Earnings 2 nd Qtr Post-Exit Employed 4 th Qtr Post-Exit Credential within 1 Year Measureable Skills Gain Indicators of Effectiveness of Service To Employers Youth Employed or In Education 2 nd Qtr Post-Exit Median Earnings 2 nd Qtr Post-Exit Employed or In Education 4 th Qtr Post-Exit Credential Rate Measureable Skills Gain Indicators of Effectiveness of Service To Employers 3

4 Origin of New Common Measures Integrated Performance Initiative (IPI) DOL funded grant from 2004 Led by Washington State Included a number of states & National Governors Assn Goal to Develop Measures Acceptable to WF, K-12, & Post-Secondary Education Championed by NGA in all WIA Reauthorization drafts Applicable to all Core Programs Title I Adult, DW, Youth, Wagner-Peyser Adult Education & Literacy Vocational Rehabilitation 4

5 Employed 2 nd Qtr Post-Exit (NonYouth) Essentially Replaces Entered Employment Includes ALL Participants, not just Unemployed Likely called Employment Rate Good Picks up instances where we help an employed person get a 2 nd Job Bad Adds 1 Qtr of Lag to the Measurement Period Counts employed people looking for new/2 nd jobs even if they stay employed at old job Dept of Ed seems to want their own definition for AEL but the NPRM doesn t provide one Would only include those in the Labor Force Might try to get it in via guidance letter 5

6 Employed or In Education 2 nd Qtr Post-Exit (Youth) Replaces Placed in Employment/Education Includes ALL Participants, not just Unemployed/Outof-School New measure counts ANY training/education not just post-secondary, counts those still in HS Good Picks up instances where we help an employed person get a 2 nd Job or help an in-school person continue their education Bad Adds 1 Qtr of Lag to the Measurement Period Counts people who stay employed at old job Similarly counts those already in school who remain in school even if we did little/nothing to support that 6

7 Median Wages 2 nd Qtr Post-Exit (All) Essentially Replaces Average Earnings Good Bad Shorter Lag than Average Earnings (2 nd Qtr vs. 2 nd & 3 rd ) Adds wage focused measure for Youth Median reduces influence of outliers Better for smaller Boards Harder to Statistically Model Smaller Numbers are less meaningful to Stakeholders Unknown Will it include all Exiters or just those Employed? Probably just those Employed 7

8 Employed 4 th Qtr Post-Exit (NonYouth) IPI Measure & Statutory Language focus Exclusively on 4 th Qtr Post-Exit Not a retention measure Feds want input on value of reporting Retention too Good Bad Subset of those Employed 2 nd Qtr Post-Exit If so, likely NOT an Accountability Measure Reporting on those not employed until later Counts people who stay employed at old job Dept of Ed seems to want their own definition for AEL but the NPRM doesn t provide one Would only include those in the Labor Force Might try to get it in via guidance letter 8

9 Employed or In Education 4 th Qtr Post-Exit (Youth) No analog to Existing Youth measures IPI Measure & Statutory Language focus Exclusively on 4 th Qtr Post-Exit New measure counts ANY training/education not just post-secondary, counts those still in HS NPRM doesn t mention interest in a Retention measure for Youth Good Reporting on those employed/in education later Bad Counts people who stay in school/employed at old job 9

10 Credential Within 1 Year (All) Will Replace Youth Attainment of Degree/Certificate & could Replace Educational Achievement Likely called Credential Rate Good Having a NonYouth Credential measure Bad Lots of Lag Extra Lag for Diploma/GED Only counts if also Employed or in Post-Secondary within 1 year of Exit assumedly using wage records which means more lag Unknown How the Lag will work Does Within 1 year after Exit mean 365 days or the 4 th quarter? Any Occupational Training & Education or that which leads to a recognized postsecondary credential or employment? 10

11 Measureable Skills Gain (All) Most Similar to Literacy/Numeracy Gains & AEL Educational Gain Likely called Measureable Skills Gain Rate Good Having an in progress measure for those in training/education for a specified reporting period Having a shorter term measure as a leading indicator for the longer term Credential Measure (little lag here) Unknown What is the specified reporting period? What does documented academic, technical, occupational or other forms of progress, towards such a credential or employment mean? How to measure for non-academic training? Literacy/Numeracy Gains for all? 11

12 Effectiveness Serving Employers (All) Undefined in Statute Secretaries required to consult with States, Locals, Employers, Stakeholders to develop Measure(s) to be Developed by 6/30/16 Good Bad Having Natl Measures of Effectiveness Serving Employers Possibly 1 measure/set rather than 1 per program Starting from Scratch (for Texas) Possibility that we get something like Employers Served Unknown MANY Unknowns NPRM Asked for input on Employment Connection Measure concepts 12

13 Employment Connection System ECS is a system that uses UI Wage Records to identify new Employment Connections between JS & ER EC assumed if wages reported in Current Q but not Prior Q Indicator of Effectiveness Serving Employers If Employer s need is qualified workers then the EC is 1 st step But turnover is a killer for Employers Maintaining Employment Connection could be Indicator of Effectiveness Employment Connection lasting at least 2 Qtrs after EC MEC run for ALL Employment Connections in Texas Not Just WF System Customers Performance Standards based on REAL WORLD PERFORMANCE Can do analysis by industry! Do our connections last longer than the general populace? 13

14 State Negotiations Negotiating Targets for 2 Years Assume 2 nd year will focus on Improvement Negotiations Must Consider Government Performance & Results Act National Targets Targets of Other States Continuous Improvement & ROI ROI? How will that be measured? NPRM doesn t say! Statistical Adjustment Model Economic & Demographic Factors Automatic Target Adjustments at End of Year Must reflect actual Economic Conditions & Participant Characteristics NPRM asks whether Negotiations should encompass more than the DOL/Dept of Ed statistical models 14

15 Statistical Adjustment Models HUGE Challenges Existing DOL Models are limited WIA Only Unemployment Rate is only Economic Variable DOL has limited Wagner-Peyser Customer Data Dept of Ed has NO AEL & Limited VR Customer Data One Model for all States/Bds? ESL has a different impact in El Paso & Detroit Movie Industry Matters a lot in CA / not much in Montana Universal Coenrollment/Low % Training vs. Classic Model Models need to be Regularly Updated Impact of Oil/Gas in South Dakota 10 years ago vs. Today NPRM requests input on how to Update 15

16 Local Negotiations Negotiations Must Consider State Targets Local Economic & Participant Characteristics via Statistical Adjustment Model Where s Continuous Improvement & ROI? Not listed but essentially included by proxy via consideration of State Targets (which include such consideration) Nothing about Negotiating Targets for 2 Years Automatic Target Adjustments at End of Year Both down and up Must reflect actual Economic Conditions & Participant Characteristics 16

17 State Consequences of Failure TA & Performance Improvement Plan for 1 st Year Failure Reduction in Governor s Reserve for 2 nd Year Failure Local Doesn t matter which of 6 WIOA programs fails Failure to Report & Failure to Perform are cumulative 10% cut TA & Performance Improvement Plan 3 rd Year Failure requires Reorg Plan including: Appointment/Certification of new Local Board Prohibition of use of eligible providers and one-stop partners with poor performance Other significant action as the Governor determines appropriate Upshot is requirement to track contractor performance Definition of Failure is KEY <X% of Target on one measure? On 2 measures? 17

18 Definition of Failure is Key NPRM creates Definitions for State Level Failure to achieve an Avg of 90% of Target on All Measures on any single WIOA Program Failure to achieve an Avg of 90% of Target on a single measure across all WIOA Programs Ex: Failure to achieve an Avg of 90% of Target on Median Earnings Failure to achieve 50% of Target on any single Measure NPRM allows states to define Definition of Failure for Boards NPRM asks for feedback on proposals so these could change THIS PART OF NPRM MAKES IT CLEAR THAT THEY ARE NOT CURRENTLY THINKING ABOUT INTEGRATED REPORTING WE NEED TO COMMENT ON THAT 18

19 Annual Report More Reporting Includes 4 Years of Data Data broken out by all kinds of demographic groupings Average Cost By level of Service? Reported at a Customer Level? Administrative Cost Data Including at Local Level Section for Pay-for-Performance Contractors including EVALUATION of the design of such programs/strategies ETP Report Much the same as above but only includes 3 Employment Outcome & Credential Rate Measures Reported for each Program of Study Includes all students, not just those funded by WF system Appears ETPs send student data & TWC calculates/reports 19

20 Areas of Opportunity Core & Intensive Services now just Career Services? Except the old Intensive Services are still segregated in the Career Services section & only used if determined to be appropriate in order... to obtain or retain employment Really there are 3 Types of Services (not a Fed concept) Informational Services (LMI & Availability of/eligibility for Services) Employment Connection Services (Job Search Assistance) Employability Improvement Services (Education/Training) Definition of Participant Informational Service Only proposed to be Excluded NPRM also excludes Self-Serve-Only customers States have invested too much in Self-Service tools to do that Also, Self-Serve-Only customers have slightly better performance 20

21 Areas of Opportunity Separating the Committed from the Curious in Defining Participant Ideally a customer would be a Participant if Received: Employability Improvement Services (Education/Training); Staff-Assisted Employment Connection Services; Self-Serve Employment Connection Services over an extended period Extended Period? Maybe self-serve in 2 of 3 or 3 of 4 consecutive weeks Key is to separate the Committed from the Curious Definition of Exit Change from 90 days? NPRM proposes 90 days without Staff-Assisted Service NPRM proposes ignoring Self-Service entirely when calculating Exit but seeks input 21

22 Opportunities Bring Complications LBB/TWIC Definitions aligned with DOL Definitions Who is a Participant? When does a Participant Exit? Changes to Federal definitions Require Changes to LBB/TWIC definitions OR A 2 nd Common Measures Reporting system One for LBB/TWIC; One for Federal Problem: No idea what final definitions will be Proposed Regs due in January 2015 but were issued DATE Final Regs due in January will they be timely? Legislative Session ends May 2015; Next ends May

23 Speaking of Complications WIOA continues to require a standard reporting mechanism for all Core Programs Don t know what the elements are or if we currently capture them Proposed WISRD won t be out until late spring Common Fiscal & Management Accountability System Establish and operate a fiscal and management accountability information system based on guidelines established by the Secretary of Labor and the Secretary of Education... WIA required this as well but DOL/DOE seem far more interested in enforcing this provision, largely ignored under WIA Feds seem to envision eventually having ONE common system for all core programs covering both financial & customer data Remember Core includes Adult, DW, Youth, WP, AEL & VR Of course there s no money to pay for this... 23

24 To Be Continued WIOA brings Massive Change Even for Texas Performance Measurement & Accountability Replace Existing Common Measures Changes in State & Local Negotiations Bigger Consequences for Failure Significant Reporting Requirements Now includes ETP Reporting Opportunities Exist but Opportunities bring Complications Next Up: Reading & Replying to 2,658 Pages of Regulations in ~60 Days 24

25 Additional Resources Info on Existing Common Measures: Info on Employment Connection System: Narrated Version of this Presentation: 25