Career Pathways as a Framework for Program Design and Evaluation. David Fein Abt Associates Inc [email protected]

Transcription

1 Career Pathways as a Framework for Program Design and Evaluation David Fein Abt Associates Inc [email protected] A Webinar September 20, 2012

2 Acknowledgments Presentation based on paper developed for ISIS, a national random assignment evaluation of nine career pathways programs for economically disadvantaged adults and disconnected youth ISIS is supported by Administration for Children and Families (ACF), Office for Planning, Research and Evaluation (OPRE) at DHHS And through an innovative partnership with the Open Society Foundations Gratefully acknowledge additional support from Joyce and Kresge foundations 2

3 Introduction Career pathways are not the latest fad [but] the building blocks of a crucially needed systemic transformation Career pathways are nothing less than a pathway to the middle class (League for Innovation in the Community College 2007). Career pathways movement has gained steam in last decade What are the key ideas? What kind of research can help assess the promise of these ideas? What role can impact evaluations play? 3

4 Background Key career pathways ideas articulated many times, but not as conceptual framework useful in guiding evaluation Aim of presentation is to introduce such a framework Draws from a recent paper on this topic, available at Abt Associates-led teams applying this framework in several national random assignment evaluations: Innovative Strategies for Increasing Self-Sufficiency Evaluation (ISIS, DHHS) Health Professions Opportunity Grants Evaluation (HPOG, DHHS) Green Jobs and Health Care Impact Evaluation (DOL) 4

5 Plan for Webinar Career pathways framework Illustrative programs Theory of change Associated measurement challenges and opportunities Evaluating career pathways programs Illustrative evidence 5

6 Forces Stimulating Career Pathways Movement Growing need for second chance post-secondary training for disadvantaged adults and youth Dissatisfaction with existing approaches Major underlying reform currents: tech prep at secondary level, sectoral workforce initiatives, college enrollment and completion Strong foundation role in stimulating innovation Increasing federal programming and inter-agency coordination (ED, DOL, HHS) 6

7 Career Pathways Defined A career pathway is a series of connected education and training programs and support services that enable individuals to secure employment within a specific industry or occupational sector, and to advance to successively higher levels of education and employment within that sector. Each step is designed explicitly to prepare the participant for the next level of employment and education. (Jenkins 2006) [All career pathways definitions] stress the importance of intentionality and integration of programs and services, meeting the needs of adult learners, focusing on employment as an end goal, and creating incremental, sequential steps towards credentials and degrees with labor market value. (Pleasant & Claggett 2010) 7

8 CPs Can Be Conceived and Implemented at Two Levels As initiatives aiming to change broad systems (e.g., at state and local levels, throughout an institution such as a community colleges) As discrete programs (e.g., specific partner organizations, staff, particular services/approach, specific occupations and fields) At both levels, though emphasize relatively comprehensive approaches (consistent with theory): Designs of initiatives and programs vary (e.g., levels of training, service strategies) Broader framework useful in initial program design, classification, research design not a blueprint for programs 8

9 Prospects for good-paying, stable employment The Basic Career Pathways Model V. BA+ Programs Upper-Skilled Jobs IV. 1-2-Year Certificate to AA Programs Mid-Level Skilled Jobs III. Short-Term Certificate Programs Entry-Level Skilled Jobs II. Sectoral Bridge Programs Skilled Jobs Semi- I. Basic Bridge Programs Occupational, academic, and life skills

10 Basic Defining Elements of CP Programs Organizations and roles Target populations Occupations, credentials, pathways levels addressed Logic model relating program inputs to progression through training & career employment steps Program inputs: Four major categories of strategies emblematic of CP programs, plus a fifth category of cross-cutting strategies 10

11 Program Inputs: Signature CP Service Strategies 1) Comprehensive assessment Academic skills Non-academic skills and needs 2) Signature approaches to instruction Modularization Contextualization Acceleration Flexible delivery Active learning 11

12 Signature CP Strategies (cont.) (3) Promising support strategies Pro-active advising and guidance (academic, nonacademic) Supplemental instruction (academics, also skills for success at school, work, life generally) Social supports Supportive services Financial assistance 12

13 Signature CP Strategies (cont.) (4) Promising strategies for connecting to employment Provide employment experiences to students Arrange post-training job opportunities (5) Meta strategies for Integrating a relatively wide range of services Creating a continuous, data-driven improvement ethos Moving towards scalability and sustainability 13

14 Illustration #1: Pima Community College Pathways to Healthcare Program (Arizona) Basic elements In partnership with county One-Stop, college provides training and credentials in 16 health care occupations, organized into 5 broad pathways (+ 3 broad levels), programs varying from 1-24 months. Targets low-income adults, substantially Latino population. Funded by federal HPOG program. Signature strategies Instruction: Upfront assessment and training planning, w/10-week contextualized basic skills development program if below required entry skill levels. Many programs offer streamlined/compressed formats, translate clock-hours at lower levels to prerequisites to higher levels. Supports: More intensive and coordinated case management (via One- Stop) and academic advising and services (via college), financial support covers training and related expenses. Employment: Outreach to engage local employers in training, internships, employment; use One-Stop services; MOUs linking to various county and state agencies. 14

15 Illustration #2: Year Up (National) Basic elements National non-profit operating in 8 urban areas. Partners with community colleges and major employers in financial services and information technology. Targets disadvantaged youth, 18-24, with high school diploma/equivalent. Signature strategies Instruction: Six-month learning development phase provides customized training in technical and contextualized basic and professional skills (e.g., communication, behavior). Earn 14+ college credits through agreement w/local colleges. Supports: High support, high feedback environment learning communities, pro-active counseling, structured group sessions, supportive services, and weekly performance-based stipend of up to $260 throughout the year. Post-program follow-up and engagement. Employment: Second six-month internship phase places students in entry-level financial services/it positions with major employers, who contribute about half of total program costs per participant. 15

16 Broad Theory of Change Literature on CP programs and strategies suggests broad outline of major intended causal pathways Useful in developing program logic models, research hypotheses, measures, and analyses Logic models for specific programs describe how selected strategies and design intended to influence specific outcomes often a subset of domains in the following figure: 16

17 TO NEXT STEP Initial Targeting & Placement Decisions Participant Characteristics Demographic Educational Economic Theory of Change for Career Pathways Take First/ Next Step In Career Pathway/Lattice Comprehensive Assessment Core Curriculum Supports Foundational Academic Skills Occupational Skills Psycho-social Factors Career Orientation and Knowledge Increase Performance & Persistence in Training Certificate/Diploma 2- year, 4- year Degree Improve Performance & Advancement in Jobs Earnings Benefits Job security Resource Constraints Employment Connections Other Personal and Family Challenges Improve Other Outcomes Income & assets Child & adult well being Local economic growth Contextual Factors: Institutional, Economic, Social Program Inputs Intermediate Outcomes Primary Outcomes

18 Measurement Challenging concepts Workable definitions of signature strategies (for program and individual-level measures) Occupational skills, psycho-social factors, career orientation and knowledge Progress in career pathways Data sources needing attention/development Observation protocols (e.g., classroom) Assessment tools Survey items Administrative systems, e.g., student records, wage records 18

19 Research & CP Continuous, data-driven improvement is a key CP principle Varied tools for varying purposes Program design: analyses of occupational demand, associated skills and training requirements, curriculum mapping Outcomes/performance monitoring: longitudinal tracking: pipeline, milestone, momentum point analyses (college & wage records systems) Implementation research: how do intended and actual program designs map to CP framework lessons for replication? Impact and cost-benefit evaluations 19

20 Contribution of Random Assignment Experiments Though many emerging CP approaches seem promising, easy to be deceived by faulty comparisons The surest way to generate a valid comparison is to randomly assign eligible individuals (or groups of individuals) to receive or not receive a given CP intervention, then measure and compare average outcomes The rewards: Better knowledge for self-improvement Funding increasingly based on experimental evidence Paper explores challenges and opportunities for useful experimental impact evaluations w/in CP framework Here, look at a few illustrations of recent experiments to begin to suggest how CP framework can begin to help organize knowledge 20

21 Illustrative Evidence: Comprehensive, well-targeted first step training programs can have substantial impacts Sectoral training (Maguire et al. 2010) 3 experienced CBOs provide customized short-term training in varied high-demand fields Disadvantaged adults, HS+, careful screening (n=1,014) Year Up (Roder & Elliott 2011) National org provides 6 months customized training + 6 month paid internship in IT and finance sectors Disadvantaged urban youth 18-24, HS+ Program articulates w/local colleges, completers earn 14+ credits Results from small initial experiment (n=164) 21

22 Other Recent Experimental Evidence on Signature Strategies Learning communities As bridge program (Fein et al. 2006) By linking courses (Sommo et al. 2012; Visher et al. 2012) More intensive, specialized personal guidance and support Coaching (Bettinger & Baker 2011) Help with financial aid applications (Bettinger et al. 2012) Student success courses + support (Weiss et al. 2011) Increased financial support Performance-based scholarships (Patel & Richburg-Hayes 2012) Unconditional needs-based grants (Goldrick-Rab et al. 2011) Psycho-social interventions E.g., implicit theories of intelligence, normalizing worries about fitting in and success (Yeager & Walton 2011) 22

23 Looking Ahead CP framework can help make sense of diverse findings, so whole > sum of parts And thereby inform work on needed policies, initiatives and programs helping break down silos As CP paradigm takes hold in policies & practices, calls for a purposeful evaluation agenda studying more comprehensive programs and signature strategies. Several major CP studies in the pipeline, more needed. A few larger questions for this agenda: Can initiatives based on CP principles succeed where traditional approaches have failed for disconnected youth and low-income adults? What does it take to successfully implement such programs and wider system changes? What models and strategies work for populations at varying skill levels? What middle-skill jobs are accessible for adults at varying skill levels, and how difficult is it to advance beyond such jobs? What more general skills are needed to move across jobs and occupational sectors, and to what degree can programs focused on skills for specific jobs also address these more general skills? What will be the impacts on success of shifts in demand for certain occupations? Of better and worse economic environments generally? 23

24 Closing Thank you for attending! Stay in touch with the ISIS team! 24