Evaluation of the Federal Talent Search Program: Phase II and Phase III Feasibility Report

Transcription

1 Cotract No.: ED-98-CO-0073 MPR Referece No.: MATHEIV ATICA Policy Research, Ic. Evaluatio of the Federal Talet Search Program: Phase II ad Phase III Feasibility Report Fial Report October 2000 Myles Maxfield Maggie Cahala Tim Silva Justi Humphrey Melissa Thomas Submitted to: Submitted by: U.S. Departmet of Educatio Mathematica Policy Research, Ic. Plaig ad Evaluatio Service 600 Marylad Aveue, S.W. 400 Marylad Aveue, S.W. Suite 550 Room 6W200 Washigto, DC Washigto, DC Project Officer: Sadra Furey Project Director: Maggie Cahala

2 ii DISCLAIMER The views, opiios, ad approaches preseted herei represet the opiios of the authors ad do ot ecessarily represet the policies or positios of the U.S. Departmet of Educatio. Disclaimer

3 ACKNOWLEDGEMENTS The authors would like to ackowledge the cotributios of Sadra Furey ad David Goodwi of the Plaig ad Evaluatio Service, U.S. Departmet of Educatio. Their commets o the techical approaches preseted herei were very helpful. Ay errors ad omissios are the sole resposibility of the authors. Ackowledgemets

4 CONTENTS Chapter Page EXECUTIVE SUMMARY xv INTRODUCTION 1 A. THE CONGRESSIONAL MANDATE FOR THE STUDY 1 B. TALENT SEARCH PROGRAM BACKGROUND 2 1. Goals ad Services 3 2. TS i the Cotext of Other TRIO Programs 4 C. CONTEXT FOR THE STUDY: CHANGES SINCE THE 1960S IN NATIONAL STATISTICS RELATED TO EDUCATION ENROLLMENT 6 D. PREVIOUS STUDIES OF TALENT SEARCH 7 E. STUDY OBJECTIVES, COMPONENTS AND RESEARCH QUESTIONS 8 1. Compoets of the Study 9 2. Study Questios 9 F. THE CONCEPTUAL FRAMEWORK FOR THE STUDY 10 G. STRUCTURE OF THE REMAINDER OF THE REPORT 13 II STUDIES USED TO INFORM THE FEASIBILITY REPORT 15 A. PERFORMANCE REPORTS AND TARGET SCHOOL ANALYSIS Project ad Participat Demographics Target Schools ad Percetage of Eligible Served Services Provided by Talet Search Objectives ad Outcomes 25 B. Project Feedback O The Study Project Director Feedback Discussios ad COE Fax Resposes Project Survey 29 C. CASE STUDIES 32

5 vi CONTENTS Chapter Page D. EXPLORATION OF THE POTENTIAL USE OF FEDERAL AID RECORDS AS A WAY TO TRACK PARTICIPANT OUTCOMES Usig the Aid Files as a Idicator of Performace 34 E. USING STATE-LEVEL DATABASES AND ADDITIONAL SOURCES OF POST-SECONDARY DATA 37 F. CONCLUSIONS FROM FEASIBILITY WORK THUS FAR 37 II STUDIES USED TO INFORM THE FEASIBILITY REPORT 15 A. PERFORMANCE REPORTS AND TARGET SCHOOL ANALYSIS Project ad Participat Demographics Target Schools ad Percetage of Eligible Served Services Provided by Talet Search Objectives ad Outcomes 25 B. Project Feedback O The Study Project Director Feedback Discussios ad COE Fax Resposes Project Survey 29 C. CASE STUDIES 32 D. EXPLORATION OF THE POTENTIAL USE OF FEDERAL AID RECORDS AS A WAY TO TRACK PARTICIPANT OUTCOMES Usig the Aid Files as a Idicator of Performace 34 E. USING STATE-LEVEL DATABASES AND ADDITIONAL SOURCES OF POST-SECONDARY DATA 37 F. CONCLUSIONS FROM FEASIBILITY WORK THUS FAR 37 Cotets

6 vii CONTENTS Chapter Page IV PHASE II NATIONAL-LEVEL IMPACT STUDY DESIGN 73 A. POLICY QUESTIONS 73 B. AGGREGATE SCHOOL DISTRICT APPROACH Experimetal Desig Propesity Model Database Costructio Samplig Aalysis : 75 C. ALTERNATIVE APPROACHES INVESTIGATED NELS 88/ Geo-Codig Idividual FFA Records Lists of HS Completers from School Districts 78 D. SUM IARY 78 E. REPORT AND BRIEFING 79 V PHASE III SERVICE MIX DEMONS.RATION EVALUATION DESIGN 81 A. PURPOSE OF THE STUDY AND HOW IT RELATES TO PHASE II 81 B. ES'IED SUPPLEMENTAL SERVICE MIXES 82 C. EXPERIMENTAL DESIGN 84 D. TIMELINE 85 E SAMPLE DESIGN AND RANDOM ASSIGNMENT 87 F. DATA ACQUISITION PLAN 91 G. IMPLEMENTATION STUDY 93 Cotets

7 viii CONTENTS Chapter Page ESTIMATES OF COST AND TASK SCHEDULE FOR PROPOSED PHASE II AND PHASE III STUDIES 95 A. OVERVIEW OF STUDIES AND COSTS State impact studies usig state secodary ad postsecodary databases, project participat level records, ad federal aid applicatio ad receipt files Demostratio studies of alterative iovative services 98 B. SUMMARY OF COSTS 99 C. LIST OF TASKS AND DELIVERABLES FOR PHASE II 100 D. DETAIL OF TASKS FOR THE PROPOSED PHASE III DEMONSTRATION STUDIES 100 REFERENCES 103 APPENDIX A: HIGHLIGHTS OF HIGH SCHOOL COMPLETION AND COLLEGE ENROLLMENT STATISTICS: APPENDIX B: 1992 TALENT SEARCH DESIGN CONFERENCES SUMMARY OF CONCLUSIONS 111 APPENDIX C: SUMMARY OF FIRST TECHNICAL REVIEW PANEL MEETING FOR THE NATIONAL EVALUATION OF TALENT SEARCH 117 APPENDIX D: MINU IES OF PROJECT DIRECTOR TELECONFERENCE 129 APPENDIX E TALENT SEARCH DATABASES IN FIVE STATES 133 Cotets

8 TABLES Table Page 1.1 TRIO FUNDING, NUMBER OF GRANTS, AND AVERAGE AWARD, AMOUNT PER PERSON SERVED AND NUMBER FUNDED TO SERVE TRIO FUNDING LEVELS IN CONSTANT 1999 DOLLARS STUDIES OF TALENT SEARCH AND EOC NUMBER OF TALENT SEARCH PROJECTS AND NUMBER OF PARTICIPANTS BY TYPE OF HOST INSTITUTION: NUMBER OF TALENT SEARCH TARGET SCHOOLS AND ESTIMATED NUMBER OF SCHOOLS SERVING TARGET AGE GROUP WITHIN THE STATE NUMBER OF MIDDLE AND HIGH SCHOOL STUDENTS SERVED BY TALENT SEARCH, NUMBER ELIGIBLE FOR THE FREE LUNCH PROGRAM IN GRADES 6-12 IN TARGET SCHOOLS AND ALL SECONDARY SCHOOLS, AND NUMBER SERVED BY TALENT SEARCH AS A PERCENTAGE OF THE NUMBER ELIGIBLE FOR THE FREE LUNCH PROGRAM IN THE TARGET SCHOOLS AND ALL SECONDARY SCHOOLS, BY STATE: ESTIMATES OF THE PERCENTAGE OF INCOME-ELIGIBLE CHILDREN SERVED BY TALENT SEARCH IN THE UNITED STATES USING CENSUS DATA ON POVERTY PERCENTAGES OF PROJECTS OFFERING SERVICE TO PARTICIPANTS, BY GRADE LEVEL: PERCENTAGES OF PARTICIPANTS RECEIVING SERVICES, AND AVERAGE NUMBER OF SESSIONS PER PROJECT: PERCENTAGES OF PARTICIPANTS RECEIVING EACH TYPE OF SERVICE, COMPARING PERFORMANCE REPORT DATA FROM AND COMPARISON OF APPROVED OBJECTIVES AND ACTUAL ACHIEVEMENTS FOR MIDDLE SCHOOL AND HIGH SCHOOL OUTCOMES :

9 TABLES Table Page 11.9 COMPARISON OF APPROVED OBJECTIVES AND ACTUAL ACHIEVEMENTS FOR POSTSECONDARY OUTCOMES: DISTRIBUTION OF POSTSECONDARY PLACEMENT OF PARTICIPANTS BY SECTOR: TALENT SEARCH PROJECT DIRECTOR RESPONSES TO QUESTION ASKING WHICH SERVICES THEY WOULD STUDY TYPE OF RECORDS ON PARTICIPANTS KEPT BY TALENT SEARCH PROJECTS TALENT SEARCH PROJECT VARIABLES PRESENT ON THE FEDERAL AID FILES RESULTS OF MATCHING STUDENT SUPPORT SERVICES NATIONAL STUDY STUDY WITH FINANCIAL AID FILES TALENT SEARCH PROJECTS IN TEXAS VARIABLES INCLUDED IN THE TEXAS PUBLIC EDUCATION INFORMATION MANAGEMENT SYSTEM VARIABLES INCLUDED IN THE TEXAS HIGHER EDUCATION DATABASE TALENT SEARCH PROJECTS IN INDIANA DATA COLLECTED ON THE STATE OF INDIANA'S ICPAC PROGRAM ENROLLMENT FORMS TALENT SEARCH PROJECTS IN FLORIDA FLORIDA DEPARTMENT OF EDUCATION (DOE) INFORMATION DATABASE REQUIREMENTS AUTOMATED STUDENT DATA ELEMENTS THE STATE UNIVERSITY SYSTEM BOARD OF REGENTS DATABASE (FLORIDA) - STUDENT DATA ELEMENTS 59 Cotets

10 xi TABLES Table Page THE STATE BOARD OF COMMUNITY COLLEGES DATABASE (FLORIDA) STUDENT DATA ELEMENTS TALENT SEARCH PROJECTS IN MASSACHUSETTS SUMMARY OF 11EMS INCLUDED IN THE MASSACHUSETTS STATEWIDE SCHOOL AND STUDENT MANAGEMENT INFORMATION SYSTEM TALENT SEARCH PROJECTS IN NORTH CAROLINA SUMMARY OF 11'EMS INCLUDED IN THE NORTH CAROLINA STUDENT INFORMATION SYSTEM SUMMARY OF ITEMS INCLUDED IN THE NORTH CAROLINA HIGHER EDUCATION DATABASE 67 V.1 EXPERIMENTAL DESIGN OF THE PLAN VARIATION STUDY 86 V.3 MINIMUM DETECTABLE DIFFERENCES (MDD) 91 V.4 SERVICE MIX DEMONSTRATION DATA COLLECTION PLAN 94 VI. 1 SUMMARY OF COST ESTIMATES 99 VI.2 TASKS AND DELIVERABLE SCHEDULE FOR PHASE II 100 VI.3 TASK STRUCTURE AND SCHEDULE FOR PHASE III 102 Cotets

11 FIGURES Figure Page I.1 EDUCATIONAL TALENT SEARCH WITHIN THE EDUCATIONAL SYSTEM POTENTIAL PATHWAYS FOR TALENT SEARCH TO INFLUENCE STUDENTS' PURSUIT OF POSTSECONDARY EDUCATION 12 II.1 PARTICIPANT DISTRIBUTION BY ELIGIBILITY STATUS, PARTICIPANT DISTRIBUTION BY GRADE LEVEL: PARTICIPANT DISTRIBUTION BY STATUS AND TYPE OF HOST INSTITUTION: III.1 TEXAS TALENT SEARCH PROJECTS BY NUMBER OF PARTICIPANTS AND TARGET LOCATIONS INDIANA TALENT SEARCH PROJECTS BY NUMBER OF PARTICIPANTS AND TARGET LOCATIONS FLORIDA TALENT SEARCH PROJECTS BY NUMBER OF PARTICIPANTS AND TARGET LOCATIONS MASSACHUSETTS TALENT SEARCH PROJECTS BY NUMBER OF PARTICIPANTS AND TARGET LOCATIONS NORTH CAROLINA TALENT SEARCH PROJECTS BY NUMBER OF PARTICIPANTS AND TARGET LOCATIONS OUTLINE OF THE STATE-LEVEL IMPACT REPORT 72 IV.1 OUTLINE OF THE NATIONAL IMPACT REPORT 80 V.1 DEMONSTRATION TIMELINE 87 VI. 1 LISTING OF TASKS AND TIME SCHEDULES 96

12 EXECUTIVE SUMMARY This report assesses the feasibility of three studies of the Talet Search (TS) program's impact o its three primary outcomes high school (HS) graduatio, applicatio for federal fiacial aid (FFA), ad erollmet i a postsecodary educatioal istitutio (PSE). The three studies are: The impact of TS withi each of several states. The impact of TS atioally. The impact of two specific supplemetary TS services. The first two studies fall ito Phase II of the Evaluatio of the Talet Search program (Cotract No. ED-98-CO-0073 with Mathematica Policy Research, Ic.), ad the third study would fall ito Phase III of that cotract. I summary, two of the three studies are foud to be feasible. To assess their feasibility, we prepared a desig for each study, icludig a careful statemet of the policy ad research questios addressed by the study, a experimetal desig, a samplig pla, a data collectig pla, a aalysis pla, ad a reportig pla. We assessed the feasibility of each compoet of the desig. For example, for the state impact study, we collected iformatio o the kidergarte through 12 th grade (K-12) database ad the higher educatio database maitaied by each state. If the state databases cotaied the eeded data coverig the appropriate populatio ad time period-- ad those databases were available to researchers, we deemed this part of the desig feasible. Each feasibility assessmet produced as a by-product the outlie of a desig for each study. These desigs should serve as the guidelies for coductig the studies i the future.

13 xvi A. STATE IMPACT STUDY Both the state ad atioal impact studies are desiged to focus o the impact of TS as a whole with limited aalysis of the impact of specific services. They address the questio, "Does TS improve the likelihood of graduatio, of applyig for FFA, ad of its participats' erollig i PSE, relative to the likelihood for studets who did ot participate i TS but may have received similar services from other programs?" The state impact study estimates TS's impact i each of several states. We examied the K-12 ad higher educatio databases i Massachusetts, Idiaa, Texas, Florida, ad North Carolia. Those states were amog 14 states cited by the report o state data systems by the State Higher Educatio Executive Officers (SHEEO, May 1999) as possibly havig databases sufficiet to support these studies. We called the state educatio agecy i each of the 14 states, idetifyig the five listed above as appearig to have sufficiet K-12 ad higher educatio databases. A review of the database documetatio from the five states revealed that the databases i Texas, Florida ad North Carolia appear able to support a impact study. Massachusetts implemeted a automated statewide K-12 database just this year. Sice it will ot cover the relevat time period, we do ot recommed coductig a impact study there. Idiaa's database is missig some variables pertaiig to HS performace ad family icome. However, we do ot view these shortcomigs as fatal, ad propose to estimate impacts i Idiaa. We foud the state impact study to be feasible, although ot without some risk. The risk is that we may ot be able to detect the true impact of TS because of limitig ourselves to admiistrative data. We believe the risk to be of a acceptable magitude, ad thus the study is feasible. B. NATIONAL IMPACT STUDY The atioal impact study addresses the same research ad policy questios as does the state impact study ad is iteded for the same audiece. The reaso for coductig both the state ad the atioal impact studies is that they have offsettig advatages ad disadvatages. The atioal impact study has the limitatios of havig a arrow rage of outcome variables ad less statistically power tha the state study but has the advatage of beig atioal i scope. The state impact study is more powerful--meaig it ca detect the true impact of TS eve if it is small--but it caot be geeralized to the atio as a whole. Uder this approach, studet addresses are culled from the FFA database ad liked to school districts by geo-codig school district boudaries, which we are ow doig. Oce FFA filigs are liked to school districts, we will lik TS projects to districts. Our recet survey of project directors recorded the umber of TS participats by target school. Executive Summary

14 xvii We will use the Commo Core of Data (CCD) to lik TS target high schools to school districts ad will assume all other high schools have o TS presece. The depedet variable is the proportio of graduatig seiors who submit a FFA applicatio. We compute the umerator of the proportio for a idividual district by coutig the umber of FFA applicatios liked to the school district. The deomiator of the proportio is the umber of HS completers, which we will extract from the CCD. We foud the atioal study ot to be feasible i the sese that the risk of ot detectig the true impact of TS is uacceptably large. This risk stems from the fact that we are limited to atioal admiistrative data. C. SERVICE MIX DEMONSTRATION The service mix demostratio has a somewhat differet purpose tha the state or atioal impact studies do. It does ot produce a estimate of the impact of TS as a whole. Istead, it estimates the impact of oe TS service mix relative to aother service mix. This meas the audiece for the service mix demostratio is TS project directors. The fidigs of the demostratio will provide -directors guidace o how best to desig their service packages. The demostratio estimates the impact of oe service ehacemet for program participats ad oe service ehacemet for eligible oparticipats. Oe ca thik of the service mix demostratio as two demostratios, oe for TS participats ad oe for eligible oparticipats. We propose that the fial choice of service mixes be decided i cosultatio with TS projects, PES, the Office of Federal TRIO programs ad the MPR study team. For purposes of discussio, we propose that the service ehacemet be metorig/tutorig. We combie metorig ad tutorig because collectively they represet services admiistered to a idividual, as distict from services admiistered to groups of studets. Providig such itesive, oe-o-oe services to participats with special eeds or barriers preset is the program desig issue most ofte metioed by TS project directors i our survey, ad is suggested as a critical compoet of youth programs by researchers i a wide rage of disciplies, icludig educatio, employmet ad traiig, juveile justice, ad adolescet psychology. Oe of the key aspects of the metorig/tutorig study is that both metorig ad tutorig will be defied i detail by ED's Plaig ad Evaluatio Service (PES) ad will be cosistet across projects. For the purposes of this study, metorig/tutorig will be provided by a sigle metor/tutor to a sigle studet, delivered over a exteded period, ad comprehesive, that is, it addresses i some fashio whatever barriers the youth faces, icludig health, substace abuse, deliquecy, housig, ad family problems. Executive Summary

15 xviii Typically, TS projects do ot provide services to eligible oparticipats. The eligible oparticipat demostratio estimates the impact of mailig to such youth a packet of iformatio o FFA ad college, ad followig up with a phoe call. The project would provide this service to eligible oparticipats i target schools. Thus, this portio of the demostratio assesses the efficacy of projects adoptig this more aggressive outreach to studets who are eligible for TS but have ot applied. The otio of TS projects possibly providig a low level of services to eligible oparticipats i target schools was suggested by project directors, cocered that may eligible youth were ot beefitig from what the projects had to offer. Ulike the state ad atioal impact studies, the service mix demostratio does ot rely o existig admiistrative data. It ivolves at least two surveys, a baselie survey ad a follow-up survey. Data from these surveys will be supplemeted by FFA data ad school records. A secod differece betwee the demostratio ad the state ad atioal impact studies is that the demostratio uses radom assigmet, rather tha a compariso group desig based o propesity score matchig. If possible, we will draw the cotrol group from the same target schools that treatmet studets are draw from. If that is ot possible, either because of high TS peetratio rates i target schools or because the project directors determie such a procedure to be logistically ifeasible, we will idetify a compariso target school for each treatmet school ad draw the cotrol studets from the compariso school. The service mix demostratio estimates the impact of ehaced services provided by TS projects ad does ot estimate the impact of TS as a whole, that is, operatig TS versus ot operatig TS. Ulike the state ad atioal studies, the demostratio uses survey data ad relies o a true cotrol group geerated by radom assigmet. Such a experimetal desig will give the demostratio the statistical power to detect eve weak impacts of the alterative services. We fid that the demostratio is feasible, is the most statistically powerful of the three studies, ad is also the most costly of the three studies. Executive Summary

16 CHAPTER I INTRODUCTION alet Search is oe of the oldest of the federal higher educatio programs desiged to icrease college access for low-icome youth ad o a yearly basis, TS reaches out to more studets tha ay of the TRIO-programs. Yet, util the curret plaed evaluatio, there has ot bee a impact assessmet of Talet Search (TS). I this documet we: (1) summarize results from our feasibility studies, ad (2) outlie the desigs for a three-proged approach to the outcomes evaluatio. The three parts of the proposed study are: State-level impact studies usig project, state, ad federal records Natioal impact study usig federal databases ad performace reports Selected site evaluatios of effectiveess of alterative services focused o a iformatio ad metorig/tutorig-based iitiative for juiors ad seiors We describe our ratioale for the proposed approaches ad outlie plas for implemetig the study. A. THE CONGRESSIONAL MANDATE FOR THE STUDY Cogress has authorized ad requested the U.S. Departmet of Educatio (ED) to evaluate TRIO programs, ad this study is beig plaed i respose to this cogressioal madate. The Natioal Study of Talet Search is the first study of Talet Search sposored by

17 2 the Departmet of Educatio i 20 years ad as idicated above it will be the first study to coduct a outcomes or impact evaluatio. The Higher Educatio Act of 1998, the authorizig legislatio for the TRIO programs, cotais the followig statemet cocerig evaluatios: I Geeral For the purpose of improvig the effectiveess of the programs ad projects assisted uder this chapter, the Secretary may make grats to or eter ito cotracts with istitutios of higher educatio ad other public ad private istitutios ad orgaizatios to evaluate the ffectiveess ofthe programs ad projects assisted uder this chapter. Practice The evaluatios described i paragraph (1) shall idetify istitutioal, commuity, ad program or project practices that are particularly effective i ehacig the access of low-icome idividuals ad first-geeratio college studets to postsecodary educatio, the preparatio of the idividuals ad studetsforpostsecodary educatio, ad the success of the idividuals ad studets i postsecodary educatio. Such evaluatios shall also ivestigate the effectiveess of alterative ad iovative methods withi Federal TRIO programs of icreasig access to, ad retetio of, studets i postsecodary educatio (H. R. 6, Sec. 402H). Evaluatios that focus o program improvemet are ot ew to TRIO programs, but the 1998 reauthorizatio added laguage callig for ivestigatio of the effectiveess of "alterative ad iovative methods withi TRIO programs." This laguage has iflueced the desigs we have put forth. The studies described i this report take together address (1) the estimated impact of Talet Search i icreasig college access, (2) the effectiveess of existig services i icreasig college goig, ad (3) the potetial impact of itroducig iovative services o college-goig outcomes. B. TALENT SEARCH PROGRAM BACKGROUND I the 1965 legislatio that first authorized the TRIO programs, Cogress oted that fiacial aid aloe would ot esure equal educatioal opportuity for disadvataged studets. So Cogress sposored the developmet of supplemetal services to prepare disadvataged studets for college ad help them to succeed oce there. I additio, Cogress called for the developmet of higher educatio istitutioal policies desiged to serve a more diverse populatio. I this regard, TRIO 's role was to be ot oly to provide direct services but also to serve as a model ad catalyst for the developmet of other similar services at the state ad local level. Talet Search is oe of the origial three TRIO programs, begiig i 1967 alog with Upward Boud. The third program, Studet Support Services, bega i Educatioal Opportuity Ceters (EOCs) started i 1974, ad staff-traiig grats were iitiated i The McNair Fellowship Program, which bega i 1989, is the most recet direct service I. Itroductio

18 3 TRIO program. I additio, ED fuds staff traiig ad program iovatio dissemiatio programs. 1. Goals ad Services The goals of Talet Search are to: Idetify qualified youths with potetial for postsecodary educatio, to ecourage them to complete secodary school ad to eroll i postsecodary educatio programs, to publicize the availability of studet fiacial aid, ad to ecourage secodary ad postsecodary school dropouts to reeter a educatioal program (Departmet of Educatio Aual Report 1997). EOCs were begu i 1974 with goals ad services similar to Talet Search but with a focus o servig a adult populatio icludig those that had left high school before completio. While the descriptive compoet of this study icludes a survey of EOC projects, the plaed impact ad demostratio studies are limited to Talet Search. I 1998 ew grats were made, icreasig the umber of Talet Search projects from 319 to 361. Durig the program year, 361 Talet Search projects atiowide provided services to about 300,000 participats, with total federal fudig of $98.5 millio. I the same period there were 83 Educatioal Opportuity Ceters servig about 157,000 participats with fudig of about $30 millio. Talet Search projects ad Educatioal Opportuity Ceters are operated by two- or four-year colleges, public or private oprofit agecies or orgaizatios, or combiatios of these sposors. Talet Search (TS) participats must be 11 to 27 years of age' ad have completed the fifth grade. I each local TS project, two-thirds of the participats must be low-icome studets who are potetially first-geeratio college studets. Ulike the other TRIO programs, the other oe-third, of participats eed ot meet the low-icome or first-geeratio criteria. TS services iclude: Academic ad persoal couselig Career exploratio ad aptitude assessmet Assistace with re-etry to high school ad college Iformatio o postsecodary educatio p Projects may serve cliets aged 28 or older if o Educatioal Opportuity Ceter is available to serve them ad doig so will ot dilute the services they provide to the mai target group. I. Itroductio

19 4 Iformatio o fiacial assistace Assistace i completig college ad fiacial aid applicatios, ad i preparig for admissios tests Exposure to a rage of career optios Tutorial services Cultural evets The major Departmet of Educatio Performace Idicator goal for TS is to icrease the "ratio of TS participats who apply to college or apply for fiacial aid compared to umber served." I recet years, two major iitiatives have bee udertake to reshape Talet Search. First, there has bee a icreased emphasis o project accoutability as maifest by ew performace reportig requiremets, while allowig projects greater flexibility i decidig how to deliver services. Secod, TS proejcts have bee ecouraged by ED to place more emphasis o icreasig the participatio of youger studets i the "college pipelie." 2. TS i the Cotext of Other TRIO Programs I preparig a desig for this study, it is useful to look at TS i the cotext of the other TRIO programs because TRIO programs are desiged to work together. TS ad EOCs are the least itesive of the TRIO programs ad serve the most participats. Talet Search was desiged to be a outreach program that cast a wide et to complemet other TRIO ad o-trio services. The cetral features of TS are the emphasis o school ad commuity outreach ad the heavy use of persoal, academic, ad fiacial aid couselig. As show by ED summary statistics, TS serves more studets a year tha ay other TRIO program. I 1999, the average TS project was fuded to serve 896 studets, ad the average EOC project served 2,117 people (Table 1.1). TS fudig averages $304 per studet. Withi TS, the typical staff-to-studet ratio is about 1 to 183 (COE 1993). As such, the outreach quality of the program both challeges the desigers of a impact evaluatio ad presets a opportuity for creative approaches. I. Itroductio

20 5 Table 1.1. TRIO Fudig, Number of Grats, ad Average Award, Amout Per Perso Served ad Number Fuded to Serve 1999 Average umber fuded to serve per project Number of Number Average Amout per 1999 Grat amout grats served award perso served Talet Search $98,450, ,541 $272,717 $ Educatioal Opportuity Ceters $29,794, ,063 $363,346 $ McNair $32,114, ,734 $205,859 $8, Studet Support Services $178,916, ,099 $224,770 $1, Upward Boud $220,500, ,960 $285,623 $4, Upward Boud Math Sciece $29,276, ,200 $236,099 $4, The fudig history of TRIO programs (Table 1.2) shows that while TS started from the smallest base compared with the other programs, fudig has icreased at a rate as high or higher tha the other TRIO programs. I costat dollars, TS fudig icreased more tha four-fold betwee 1970 ad _ The periods of icreased fudig for TS are similar to those of other TRIO programs. However, the icreases aroud 1991, whe servig youger studets i the seveth ad eighth grades became a priority, are especially apparet. TS ad EOC programs doubled fudig i costat dollars i I 1993, TS regulatios were further revised to allow studets who had completed at least the fifth grade to be eligible for participatio. While TS ad EOC do ot offer itesive services, practitioers have observed that these smaller itervetios ca be critical to a disadvataged studet. Table 1.2: TRIO Fudig Levels i Costat 1999 Dollars Year TS $12.43 $21.5 $18.6 $30.9 $32.1 $34.5 $85.7 $98.5 EOC $9.3 $15.6 $14.2 $15.2 $26.9 $29.8 UB $139.7 $127.1 $118.6 $126.4 $114.0 $128.2 $208.3 $220.5 SSS $42.9 $71.2 $121.3 $108.5 $115.9 $156.9 $178.9 McNair $3.8 $20.9 $32.1 Traiig $4.0 $2.0 $2.0 $2.2 $4.5 UB Math $29.1 SOURCE: Calculated from iformatio i U. S. Departmet of Educatio Aual Report, 1997; CPI 1997 US Statistical Abstract, Table 752. I. Itroductio

21 6 C. CONTEXT FOR THE STUDY: CHANGES SINCE THE 1960S IN NATIONAL STATISTICS RELATED TO EDUCATION ENROLLMENT Sice the implemetatio of the TRIO programs i the 1960s, a umber of social ad demographic chages have occurred i the atio, ad these will costitute the larger cotext i which the TS impact evaluatio is coducted. Viewed from a broad societal perspective, the ultimate goal of TRIO ad TS is to reduce the gap betwee low-icome ad high-icome studets i high school graduatio, college erollmet ad graduatio rates. Appedix A summarizes atioal statistics o high school completio ad college etrace sice the iitiatio of the TRIO programs. These data show that the rate of college cotiuatio of high school graduates has icreased substatially amog all socioecoomic status (SES) groups sice the late 1960s, from about 45 percet to a record high of 66 percet by the ed of the i 1990s (Postsecodary Educatioal Opportuity Jue 1999). While the percetage icrease i college-goig rates amog low-icome youth has bee larger tha amog high-icome youth, the gap betwee the rates of college attedace for low-icome ad high-icome youth remais large. Data from NELS88 show a gap of about 42 percetage poits betwee low ad high SES youth i college erollmet withi two years of high school graduatio. Usig NELLS88 data, the reported erollmet rates were 49 percet for low SES youth ad.91 percet for high SES youth i Data from the High School ad Beyod Study i the 1970s reported rates of 42 percet for low SES youth ad 85 percet for high SES youth. Cesus Bureau data from 1998 idicate that amog high school graduates, about 86 percet of high-icome youth ad about 56 percet of low-icome youth atteded college withi two years. Differeces were larger if oe cosiders the total age cohorts ad the chaces of college completio by age 24. Estimates are that about 5 percet of those i the lowest ecoomic quartile will receive a bachelor 's degree by age 24, while about 50 percet of those i the highest quartile will receive a bachelor ' s degree by age 24 (Postsecodary Educatio Opportuity December 1998). Similarly the type of college oe atteds is highly correlated with icome. There is also evidece that the type of college atteded has become more stratified by icome over time (McPherso ad Schipiro 1998). At a time whe less tha 10 percet of low-icome studets ca expect a bachelor's degree, surveys of college aspiratios repeatedly fid that low-icome studets report high aspiratios for college. A recet survey of all 11 th graders i Idiaa foud that more tha two-thirds expected to obtai a bachelor ' s degree ad oly 5 percet had o expectatios to obtai ay postsecodary educatio (Idiaa Guidace Report for 11 th Grade Survey 1998). At the same time, aalyses have show that low-icome studets are much less likely to take college preparatory courses ad that these courses are highly correlated with eterig ad succeedig i college (Natioal Ceter for Educatio Statistics 1997; Adlema 1999). Sigificat shifts have, however, occurred i the college cotiuatio rates of wome, ad i the icreased college participatio of racial ad ethic miorities. College cotiuatio rates for wome are ow at about 70 percet compared with 61 percet for I. Itroductio

22 7 me. Racial ad ethic miorities take together ow make up about 30 percet of udergraduate erollmets compared with about 17 percet i the 1970s (HEGIS 1974; IPEDS 1996). These treds mea that the demographic compositio of college erollees has chaged sigificatly sice the iitiatio of the TRIO programs. A equally importat chage i the last 20 years is the rapid itroductio of persoal computers ad Iteret commuicatio, that are sigificatly chagig the atio's educatioal system ad are affectig how TS ad EOCs provide services. The umber of computers i the atios' public schools wet from 125 studets per computer i 1983 to 9 studets per computer i 1996 ad is droppig rapidly (QED, Glea ad Melmed 1996). The proposed TS evaluatio will be coducted i the cotext of these treds ad chages. Because there has ot bee a atioal impact evaluatio of the TS program, we do ot kow what role the program has played i these statistics. Commo sese ad the testimoy of studet participats tell us that it has played a positive role but they also tell us that the program is actig withi a system i which there are other similar programs ad other factors that exert both positive ad egative iflueces o college-goig rates. It may be that the impact of TS ad other, similar programs has bee to help keep the gap betwee the high- ad low-icome rates of college attedace withi a certai rage. It may also be that the role played by TS i the previous 35 years eeds to be reexamied to esure that it is the model best suited for the future. D: PREVIOUS STUDIES OF TALENT SEARCH Table 1.3 lists the major studies coducted of Talet Search over its history. The plaed study is uique i that it would be the first atioal study of TS that icludes the possibility of a impact study. Previous atioal studies of TS have bee descriptive studies, or they have bee combied with studies of other programs, such as the evaluatio of Upward Boud ad Talet Search coducted i the 1970s (Burkheimer, Riccoboa, Wisebaker 1976) ad the College Board study of the mid-1980s (Frakli 1985). I 1992, ED commissioed six desig papers lookig at the potetial for a impact study of Talet Search. A summary of these papers is icluded i Appedix B. These studies poited out the difficulty i implemetig a radom assigmet study for Talet Search, but oted the potetial for a compariso group desig focused o short-term impacts. I 1994, ED cotracted with Decisio Iformatio Resources (DIR) to coduct a small descriptive study of TS to look at feasible measures of program performace criteria (Jackso, Arboa, ad Brach 1994) ad to prepare a review of target populatio eeds ad effective itervetios (Arboa ad Jackso 1994). This study cocluded that it would be possible for projects to keep records of participats ' services ad do limited trackig of outcomes. I additio, the Natioal Coucil of Educatioal Opportuity Associatio (NCEOA) sposored a literature review ad a survey of Talet Search ad Upward Boud i the early 1990s (NCEOA 1993; Nettles ad GetzFeld 1992). Results of these studies are summarized i the literature review documet prepared for the atioal study (Silva ad Kim 1998). I. Itroductio

23 8 Table 1.3: Studies of Talet Search ad EOC Author(s) Ad Date Brief Descriptio of Study Pyecha et al Frakli 1985 Coles 1992 This was the first atioal evaluatio of Talet Search, coducted for the U.S. Office of Educatio by Research Triagle Istitute. It icluded a survey of all 114 project directors (respose rate 92 percet), a survey of the postsecodary istitutios o the erollmet status of a sample of former program participats (respose rate 93 percet), ad case studies of 20 projects. Primarily focused o 11 purposively selected Talet Search projects, usig a mail survey, telephoe iterviews, ad documet review. Also drew o a limited amout of atioal level data--aual Performace Reports--from ED for (percetages obtaied ureported). Coducted for the College Etrace Examiatio Board. I-depth iterviews with the directors of 19 purposively selected Talet Search projects. Prepared for the Desig Coferece for the Evaluatio of the Talet Search Program, hosted by the U.S. Departmet of Educatio, Office of Policy ad Plaig, September 30, Eiser 1992 A review of data from Aual Performace Reports for ad Data obtaied for a estimated two-thirds of the Talet Search fuded i ad for 92 percet of the 177 projects fuded i Icluded i the report from the Talet Search Desig Coferece. Lee ad Clery 1993 A mail survey of all 294 Talet Search projects operatig i 1992 (respose rate 72 percet). Coducted for the Natioal Coucil of Educatioal Opportuity Associatios. Decisio Iformatio Case studies of seve purposively chose Talet Search projects, focusig o Resources (DIR) curret ad potetial program performace measures. Coducted for the U.S Departmet of Educatio, Office of Plaig ad Evaluatio. E. STUDY OBJECTIVES, COMPONENTS AND RESEARCH QUESTIONS As stated i the statemet of work for the project, the study has the followig overall objectives: To cotribute to strategic thikig of how best to improve high school graduatio rates ad access to college for disadvataged studets To uderstad what role federal policy ca play i improvig ad promotig educatioal opportuity for disadvataged studets I. Itroductio

24 9 To provide updated iformatio o the operatios ad services that TS ad EOC provide. 1. Compoets of the Study To achieve these objectives the overall study ivolves three major elemets: 1) A descriptive study of TS ad EOC utilizig iformatio from: - Descriptive surveys of all TS ad EOC projects that are ogoig ad expected to be completed i September of Studet-cetered case studies of 14 sites doe i the period of Sprig 1999 to March 2000 Aalysis of the most recet performace reports curretly beig fialized Aalysis of the characteristics of Target Schools from matchig with Commo Core of Data (i process) Iterviews with former_ project directors from sites that are o loger fuded 2) A study to look at the feasibility of coductig a impact assessmet for TS utilizig iformatio from the above oted compoets (for which this documet is the report) 3) If feasible, a impact study of the program. The study is cumulative i that the descriptive phase iforms the impact study phase. Both phases will build o the fidigs of related work o servig disadvataged studets. The focus of the impact study will be the implemetatio of a impact desig to observe short-term impacts. 2. Study Questios The study's desig would address the followig questios, which are derived from the cogressioal madate ad from our ow assessmet of what might be the best use of the evaluatio resources set aside for the program: What role does the Talet Search Program ow play i icreasig college access ad success of low-icome youth? How ca the program be improved? How effective are selected alterative services i icreasig college access? What is the most effective timig ad mix of services? I. Itroductio

25 10 How do Talet Search projects act i cosort with other college access iitiatives ad programs? F. THE CONCEPTUAL FRAMEWORK FOR THE STUDY A key compoet of the study's framework is recogizig that Talet Search operates withi a system ad that the impact study must look at the role of TS withi that system. Figure 1.1 graphically locates Talet Search withi the educatioal systems i which it operates. This figure lists other services ad activities that may also affect a studet's college decisio. TS projects are desiged to operate i cosort with these services. The figure also lists obstacles that TS ad other programs face i attemptig to foster icreased postsecodary attedace. Figure 1.2 portrays the pathways through which Talet Search iflueces a studet's pursuit of postsecodary educatio, icludig family factors, regular school guidace, other precollege programs, peers, commuity orgaizatios, ad postsecodary istitutios themselves. This coceptual framework relates to selected metaphors from a systems theory approach. These iclude viewig TS programs as part of a iteractig system with recogitio that: Relatively small chages i coditios may result i sigificat differeces i outcome. The timig of service may be a importat determiat of outcome. The targetig of services to specific groups (for example those who are udecided i later years i high school as to whether they wish to atted college or ot) may be a way to icrease the impact of TS. The matchig of services to umet eeds may be a way to icrease the impact of Talet Search. There is a eed to address the iteractio of TS with other pre-college services ad to assess the role that TS has played i fosterig istitutioalizatio of services withi the schools served. I. Itroductio

26 11 Figure 1.1: Educatioal Talet Search Withi the Educatioal System Postsecodary Educatio System Related Services TRIO (UB, SSS) 4 Other pre-college services Educatoal Talet Search Program Academic skills Decisio makig skills Motivatio developmel Resources iformatio 4-Year College 11th ad 12th Grades 2-Year College 4 Local School System Preschool Elemetary School Middle School Academic skills College awareess Metorig Tutorig Couselig Early High School Trasitio to HS Course selectio Decisio makig Academic skills SAT/ACT prep Applicatios for fiacial aid Career goals College matchig Couselig Course selectio Other Optios rr EOC Program Low self esteem Negative school experieces Trackig out Drugs/alcohol Family factors Pregacy Need to work Obstacles Lack of skills Lack of motivatio Lack of resources Poor metal health Poor physical health Behavior problems Lack of iformatio Other competig optios Academic skills Study skills Decisio makig skills Retur couselig Motivatio developmet Resources Iformatio GED I. Itroductio

27 12 Figure 1.2: Potetial Pathways for Talet Search to Ifluece Studets' Pursuit of Postsecodary Educatio Family Factors Regular School Guidace ad Assistace Other Precollege Services Peer Group Factors - Commuity Groups/ Orgaizatios Studets ad Outof-School Youth SHORT-TERM STUDENT OUTCOMES Educatioal Expectatios Attitudes Toward School Kowledge of Prerequisite Courses ad Skills Idetificatio of College Optios ad Career Iterests Erollmet i Academic Program, Receipt of Academic Credits/H.S. Degree Receipt of Special Services to Improve Skills, (Tutorig, AP Class, SAT/ACT Prep.) Awareess of Fiacial Aid Optios ad Procedures Applicatio for Fiacial Aid Applicatio for College Admissio, Takig College Admissio Tests Acceptace by College/Receipt of Fiacial Aid I. Itroductio

28 13 We discuss these poits i more detail i Chapter II whe we describe the coclusios of the feasibility studies. G. STRUCTURE OF THE REMAINDER OF THE REPORT Chapter II cotiues the itroductory material by summarizig results from the feasibility studies that have bee used to iform the desigs preseted i Chapters III to V. Chapters III ad IV preset desigs for a Phase II of the study callig for a impact study that utilizes existig records data. Chapter III presets plas for a Phase II study i four selected states. Chapter IV discusses a pla for a exploratory atioal impact study usig federal databases ad existig atioal data from performace reports ad the project survey. This is referred to as the Natioal Study compoet of Phase II Chapter V presets a pla for a Phase III evaluatio of the impact of alterative services desiged to icrease the targetig outreach of TS ad the itesity of services for a radomly assiged groups of eligible studets. I. Itroductio

29 CHAPTER II STUDIES USED TO INFORM THE FEASIBILITY REPORT To iform the feasibility assessmet ad the desigs preseted i Chapters III to V, MPR collected ad utilized iformatio o Talet Search from a umber of sources. I this chapter we summarize this iformatio ad idicate how these ivestigatios led us to the proposed research desigs. The sources for this chapter are: Aalysis of Performace Reports for 1999 ad aalysis of Talet Search target schools Cosultatio with Talet Search project directors ad Project Director Survey results Case studies of 14 projects Exploratory matchig of TRIO participat samples from the Natioal Evaluatios of Studet Support Services ad Natioal Evaluatio of Upward Boud with atioal federal aid files Exploratio of state-level trasitio databases that track the progress of studets from high school to postsecodary educatio. A. PERFORMANCE REPORTS AND TARGET SCHOOL ANALYSIS I 1999 Talet Search Projects completed a ewly revised performace report form. These reports provide iformatio o Talet Search participats, services, objectives ad self-reported achievemet of outcomes. I additio, projects submitted ames of each

30 16 Target School served, ad Mathematica Policy Research (MPR) matched this iformatio with the U.S. Departmet of Educatio (ED) Commo Core of Data (CCD). This matchig eabled us to look at the characteristics of schools served ad allowed us to estimate the proportio of eligible studets beig served. A full report has bee prepared by MPR (Kim ad Cahala 2000) ad is uder ED review. Here, we ote some highlights that have helped iform the feasibility study. 1. Project ad Participat Demographics I 1999, there were 361 Talet Search projects. Forty-four percet were hosted by 4-year-ad-above degree-gratig postsecodary istitutio ad 34 percet were located at 2-year degree-gratig postsecodary istitutios. The remaiig 19 percet were hosted by commuity orgaizatios ot affiliated with a college or uiversity (Table 11.1). Together the projects served 307,451 participats i Table 11.1: Number Of Talet Search Projects ad Number of Participats By Type Of Host Istitutio: Sector Total projects i 1998 Percetage of total TS projects Project respose rate Number of participats reported Percet distributio Public 4-year % 98% 104,372 34% Private 4-year 48 13% 96% 36,480 12% 2-year % 97% 90,217 29% Commuity orgaizatios 68 19% 98% 76,382 25% All projects % 98% 307, % Source: Calculated from Uited States Departmet of Educatio, Office of Federal TRIO Programs, Talet Search Performace Reports, ad Itegrated Postsecodary Educatio System, Istitutioal Characteristics File, Eligibility Status. As discussed i Chapter I, Talet Search eligibility regulatios require that that two-thirds of participats be low-icome ad potetially first-geeratiocollege or disabled. Ulike the other TRIO programs that require that the other oe-third be either low-icome or first geeratio, Talet Search does ot have specific requiremets for the last oe-third of participats. Nevertheless, examiatio of the performace reports idicates that overall 95 percet of Talet Search participats were low-icome or potetially first-geeratio college studets. I 1999, 74 percet of participats were both low icome ad first geeratio. Seve percet of TS studets were low-icome oly ad 14 percet were first-geeratio oly. Oly 5 percet met either of the criteria (Figure 11.1). II. Studies Used to Iform the Feasibility Report

31 17 Figure Participat Distributio by Eligibility Status, 1999 First geeratio oly Other 14% 5% Low-icome & first geeratio college 74% Source: Calculated from Uited States Departmet of Educatio, Office of Federal TRIO Programs, Talet Search Performace Reports, Sex ad Ethicity. As with the other TRIO programs, more females participated tha males. I 1999, 60 percet of those served were female. Just over oe-third (36 percet) were black or Africa America, 32 percet were white, 22 percet Hispaic or Latio, 4 percet Asia, 4 percet America Idia or Alaska Native ad 1 percet Native Hawaiia or Other Pacific Islader. Grade Level of Participats. I the last decade, ED has ecouraged Talet Search projects to exted services to middle schools, ad i 1999 about oe-third of participats were from grades 6 to 8. The majority of the participats, however, cotiue to be i high school. I 1999 two-thirds of participats were i seior high ad almost oe-fourth were i the 12`h grade (22 percet). About 5 percet were high school or postsecodary dropouts or high school graduates (Figure 11.2). II. Studies Used to Iform the Feasibility Report

32 18 Figure 11.2: Participat Distributio by Grade Level: Postsecodary dropout High school (or GED) graduate Secodary school dropout 12th grade oly 9th-11th grade 6th-8th grade 0% 10% 20% 30% 40% 50% Source: Calculated from Uited States Departmet of Educatio, Office of Federal TRIO Programs, Talet Search Performace Reports, Participat Status. Just uder half of the participats i 1999 were ew participats (48 percet) ad just over half (52 percet) were cotiuig participats, meaig they had bee i the program at some time prior to the reportig year (Figure 11.3). Figure 11.3: Participat Distributio By Status ad Type of Host Istitutio: % 80% 60% 40% 20% 0% Public 4- Private 4-2-year Commuity All projects year year orgs. Source: Calculated from Uited States Departmet of Educatio, Office of Federal TRIO Programs, Talet Search Performace Reports, II. Studies Used to Iform the Feasibility Report

33 19 2. Target Schools ad Percetage of Eligible Served Usig iformatio from the Performace Reports, we were able to determie which schools atioally were served by Talet Search (see Table 11.2). I 1999, Talet Search served about 5,094 high schools across the atio a average of 14 per project. This umber represets about 13 percet of the schools i the atio servig the target age populatio of studets (defied here as ay school servig ay studets i the 7th grade or higher.) Table 11.2: Number of Talet Search Target Schools ad Estimated Number of Schools Servig Target Age Group Withi the State Estimated umber of State Number of Talet Search target schools middle ad seior high schools i the state* Estimated percetage of middle ad seior high schools served by Talet Search* Alabama % Alaska % Arizoa % Arkasas % Califoria 306 3,536 9% Colorado % Coecticut % Delaware % District of Columbia** 149** 53** 281%** Florida 123 1,294 10% Georgia % Hawaii % Idaho % Illiois 196 2,289 9% Idiaa % Iowa % Kasas % Ketucky % Louisiaa % Maie % Marylad % Massachusetts % Michiga 91 1,466 6% Miesota % Mississippi % Missouri % Motaa % Nebraska % Nevada % New Hampshire % New Jersey % New Mexico % New York 150 1,721 9% North Carolia % North Dakota % Ohio 134 1,613 8% Oklahoma 185 1,083 17% Orego % II. Studies Used to Iform the Feasibility Report

34 20 Table 11.2: Number of Talet Search Target Schools ad Estimated Number of Schools Servig Target Age Group Withi the State Estimated umber of middle ad seior high schools i the state* Estimated percetage of middle ad seior high schools served by Talet Search* Number of Talet State Search target schools Pesylvaia 147 1,200 12% Rhode Islad % South Carolia % South Dakota % Teessee % Texas 283 3,312 9% Utah % Vermot % Virgiia % Washigto % West Virgiia % Wiscosi % Wyomig % Outlyig Areas % Total ,817 13% *All public schools that have ay erollmet at the 7th grade or higher were icluded i this cout derived from the Commo Core of Data for The cout icludes K-8 ad K-12 schools. Natiowide there were about 20,000 secodary schools, 10,000 middle schools, 51,000 elemetary schools about 5,000 combied ad other schools i the atio (Digest of Educatio Statistics, 1998). **Figures for the District of Columbia reflect that a program admiistered i DC has target schools across the atio Percetage of Low-Icome Studets Served. Matchig the iformatio o Target Schools from the performace reports with the CCD allowed us to estimate the percetage of low-icome studets (defied as eligible for free luch) served by Talet Search withi the Talet Search Target Schools. We also estimated the percetage of low-icome studets served by Talet Search i all schools servig the target age groups withi the state ad the atio (Table 11.3). II. Studies Used to Iform the Feasibility Report