Agenda. American Institutes for Research. Oregon 21st Century Community Learning Center (CCLC) Needs Assessment. Findings from the First Year

Transcription

1 Oregon 21st Century Community Learning Center (CCLC) Needs Assessment Findings from the First Year Jessica Newman Research Associate Fausto Lopez Technical Assistance Consultant Agenda Introductions Understanding continuous improv ement processes The current needs assessment project Background Design Findings from the first year Next steps 2 American Institutes for Research Established in 1946, with headquarters in Washington, D.C., American Institutes for Research (AIR) is an independent, nonpartisan, not-for-profit organization that conducts behavioral and social science research and delivers technical assistance both domestically and internationally. As one of the largest behavioral and social science research organizations in the world, AIR is committed to empowering communities and institutions with innovative solutions to the most critical challenges in education, health, workforce, and international development. 3 1

2 Learning Supports Network AIR s Learning Supports Network represents the system of supports for education agencies, schools and districts, f oundations, and other stakeholders to ensure that young people s educational experience fosters positive growth and dev elopment. The network includes services related to afterschool and expanded learning, social and emotional learning, and the Good Behavior Game AIR s classroom behav ior management strategy. 4 Afterschool and Expanded Learning The Af terschool and Expanded Learning team at AIR has more than a decade of experience in supporting the implementation of high-quality opportunities for young people, evaluating afterschool initiatives using qualitative and quantitative techniques, and supporting informed policy decisions. Team members are experts in building continuous improvement systems, and they strive to prov ide practitioners with meaningful linkages between research and practice in afterschool and expanded learning programs. 5 Who is in the room? 6 2

3 Background: AIR and ODE 2011: AIR and Gibson Consulting Group, Inc. conduct a comprehensiv e ev aluation of the 21st CCLC program Designed to answer questions about program implementation and program impact: What organizational processes are drivers of instructional/point-of-service quality at high-performing centers? To what extent are participants demonstrating better performance on the outcomes of interest when compared with similar students who are not participating in the program? 7 Understanding Continuous Improvement Processes 8 What is continuous improvement anyway? Develop programs using sound processes Implement those programs by employ ing quality practices Evaluate the implementation of the program Use the data f rom the ev aluation activ ities to continually improve the program. 3

4 Continuous Improvement Components 1. Vision 2. Assessment 3. Data 4. Prof essional Dev elopment 10 Continuous Improvement The driv ing f orce behind continuous improv ement is data Formal data (evaluation, attendance tracking system) Informal (anecdotal data) What Data Do We Collect? 4

5 Data for Continuous Improvement Qualitative Anecdotal success stories Focus groups Interviews Observ ations Self -assessment Other documents (newsletters, meeting minutes, and other sources of inf ormation Quantitative Surv eys Collection of demographic inf ormation about participants in the program Inf ormation reports (grades, test scores, comparisons of crime statistics, detention reports) Data for Continuous Improvement Make an ef f ort to include inf ormation about each of the f ollowing categories: Program deliv ery (e.g., attendance, activ ities, etc.) Key stakeholder satisf action (e.g., students, teachers, parents) Program outcomes (both short-term and long-term) How can and do we use the data? 5

6 Oregon s 21st CCLC Needs Assessment 16 Needs Assessment Questions What opportunities and quality improv ement ef forts currently exist among 21st CCLC grantees in Oregon? What do grantees need (in terms of capacity and other supports) in order to ef f ectively implement continuous improv ement processes in all programs? What supports can the Oregon Department of Education and their partners prov ide to grantees to support the implementation of continuous improv ement processes? 17 Project Design Partners ODE Education Northwest Needs assessment advisory board Timeline and Plans Year 1: Data collection and findings Year 2: Recommendations, pilot* implementation, ongoing support, and feedback loop 18 6

7 Data Collection Efforts to Date Summer, 2013: Needs sensing activities (Education Northwest) October, 2013: Fall conference listening sessions (AIR) February, 2014: Needs assessment survey (AIR) 19 Fall Conference Listening Sessions Four listening sessions throughout the day 71 participants Variety of roles, primarily PDs and SCs Dichotomy of needs between f irst and f if th year grantees 20 Needs Assessment Survey Online 181 respondents (est. response rate of 56%) First-y ear grantees (81%) Site coordinators (41%), project directors (24%), Elementary schools School-based programs Distributed ev enly across Oregon; slightly higher in Portland Rural () 21 7

8 Needs Assessment Findings Current Program Implementation 22 Visioning A v ision is the big picture, v alues-based idea y ou and y our stakeholders identif y. It is the ideal picture of what you will accomplish, either as an organization or at the programlev el. Many groups of ten go through a v isioning process to determine what that big picture is. This process may include activ ities such as identif y ing the mission, dev eloping organizational or program goals, creating a logic model or theory of change, or coming up with a program plan. 23 Visioning: Questions Hav e y ou participating in v isioning activ ities? What ty pes of activ ities? What was the v ision based on? How were y ou inv olv ed? Who else was inv olv ed? What ty pes of support would y ou like in this area? 24 8

9 Visioning: Findings Grantees hav e experience in some f orm of v isioning activ ities (74% according to surv ey ) 25 Visioning: Findings Participation in Visioning-Related Activities 66% 51% 37% 35% 25% Meetings Feedback on vision/concepts Document revi ew Engaging partners Leading the process 26 Visioning: Findings Sources of Information for Visioning Activ ities 56% 56% 54% 47% 42% 31% Gr ant/ fund ing Nee ds Staff Pare nt Yout h Com mu nity re quir eme nts asse ssme nt sur veys/f eed back sur veys/f eed back sur veys/f eed back sur veys/f eed back re sults 27 9

10 Reflection 28 Program Quality Assessment Also known as observ ation, ref lection, or self -assessment tools. Program quality assessment tools allow programs to identify their strengths and weaknesses using a structured and v alidated instrument. Programs can use tools to do a self -assessment, which prov ides opportunities f or ref lection, or they can work with an external observ er to conduct the assessment. 29 Program Quality Assessment: Questions Do y ou use a f ormal tool? What do y ou use? What other tools are y ou f amiliar with? How do y ou use the tool? Why did y ou choose the tool? Successes? Challenges? Areas of support? 30 10

11 Program Quality Assessment: Findings Use of a f ormal assessment tool 35% are using a formal tool 65% are not but are using data for improvement efforts Data is used primarily f or program planning (64%), improv ement (62%), and prov iding f eedback () 31 Program Quality Assessment: Findings 47% 36% Formal Tool Use Among Grantees Other OregonASK Pr ogram Obser vation Tool (POT) - National After School Association 7% 5% 3% 2% 2% 2% Out-of-School Time Obser vation Tool (OST) - Policy Studies Associates, Inc. Pr omising Pr actices Rating Scale (PPRS) - Wisconsin Center for Education Research and Policy Studies Associates, Inc. Youth Program Quality Assessment (YPQA) - David P. Weikart Center for Youth Program Quality Pr ogram Quality Obser vation Tool (PQO) - Debor ah Lowe Vandell and Kim Pierce Quality Assurance System (QAS) - Foundations, Inc. 32 Program Quality Assessment: Findings Reasons for Choosing Formal Assessment Tool 38% 33% 24% 19% 12% Existing tool Prior knowledge Preference Funding requi rement Cost 33 11

12 Reflection 34 Data Management Includes data collection, data use, and information sharing. Data can include any information on the youth in your program. Examples are attendance, student satisfaction, student academic inf ormation (grades, assessment scores), behavior incidents, among others. Inf ormation sharing can include providing data on youth progress or attendance to partners, sharing findings or analysis from an ev aluation with partners, funders, or families, among other instances. This is a broad category that can include a variety of ty pes of information and scenarios 35 Data Management: Questions Do y ou collect data? What do y ou collect and how do y ou use it? Why do y ou collect it and who does the collecting? Do y ou share inf ormation, why, and with whom? Successes and challenges? Areas of support? 36 12

13 Data Management: Findings 97% of surv ey respondents indicate their program collects data Programs primarily use computer programs (e.g., Microsof t Excel or Access); some store data on paper, some use a Management Inf ormation Sy stem (MIS) Internal data work with some external supports Site coordinator is main person responsible 37 Data Management: Findings Types of Data Collected by Programs 92% 66% 62% 59% 55% 55% 43% Progr am School-d ay Studen t grad es Discipline Youth Paren t Youth atte ndance atte ndance satisfactio n satisfactio n eng ageme nt 38 Data Management: Findings Data Use Among Programs that Collect Data 87% 51% 51% Evaluating program effe ctiveness Progr am imp rovem ent Storyte ling Mee ting pa rent a nd stud ent re quests 39 13

14 Reflection 40 Professional Development Prof essional dev elopment opportunities are opportunities that are designed to improve staff members or programs capacities in a v ariety of way s. Prof essional dev elopment includes technical assistance and coaching opportunities. These can occur in a variety of formats such as conf erences, workshops, webinars, indiv idual coaching, and peer-to-peer learning, among others. These opportunities can hav e a specif ic f ocus in terms of content area or skill (like STEM or data use) or can be much more general. 41 Professional Development: Questions Hav e y ou had opportunities to participate in PD? What ty pes hav e y ou participated in and what was most usef ul? Hav e y ou had opportunities f or peer-to-peer learning? Technological capacity and pref erences? What do y ou want more of? What could be improv ed? Successes and challenges? 42 14

15 Professional Development: Findings Almost all hav e access to appropriate technology Almost all hav e participated in PD/TA during tenure with 21st CCLC program Pref erence f or in-person PD/TA 43 Professional Development: Findings Participation in Different PD/TA Activ ities 71% 67% 57% 53% 49% 26% 15% 13% Presen tations at confe rence s Small gro up sessions a t confe rence s Worksho ps Webinar s Peer- to-pe er lear ning Te chnical assistance Individu alized coach in g Classes 5% Othe r 44 Professional Development: Findings Participation in Different PD/TA Activ ities 71% 67% 57% 53% 49% 26% 15% 13% Presen tations at confe rence s Small gro up sessions a t confe rence s Worksho ps Webinar s Peer- to-pe er lear ning Te chnical assistance Individu alized coach in g Classes 5% Othe r 45 15

16 Professional Development: Findings Most Useful PD/TA Activ ities 88% 77% 72% Individualized coaching Peer-to-peer learni ng Workshops 46 Professional Development: Findings Most Useful PD/TA Activ ities 88% 77% 72% Individualized coaching Peer-to-peer learni ng Workshops 47 Professional Development: Findings I f eel that more practical inf ormation can be gleaned f rom observ ation in a peer-to-peer f ormat than f rom a presentation at a conf erence. Workshops also prov ide opportunities f or hands-on learning, and one-on-one interactions. I f ind the webinars that we hav e been inv olv ed with hav e been unhelpf ul On the other hand, small workshops [ hav e giv en] me real lesson plans and kits that I used in my planning

17 Peer-to-Peer Learning: Findings Peer-to-Peer Learning Opportunity Formats 66% 54% 44% 41% 33% Staff meetings Small group conference sessions Informal networking Conference networking Peer learning groups Site visit partners 49 Peer-to-Peer Learning: Findings Peer-to-Peer Learning Cohort Attributes 58% 57% 46% 38% 35% 29% 21% 15% Staff role Academic focus Pr ogram type Grade level served Grant cycle Region Size of program/school Geographic characteristics 50 Peer-to-Peer Learning: Findings 3% 31% 66% Learner Only Learner & Mentor NA 51 17

18 Professional Development: Findings Nearly one quarter hav e not had an opportunity to participate Lack of awareness Limited time Irrelevant opportunities Limited funds 52 Reflection 53 Needs Assessment Findings Grantee Needs 54 18

19 Grantee Needs Time Plans and protocols Ongoing opportunities f or prof essional dev elopment 55 Visioning: Needs Time bef ore the start of the program Process f or dev eloping a v ision Training f or grantees 56 Program Quality Assessment: Needs More time Plans f or tool selection, implementation, and associated data management More training Training Needs (Not (Currently Using Using a Tool) a Tool) 83% 82% 69% 57% 41% 35% Gaining funding Using data Tool use Iden tifying a to ol To ol use Using da ta Gaining funding 57 19

20 Data Management: Needs More time Specif ic tools f or data collection Processes f or data collection and management Streamlined data collection processes among f unding agencies and other stakeholders More training Data collection instruments Training Needs 36% 34% 34% Ensuring data Knowing what is reliable data to collect Data analysis 58 Data Management: Needs More time Specif ic tools f or data collection Processes f or data collection and management Streamlined data collection processes among f unding agencies and other stakeholders More training Training Needs: Recommended tools and processes Best practices and success stories 59 Professional Development: Needs Specif icity and relev ance Hands-on and interactiv e Usable tools and resources Peer-to-peer learning 60 20

21 Professional Development: Needs STEM /S TE AM Social and emotional wellness Math Positiv e Youth Developm ent Evaluation Data use Quality improvement Comm on Cor e Reading/E LA ELL School partnerships Health and healthy behaviors Grade-specific Safety tr aining 3% 12% 22% 18% 18% 18% 17% 16% 16% 15% 27% 26% 61 Next Steps Dev elop and ref ine a set of recommendations f or continuous improv ement processes f or all grantees Construct plan f or implementation Identif y pilot group* f or implementation Prov ide ongoing support f or implementation (Ed NW, AIR) Conduct f ollow-up interv iews *Pilot group is dependent upon recommendations 62 Questions? 63 21

22 Jessica Newman Fausto Lopez North Wacker Drive, Suite 1231 Chicago, IL General Information: