Aligning Grant Management and Evaluation to Achieve Desired Program Outcomes

Transcription

1 Aligning Grant Management and Evaluation to Achieve Desired Program Outcomes

2 Convener: Sandi Sheppeard PA State GEAR UP Director PA State System of Higher Education, Harrisburg, PA Presenters: Diana Fischer PA State GEAR UP Coordinator PA State System of Higher Education, Harrisburg, PA David Jochelson Project Manager College Access Programs FHI 360, Washington, DC

3

4 What is your most important lesson learned regarding GEAR UP Evaluation or Grants Management?

5 Learning Objectives Understand the multiple evaluation data sources and strategies Understand integrated quarterly evaluation and grant management process Understand SMART Goals used in action plans Learn how evaluation report results are shared Understand future evaluation process changes

6 Overview of the Session Introduction and Background Information Planning and Implementation Reporting and Utilizing Feedback from Reports Conclusion Questions and Discussions

7 Pennsylvania State GEAR UP Snap Shot State grant 2 nd cycle began in 2008 Completing Year 6 of grant Single cohort model Graduated in Serving: 3 School Districts 8 High Schools 3,000+ Students

8 Background of Evaluation Process 2011 GEAR UP Director retired and New Director hired Semi-annual evaluation reports in place Data reports too late to make effective changes that year Developed quarterly evaluation reports to: Guide program changes Keep partners on track to meet target goals

9 Next Steps PASSHE GEAR UP Staff and Evaluation Team: All day planning session to develop quarterly evaluation process, report format, and action plan changes to SMART goals Adjustments and changes were made as needed: Modification Forms Data Review Meetings

10 FHI 360 s Role (as External Evaluator) Overall Program Evaluation Support Federal Annual Performance Reporting (APR) Pennsylvania State System of Higher Education (PASSHE) External Program Evaluators (FHI 360) Allentown School District GEAR UP Harrisburg School District GEAR UP School District of Philadelphia GEAR UP 2 GEAR UP High Schools 2 GEAR UP High Schools 4 GEAR UP High Schools

11 Planning and Implementation Action Planning and Participation Target Setting Coordinators work with districts annually to set targets referring to baseline data SMART Goals (Why beneficial?) Evaluator Review of Targets (Annual review prior to program year to ensure that objectives are SMART ) Management of District Program Modification Forms (Process to modify plans or targets)

12 Action Planning Action Plan development meetings are scheduled in the spring between PASSHE and district GEAR UP teams Each GEAR UP activity is aligned with 1. PA State GEAR UP s objectives and performance measures, 2. PA Career Education and Work Standards, and 3. USDE s Federal Service Categories Activities include the following components: 1. Federal Service Category Alignment, 2. Description of the Activity, 3. Performance Measure Alignment, 4. Resources/Budget, 5. Who is Responsible? 6. Timeline 7. Participation Target (SMART GOALS)

13 Action Planning (cont.) Once a draft action plan has been developed, it is presented to the PA State GEAR UP Director and grant evaluators (FHI360) for final review. Emphasis on SMART GOALS. After all edits/updates have been incorporated, a final version is submitted to the PA State GEAR UP Director for approval. Once approved, the action plan acts as a framework for the district s GEAR UP program.

14 Participation Target Setting PASSHE analyzes prior years participation data using our online database system (Cayen Systems) to determine baselines for GEAR UP activities. Baselines are used to create reasonable and achievable participation targets for each GEAR UP activity listed in the action plan. Participation targets are set to ensure each activity has a measurable goal which can be evaluated to determine the district s ability to reach the GEAR UP population and achieve PA State GEAR UP s objectives.

15 Action Plan SMART goals/objectives SMART Plans SMART Plans Specific Measurable Attainable Realistic Time bound

16 SMART Goals & Objectives A systematic way to think about and identify the key elements of the evaluation plan and measurement Indicators & performance measures Used to guide program implementers in setting objectives for: Employee performance management; Personal development plans; Performance indicators; or Action plans

17 Management of District Program The action plan is the main management tool used. Action Plan = Road map for the entire year All GEAR UP staff are provided a yearly program manual which includes: 1. Action Plan, 2. Budget 3. Participation Targets Chart, 4. Year-at-a-Glance Calendar, 5. Data Entry and Reporting Schedules, 6. Data Entry Guidelines, 7. Grant Procedures PASSHE s State Coordinator and each GEAR UP team conduct monthly meetings to assess the district s ability to successfully complete the activities outlined in the action plan.

18 Management of District Program (cont.) The action plan is a LIVING DOCUMENT. Remember: Plans Change. Action Plans must be modified when needed for the program to remain successful. PASSHE developed two types of Modification Forms for districts in the event an action plan activity needed to be changed: Modification Form #1 Approved by State Director - to be used for activity description changes, adding new activities, or deletions before the activity occurs. This includes activity budget and match changes. Modification Form #2 Approved by District Manager - to be used to change dates/locations of activities or participation targets before the activity occurs.

19 Management of District Program (cont.) Approved modification forms are collected and incorporated into the action plan and budget each marking period by the State Coordinator. Modifications to the action plan are listed in an Appendix of the action plan. The updated action plan is shared with the PA State GEAR UP Director, FHI360, and the district team after each marking period. Updates to the action plan are reflected in the next quarterly evaluation report.

20 Reporting and Utilizing Feedback from Reports Format and Frequency of Reports Data Sources (Quantitative and Qualitative) Analytical and Tracking Tools Sharing findings with Stakeholders Managing the process (Using Project Management Tool)

21 Format and Frequency of Reports Evaluation Report Format Outline (handout) Frequency: After each marking period (4x/year for each school district)

22 Data Sources (Quantitative and Qualitative) District Data Demographics School Attendance Course Grades Participation Data (from online database) Site Visit Observations Phone Interviews Survey Results

23 Analytical and Tracking Tools Statistical Software (SAS, SPSS, STATA) Qualitative Software (Atlas.ti, NVivo) GEAR UP Tracking Database (Cayen Systems, Xcalibur Scribe, Cobro Compass)

24 Sharing Findings with Stakeholder Quarterly evaluation meetings with report review as only agenda item Schedule face-to-face or virtual meetings Invite as many stakeholders as possible GEAR UP staff District staff Other partners Share report via to other stakeholders as appropriate

25 Project Management Tool

26 Future Changes to Improve the Process Internal Evaluator Track quarterly data toward targets Produce quarterly data reports without analysis External Evaluators Semi-annual reports with in-depth analysis One on-site visit at the end of the semester

27 Questions and Discussion

28 Contact Information: Sandi Sheppeard PA State GEAR UP Director PA State System of Higher Education Harrisburg, PA Tel Diana Fischer PA State GEAR UP Coordinator PA State System of Higher Education, Harrisburg, PA Tel David Jochelson Project Manager College Access Programs FHI Washington, DC Tel [email protected]

29 Pennsylvania State GEAR UP Program The PA State GEAR UP (Gaining Early Awareness and Readiness for Undergraduate Programs) grant is designed to assist underserved school districts with making students' dreams of attending college become a reality. The program ensures that students and their families have access to the necessary resources in order to graduate from high school and continue on to post-secondary education.