Idaho Center of Excellence Healthcare Partnership Program Evaluation Plan The following timeline will be used for submission of quarterly evaluations unless otherwise directed by the U.S. Department of Labor 1 Table 1. TAACCCT Reporting Schedule and Quarterly Narrative Progress Report Quarter End Dates Quarterly Data-Supported Narrative Progress Report Submission DUE DATE Report Activities that Occurred During: December 31, 2014 February 15, 2015 October 1 December 31, 2014 March 31, 2015 May 15, 2015 January 1 March 31, 2015 June 30, 2015 August 14, 2015 April 1 June 30, 2015 September 30, 2015 November 14, 2015 July 1 September, 2015 December 31, 2015 February 15, 2016 October 1 December 31, 2015 March 31, 2016 May 15, 2016 January 1 March 31, 2016 June 30, 2016 August 14, 2016 April 1 June 30, 2016 September 30, 2016 November 14, 2016 July 1 September, 2016 Interim Report December 15, 2016 October 1, 2014-September 30 2016 December 31, 2016 February 15, 2017 October 1 December 31, 2016 March 31, 2017 May 15, 2017 January 1 March 31, 2017 June 30, 2017 August 14, 2017 April 1 June 30, 2017 September 30, 2017 November 14, 2017 July 1 September, 2017 December 31, 2017 February 15, 2018 October 1 December 31, 2017 March 31, 2018 May 15, 2018 January 1 March 31, 2018 June 30, 2018 August 14, 2018 April 1 June 30, 2018 Final Report December 15, 2018 October 1, 2014-September 30 2018 Evaluation Plan Summary: ICE Healthcare Partnership will expand the ability of North Idaho College, Eastern Idaho Technical College, Idaho State University and Lewis-Clark State College to respond to economic and workforce development needs in northern, eastern, and central Idaho by increasing the education and skill attainment of TAA-eligible, veterans and other dislocated workers. The program seeks to enroll 360 unique participants over three years into an effective series of courses and industry recognized credentials, leading to certificates, job placement and degree attainment in the healthcare sector. North Idaho College will provide a program director that will be responsible for collecting data from 1 U.S. Department of Labor. Employment and Training Administration, Division of Strategies Investments. TAACCCT Reporting Schedule. 2012. Retrieved from https://etagrantees.workforce3one.org/view/2001210248564550404 May 30, 2014. 1
the partner institutions, and retaining a third party evaluator with demonstrated experience in developing similar grant deliverables. Methodology: Through a request for proposals, an external evaluator will be hired to coordinate the evaluation of all project goals, outcomes, and activities. NIC will adhere to the requirements of the USDOL to assure that the evaluator has the required documented and demonstrated expertise, experience, and qualifications to represent the consortium project. The external evaluator will support the program director and provide further expertise and support in evaluation design, data collection, analysis of data, and reporting that is consistent with USDOL guidelines and requirements. The program director will be supported by the Office of Institutional Research at each partner institution and will focus on formative assessment of the program. This formative assessment will monitor the implementation and impact of the proposed programs. The external evaluator will use data collected by the program director, as well as independent measures, to develop the program summative assessment. The external evaluator will provide a bias-free perspective of data and achievement toward goals which will be reported to stakeholders. North Idaho College has reviewed the recommended TAACCCT Project Characteristics as illustrated in Appendix O: Framework of Evaluation Methodologies. A Matched Comparison Group design will be used to measure the impact of various program components on its participants by contrasting participants with non-participants. Where data is available and matched, the program director will work with the third-party evaluator to develop the comparison groups across the programs identified in the work plan. Data gathering will occur through each partner s student management system, through a collaborative partnership with the Idaho Department of Labor, and through established job placement surveys at all partner institutions. All consortium colleges utilize enterprise data systems such as Banner or Colleague to record and track student data for state and federal reporting. Evaluation Timeline: Year One (Fall 2014-Fall 2015) will focus on program development and securing the third party evaluator to continue evaluation planning. Data collection tools and methods and 2
comparison groups will be finalized. Quantitative data collection will begin in spring (quarter 2) of Year One, as participants are selected for the program and comparison groups. Qualitative data collection will begin in the first quarter with quarterly narratives submitted to U.S. Department of Labor. An Interim Summative Report will be submitted by December 15, 2016. A Final Summative Report will submitted by December 15, 2018. Although program funding ends at the close of Year Three, the third party evaluator will complete a final report during the quarter after the final quarter of Year Four. Participant Impact or Outcomes The program Sr. Administrative Assistant and the Program Director will work with NIC s Institutional Effectiveness office to ensure consistent data collection and reporting activities. The outcome measures will inform the evaluation of the proposed program. Table 2 outlines potential demographic data that will be used to develop comparison groups. Specific outcome measures will be further developed by the program director and the third party evaluator. Table 2. Potential Outcomes Data Social Security Number Gender Selective Service Status Eligible Veterans Status Highest Level of Education Completed Entry and Exit Dates Employment Post Training Recognized Industry Certifications Achieved Date of Birth Race Disability Status Employment Status Training Prior Learning Completion Data Advancements in Employment GPA Program Implementation Measures: Implementation and progress measures are aligned with the Core Elements outlined in the Work Plan. The data will monitor the progress of the program strategies and inform the overall evaluation of the program. Table 3 provides details of the measures to be used to evaluate the implementation, progress, and overall impact of the proposed program. 3
Table 3. Implementation and Progress Measures Strategy Implementation Measures Progress Measures Core Element 1: Evidenced Based Design 1.1 Number of faculty and staff trained to provide comprehensive, intrusive student advising/services Establishment of comprehensive, intrusive student advising/services protocol Enhanced student engagement programming and activities established 1.2 ICE Healthcare Partnership Leadership Team key agencies involved in pathway 1.3 Number of staff and faculty trained in implementing credit for prior learning process Establishment of process to award credit for prior learning and/or non-credit training for veterans Core Element 2: Career Pathways 2.1 Approval of course competencies by employers 2.2 Approval of course competencies by employers 2.3 Approval of course competencies by employers 2.4 Approval of course competencies by employers 2.5 Approval of course competencies by employers Core Element 3: Advanced Online and Technology-Enabled Learning 3.1 Establishment of distributive hybrid courses/program 3.2 Establishment of hybrid and online courses/program 3.3 Establishment of enhanced student engagement programming and activities into courses/program Number of students that complete initial Percentage of credit hours completed based on. Number of enhanced student engagement programming and activities received by participants List of members of ICE Healthcare Partnership Leadership and the names and roles in operations of their agencies Number of military medics that complete initial Number of military medics that complete initial Number of students that enroll in distributive hybrid courses Number of students that enroll in hybrid and online courses Number of students that enroll in courses offering enhanced student engagement programming and activities 4
3.4 Establishment of professional development training plan and schedule Number of faculty trained to integrate hybrid instruction tools, activities and resources into design and delivery of distributive hybrid, hybrid, or online courses/programs Core Element 4: Strategic Alignment with the Workforce System and Other Stakeholders 4.1 ICE Healthcare Partnership Leadership Team key agencies Regular leadership team meetings held with most members in attendance Percentage of leadership team members attending each meeting/all meetings each 4.2 Funding and administrative policy barriers to successful pathway implementation identified (e.g. siloed funding, data collection, and service delivery) Solutions to identified barriers developed and implemented Core Element 5: Alignment with Previously-Funded TAACCCT Projects 5.1 ICE Healthcare Partnership Implementation Team key agencies involved in pathway operations Regular pathway operations team meetings held with most members in attendance Research compiled from previous TAACCCT grant grantees including lessons learned and best practices 5.2 Visual road map for the pathway(s) developed and implemented Plan for implementing the ICE Healthcare Partnership website developed Core Element 6: Sector Strategies and Employer Engagement 6.1 Employers participate regularly in pathways activities Employers participate in curriculum design List and description of each identified barrier Description of how identified barrier(s) was addressed and effect on the career pathways system List of members of ICE Healthcare Partnership Implementation Team and the names and roles in operations of their agencies Percentage of implementation team members attending each meeting/all meetings each Research utilized by ICE Healthcare Partnership Leadership Team in planning Document that presents the road map Copy of the plan Website completed Number and percentage of such participating employers from targeted healthcare sector in any given Number of employers that review education/training program curricula each 6.2 Employers host training/clinical learning sites Number of employers that host training/clinical sites for pathway programs each The evaluation plan will determine whether the core elements of the program are fully operational and will assess the alignment of education and industry and the alignment of two- and four-year postsecondary institutions to determine how alignment can be enhanced. The primary stakeholders of this grant-funded program include the USDOL, the ICE Healthcare Partnership, grant funded faculty and staff, grant-funded program participants, employer partners, and economic development organizations and other interested community partners. 5