Formative Assessment for Evaluation: Collaborative, community-based design of evaluation tools and systems Dr Amy Gullickson Hadeel Al-Nawab Centre for Program Evaluation September 2015 AES International Evaluation Conference 2015
Agenda Context Plan Components (Webinars and workshop)
Advanced Technological Education www.nsf.gov/ate
Funding Levels Funding Levels Centers 177 Projects
Funding Levels 38 Centers 177 Projects
Streams of Activity Program Development and Improvement Curriculum Development Professional Development Targeted Research Leadership Capacity Building Teacher Preparation Business and Entrepreneurial Skills Development for Students
Discipline Areas
Roles Roles Principal Investigator (PI) Co-Principal Investigator (Co-PI) Staff Evaluator
Levels of Experience Principal Investigator (PI) Co-Principal Investigator (Co-PI) Staff Evaluator
Streams of Activity Program Development and Improvement Curriculum Development Professional Development Targeted Research Leadership Capacity Building Teacher Preparation Business and Entrepreneurial Skills Development for Students
Streams of Activity Streams of Activity Program Development and Improvement Curriculum Development Professional Development Targeted Research Leadership Capacity Building Teacher Preparation Business and Entrepreneurial Skills Development for Students
Discipline areas Discipline Areas
Discipline areas Discipline Areas Program Development and Improvement Curriculum Development Professional Development Targeted Research Leadership Capacity Building Teacher Preparation Business and Entrepreneurial Skills Development for Students
Discipline areas Discipline Areas Program Development and Improvement Curriculum Development Professional Development Targeted Research Leadership Capacity Building Teacher Preparation Business and Entrepreneurial Skills Development for Students
Discipline areas Discipline Areas Program Development and Improvement Curriculum Development Professional Development Targeted Research Leadership Capacity Building Teacher Preparation Business and Entrepreneurial Skills Development for Students
Discipline areas Program Development and Improvement Curriculum Development Professional Development Targeted Research Leadership Capacity Building Teacher Preparation Business and Entrepreneurial Skills Development for Students
Demo
FORMATIVE ASSESSMENT SYSTEMS FOR ATE COMMUNITY BASED DESIGN OF PROJECT EVALUATION TOOLS / SYSTEMS
VISION VISION A system of tools that ATE PIs can adapt and use to identify and collect real-time data for day to day learning and decision making on their projects. COMMUNITY BASED DESIGN OF PROJECT EVALUATION TOOLS / SYSTEMS FORMATIVE ASSESSMENT FOR ATE
Strategy Strategy
Streams of Activity Streams of Activity Program Development and Improvement Curriculum Development Professional Development Targeted Research Leadership Capacity Building Teacher Preparation Business and Entrepreneurial Skills Development for Students
Recipe for Success Recipe for Success Benefits for Participants Activity Benefits for the Project Logic Models (LM) Webinar 1 Workshop Recruiting Backwards Design Webinar 2 LM for Workshop Workshop
Recipe for Success Recipe for Success MOA for participants
Project Overview Webinars Workshop Synthesis Individual Project Across Projects Blogs Logic Model High Level Logic Models Strategy and Prioritising Evaluative Activities High Level Evaluation Data/Tool Needs System Tools, Activities, and Priorities
Leadership Team, Advisory Committee & Keynote Speakers Executive Group Arlen Gullickson Amy Gullickson Lori Wingate Gordon Snyder Dee Al Nawab Advisory Committee Elaine Craft Mike Lesiecki Joyce Malyn Smith Helen Sullivan ATE PIs & Senior Staff Workshop Keynote Speakers Janet Clinton John Hattie
Project Overview Webinars Workshop Synthesis Individual Project Across Projects Blogs Logic Model High Level Logic Models Strategy and Prioritising Evaluative Activities High Level Evaluation Data/Tool Needs System Tools, Activities, and Priorities
Webinars Overview 85 56
Webinars Planning & Implementation Content Design Logistics Blackboard Online vs PDF versions
Blackboard Ba Chat Box
Recipe for Success Recipe for Success Benefits for Participants Activity Benefits for the Project Logic Models (LM) Webinar 1 Workshop Recruiting Backwards Design Webinar 2 LM for Workshop Workshop
Recipe for Success Benefits for Participants Activity Benefits for the Project Logic Model (LM) Webinar 1 Workshop Recruiting Backwards Design Webinar 2 Workshop Logic models and early identification of formative needs
Project Overview Webinars Workshop Synthesis Individual Project Across Projects Blogs Logic Model High Level Logic Models Strategy and Prioritising Evaluative Activities High Level Evaluation Data/Tool Needs System Tools, Activities, and Priorities
WORKSHOP Workshop Moving toward Real Time Data for ATE Projects
Workshop Attendees Workshop Attendees Principal Investigator (PI) 11 Co-Principal Investigator (Co-PI) 6 Staff 4 Facilitator 5 Evaluator 10
Working Groups Working Groups
Working Groups Working Groups Program Development and Improvement Curriculum Development Professional Development Targeted Research Leadership Capacity Building Teacher Preparation Business and Entrepreneurial Skills Development for Students
Activity Structure Activity Structure Task 1 Logic Models Refining Individual Logic Models, Creating High Level Logic Models INPUTS ACTIVITIES OUTPUTS Facilitator Techniques Participant Expertise Each person/team gets a big post it and a pack of small post its. Use the big post it to create a logic model of your project or center in the activity stream you are assigned Logic Model for each participant / team Post -its 15 Minutes
Recipe for Success Benefits for Participants Activity Benefits for the Project Draft logic model Webinar 1 Workshop Recruiting Backwards design: identify data collection needs Refined logic model & data mapped onto management needs and project success throughout Webinar 2 Workshop Logic models and early identification of formative needs
Recipe for Success Benefits for Participants Activity Benefits for the Project Draft logic model Webinar 1 Workshop Recruiting Backwards design: identify data collection needs Refined logic model Data mapped onto management needs and project success throughout Webinar 2 Workshop Logic models and early identification of formative needs Logic models for activity streams, Prioritised needs for tools and systems, Collaborators & next steps
Project Overview Webinars Workshop Synthesis Individual Project Across Projects Blogs Logic Model High Level Logic Models Survey Evaluative Activities High Level Evaluation Data/Tool Needs Strategy, System Tools, Activities, and Priorities
Blogs
SURVEY Survey
Survey Results - Highlights SURVEY Results Highlights Value of Ideas Presented Outcome of Participation Subsequent Action
Next Steps Continued collaboration Dashboard development Common items for surveys Evaluation checklist for PD projects
Q&A
Thank you amy.gullickson@unimelb.edu.au hadeel.al@unimelb.edu.au Copyright The University of Melbourne 2015