EVALUATION FROM START-UP TO SCALE

Transcription

1 TRAINING REPORT EVALUATION FROM START-UP TO SCALE 9 TH FEBRUARY 2015 CENTER FOR EDUCATION INNOVATIONS P.O. BOX NAIROBI Prepared by:

2 ACKNOWLEDGEMENT This capacity building training report was developed through the effort of a team from Center for Education Innovations (CEI) and facilitated by Shape Afrika. The consultant, Shape Afrika sincerely acknowledges the support and contributions of departments and individuals who made the training and the development of this report possible. Special acknowledgments to: 1. Dr. Gordon Carver; Project Director, Caroline Jordan; Project Manager, Namulanta Kombo; Project Officer and Events Manager and Esther Sifuma; Project Officer for their leadership and logistical support in the implementation of the capacity building workshop. 2. Prof. David Macharia and Dr. Kinyanjui Nganga, the lead trainers, for effective facilitation of the training program. 3. The training participants for their active participation during the training and without whom the training exercise would not have taken place. 4. The various organizations and institutes that allowed their staff members to participate in the training program. ii

3 CONTENTS TABLE OF CONTENTS PAGE ACKNOWLEDGEMENT... 2 TABLE OF CONTENTS... 1 ABBREVIATIONS... 3 EXECUTIVE SUMMARY... 4 CHAPTER ONE: INTRODUCTION Introduction of Chapter One Background of the Training Objectives of the Training Significance of the Training Limitations of the Training Organization of the Training Report... 7 CHAPTER TWO: ORGANIZATIONAL BACKGROUND Introduction of Chapter Two Organizational Historical Background Organizational Vision Organizational Mission Organizational Approach Organizational Network CHAPTER THREE: TRAINING METHODOLOGY Introduction of Chapter Three Training Approach Training Participants Training Duration

4 3.5 Training Modules Training Evaluation CHAPTER FOUR: TRAINING FINDINGS AND ANALYSIS Introduction of Chapter Four Findings from the Training Analysis of the Training CHAPTER FIVE: CONCLUSIONS AND RECOMMENDATIONS Introduction of Chapter Five Conclusions on the Training Recommendations APPENDICES...28 (i) Appendix I: List of Participants (ii) Appendix II: Training Program (iii) Appendix III: Feedback Form

5 ABBREVIATIONS CEI CHMI DFID ECD LFA M&E NGO OVI R4D SGA ToR Center for Education Innovations Center for Health Market Innovations Department for International Development Early Childhood Development Log Frame Approach Monitoring and Evaluation Non-Governmental Organization Objectively Verifiable Indicator Results for Development Skills Gaps Analysis Terms of Reference 3

6 EXECUTIVE SUMMARY Center for Education Innovations (CEI) embarked on training program implementers and associates across the ECD (Early Childhood Development) and Skills education sectors on project/program evaluation. The training was expected to provide participants with practical guidance for understanding evaluation and its application in respective education programs. Forty four participants were targeted in a two-days training conducted on 3 rd and 4 th February 2015 in Nairobi, Kenya. The training approach was participatory through analysis of case studies in focus group discussions with the aid of PowerPoint presentations and flip charts to assess knowledge acquisition. Under the training theme of Evaluation from Start-up to Scale, participants were trained on the following six modules: introduction to evaluation; evaluation stakeholders; evaluation indicators; evaluation frameworks; evaluation report; and evaluating for program results. Learning materials and participation certificates were given to participants after the training. In addition, participants were given an opportunity to evaluate the effectiveness of the training by filling a feedback form. Findings and analysis of the training indicated that participants appreciated the concept of evaluation and were able to link evaluation with performance. As such, participants appreciated evaluation as an important requirement for project/program performance. In addition, participants enhanced their skills on designing evaluation indicators and evaluation frameworks in a manner that can guide the performance of projects/program. Moreover, participants understood the concept of conducting an evaluation including writing and presenting evaluation reports to the satisfaction of stakeholders. Based on the findings and analysis of the training, it was recommended that more time be allocated for similar trainings in future. It was also recommended that participants be trained in project management and in M&E and projects/programs be assisted in conducting evaluations. In addition, it was recommended that a more thorough skills gaps analysis be done prior to implementing such trainings to profile respondents capacity. 4

7 CHAPTER ONE INTRODUCTION 1.1 Introduction of Chapter One This chapter examines the background of the training; objectives of the training; significance of the training; limitations of the training; and organization of the training report. 1.2 Background of the Training Center for Education Innovations (CEI) seeks to identify, analyse and connect innovative education programs globally. As part of its global network, CEI has a fully established office in Kenya GEMS Education Solutions, that is serving the East African region. Based on this mandate, CEI organizes capacity building events for innovative education programs in the region that have been profiled in the CEI website as part of the CEI network. These programs range from small to large for-profit start-ups and small NGOs. For programs to be profiled on the CEI network, it is required that they demonstrate existence of some form of Monitoring and Evaluation (M&E) in their organization as well as education innovation. The training participants were program implementers and associates across the ECD (Early Childhood Development) and Skills education sectors. The reason for selecting these two thematic areas is that they are a focus of the work at CEI and the workshop built upon Thematic Forums held last year for Skills and ECD implementers. The workshop was expected to provide participants with practical guidance for understanding evaluation and its application in respective education programs. 5

8 1.3 Objectives of the Training The specific objectives of the training were:- i. Help participants appreciate the purpose of project/program evaluation and link evaluation to overall project/program performance to the satisfaction of all stakeholders; ii. iii. Help participants conduct project/program evaluation using standard tools, frameworks and objectively verifiable indicators; Help participants to write a standard evaluation report and effectively present findings to key stakeholders for continuity of the project/program. 1.4 Significance of the Training Training on Evaluation from Start-up to Scale was expected to help participants to conduct evaluation and write a standard evaluation report and present the same in a convincing manner to stakeholders. In addition, the training was expected to help participants to design evaluation indicators that can access the progress of the projects/programs. Further, the training was expected to equip the participants with skills to design evaluation frameworks that would not only summarize projects/programs progress but also help project/program implementers to be specific in identifying project/program direction in respect to desired project/program results while designing and planning for project/program implementation. The training was also expected to help participants to appreciate and undertake projects/programs stakeholders analysis. 1.5 Limitations of the Training The duration of the training, two-days, was relatively short in respect to anticipated objectives of the capacity building exercise especially since the training methodology 6

9 included the use of interactive sessions by participants. The trainers, however, made the most of the duration allocated by triangulating training methodology with the help of training aids like LCD projector, flip charts and focus group discussions. In addition, the learners educational level and M&E skills varied significantly which in turn limited the level of engagement in the training. The trainers, however, implemented the exercise in a manner that accommodated the majority of the participants comfortably and the participants were able to learn from one another during the focus group discussions. 1.6 Organization of the training report This report is organized in five chapters. Chapter one of this report is an introduction and it focuses on the background of the training; objectives of the training; significance of the training; limitations of the training; and organization of the training report. Chapter two of this report is on organizational background and it focuses on organizational historical background; organizational vision; organizational mission; organizational approach; and organizational network. Chapter three of this chapter is on training methodology and it focuses on the training approach; training participants; training duration; training modules; and training evaluation. Chapter four of this report focuses on findings from the training; and analysis of the training based on the training objectives. Chapter five of this report focuses on training conclusions and recommendations. The list of participants, participants feedback form and the training program are appended in the last section of this report. 7

10 CHAPTER TWO ORGANIZATIONAL BACKGROUND 2.1 Introduction of Chapter Two This chapter examines the organizational historical background; organizational vision; organizational mission; organizational approach; and organizational network. 2.2 Organizational Historical Background Center for Education Innovations (CEI) is a go-to resource on education innovations in low-income communities; providing data, analysis and connections as a global public good. Launched in 2013, CEI is an initiative of Results for Development (R4D) and modelled on the sister program Center for Health Market Innovations (CHMI). CEI aims to identify, analyse, and connect innovative education programs in developing countries so as to increase access to quality education and improve learning outcomes, especially for the poorest and most vulnerable. CEI pursues this goal using two mutually-reinforcing mechanisms: a virtual platform and a network of partners that drive in-country activities. GEMS Education Solutions implement CEI in East Africa, currently covering Kenya and Uganda. By mapping the education landscape, CEI East Africa has actively engaged with implementers, funders, policymakers and researchers in the region and developed analysis tools and connect opportunities meeting stakeholder needs. Thematic Forums have been held and what works syntheses produced in select thematic areas of ECD and Skills. This Workshop builds upon the connections established and knowledge shared to deliver capacity building to ECD and Skills implementers. 8

11 2.3 Organizational Vision CEI promotes programs, policies, and practices that increase access to quality, affordable, and equitable education for the world s poor. The vision of CEI is for education systems around the world to capitalize on innovation so as to increase access to quality education and improve learning outcomes, especially for the poorest and most vulnerable Why focus on education innovations? The development of CEI was based on the understanding that there is an incredible amount of innovation and activity in education, but it is not systematically documented and there is little evidence about the most effective ways to improve access to quality education for the poor. CEI was developed in order to address this gap What makes a program innovative? CEI realize that effective education systems are complex. They include delivery, but also policies and financing mechanisms that foster equity and quality, as well as support products and services for schools, teachers, and students. When a program for the poor addresses any one of these system components using approaches that are new or combines approaches in a new way CEI consider it innovative. CEI see innovation as relative: what is commonplace in one community may not be in another; an application of an existing approach in a new place or sector is innovative. 2.4 Organizational Mission CEI increases access to quality education for the poor by identifying, analyzing, and connecting non-state education innovations. 9

12 2.5 Organizational Approach CEI works to create impact expanded access to quality, affordable education for the poor by providing information, analysis, and connections for program managers, funders, researchers, and policymakers who work to enable the scale-up, replication and improvement of promising innovations in education Identify CEI identifies and profiles innovative education programs, building a global network of innovators. CEI team members around the world conduct interviews with every program they profile so they can fully understand the model, the challenges it faces, and its plans for the future. The CEI profiles provide information on the program s approach, target beneficiaries, financing, scale, monitoring and evaluation strategy, results achieved and external evidence Analyze The CEI website includes the Research & Evidence Library, a searchable database of over 600 analytical products about education in developing countries. The Library includes country landscapes, evaluations, case studies, policy documents, metaanalyses, global initiative reports, and more. With over 500 entries in the Programs Database, CEI is now beginning to focus on using the database as a starting point for understanding and sharing lessons about what works in education. Throughout 2014, CEI produced topic and country briefs that will surface common themes and developments from an analysis of CEI data, on top of existing literature and available evidence. 10

13 CEI works with global partners such as UNICEF to better understand what types of innovative programs and interventions have the greatest potential for scalability and impact. CEI also works with partners to identify programs interested in formative evaluation and works with them to identify challenges, strategize about solutions, develop monitoring and evaluation capacity, and ultimately share lessons about their experience Connect The CEI website enables users to directly contact managers of innovative programs, enabling direct engagement with a diverse community of innovators. CEI hosts customized virtual collaboration spaces for various groups involved in education innovation, including the Girls Education Challenge (GEC), the Partnership to Strengthen Innovation and Practice in Secondary Education (PSIPSE) and STIR Education. CEI s most critical connections are facilitated by our network of regional partners working select geographies to foster connections and learning among innovative programs and link them to funding, operational and research opportunities. Read more about CEI s regional partners here. Recognizing the value partners can offer high-potential education programs, CEI collaborates with a wide range of organizations seeking to scale up and improve education innovations by connecting them with essential financial, operational, technical, and advisory support. CEI builds upon the example of the Center for Health Market Innovations (CHMI), a companion R4D initiative that is now the world s largest freely accessible information resource of innovations in the non-state health sector. CEI is funded by the United Kingdom s Department for International Development (DFID) and UNICEF. 11

14 2.6 Organizational Network CEI is coordinated by the Results for Development Institute (R4D) and relies on a broad network of regional partners and global collaborators Regional Partners To foster the uptake of promising education innovations, country-based activities that complement the online platform are driven by both the CEI team at R4D and a network of CEI regional partners: i. INDIA Catalyst Management Services ii. iii. iv. EAST AFRICA GEMS Education Solutions SOUTH AFRICA University of Cape Town, Graduate School of Business, Bertha Centre for Social Innovation and Entrepreneurship NIGERIA TEP Centre These organizations represent CEI in their country and region, carrying out a range of activities such as convening innovators with high potential business models, brokering relationships between innovators and policymakers to initiate cooperation between the public and private sectors, and building relationships between innovators and donors, investors, and others that can help them grow and scale up their activities Global Collaborators For profiled programs, CEI provides access to forums, competitions, fundraising platforms, and networks run by our Global Collaborators shown below. CEI provides Global Collaborators with a pipeline of excellent candidates for these opportunities. Collaborators often search CEI s database to identify candidates. CEI may share relevant programs that meet the nomination criteria thematically, geographically, or are reporting specific results. It is important for programs to keep their profiles up to date, so 12

15 that they can be considered for potential opportunities. Global collaborators include: UNICEF; GLOBAL GIVING; OECD; STIR EDUCATION; GIRLS EDUCATION CHALLENGE; UKAID; TONIC; ACUMEN; IDEAS; WISE; AND SKOLL FOUNDATION. 13

16 CHAPTER THREE TRAINING METHODOLOGY 3.1 Introduction of Chapter Three This chapter examines the training approach; training participants; training duration; training modules; and training evaluation. 3.2 Training Approach The scope of the training as well as the Terms of Reference (ToR) for the training was deliberated on by a project team comprising officials from CEI and senior trainers from Shape Afrika, an organizational research and corporate training private firm. The training methodology, objectives, modules and the program of the training were all duly agreed upon by the two parties prior to preparation of the training materials. Training on Evaluation from Start-Up to Scale was facilitated by two senior trainers from Shape Afrika: Prof. David Macharia and Dr. Kinyanjui Nganga, with practical expertise and experience in capacity building in M&E and project management. The training was done in accordance with the norms of adult learning through interactive sessions and focus group discussions with the aid of flip charts and LCD projector. Participants, therefore, would be given space not only to clarify issues during the training but also to learn from one another through focus group discussions to ensure the acquisition of actual practical skills on the subject matter under examination. 14

17 3.3 Training Participants There were forty four (44) participants in the training. These participants were program implementers and associates across the ECD (Early Childhood) and Skills education sectors who were expected to use the skills acquired from the training in project management and specifically in project evaluation. The list of the participants and the respective organizations where the participants work is appended in this report as Appendix i. 3.4 Training Duration The training was conducted in two working days on Tuesday 3rd and Wednesday 4th February 2015 from 8.00am to 4.30pm on both days. The training program is appended as Appendix ii in this report. 3.5 Training Modules The modules that were delivered in the training as well as the learning objectives and learning outcomes for each of the training days are as shown in Table 3.1. Table 3.1: Training Modules (i) Modules for day I Module Learning Objectives Learning Outcomes Module I Introduction to Evaluation Evaluation as a key phase of the project life cycle Purpose of evaluation Self-efficacy in evaluation Components of evaluation Appreciation of evaluation as a prerequisite of performance Taking personal responsibility for projects / programs outcomes Appreciation of the concept of evaluation and various 15

18 Module II Evaluation stakeholders Module III Evaluation Indicators Types of program evaluations Evaluation ethics Identifying evaluation stakeholders Stakeholders needs analysis (cost, time, program efficiency, program impact etc) Purpose of evaluation indicators Designing evaluation indicators Critiquing evaluation indicators views about evaluation Integrity in programs implementation Appreciation that programs comprise various stakeholders with dynamic interests that must be satisfied hence the focus of the evaluator should not be her/himself but program stakeholders Participants should appreciate the importance of evaluation indicators Learners should be able to design Key Performance Indicators (KPIs) Learners should allow other evaluators to critique whether the indicators they have developed meet desired project/program objectives (ii) Modules for day II Module Learning Objectives Learning Outcomes Module IV Evaluation Frameworks Purpose of evaluation frameworks Types of frameworks used in evaluation Contextualizing evaluation frameworks With or without a full evaluation report, participants should be able to summarize their evaluation findings in recognized evaluation frameworks Appreciation and acceptance of various evaluation frameworks that are used in projects/programs Learners should be able to contextualize existing evaluation frameworks in their program setting 16

19 Module V Evaluation report Module VI Evaluating for program results Conducting a program evaluation Designing program evaluation instruments Writing an evaluation report Handling evaluation findings Communicating evaluation findings Utilization of evaluation report Skills to conduct an evaluation cognizant of multiple realities Ability to design instruments that are valid and reliable for evaluation Ability to write a standard evaluation report Integrity in accepting evaluation results as scientific findings Communication / Presentation skills in dispensing evaluation report in a convincing manner to all stakeholders Ability to apply recommendations from evaluation reports in implementation of similar future projects 3.6 Training Evaluation At the end of the training, participants were asked to evaluate the program including the trainers ability to effectively train, mode of delivery, content of training materials, relevance of the training and to make any other comment especially on future trainings and areas of improvement. The feedback from the participants was used in compiling this training report and formed part of the recommendations herein. In addition, participants were awarded with participation certificates on successful completion of the course. Participants were also given the training materials. 17

20 CHAPTER FOUR TRAINING FINDINGS AND ANALYSIS 4.1 Introduction of Chapter Four This chapter presents findings from the training; and analysis of the training based on the training objectives. 4.2 Findings from the Training The findings from the training were identified and categorized in accordance with the training s specific objectives Findings from the Training on Training Objective 1 Training objective 1 was to help participants appreciate the purpose of project/program evaluation and link evaluation to overall project/program performance to the satisfaction of all stakeholders. Based on both the participants participation in the workshop and feedback received from the evaluation forms, indications were that participants appreciated the purpose of project/program evaluation including linking project/program evaluation to the overall project/program performance. In addition, based on class assessments in the workshops on stakeholders analysis, it can be argued that participants appreciated the concept of stakeholder analysis in respect to projects/programs evaluation. As such, indications from the training are that participants appreciated the concept of self-efficacy in evaluation, taking responsibility for project/program performance and appreciation of evaluation as a necessary phase in the project life cycle to quantify and enhance performance rather than witch hunt from senior management and project/program sponsors. 18

21 4.2.2 Findings from the Training on Training Objective 2 Training objective 2 was to help participants conduct project/program evaluation using standard tools, frameworks and objectively verifiable indicators. Based on assignments allocated in the focus group discussions in designing project/program indicators, it can be argued that majority of the participants could individually and collectively design Objectively Verifiable Indicators (OVIs) that can be helpful in measuring actual project/program performance. In addition, on-job training assessments and feedback given by participants on the training indicate that participants appreciated the types of evaluation frameworks available for project managers including the ability to decide and design relevant evaluation frameworks for given project/program context Findings from the Training on Training Objective 3 Training objective 3 was to help participants to write a standard evaluation report and effectively present findings to key stakeholders for continuity of the project/program. Informed by this training objective, participants were trained on how to conduct an evaluation including writing and presenting an evaluation report. Following the training on conducting an evaluation, participants appreciated the need for conducting project/program evaluations, basis for selecting internal or external evaluators, conceptualizing a project/program evaluation, tools used in conducting project/program evaluation, evaluation designs, analysis of project/program findings, ethical issues in evaluation and compiling and presenting an evaluation report to the satisfaction of key project/program stakeholders. In addition, in the event participants chose to outsource evaluation from external evaluators, the participants were equipped with requisite skills that would enable them to draw clear evaluation Terms of Reference (TORs) for the said evaluators based on appreciation of the evaluation conceptual framework. On the same note, the knowledge acquired from the training would enable participants not only be able to follow what an external evaluator is doing on behalf of the project/program but also be able to assess whether an external evaluator understands the tasks allocated in respect to project/program objectives. 19

22 4.3 Analysis of the Training This section is on the analysis of the training. The analysis of the training was done qualitatively based on class participation and assessments and feedback from participants. Analysis of the training was carried out in accordance with the training objectives Analysis of the Training on Training Objective 1 Training objective 1 was to help participants appreciate the purpose of project/program evaluation and link evaluation to overall project/program performance to the satisfaction of all stakeholders. While participants appreciated evaluation as a key phase of the project life cycle prior to the training, findings from the training indicate that many participants had not previously appreciated evaluation as a direct prerequisite to project/program performance. Evaluation was previously perceived by many participants just as a requirement by sponsors and senior management. In addition, prior to the training, most participants viewed the project life cycle linearly rather than inter-connected phases with M&E taking place throughout the life of a project. Additionally, while most participants appreciated the broad concept of evaluation prior to the training, most of the participants assumed that evaluation could only be done at the end of a given project. The training, therefore, helped the participants appreciate various types of evaluations including their diverse applications. Further, while most participants understood that evaluation was core to project funders, the fact that individual career growth and development was, to a great extent, linked to project/program outcomes was not clear to many participants prior to the training. In addition, many participants were not confident in respect to project/program evaluations. While self-efficacy in evaluation focused on demystifying evaluation, the entire course, to a large extent, served the same purpose of building confidence in participants in respect to project/program evaluation. 20

23 4.3.2 Analysis of the Training on Training Objective 2 Training objective 2 was to help participants conduct project/program evaluation using standard tools, frameworks and objectively verifiable indicators. Assessments from the training indicate that participants appreciated the importance of evaluation indicators prior to the training but most participants could not design objectively verifiable indicators that could guide the project/program direction in a manner that could aid in objective project/program evaluation prior to the training. In addition, indications from the training revealed that many participants did not prior to the training expose their evaluation indicators to peer review which undermined in-depth understanding of the concept of evaluation indicators. Although participants appreciated the value of evaluation frameworks prior to the training, weak designs of evaluation indicators compromised the quality of evaluation frameworks previously designed by the participants. While there were time constraints in working through evaluation frameworks in the workshops, participants were exposed to various evaluation models in use as required by various development partners. However, while participants were able to work through focus group discussions on case studies chosen by individual groups, the time allocated for the training was not adequate for participants to be tested on ability to contextualize evaluation frameworks in respect to the projects/programs presented in the workshops. 21

24 4.3.3 Analysis of the Training on Training Objective 3 Training objective 3 was to help participants to write a standard evaluation report and effectively present findings to key stakeholders for continuity of the project/program. While participants knew the importance of conducting project/program evaluation, not as many could effectively undertake a project/program evaluation prior to the training. While it was not possible to quantify actual acquisition of skills to conduct a project/program evaluation during the workshop, participants were equipped with skills to conduct an evaluation cognizant of multiple realities. Although learners were trained on designing evaluation data collection instruments, time allocated for the training could not allow a critique of important aspect of evaluation data collection instruments like the practical usefulness of validity and reliability of evaluation data collection instruments. Similarly, while evaluation designs and evaluation statistical tools of analysis were included in the module on conducting an evaluation, the training was limited in helping participants appreciate the concepts through contextualizing the skills in a project case study. While some of the participants could comfortably write an evaluation report, others could neither conduct nor write an evaluation report nor disseminate evaluation findings. The training, however, was able to unearth learners capacity in respect to conducting an evaluation which formed the basis for the report s recommendations. While dissemination of evaluation findings had been identified as a key deliverable from the training, the same was the most adversely affected in respect to time allocation due to time constraints. However, in as much as this was not done as a stand-alone module as originally envisioned, the concept was intertwined in the entire course. Assessments from the workshops indicated that most participants were relatively comfortable in dispensing evaluation report in a convincing manner to all stakeholders to buy into the evaluation results. However, that conducting evaluation must be scientific in approach for results to be reliable and valid for utilization by project stakeholders needed more time for internalization by participants. This way, evaluation findings are accepted as having the required integrity threshold to quantify either an on-going or a concluded project and to form the basis for implementation of future similar projects. 22

25 CHAPTER FIVE CONCLUSIONS AND RECOMMENDATIONS 5.1 Introduction of Chapter Five This chapter presents conclusions and recommendations made based on the training objectives. 5.2 Conclusions on the Training This section is on conclusions made from the training based on the training objectives Conclusions on Training Objective 1 Training objective 1 was to help participants appreciate the purpose of project/program evaluation and link evaluation to overall project/program performance to the satisfaction of all stakeholders. Based on the findings and analysis of training objective 1, it was concluded that participants appreciated evaluation as a prerequisite of performance rather than a fault finding mission. It was also concluded that participants, based on the training, appreciated taking personal responsibility for projects/programs outcomes informed by the conventional usage of the concept of evaluation and various scholarly views about project/program evaluation. In addition, it was concluded that participants appreciated integrity in programs implementation and the fact that programs comprise various stakeholders with dynamic interests that must be satisfied. It was, therefore, concluded that participants appreciated that the focus of the evaluator should not be personal interests but satisfaction of all project/program stakeholders. 23

26 5.2.2 Conclusions on Training Objective 2 Training objective 2 was to help participants conduct project/program evaluation using standard tools, frameworks and objectively verifiable indicators. Based on the findings and analysis of training objective 2, it was concluded that participants appreciated the importance of evaluation indicators and were able to design Key Performance Indicators (KPIs) and to allow other evaluators to critique whether the indicators they develop meet desired project/program objectives. Based on the training, participants were also able to summarize and contextualize evaluation findings in acceptable and recognizable various evaluation frameworks Conclusions on Training Objective 3 Training objective 3 was to help participants to write a standard evaluation report and effectively present findings to key stakeholders for continuity of the project/program. Based on the findings and analysis of training objective 3, it was concluded that participants acquired necessary skills to conduct a standard evaluation cognizant of multiple realities including ability to design simple data collection evaluation instruments that are valid and reliable for evaluation. In addition, it was concluded that participants could write and present a standard evaluation report based on skills acquired from the training. Further, it was concluded that participants appreciated the concept of integrity in accepting and utilization of evaluation results as scientific findings useful for advising the implementation of similar future projects 24

27 5.3 Recommendations This section is on training recommendations and it was organized in accordance with the training objectives Recommendations from Training Objective 1 Training objective 1 was to help participants appreciate the purpose of project/program evaluation and link evaluation to overall project/program performance to the satisfaction of all stakeholders. Based on the findings and analysis of training objective 1, the following was recommended:- i. Training on performance management to be undertaken not only to the project/program managers that participated, but also to other project/program officers from the institutions/organizations that participated to help project team members link their individual performance to the broader project/program performance through performance measurement. ii. iii. iv. Training on project management to precede trainings on evaluations so that participants can link evaluation to project/program performance. Further, participants ought to appreciate that evaluation is not a stand-alone phase in the project life cycle and hence interlinkages with other project phases ought to be appreciated prior to zeroing on project/program evaluation. Participating organizations need capacity building in the area of understanding the various types of evaluations that can be executed rather than just undertaking evaluations at the end of a project. Based on findings and analysis done from the training and in view of the fact that evaluation is both a science and an art, it was recommended for project/program managers to be trained on soft skills and in particular team work in project/program evaluations, confidence in projects/programs evaluations, management of project/program stakeholders and self-efficacy in project/program evaluations. 25

28 5.3.2 Recommendations from Training Objective 2 Training objective 2 was to help participants conduct project/program evaluation using standard tools, frameworks and objectively verifiable indicators. Based on the findings and analysis of training objective 2, the following was recommended:- i. While indicators were taught through presentations and focus group discussions, time allocated for the same was inadequate for satisfactory appreciation of this important concept in evaluation. It was, therefore, recommended that more time to be allocate for future similar practical hands-on training on designing and developing evaluation indicators and linking OVI with project/program evaluation and performance. ii. Although learners appreciated evaluation frameworks and were able to design the same through focus group discussions, it would have been more efficient to allow participants to individually design an evaluation framework based on a given project case study. In addition, time available for the workshop only allowed participants to work through one evaluation framework: Log Frame Approach (LFA) Model. While it is generally argued that LFA has wider usage than other evaluation models, it would have been better to allow participants to work through various frameworks collectively and individually both for knowledge acquisition and to appreciate requirements from different development partners on different evaluation frameworks. It was, therefore, recommended that more time, say, a whole working day, to be allocated in future similar trainings on evaluation frameworks. 26

29 5.3.3 Recommendations from Training Objective 3 Training objective 3 was to help participants to write a standard evaluation report and effectively present findings to key stakeholders for continuity of the project/program. Based on the findings and analysis of training objective 3, the following was recommended:- i. While participants were trained on conducting an evaluation including writing an evaluation report, no time was allocated to undertake either individual or group assignment on this particular training objective partially because of the complexity of the task. While more time may be allocated in future similar trainings, an evaluation cannot be conducted from a workshop context. As such, it was recommended that evaluation experts be attached to the projects/programs that are in the CEI network to help project/program managers on the ground understand the entire process of conducting project/program evaluation from conceptualization to writing and presentation of evaluation report. ii. Since conducting a project/program evaluation is conventionally a rigorous scientific process, it was recommended that a pre-training Skills Gap Analysis (SGA) be undertaken to profile participants entry behavior. This would assist trainers to engage participants at their skills level in such a manner that all participants can effectively be trained on how to conduct an evaluation depending on their educational level, learning capacity and specific skills in M&E. In addition, this would assist in development of the training curriculum and materials in respect to content depth, breadth and mode of delivery. iii. While planning for future trainings, it is recommended that conducting an evaluation to be done as an entire course on its own merit. In this case, the modules in this course would range from designing an evaluation, developing evaluation objectives, evaluation designs, evaluation approaches, evaluation respondents, evaluation tools of data collection, evaluation statistical tools of analysis, analyzing evaluation findings, writing an evaluation report and presenting an evaluation report. 27

30 APPENDICES (i) Appendix I: List of Participants No. Organisation Forename Surname 1 Center for Education Innovations Caroline Jordan 2 Center for Education Innovations Esther Sifuma 3 Center for Education Innovations Namulanta Kombo 4 Daraja Civic Godfrey Otieno 5 DOT Trust Antony Omutobe 6 Educate Africa Nuala Alibhai 7 Emerging Leaders Foundation (ELF)Africa Caleb Odhiambo Ouma 8 Equity African Leaders Program (EALP) Michael McCreary 9 Future First Pauline Wanja 10 Global Education Fund Daniel Masawi 11 GRADIF Kenya Gladys Miriti 12 Hope for Teenage Mothers Lucy Ndungu 13 ICRI - International Child Resource Institute Stephen Waweru 14 Innovate Kenya Richy Bikko 15 Kesho Kenya Dr. Kate Nokes 16 Kibera Girls Soccer Academy (KGSA) Richard Teka 17 Kidogo Janet Mwitiki 18 Kids Comp Camp Caleb Ndaka 19 Kindergarten Experts Peter Odour 20 Kito International Wiclif Otieno 21 Koinoina Community Okada Buluma 22 Livelyhoods Millicent Chepkemoi 23 Mentor Coach Empower (MCE) Uganda James Katumba 24 Mobjap Children Center Bernard Asanya 25 Mortely Youth Group Stephen Kimani 26 Nairobi Parenting Clinic Vivianne Matuku 27 Nairobits Fatuma Ramadhan 28 NAPS Kenya Juliet Mwangi 29 PACEMaker International Doris Kiogora 30 PACT Global Business Wangari Kabiru 31 Riziki Kenya Mwirigi Gatobu 32 Safe Spaces Immaculate Agusta 33 Samburu Girls Foundation Wanjiru Wahome 34 Samsung Engineering Academy Beverlyne Mudeshi 28

31 No. Organisation Forename Surname 35 Sense of Worth Trust Peggie Kalie 36 Spire Education Pauline Atieno 37 Star of Hope Daycare Centre Miriam Nyongesa 38 STEM Africa Dr. Susan Musembi 39 The Kuza Institute Gilbert Mitullah 40 The Supply John Ng'ang'a 41 Tushinde Children's Trust Paul Otieno 42 Ufahamu Youth Okumu Churchil VICDA - Volunteer International Community 43 Development Africa Irene Ngatia 44 Youth Fund Sharu Huka 29

32 (ii) Appendix II: Training Program Day One: Time Module Learning Objectives Learning Outcomes a.m. REGISTRATION & OPENING REMARKS a.m a.m. Module I Introduction to Evaluation Evaluation as a key phase of the project life cycle Purpose of evaluation Self-efficacy in evaluation Components of evaluation Types of program evaluations TEA BREAK Appreciation of evaluation as a prerequisite of performance Taking personal responsibility for projects / programs outcomes Appreciation of the concept of evaluation and various views about evaluation p.m. Module II Evaluation stakeholder s Evaluation ethics Identifying evaluation stakeholders Stakeholders needs analysis (cost, time, program efficiency, program impact etc) p.m. LUNCH Integrity in programs implementation Appreciation that programs comprise various stakeholders with dynamic interests that must be satisfied hence the focus of the evaluator should not be her/himself but program stakeholders p.m. Module III Evaluation Indicators Purpose of evaluation indicators Designing evaluation indicators Critiquing evaluation indicators Participants should appreciate the importance of evaluation indicators Learners should be able to design Key Performance Indicators (KPIs) Learners should allow other evaluators to critique whether the indicators they have developed meet desired project/program objectives p.m. PLENARY SESSION & REFRESHMENTS 30

33 Day Two: Time Module Learning Objectives Learning Outcomes a.m a.m. Module IV Evaluation Frameworks Purpose of evaluation frameworks Types of frameworks used in evaluation Contextualizing evaluation frameworks TEA BREAK With or without a full evaluation report, participants should be able to summarize their evaluation findings in recognized evaluation frameworks Appreciation and acceptance of various evaluation frameworks that are used in projects/programs Learners should be able to contextualize existing evaluation frameworks in their program setting p.m. Module V Conducting and reporting evaluation Conducting a program evaluation Designing program evaluation instruments Writing an evaluation report p.m. LUNCH Skills to conduct an evaluation cognizant of multiple realities Ability to design instruments that are valid and reliable for evaluation Ability to write a standard evaluation report p.m. Module VI p.m. Evaluating for program results Handling evaluation findings Communicating evaluation findings Utilization of evaluation report Integrity in accepting evaluation results as scientific findings Communication / Presentation skills in dispensing evaluation report in a convincing manner to all stakeholders Ability to apply recommendations from evaluation reports in implementation of similar future projects CLOSING REMARKS, AWARDING CERTIFICATES & REFRESHMENTS 31

34 Strongly Agree Agree Neutral Disagree Strongly Disagree (iii) Appendix III: Feedback Form Introduction Evaluating your Education Program from Start-Up to Scale CEI Evaluation Training Workshop 3 rd and 4 th February 2015 Participants Feedback Form Following your successful completion of the Evaluation from Start Up to Scale course, we would like to request you to fill this Participants Feedback Form to help us improve the implementation of future similar undertakings. Section A: Course Relevance 1) Based on the course that you have been trained, kindly rate the following factors / statements using a scale of SA Strongly Agree; A Agree; N Neutral; D Disagree; and SD Strongly Disagree Parameters a) I am happy that I attended the training (SA) (A) (N) (D) (SD) b) The training is relevant in my work c) The training has enhanced my evaluation skills d) The course content is relevant in my work e) The course will help me improve in program evaluation f) The time allocated for the training was satisfcatory g) I understood what was being taught h) The course has met my expectations 32

35 Strongly Agree Agree Neutral Disagree Strongly Disagree Section B: Competence of Trainers 2) Based on the course that you have been trained, kindly rate the following factors / statements using a scale of SA Strongly Agree; A Agree; N Neutral; D Disagree; and SD Strongly Disagree Parameters a) I enjoyed the training (SA) (A) (N) (D) (SD) b) Training delivery was satisfactory c) There was clarity in the trainers language d) The training was participatory e) The trainers were competent in the subject matter f) I would be happy to be trained by the same trainers again Section C: Improvement Strategies 3) Which was the most useful component of the workshop and why? ) What would you like to see included in future workshops?

36 5) Suggest future workshops that you would like to attend facilitated by CEI ) Are there any stakeholders you would like CEI to help you make linkages with? (Either attendees or others) ) Is there any support you would want in your organization in light of the training that you have undergone? (e.g. conducting an evaluation, reviewing your evaluation framework, training other staff members on evaluation etc) ) Any other comment Thank you. 34