JOBS FUND PROJECT EVALUATION GUIDELINES FOR IMPLEMENTING PARTNERS

Transcription

1 JOBS FUND PROJECT EVALUATION GUIDELINES FOR IMPLEMENTING PARTNERS

2 Contents 1 Introduction Why evaluation of the Jobs Fund Project Purpose of the Evaluation Guidelines Planning for the Final Evaluation Elements of the Evaluation Plan Background and Context Evaluation Purpose Evaluation Scope and Objectives Evaluation Methods Evaluation Baseline Data Example Planning Matrix for Monitoring & Evaluation Data Quality Evaluation Criteria and Evaluation Questions Evaluation Management Identifying Evaluation Stakeholders Use of External Evaluation Specialists Evaluation Report Evaluation use and follow-up P a g e

3 1 Introduction The Jobs Fund Grant Agreement signed with project implementing partners calls for a Final Evaluation to be conducted for all projects falling under the 3 rd Call for Proposals. The Grant Agreement states that the Grantee shall undertake the following: Commission an independent Evaluation of the Project and ensure that this Evaluation is reflected in the ABC PIMP; Provide the JOBS FUND with all Evaluation reports, access to Project data, learning's and beneficiary experiences of the Project; and Consult with the JOBS FUND on their Evaluation approach and methodology and ensure that the Evaluation reports are made available to the JOBS FUND. The evaluation activities shall measure the extent to which projects were successful in achieving both anticipated outputs and outcomes as per the project theory of change. This will be done in terms of assessing these theories Relevance, Efficiency, Effectiveness, Outcomes and Sustainability factors. The evaluation activities shall, therefore, also take a broad view of projects implementation success, taking into account, the manner in which projects were implemented, both intended and unintended outcomes and the extent that results can be attributed to the projects and ultimately the Jobs Fund programme. 1.1 Why evaluation of the Jobs Fund Project As stated in the Jobs Fund Monitoring and Evaluation Strategy, the Jobs Fund Projects are evaluated for two reasons: To find out whether a particular job creation intervention is effective, i.e. are actual new sustainable jobs being created as a result of the JF grant, and what change has occurred in the economy to stimulate the growth and job creation? Secondly, projects are evaluated to try and find out how a particular project has been able to catalyse growth, and why or why not it been successful? How sustainable is a particular initiative (will it end when the JF grant is over)? And importantly, how can a particular job creation initiative be replicated to another industry, geography, or for additional jobs? 1.2 Purpose of the Evaluation Guidelines The purpose of this document is to articulate the evaluation framework for all projects supported through the Jobs Fund programme, including: 3 P a g e

4 To provide the basic understanding of the purposes, processes, norms, and guiding principles for planning and executing evaluations for Jobs Fund projects. To provide knowledge of the essential elements of the evaluation process in the Jobs Fund: developing an evaluation plan; managing, designing and conducting quality evaluations; and using evaluation for managing job creation results, learning and accountability. To enhance the results-based culture among Jobs Fund partners and improve the quality of planning, monitoring and evaluation for job creation. The oversight of project evaluations will be carried out by the Jobs Fund s Monitoring and Evaluation function within the Programme Management Unit. 2 Planning for the Final Evaluation Elements of the Evaluation Plan Background and Context The Background and Context element spells out what is to be evaluated and identifies the critical social, economic, political, geographic and demographic factors within which the project operates that have a direct bearing on the evaluation. The section should include the following: Description of the Jobs Fund project being evaluated, its purpose and objectives, who it is intended to benefit and what outcomes or outputs it is intended to achieve. The duration of the project and its implementation status within that time frame. The scale and complexity of the project and its geographic context. The total financial resources, including the Jobs Fund grant and the match funding. Key partners involved in the project, including the implementing agencies and other key stakeholders Evaluation Purpose This section explains clearly why the evaluation is to be conducted, who will use or act on the evaluation results and how they will use or act on the results. The purpose must include motivation for the need of the final evaluation and how the evaluation forms an integral part of the larger Jobs Fund Programme. Such a clear statement of purpose will ensure a firm foundation for a well-designed evaluation Evaluation Scope and Objectives The evaluation scope and objectives defines the parameters and focus of the evaluation. The section addresses the aspects of the intervention that are to be covered by the evaluation. This would include the timeframe, implementation phase, geographic area, and target beneficiaries to be considered, and as applicable, which of the Jobs Fund Outcomes and project specific output are to be included. This section should also include the project specific primary issues of concern to both the Jobs Fund Programme and the Grantee that the evaluation needs to address. In this regard, the Jobs Fund is 4 P a g e

5 2.1.4 Evaluation Methods The evaluation plan must give details of the specific methods the evaluation will use to collect the information needed to address the evaluation criteria and answer the evaluation questions. The data to be collected and the methods for collecting the data must be determined by the following: The evidence needed to address the evaluation questions; The analyses that will be used to translate the data into meaningful findings in response to the evaluation questions; and Assessment of what data are feasible to collect given constraints of time and resources. The evaluations will draw on data (performance indicators) generated in of terms of the Project Implementation and Monitoring Plan (PIMP) during the project implementation cycle. It must be noted that while the project indicators are useful, they also have limitations. This is because indicators only indicate but they do not explain. Indicators will not likely address the full range of questions the evaluation seeks to address. While indicators provide a measure of what progress has been made, they do not explain why that progress was made or what factors contributed to the progress. It is for this reason that the Jobs Fund recommends that project evaluations should utilize the mixed methods approach so that other data sources and information give meaning to what performance indicators tell us about the project. The data collection methodologies and procedures must be methodologically rigorous and defensible and produce empirically verified evidence that is valid, reliable and credible Evaluation Baseline Data The Jobs Fund Grantee is expected to consult with the JOBS FUND in respect of the baseline Project data to be collected, which data is to be collected at the start of the Project. Baseline data is basic information gathered before the project begins primarily indexed on the contracted indicators in terms of the project s PIMP. Quality baseline data that measure the employment status of the target population are an essential component of effective monitoring and impact evaluation. Without this reference information, it is very difficult to assess how well the project has performed and how effectively it has achieved its objectives. Baseline measurement shall be conducted to establish the base status or status quo prior to the beginning of the grant support for the project against which impacts/changes can be assessed. There are a number of ways in which Jobs Fund projects can acquire baseline data. These include: The project s monitoring and evaluation system (PIMP) Rapid assessment studies Surveys commissioned at the start of the project Secondary data sources. 5 P a g e

6 2.1.6 Example Planning Matrix for Monitoring & Evaluation Outcome Indicator Target Data Source New permanent jobs created as a result of the funded initiatives Number of new permanent jobs: Data Gathering Methods Frequency Information Use Responsibility Output 1.1: Indicator 1.1: Output 1.2: Indicator 1.2: Beneficiaries employed in permanent positions beyond project partners Number of beneficiaries employed in permanent positions beyond project partners Output 1.1: Indicator 1.1: Output 1.2: Indicator 1.2: Beneficiaries employed in permanent positions with project partners Number of beneficiaries employed in permanent positions with project partners Output 1.1: Indicator 1.1: Output 1.2: Indicator 1.2: New short term jobs Number of new short term jobs Etc. Etc.

7 2.1.7 Data Quality Quality data is the cornerstone of M&E. Without it, any results obtained will lack sufficient reliability to form an accurate picture of the efficacy of a programme. A Data Management Plan should be completed for each project, pertaining to all indicators. Data quality can generally be broken down into six categories: Validity: There are precise definitions for all data collected, data sources are validated and there are no sampling or administration errors. Reliability: The survey process and protocols are well defined and consistently applied, with all documentation formulated and correctly implemented. Integrity: Statistical practices are correctly implemented, there is transparency (no perverse incentive to bolster numbers) and ethical standards regarding collection and storage of data are adhered to. Precision: Data contains no bias or errors (transcription and manipulation are common), an error log is maintained. Timeliness: Recording and collection of data follows implementation plan, data is collected inside a timeframe that is appropriate for its subsequent use. Completeness: All data is received as expected and where not, this is recorded and accounted for; procedures are in place for addressing missing data. 2.2 Evaluation Criteria and Evaluation Questions The evaluation criteria help to focus evaluations by making clear the aspects of the Jobs Fund project that will be considered when assessing its performance. Evaluation questions reflect the underlying chain of assumptions about how the project is expected to operate within its contexts as it pursues its intended outputs and outcomes. The questions chosen for an evaluation should be based on a good understanding of the project s operations, intentions and context and should be selected for their role in meeting the evaluation purpose, objectives and relevant evaluation criteria. The Jobs Fund recommends the application of the following most commonly used evaluation criteria, and in terms of which the evaluation questions should be formulated: Relevance: The extent to which the objectives of the Jobs Fund supported project address identified needs of target beneficiaries as well as local and country socio-economic priorities. To what extent are the objectives of the project consistent with the needs and priorities of the target beneficiaries, and relevant to the local and country needs? To what extent have stakeholders been involved in the planning and implementation of the project?

8 Is the project strategy, objectives and assumptions appropriate for achieving the planned results? Are the activities and outputs of the project consistent with the intended impacts and effects of the Jobs Fund? Effectiveness: The extent to which the project s contracted targets and objectives were achieved, or are expected to be achieved, including the extent to which the originally identified problem statement has been successfully addressed through the nominated strategies and models employed by the project. (NB: Attach the Reporting PIMP for the entire project.) How well has the project achieved its stated objectives? What are the reasons for the achievement or non-achievement? How effective has each of the project interventions been (e.g. training, enterprise development, access to finance, and infrastructure development)? How have outputs been transformed into outcomes? What adjustments were made to the project implementation process, the target indicators and why? Has the project effectively and efficiently succeeded in mainstreaming gender equality and youth development (Jobs Fund disaggregated indicators) in its areas of work (outputs) and its processes? Has the management and governance arrangements of the project been appropriate? Has the management and governance arrangements of Jobs Fund been supportive of the implementation process of the project? Efficiency: Efficiency measures the extent to which the project has cost-effectively converted inputs into activities which in turn covert into both qualitative and quantitative outputs. It seeks to ascertain whether the project has achieved good value for money in the realization of its desired outcomes. This generally requires comparing alternative approaches, where applicable, to achieving the same outputs, to see whether the most efficient process has been adopted. To what extent can the costs of the project be justified by its actual results, taking relevant alternatives into account? To what extent has the project been able to build on other initiatives and create synergies with other projects, partners and programmes? 8 P a g e

9 Were the available technical and financial resources adequate to fulfill the project plans? Impact: The Impact criterion seeks to assess the positive and negative, intended and non-intended changes and consequences produced by the Jobs Fund supported project. This should involve impacts and effects resulting from the project on the local social, economic, environmental and other development indicators. In addition to the impact expectations set out in the Jobs Fund grant agreement, which should be assessed, what other impacts can be identified, positive and negative, intended and unintended? Are there any innovative elements of the project with scope to be replicated and/or scaled-up, and if so, which ones? What catalytic effects has the project had in job creation within or outside the sector, in the local area or outside the local area? Sustainability: Sustainability is concerned with measuring whether the benefits of the project are likely to continue after the Jobs Fund support has come to an end. This enquiry therefore is concerned with the project s ability to continue to provide and grow the benefits it would have been delivering during its partnership with the Jobs Fund. What is the sustainability of the permanent jobs created as a result of the funded initiative? How does project growth (in terms of operations, profits, demand and supply, labor relations, etc.) translate into the sustainability of jobs created? To what extent are skills developed, the enterprises created, or infrastructure developed (durability and usage) through the project sustainable? Where should the Jobs Fund focus its future interventions in order to achieve sustainable impacts in job creation? Is the project supported by other local or national institutions? Do these institutions demonstrate leadership commitment and technical capacity to continue to work with the project or replicate it? 9 P a g e

10 3 Evaluation Management The Jobs Fund recommends that the Grantee puts together a management structure for the evaluation, which should be complete with roles and responsibilities to ensure successful conduct of the evaluation. This will also foster the principles of participation and consultation, as well as transparency and accountability. All Jobs Fund projects should appoint an evaluation manager who will assume the day-to-day responsibility for managing the evaluation. The project manager may assume this role in the situation where the Grantee does not have an M&E functionary in its organizational structure or alternatively an external Task Manager may be hired to manage the evaluation process. Day-to-day evaluation management responsibilities will include: Serve as the central point of contact for and during the evaluation Identify, lead and define the roles of other key players, in consultation with stakeholders Lead the development of the evaluation terms of reference Receive and manage the distribution, discussion and acceptance of interim evaluation deliverables or products (such as data collection tools, status or progress reporting, etc.) Ensure the quality and integrity of the evaluation process Facilitate and ensure the use of evaluation Lead the process of responding to evaluation results/recommendations. 3.1 Identifying Evaluation Stakeholders Stakeholders are any persons or groups who have an interest in the project. Analysis of stakeholders of the project is important so that it brings to the fore their interests, needs and influence on the project. This ensures that the evaluation does not address one-sided interests and needs, and enhances the acceptability and use of the evaluation findings and recommendations. The evaluation process should involve the Jobs Fund, the project partners, other project funders, entities beyond project partners (as applicable) as well as beneficiaries of project. Such broad-based involvement of stakeholders will enhance not only the ownership of and mutual accountability for results, but also the credibility and transparency of the evaluation exercise. Stakeholders of the evaluation should be consulted and engaged, when appropriate, in developing an evaluation plan, drafting the evaluation Terms of Reference, appraising the selection of evaluators, providing the evaluators with information and guidance, reviewing the evaluation draft, preparing and implementing the management response, and disseminating and use of the knowledge generated from the evaluation. 10 P a g e

11 3.2 Use of External Evaluation Specialists The recruitment and selection of external evaluation professionals will be guided by the Grantee s contracting policies and procedures. Consider carefully all of the expertise that may be required to do the evaluation based on the nature of the project and its context. Emphasis should be placed on evaluation expertise, knowledge of the sector in which the jobs are being created, expertise in the subject of the project, relevant language skills, and experience in and sensitivity to the cultural and social environment. This expertise may not be found in a single individual, but rather as the collective expertise of the evaluation team. The main deliverables of the evaluation team should include the following: Inception Report: The inception phase and resultant inception report merges the Terms of Reference and the evaluation team's proposal into a definitive evaluation design, elaborating on what will happen and when within the evaluation. The inception phase provides an opportunity for the evaluation to be fine-tuned in approach, methods, sampling, timing, and roles and responsibilities. Importantly, the Inception Report should be very clear on how the evaluation team will report to and engage with the evaluation manager and the stakeholders throughout the evaluation process. Information and data collection: Information and data collection is undertaken by the evaluation team. The manager and stakeholders will be called upon to facilitate this process by providing background documentation, identifying contacts, and potentially providing access to systems and work sites. Analysis of information and data should occur throughout the data collection period. Final report: Once all information and data has been collected, an analytical process is undertaken that involves the integration, comparison and synthesis of information and data derived across and through all methods. The analysis includes an assessment of what the information is saying about each of the evaluation questions. The draft and final reports are derived from this analytical process. 3.3 Evaluation Report The evaluation report should ideally include the following: 1. Title page, table of contents, acronyms 2. Executive summary 3. Purpose of the evaluation 4. Evaluation objectives and scope 5. Evaluation Methodology 6. Context and description of the project 7. Findings 8. Analysis 9. Conclusions 10. Recommendations 11. Lessons learnt 12. Annexes 11 P a g e

12 4 Evaluation use and follow-up Through the generation of evidence and objective information, evaluations will enable the Jobs Fund and its partners to make informed decisions and plan strategically. When evaluations are used effectively, they support programme improvements, knowledge generation and accountability. The intended users of evaluation are those who have a vested interest in the evaluation results and are in a position to make decisions or take action based on the evaluation results. The following table summarizes the use of the evaluations: Information sought through the evaluation: Users of the Evaluation Results: How the Evaluation Results will be used: The relevance of intended outputs or outcomes The validity of the results framework and results chain The status of the Jobs Fund outcomes and factors affecting them The effectiveness of the Jobs Fund implementation strategy The status of project implementation The cost of the project relative to the observed benefits Lessons learned. The National Treasury, GTAC and others involved in the design and implementation of the Jobs Fund The Jobs Fund Programme and other oversight bodies Government entities, policy makers, strategic planners The Jobs Fund Grantees their partners Other funders of similar projects The public and project beneficiaries To design or validate job creation models and strategies To improve project design and implementation To ensure accountability To make funding decisions To increase knowledge and understanding of the benefits and challenges of programmes and projects intended for the creation and enhancement of sustainable jobs. 12 P a g e