Key elements of evaluation as a decision making tool: the Specific Performance Evaluation (EED) experience in Mexico. Author(s): Hortensia Pérez, Analysis and Results Follow-Up Director, Consejo Nacional de Evaluación de la Política de Desarrollo Social (CONEVAL); Thania de la Garza, General Director of Evaluation, CONEVAL. Presenting Author: Hortensia Pérez, Analysis and Results Follow-Up Director, hperez@coneval.gob.mx, +52.55.54.81.72.18 Sub-topic: Factors that affect the use of evaluation and how to enhance use/ National Learning Culture and absorptive capacity for use/ Socialization of evaluation and its results. Developing an effective results-oriented budgeting system is even more difficult than theory suggests. It is not only needed to build and reinforce the evaluation culture, but to create comprehensive and practical tools to display evaluation outcomes. Experience also implies that the use of evaluation results will increase if the utility of the information in the decision making process is evident to the final clients, the decision makers. During the seventies, Mexico government made the first systematic attempts on evaluation, with the assistance of international organisms. Several isolated exercises were done for the evaluation of federal programs. Later, during the nineties, there was a major step forward on the evaluation of social programs with the beginning of the Progresa program, now called Oportunidades, because its evaluation strategy was designed at the same time of its implementation. In the year 2001, in order to obtain information about the achievements of public programs, Mexican Congress determined that autonomous agents of the ministries that operate the programs should conduct evaluations about its design, implementation, satisfaction of beneficiaries and impact. Although this effort allowed the improvement in the efficiency of social programs, it was still not possible to identify complementarities and similarities between them and was not clear how to use the results information for planning or budgeting decisions. Since 2004, the evaluation process in Mexico is institutionalized through the approval of two laws: the General Law of Social Development and the Budget and Fiscal Responsibility Law. These laws clearly identify the actors in charge of the evaluation of public programs which are the Consejo Nacional de Evaluación de la Política de Desarrollo Social (CONEVAL), the Ministry of Finance and the Audit Ministry. CONEVAL is a federal decentralized public organization, with autonomy and technical capability to generate objective information on the social political situation and poverty measurement in Mexico. CONEVAL has two main functions:
1. Regulate and coordinate the evaluation of national policy on social development and of social policies, programs and actions executed by public dependencies. 2. Establish the guidelines and criteria used to define, identify and measure poverty, guaranteeing transparency, objectivity and technical rigor. Both functions intend to provide valid and reliable information to decision-makers in order to make policies and programs efficient and to inform the citizens about the results of social policy. Focusing on monitoring and evaluation, CONEVAL has being dealing with some challenges 1 : A. Institutional, which implies the definition and implementation of game rules for the agents involved, in order to establish the actions and responsibilities for the creation and use of information. B. Challenges in matter of linking the national priorities and needs with the public policy instruments through a casual logic scheme in which the results that are expected to obtain are well explained. C. Technical challenges related to the definition of appropriate methodologies for evaluating policies and programs, the collection of necessary information to make the evaluations and the identification of evaluation organisms. In order to face the challenges some strategies have been made, including the definition of a Monitoring and Evaluation System 2 that consist in planning and evaluating with a results oriented view. The planning step tries to align the social programs indicators (according to the Logical Framework Methodology) with the government strategic goals. On the other hand, the evaluation phase includes these seven types of evaluation for programs and policies: Consistency and Results Evaluation, which is an assessment of the institutions' capacity to achieve goals. This evaluation allows counting with a diagnosis on the institutional, organizational and management capability of 1 Hernandez, Gonzalo. CONEVAL s M&E System presentation. 2 See Appendix 1. Monitoring and Evaluation System Diagram
results-based programs. CONEVAL issued the terms of reference model with 51 questions and among its main objectives are the analysis of the design of the programs based on the results matrix, the obtainment of relevant information regarding the operation of the programs and the results that have been gained. Processes Evaluation, which analyzes the contribution of operational processes to the purpose of the program. This evaluation detects the strengths, weaknesses, opportunities and threats of the regulatory framework, structure and functioning of programs, focusing in determine strategies that may enhance operative effectiveness and enrich the programs design. Indicators assessments, which analyses relevance and range for the indicators. Impact Evaluations, these measure the effects attributable to programs. One of the main challenges in matters of evaluation is determining if social programs meet the objectives for which they were designed. Therefore, the development of impact evaluations is relevant, using a rigorous methodology to determine if the program had an impact on economic features or those relative to the wellbeing of their beneficiaries, by means of the use of the appropriate indicators. Complementary assessments including additional topics, which are not addressed in defined evaluations. Strategic assessments which are policy evaluations of a set of programs. Specific Performance Evaluation, this evaluation compiles the information of all of the above and some other available information. It will be analyzed more in the following paragraphs, because of it experience as a decision making tool. The Specific Performance Evaluation (EED, by its Spanish acronym) is a summarized assessment about the annual performance of social programs, which is presented through a unified format. EED presents the progress of the planned objectives and goals of a social program, based on a summary of the information contained in the Performance Evaluation System 3 (SED, by its Spanish acronym) such as external evaluations, internal reports, coverage data, and through the analysis of results, services and management indicators. 3 The Specific Performance System (SED) was created upon the Budget and Fiscal Responsibility Law, as an element of the results based budget. The system is operated by the Ministry of Finance, but the EED has its own module managed by the CONEVAL.
This evaluation is aimed at decision making actors as ministers, programs managers, congressmen and policy analysts, and it tries to trim and describe the most relevant results and findings of each program, interpreted by the expert opinion of the external evaluator. In 2008, the challenge was to create a practical tool or report that could be used in the budgeting process, attractive to the reader, comprised of no more than ten pages, comparable within programs, amicable and easy to understand even if you are not in the evaluation field. Since then, CONEVAL has coordinated around 130 EED per year 4 (2009-2011), improving the format design and content evaluation as a result of feedback sessions with the evaluators, the evaluated programs, the participant agencies and the principal users. The evaluation is nine pages long and content the following general subjects: Results. The advance in responding to the problem or need which the program was created for. Management. The efficiency of delivering goods and services to the beneficiary population and budgetary accounting. Coverage Population. The number of inhabitants serviced and their geographic location. Follow-up of recommendations. The use of findings in evaluations reports to improvement and the achievement of those improvements. The EED development cycle has restricted times; it starts in April, when the official information of the last fiscal year is published (Cuenta Pública), therefore, all the indicators values, coverage data and budgeting information are available and validated; and it ends in June, so it results can be use in the budgeting process of the next year. Because of time restrictions, and in order to have reports and systematic information, the EED is done in an internet based CONEVAL software denominated 4 The EED Reports per year can be consulted in www.coneval.gob.mx
Development Evaluation System (SIEED, by its Spanish acronym), that generate a homogenous format for the final informs. As it was established before, this evaluation is designed for decision makers, but it also responds to three major goals: support the decision making process, inside or outside the program; improve programs and policies, by knowing their results, opportunities and limitations, and contribute to the government accountability. Some of the main achievements regarding this three-year evaluation experience are: the increasing of evaluation readers, people are actually reading the reports, because of the format and summarized information; it has also increase the programs interest in developing better information and more accurate data, and it has become an input to the Ministry of Finance for the budget negotiation. Based on the EED results, there are some tools generated for the budgeting process. One of them is a traffic light system (a simple way to identify programs strengths or red lights ); another is a one page program summary (useful to identify relevant information), and also databases with all the information available. In this time, the EED has evolved and matured, especially in the implementation mechanism. We reaffirmed that the participation of all the stakeholders is vital, hence since the second year there are two mandatory meetings between the evaluator and the program s officers, and the constant communication is strongly recommended. There are some lessons learned or even remembered about the evaluation process and culture. First, evaluation is complex, and it is not directly related to the increasing or decreasing of a program budget, but it can certainly help to identify straights and weaknesses, that analyzed with the priorities and governmental context will support the decisions that have to be made. Second, it is important to know your users, to define the purpose of each evaluation, and to make sure that all the actors understand the concepts and the evaluation practical language. Third, evaluation for decision making processes can be less cost effective, in terms of timing and effort, if it is not accompanied by enhancing use strategies. Finally, sometimes the non evaluated programs have an advantage, because they do not make their results information as clear and available as the evaluated ones, and that may discourage the programs participation.
Thus, CONEVAL tries to enrich and support the decision-making process by delivering the EED Report to strategical actors. The learning process in the development of the EED has allowed innovation in different key elements related to the report s design, the implementation method and the coordination between actors during the evaluation process, which all together could be a useful experience to share with other evaluation agencies and countries.
Appendix 1 Monitoring and Evaluation System Diagram CONEVAL. Informe de Evaluación de la Política de Desarrollo Social 2008.