Evaluation Capacity Building in the EU: Counterfactual and Theory Based Approaches for Evaluating Impact

Transcription

1 International Seminar Evaluation Capacity Building in the EU: Counterfactual and Theory Based Approaches for Evaluating Impact Barcelona (ES), November 2015 European Institute of Public Administration Institut européen d administration publique Introduction Target group Managing authorities at all levels, Directors or civil servants responsible for designing evaluation systems, managing and monitoring evaluations carried out by external evaluators; as well as those responsible for controlling public administrations at different governmental levels, and who need to develop or are interested in developing their knowledge about public policy evaluation. Description The seminar does not aim to train participants on how to use evaluation techniques; however, different approaches and methods will be presented and explained with the aim of analysing the advantages and disadvantages that justify the use of some above others. The aim is to raise awareness on the wide range of approaches that are deployed in the name of evaluation, and serve a range of different purposes (process and impact evaluation). The critical question is how these can tell policy makers what and what they cannot get from them. The examples presented in the seminar, will show how combining different types of impact evaluation in the appropriate way (counterfactual (empirical) or theory-based approaches) complement each other and help managing authorities to reach better and more useful evaluations. Objectives Does your country have qualified experts who are willing and capable of working on designing, managing and monitoring evaluation activities? Will you be able to ensure the quality of evaluation? Will evaluation results be used to improve programmes and make evidence based decisions? Both contracting authorities and service providers have been learning and growing together. The member states have launched and implemented evaluation capacity building projects, which included training for representatives from EU Structural Funds administrations and potential evaluators, drafting methodological documents and publications on evaluation, holding public discussions and events for the presentation of evaluation results. Today, we can already say that most MS have a sufficiently developed evaluation culture. We have an efficient evaluation planning and coordination system and a small but qualified market of experienced evaluators. Evaluation results are communicated to the public. However, there are still many challenges left: how to ensure better dissemination of evaluation results and improve the perception of evaluation benefits, how to promote the use of evaluation results in decisionmaking processes, how to strengthen evaluation capacities and make better use of the potential of evaluation. All these challenges should be addressed by ongoing evaluation capacity building activities. This seminar will raise your capacity to design, manage and monitor the evaluations conducted during the actual programming period. The main aim is not to train participants as evaluators, but to put them in a position to manage evaluations and understand the basics of certain tools and approaches so they can competently interact with (external or internal) evaluators and assess the quality of their work. In particular, the seminar will focus on enhancing participants understanding of two families of approaches that have recently been the focus of much debate in evaluation circles across Europe and beyond. Counterfactual (also known as empirical or (quasi) experimental) approaches attempt to tackle the question of attribution (how can I be certain that the effect I observe has been caused by the intervention I am evaluating?) by identifying or constructing a control group. While such approaches have been and are still being seen as the gold standard by many, there is increasingly a recognition that they are not feasible (and arguably also not desirable) in some cases. The seminar will attempt to provide participants with a deeper understanding of what counterfactual approaches can and cannot do, and will discuss theory-based approaches as a possible alternative to addressing causality where a suitable counterfactual cannot be found. Learning methodolgy The first sessions will cover the introduction to evaluation and evaluation design, to set the level playing field for all participants. In the afternoon we will challenge participants with the presentation on Counterfactual impact evaluation by an expert on the topic who will be presenting examples and best practices. On the second day, we will introduce Theory-based evaluation, again followed by a practical example, and then a discussion to compare the two groups of approaches and when they each might be best used. We will end with the wrap-up session and discussion on how best to combine different types of evaluation in the appropriate way with presentation of cases brought up by the audience and discussed with the experts.

2 Programme THURSDAY 19 NOVEMBER 2015 FRIDAY 20 NOVEMBER Registration of participants Opening and introduction to the seminar: presentation of the methodology to be followed. Tour de table presentation of participants Gracia Vara Arribas, Expert, Project Leader, EIPA Barcelona (ES) Main concepts of evaluation: what is evaluation about and why do we do it? Where does it stand in the public policy cycle? Gracia Vara Arribas Designing an evaluation (1): defining the purpose, terms of reference, evaluation criteria and questions Perspective from the civil servant who is designing/ commissioning an evaluation, and who has to draft terms of reference, decide on the main purpose (summative or formative), decide on the evaluation criteria (relevance, effectiveness, efficiency, etc.) and formulate evaluation questions Daniele Vidoni, European Commission Coffee break Designing an evaluation (2): Introduction to evaluation approaches and design types of evaluation, approaches, and research design From the perspective of the evaluator who has to propose an appropriate evaluation design to answer the evaluation questions: key characteristics of impact, process and economic evaluations, implications for the research design and methods used to structure evaluation. Martin Kuehnemund, SQW (UK) Theory-based evaluation: What is it, how does it work, and how can it be used to evaluate impact? Overview of theory-based approaches and techniques to conduct evaluations, and of if and how they are suitable to assessing impact and attribution / contribution. Approaches covered will include realist evaluation, contribution analysis, and Qualitative Comparative Analysis. Martin Kuehnemund Example of TBE Group discussion Coffee break Discussion and comparative experiences Discussion of specific cases and examples provided by participants, and the applicability, pros and cons of different approaches to these specific examples Lunch Combining different types of evaluation in the appropriate way: presentation of good examples Managing Authority (tbc) Wrapping up session and presentation of group conclusions Closing of the seminar Lunch break Counterfactual impact evaluation: What is it, and how does it work? Overview of quasi-experimental methods and techniques to quantitatively evaluate the impact of an intervention by comparing with a counterfactual. Approaches covered will include difference-in-difference, matching, and discontinuity design. Daniele Vidoni, European Commission Example of CIE Group discussion Moderater: Martin Kuehnemund End of the first day Welcome dinner

3 Experts (in alphabetical order) Gracia Vara Arribas (ES) Lawyer LLM, Senior Lecturer and Expert in Multilevel Governance and EU Law (EIPA Barcelona). For the last ten years Gracia has been devoted to developing the capacity of regional and local actors in building a shared method of governance between the various levels, and the use of new instruments of Multilevel Governance. This service is provided through research, training and consultancy work for various levels of administration, as well as, European institutions. Under her direction, several impact assessment and evaluation studies have been carried out, commissioned by the EU Presidencies (UK 2006 and SE 2009) and for different Member States. In 2010, she was commissioned by the Dutch Ministry of Interior to evaluate the impact of the Lisbon Treaty on the Dutch Provinces and Municipalities. At this moment she is mainly working as a consultant for the Committee of the Regions. Martin Kuehnemund (DE) Martin is an expert in all phases and facets of the (ex post) evaluation and (ex ante) impact assessment (IA) of public policies, programmes, and other legislative and nonlegislative measures, principally in the specific context of the European Union (EU), its institutions and policies. He has vast international experience across the EU member states and beyond. His academic background is in political science, economics and European law. In addition to his native German, he is fluent in English and Spanish. Martin has successfully delivered over two dozen assignments for clients from across the European Commission (EC) and other EU institutions, spanning a wide range of projects (evaluations, IAs, feasibility and other studies), subjects (policies, programmes, legislation, etc.) and policy areas (including public health, business, regional policy, employment and social affairs, customs, research policy, and better regulation) during his time with SQW, and previously The Evaluation Partnership and Coffey. European Institute of Public Administration EIPA Barcelona C/ Girona, 20 / Barcelona Project Leader Gracia Vara Arribas EIPA Expert Tel g.vara-arribas@eipa.eu Daniele Vidoni (IT) European Commission

4 General Information Programme The programme will commence on Thursday at and will finish on Friday at Seminar venue Hotel Grupotel Gran Via 678**** Gran Via de les Corts Catalanes, 678 ES Barcelona Tel Working language The seminar will be conducted in English. Registration Participation fee: 950 (includes: documentation, coffee breaks, two lunches, a welcome dinner and a certificate of attendance). Accommodation and travel costs are at the expense of the participants or their administration. EIPA offers its members a reduction of 10% off the registration fee. This reduction is available to all civil servants working for one of EIPA s member countries (i.e. AT, BE, BG, CY, CZ, DK, FI, FR, DE, GR, HU, IE, IT, LT, LU, MT, NL, PL, PT, ES, SE, UK) as well as civil servants working for an EU institution or agency. If you are eligible for a discount, please tick the box in the registration form. Please note that reductions cannot be accumulated. For more information, please visit EIPA s website: Deadline for registration: 5 November 2015 You can also submit the online registration form which can be found on EIPA s website: The number of participants is limited (we will not accept any registrations once the available places have been filled). Your name and address will be part of EIPA s database for our mailing purpose only. If you do not want to be included in our mailing database, please tick the box on the registration form. Payment before the deadline is a condition for participation. For cancellations received after the deadline we will have to charge an administration fee of 150 unless an alternative participant is nominated. Information and registration Raymond Pelzer, Programme Organiser C/ Girona, 20 ES Barcelona Tel.: Fax: r.pelzer@eipa.eu Hotel reservations EIPA Barcelona will be pleased to make reservations for you at the hotel where the seminar takes place (special rates are applied for EIPA participants): Hotel Grupotel Gran Via 678**** Gran Via de les Corts Catalanes, 678 ES Barcelona Tel Rates: 90 Single room 100 Double room Breakfast included, 10% VAT not included City tax: 1.21 per night and person Should you wish to make use of this possibility, please indicate the dates of arrival and departure on the registration form. Payment is to be made directly and personally to the hotel on checking out. Please note that if you register after the closing date, hotel reservations cannot be guaranteed. Confirmation Please note that your registration is not finalised until you receive a confirmation from us. Moreover, your participation is not confirmed until you receive another from the Programme Organiser confirming your registration and some details. Cancellation policy EIPA reserves the right to cancel the seminar up to two weeks before the starting date. EIPA accepts no responsibility for any costs incurred (travel, hotel, etc.). For EIPA s cancellation policy, please visit our website (FAQ - cancellation policy)

5 Registration Form Evaluation Capacity Building in the EU: Counterfactual and Theory Based Approaches for Evaluating Impact Barcelona (ES), November 2015 Surname:... Title:... M / F First name:... Organisation:... Department:... Current position:... Work address:... Postal code & Town:... Country:... Telephone number:... Fax number:... address:... Invoice information Organisation:... Fiscal number/nif:... Department:... Address:... Postal code & Town:... Country:... VAT number:... Your reference number:... address:... Registration fee The fee includes participation in the seminar, documentation, coffee breaks, two lunches, a welcome dinner and a certificate of attendance Discount of 10% (Please check first the conditions at (FAQ - special discounts)) 950 Method of payment Bank transfer It is necessary to state the name of the participant and the reference EIPA s Barcelona account number IBAN: ES , BIC: BBVAESMM at the BBVA, Plaça Catalunya 5, Barcelona Credit card American Express card Eurocard/Mastercard Visa card Card number:... Expiry date:... /... Card Validation Code:... Hotel reservation Please make a reservation for me at Hotel Grupotel Gran Via 678**** Double room for single use Date of arrival:... Date of departure:... Double room Number of nights:... No hotel reservation required Meals Vegetarian Fish allowed Other dietary requirement:... To ensure the optimal organisation of the seminar, please return the completed registration form by fax or before 21 September 2015 to: Raymond Pelzer, Programme Organiser, Tel.: ; Fax: ; r.pelzer@eipa.eu, The registration will be confirmed by . Your name and address will be part of EIPA s database for our mailing purpose only. Please tick if you do not want to be included in our mailing database.