Evaluation for the European Commission

Transcription

1 THEMATIC EVALUATION OF THE EC SUPPORT TO GOOD GOVERNANCE FINAL REPORT Volume 3 SYNTHESIS NOTE AND COUNTRY NOTES Contract Number: EVA/ Service Contract for the Evaluation (sectoral and thematic) of European Commission Programmes and Policies in Third Countries, relating to Social and Human Development issues. March 2006 Evaluation for the European Commission

2

3 A consortium composed of PARTICIP, Cideal, Channel Research and South Research c/o lead company PARTICIP: PARTICIP GmbH, Consultants for Development & Environment Headquarters: Hildastrasse 66, D Freiburg, Germany Brussels Branch: Avenue des Arts 50 (5th floor), B 1000 Bruxelles, Belgium Service Contract for the Evaluation (sectoral and thematic) of European Commission Programmes and Policies in Third Countries, relating to Social and Human Development issues. Centro de Investigatión y Cooperación al desarollo Madrid, Spain Thematic Evaluation of the EC support to Good Governance Channel Research Brussels, Belgium With the Collaboration of ECDPM South Research Leuven, Belgium Project Supervisor is project manager of EuropeAid Co-operation Office, Evaluation Unit H6 Contract manager is Mr René Madrid from PARTICIP GmbH International Experts National Experts Jean Bossuyt (Team leader) Emery Brusset (thematic expert) Harry Garnett (thematic expert) Marc de Tollenaere (thematic expert) Katharina Madrid (thematic expert) René Madrid (Key expert) Marion Miketta (Junior expert) Lucio Filippini (Junior expert) Frédéric Ceuppens (Junior expert) Serani Siegel (Junior expert) Helge Rieper (Junior expert) Ramon Seiffe (Domenican Republic) Mutaz Al-Taher (jordanie) Andrei Marusov (Ukraine) Lamien Ouando (Burkina Faso) Malika Ramoun (Algeria) Cassinda Ernesto (Angola) Herminio Castillo (Guatemala) Maria Pakpahan (Indoniesia) The opinions expressed in this document represent the authors points of view, which are not necessarily shared by the European Commission or by the authorities of the countries concerned. Thematic Evaluation of the EC support to Good Governance Final Report - March PARTICIP GmbH

4 Thematic Evaluation of the EC support to Good Governance Final Report - March PARTICIP GmbH

5 FINAL REPORT Volume 1 Volume 2 Volume 3 Synthesis Report Annexes Synthesis Note and Country Notes VOLUME 3 - TABLE OF CONTENTS Overall analysis Country Note 1 Country Note 2 Country Note 3 Country Note 4 Country Note 5 Country Note 6 Country Note 7 Synthesis Note Angola Burkina faso Domenican Republic Guatemala Indonesia Jordan Ukraine Thematic Evaluation of the EC support to Good Governance Final Report - March PARTICIP GmbH

6 TABLE OF CONTENT OF VOLUME 1 1 EXECUTIVE SUMMARY Purpose of the evaluation Background to the evaluation Methodology Analysis and main findings for each evaluative question Main conclusions Main recommendations INTRODUCTION Purpose of, and background to the evaluation Commission s strategies and programs : objectives and intended impact Context: Brief analysis of the political, economic, social and cultural dimensions, as well as the needs, potential for and main constraints METHODOLOGY General approach Approach to the inception phase Approach to the desk phase Approach to the field phase Synthesis phase Main Limitations Method of data and information collection Method of data and information analysis Methods of Judgement Quality assurance MAIN FINDINGS AND ANALYSIS Mainstreaming and priority area Tools and working practices, financial mechanisms and instruments Tools and working practices, financial mechanisms and instruments Tools and working practices, financial mechanisms and instruments Partner-country ownership, relevance of Commission governance support, and flexibility in adapting to different country contexts The three CS s : Coherence, complementarity and coordination Cross-cutting issues Achievement of objectives Actors and levels of governance Institutional Capacity at the level of the EC MAIN CONCLUSIONS AND LESSONS Main conclusions MAIN RECOMMENDATIONS Overall recommendation Strategic and operational recommendations Thematic Evaluation of the EC support to Good Governance Final Report - March PARTICIP GmbH

7 VOLUME 2 - TABLE OF CONTENTS Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 Annex 7 Annex 8 Annex 9: Annex 10: Annex 11 Annex 12 Annex 13 Annex 14 Annex 15: Annex 16 Annex 17 Annex 18: Annex 19 Annex 20 Annex 21 Annex 22 Annex 23 Annex 24 Terms of reference Review of EC governance objectives and Impact diagrams EC policy and institutional framework of governance Comparative analysis of the different regional policies Evaluative questions, judgement criteria and indicators Evaluation analysis methodology Method of data and information collection and analysis Approach to quality assurance and common approach Selection process for the field visits Selection process for the CSP analysis Country Strategy Paper analysis Lessons learnt from other donors evaluations and some relevant EC evaluations Integration of governance at the regional level Proposed list of weighting good governance in existing DACsectors Statistical analysis: EC budget commitments and payments analysis Delegation questionnaires: template Summary of the Questionnaire analysis Questionnaires for Delegations: management and main results Results from focus groups Links between findings, conclusions and recommendations Recommendations per Evaluative Questions Working Plan List of documents and information sources List of persons met Thematic Evaluation of the EC support to Good Governance Final Report - March PARTICIP GmbH

8 Thematic Evaluation of the EC support to Good Governance Final Report - March PARTICIP GmbH

9 THEMATIC EVALUATION OF THE EC SUPPORT TO GOOD GOVERNANCE Synthesis Note Service Contract for the Evaluation (sectoral and thematic) of European Commission Programmes and Policies in Third Countries, relating to Social and Human Development issues. January 2006 Evaluation for the European Commission

10

11 A consortium composed of PARTICIP, Cideal, Channel Research and South Research c/o lead company PARTICIP: PARTICIP GmbH, Consultants for Development & Environment Headquarters: Hildastrasse 66, D Freiburg, Germany Brussels Branch: Avenue des Arts 50 (5th floor), B 1000 Bruxelles, Belgium Service Contract for the Evaluation (sectoral and thematic) of European Commission Programmes and Policies in Third Countries, relating to Social and Human Development issues. Centro de Investigatión y Cooperación al desarollo Madrid, Spain Thematic Evaluation of the EC support to Good Governance Channel Research Brussels, Belgium With the Collaboration of ECDPM South Research Leuven, Belgium Project Supervisor is project manager of EuropeAid Co-operation Office, Evaluation Unit H6 International Experts Contract manager is Mr René Madrid from PARTICIP GmbH National Experts Jean Bossuyt (Team leader) Emery Brusset (thematic expert) Harry Garnett (thematic expert) Marc de Tollenaere (thematic expert) Katharina Madrid (thematic expert) René Madrid (Key expert) Marion Miketta (Junior expert) Lucio Filippini (Junior expert) Frédéric Ceuppens (Junior expert) Serani Siegel (Junior expert) Helge Rieper (Junior expert) Ramon Seiffe (Domenican Republic) Mutaz Al-Taher (jordanie) Andrei Marusov (Ukraine) Lamien Ouando (Burkina Faso) Malika Ramoun (Algeria) Cassinda Ernesto (Angola) Herminio Castillo (Guatemala) Maria Pakpahan (Indoniesia) The opinions expressed in this document represent the authors points of view, which are not necessarily shared by the European Commission or by the authorities of the countries concerned.

12

13 i TABLE OF CONTENT I. INTRODUCTION...1 II. APPROACH AND METHODOLOGY...2 III. CONTEXTUAL FACTORS...3 IV. PRELIMINARY FINDINGS...5 V. THE WAY FORWARD...15 Thematic Evaluation of the EC support to Good Governance Synthesis Note - January PARTICIP GmbH

14 ii Thematic Evaluation of the EC support to Good Governance Synthesis Note - January PARTICIP GmbH

15 1 I. INTRODUCTION 1. The purpose of this evaluation is to provide the relevant external co-operation services of the European Commission and wider public with an independent evaluation of the EC s support for good governance, in terms of the relevance, the efficiency, and effectiveness of that support and the sustainability of the impacts on good governance processes and in encouraging and promoting good governance approaches and practices, in third countries. The evaluation focuses on a set of fundamental issues (translated in evaluation questions) and should make possible a general overall judgement of the extent to which Commission strategies, programmes and projects have contributed to the progress towards good governance. 2. For this evaluation, a working definition was agreed upon and spelled out in the Terms of Reference: Governance refers to the structure, functioning and performance of public authorities/institutions at all levels. Governance is about the way public functions are carried out (including public service delivery), public resources (human, natural, economic and financial) are managed and public regulatory powers are exercised (including enforcement) in the management of a country s affairs. The Terms of Reference also provide an indication on the meaning of good governance, which is considered as a process and an aspiration towards governance systems adhering to a set of key values 1. It was furthermore agreed that in order to get a feasible scope, the evaluation shall focus on the support provided to four thematic governance clusters Since donor support for good governance processes and programs is relatively new, the evaluation is forward looking, providing lessons and recommendations for the continued support to governance in particular as regards: the use of dialogue in supporting governance; the efficiency and effectiveness of capacity building activities (including TA and twinning); the potentials of supporting governance through budget support and sector policy support programmes (SPSPs); the Commission as a change agent in the governance process; the Commission s capacity to adapt its support and approaches to different types and country situations. 4. The evaluation is in an advanced stage of execution. The desk study has been finalized (including a detailed regional analysis of governance strategies and approaches) while the field phase (including 8 country studies) 3.has been completed with the production of country notes. This Synthesis Note is an important step in the process of elaborating the final report. Hence, it is useful to properly understand its nature and role. According to 1 Including : (i) efficient, open, transparent (non-corrupt) and accountable public institutions at all levels, including clear decision-making procedures; (ii) sound, efficient and effective management of human, natural, economic and financial resources for the purpose of equitable and sustainable development; (iii) a democratic society managed with respect for human rights and democratic principles; (iv) civil society participation in decision-making procedures; (v) the existence of, respect for and enforcement of the rule of law and the ability to enforce rights and obligations through legal mechanism (see Terms of Reference p. 6). 2 The 4 thematic clusters are : support to public administration reform, including public finances; decentralisation and local government reform; the rule of law; and the empowerment of civil society related to good governance processes. The evaluation is not expected to focus specifically on two other governance clusters (i.e. human rights and democracy). 3 Angola, Burkina Faso and Dominican Republic (for the ACP) ; Ukraine (TACIS) ; Indonesia and Guatemala (ALA), Jordan and Algeria (MEDA). Thematic Evaluation of the EC support to Good Governance Synthesis Note - January PARTICIP GmbH

16 2 the Terms of Reference, the Synthesis Note is to be prepared when all the field missions have been conducted and before the start of the synthesis phase with the purpose of summarizing the data and information collected and presenting preliminary findings, including information gathered during the Desk Phase. It should be succinct and mainly constitute a basis for a Reference Group meeting to prepare the synthesis phase. It is also foreseen that the document will not be published as a separate note. 5. The evaluation team welcomes the opportunity to check preliminary findings and emerging conclusions with the Reference Group before starting the work on the final report. In order to keep the note succinct, it was necessary to be selective in the data and information provided. The choice was also made to present an aggregated analysis (rather than a sliced approach per evaluation question). This seems more appropriate at this stage of the process for a synthesis document aimed at facilitating a focused discussion on key preliminary findings. II. APPROACH AND METHODOLOGY 6. The Inception Report, approved by the EC, spells out in detail the approaches to be followed by the evaluation team during the different phases of the exercise (desk, field and synthesis phase). 7. From the outset, it was understood that the evaluation has to respond to both the need for accountability and for learning (including recommendations to improve the overall EC performance in this area). In order to properly assess the learning objective, the evaluation team has applied an analytical approach aimed at assessing the extent to which objectives have been reached as well as looking at the reasons and determining factors behind the observed successes and failures. As good governance should be seen as a process, the evaluation has assessed achievements in the light of changes, developments and trends rather than against fixed and standardised targets. This is needed considering the relative novelty of governance as a key area for EC support and the ongoing evolutions. Efforts were made to ensure comparison with other donor agencies and their experiences with delivering governance programmes and building their own capacity. 8. The evaluation team has been sensitive to the context-specific nature of good governance, and the different frameworks within which cooperation is conducted for the five regions. Given the lack of a clear, overall framework for (good) governance over the evaluation period, and given the regional differences, the evaluation will assess the European Commission s cooperation activities supporting good governance, relative to the general and specific objectives of the different regional cooperation and development programmes. This approach will be balanced with the need to keep a level of coherence between the different regional perspectives and case studies so as to facilitate the synthesis of the findings, conclusions, lessons and recommendations in the final report. 9. The evaluative approach was further specified by defining ten evaluation questions as well as different methods of data collection (i.e. analysis of 23 questionnaires from EC delegations ; analysis of 35 CSP ; 8 field studies literature reviews; interviews both structured and unstructured; PRA techniques ; SWOT analysis; focus groups; identification of the intervention logic; use of evaluative questions; instruments analysis; as well as a statistical analysis) that were used to cross-check the information obtained. Thematic Evaluation of the EC support to Good Governance Synthesis Note - January PARTICIP GmbH

17 3 10. Several limitations were encountered in the process of executing the evaluation: The desk study struggled with the sheer scope of a rapidly (expanding) governance agenda and the manner in which it is understood in different parts of the EC and the Delegations. It therefore had to be selective, concentrating on the most essential information. The novelty of governance meant there was a deficit of clear policies, accumulated knowledge and institutional memory in certain areas. It also proved difficult to assemble reliable statistical data (for details see Desk Report). The huge number of documents tackling one or more good governance issue, thus making the synthesis work more difficult; In the field phase, it proved difficult to convince the Delegations that the purpose of the country studies was to provide information for a study of the EC s support as a whole and not to carry out an outcome evaluation of the Delegations own support for specific governance activities). The planned country visit to Algeria (i.e. the second MEDA country to be studied) could not take place, as not agreement could be reached with the government The short period of visit did not always allow the use of all standard methods for data collection (e.g. focus group discussions). Availability of EC officials (especially RELEX staff) sometimes posed problems, both in Brussels (e.g. for the three focus group discussions that were organised 4 ) and in the field. This was compounded by the recent organizational restructuring, which led to a major rotation of staff. A shortage of indicators of outcomes used by the EC for its activities, i.e. designed, collected and analysed in a systematic manner, over and beyond the useful project monitoring reports found in some locations. A large part of EC supported programmes in the field of governance have only recently been formulated or implemented 5. It is generally too early for there to be significant outcomes on the ground (considering the nature of governance work). This imposes major limitations on the possibility of making judgments on impact. III. CONTEXTUAL FACTORS 11. Before presenting the key preliminary findings, it seems appropriate to set the scene by highlighting a number of important contextual elements related to governance. Four context factors stand out as particularly relevant in the framework of this evaluation. Each of these factors presents both opportunities and risks. 12. First, the rapid progression of governance on the development agenda. In less than a decade, governance has moved to the centre stage, both at the level of the discourse of the international donor community and in third countries. Governance is everywhere nowadays was a recurrent observation noted during interviews. It captures well the ascent of governance as a political priority in EC relations with third countries and more generally the growing concern about the delivery environment which should underpin progress made in the late 1990s in terms of policy goals and coordination in international assistance. The CSP analysis confirm that governance related issues are often at the heart 4 Respectively on EC strategies and support to difficult partnerships; on the way governance issues were considered during performance-based mid-term review processes; and on the nexus between budget support and governance. 5 This holds particularly true for the new generation of governance programmes that emerged in line with new EC policy orientations with regard to governance in the late 1990s. Thematic Evaluation of the EC support to Good Governance Synthesis Note - January PARTICIP GmbH

18 4 of the strategies (in terms of objectives and political focus). The desk study shows how governance has been integrated as a key component of the partnership with different regions (EQ 1). The 2003 Communication on Governance and Development (615/2003) provides a comprehensive analysis of the different dimensions of the governance agenda while the EC Draft Handbook on Governance presents governance as an over-arching concept (including the promotion of human rights, democracy and civil society). The statistical analysis reveals that funding (from different sources) for governance is growing. The central position of governance is likely to be further reinforced by new policy developments (e.g. the European New Neighbourhood Policy with its enhanced focus on governance); the pressure to achieve the Millennium goals; the move towards budget/sector support; the creation of a thematic unit on governance in Europeaid, etc. As shall be seen below, this rapid evolution is also the source of confusion and ownership problems among actors (see chapter IV). 13. Second, governance is no longer an issue pushed forward by the international donor community alone. In recent years, the governance agenda has been by formally embraced by many governments in third countries, as evidenced in Poverty Reduction Strategy Papers or national development plans. New initiatives on governance have been launched at national, regional and continental level (e.g. by the African Union/NEPAD). In some cases (for example in the Dominican Republic, or in Burkina Faso) it coincides with a well-established agenda which it tends to supersede, such as the reform of public administration. Interestingly, there is also a growing societal demand for good governance. People and civil society organizations thus become actors in the governance debate, contributing ideas, claiming rights, demanding accountability. This, in turn, raises a number of critical questions that external agencies will need to be carefully address: how do these different local stakeholder perceive and define governance? Who sets the agenda? To what extent are governance priorities defined through inclusive, multi-actor dialogue processes? How should the design and implementation of (EC) governancerelated be adapted to these dynamics in third countries? 14. Third, it is increasingly acknowledged that governance is not only an issue over there, in third countries. The good governance obligation also applies to external actors. This is reflected in the growing pressure on donors to practice principles of good governance in their dealings with partner countries. It invites them to respect the commitments made on alignment and harmonization. From an EU perspective, good governance also involves the need to continue improving policy coherence in all relevant areas, such as for example environment, trade and agriculture 6 or to overcome the global governance deficit 7. Yet experience suggests that effective progress on these reciprocal obligations has so far been rather slow to materialize. 15. Fourth, the ongoing policy and institutional reforms also form an important contextual factor. Several elements of the reform process are likely to have an impact (positive or negative) on the overall regulatory, financial and institutional framework to deal with governance programmes (e.g. de-concentration; the adoption and mainstreaming of new programming methods; the planned overhaul of the financial architecture for external actions, etc.). 16. All this suggests that there is strong political momentum and commitment in favor of putting governance at the centre of (EC) cooperation processes. Yet it also shows that governance is a both a complex and rapidly evolving field of action, confronting donor 6 Communication on Governance and Development, par. 6 7 Ibid., with the observation that the EC deals with global governance issues in international fora (e.g. the UN Commission on Global Governance) and in dialogue processes with important third partner countries such as China, Russia, India and Brazil. Thematic Evaluation of the EC support to Good Governance Synthesis Note - January PARTICIP GmbH

19 5 agencies (including the EC) with major strategic and operational challenges. In this context, the evaluation exercise was generally perceived to come at the right time and as a potentially useful tool to assist the EC in its ongoing search to ensure a proper match between the political priority given to governance agenda and its own overall response capacity. IV. PRELIMINARY FINDINGS 17. In order to facilitate a focused discussion with the Reference Group, the evaluation team proposes three main (closely inter-related) preliminary findings in this Synthesis Note (with supporting data and evidence from different sources): (1) There is much conceptual and operational confusion on governance. (2) The EC has made substantial progress in dealing with governance, particularly in terms of defining what are likely to be the most effective approaches to sustainable improvements to governance. (3) Major gaps exist between the policies laid out in Brussels and actual implementation practices in the field. These gaps, in turn, substantially reduce the potential contribution of EC interventions in governance-related processes. (1) There is much conceptual and operational confusion on governance 18. This is a first key finding that clearly comes out of the different analyses undertaken during the desk and field phases (e.g. questionnaire, CSP-analysis, country notes, focus group discussions). As governance moves to the forefront and starts permeating all spheres of EC cooperation processes with third countries, there is no shortage of conceptual and operational confusion among key actors and stakeholders involved at different levels. 19. This confusion relates to a wide variety of issues, including: the definition of (good) governance; the scope of the governance agenda (e.g. the distinction between political, economic and corporate governance and their possible linkages); the issues that should be included under the banner of governance in a given country programme; the linkages between governance and key development objectives (such as poverty reduction); the linkages between governance and other policy areas (e.g. peace, development and security) and situations (relief, rehabilitation, development); approaches, tools and methods to be used to promote it in different contexts, particularly in difficult partnerships roles to be played by the different actors in setting the agenda, designing and implementing programmes; the meaning (and limits) of the EC acting as a change agent ; indicators for monitoring and evaluating progress achieved; the reciprocal good governance obligations of the EC/EU Thematic Evaluation of the EC support to Good Governance Synthesis Note - January PARTICIP GmbH

20 6 20. The prevailing state of confusion manifests itself in many ways: Within the EC as a whole, a huge variety of interpretations tend to (co-) exist at different levels 8, ranging from rather narrow-technocratic approaches (focusing on the managerial aspects of public affairs, including the fight against corruption) to broad-based holistic approaches (considering governance as a cross-cutting issue) with a strong political connotation (ramifying into issues such as human rights, democracy, etc.). During the field phase, it was interesting to note that Delegation staff often started the interview with a request towards the evaluation team to define governance. The CSP-analysis indicates that the issues covered by governance vary substantially from country to country. It also reveals that a broad, multi-dimensional concept of governance is generally used in the analytical part of CSPs (encompassing the political dimensions of democracy, human rights, civil society participation). However, at programming level, this comprehensive governance agenda is often narrowed down considerably, with EC support being targeted primarily to public administration reform or improving the management of public finances (in the context of budget support provided by the EC).. In some regions (e.g. MEDA), EC officials prefer to use other conceptual frameworks (i.e. the UNDP definition), perceived to be more attuned to the governance thinking and needs in MEDA countries than EC policy frameworks and concepts. Within Delegations, there is clearly confusion on whose job it is to take care of governance. A tendency can be observed to confine governance to the specialised unit or staff directly dealing with specific governance issues (e.g. human rights, public sector reform, justice). However, there is considerable evidence of a focus on governance in other sectors and programmes funded by the Commission in the country involved. Yet these possible indirect contributions to governance are often not recognised as such, nor integrated into the overall governance strategy of the EC. At country level, there is often no reciprocally agreed definition between the EC and the government on what is the concrete meaning of governance (a situation which may also reflect a lack of societal agreement between local actors on the precise meaning of governance and the main challenges of the reform process). 21. Some of this confusion may be linked to the relative novelty of the governance debate and to the existence of region-specific needs and priorities. Yet there are important risks attached to this state of affairs, including; (i) the use of governance as a catch-all phrase, a basket concept; (ii) lack of ownership by EC officials involved in other sectors than those directly concerned with governance; (iii) possible conflicts with third countries, particularly when the consensus on broad principles is brought forward into political dialogue processes on progress achieved, into concrete action (program design, implementation) and into performance-based review processes 9 ; (iv) the difficulty for the EC to properly evaluate (including in quantitative terms) and account for its efforts in promoting governance in third countries. 8 Examples of differentiated approaches to governance can, amongst others, be found in the regional agreements; between EC headquarters and Delegations; between DG-DEV and RELEX; within different units of a Delegation; between the technical staff involved in governance programmes and the macro-management level of Delegations; between the EC and the European Parliament (e.g. in relation to the desirable focus of the European Initiative on Human Rights and Democracy), etc. 9 In this context, it is interesting to note that one of the apparent reasons for cancelling the Algeria mission was related to government sensitivities to include consultations with civil society actors in the evaluation process. Yet the use of this argument is in contradiction with a stream of jointly agreed policy documents emphasising the importance of civil society in development, governance and MEDA cooperation. Thematic Evaluation of the EC support to Good Governance Synthesis Note - January PARTICIP GmbH