of the THREE-YEAR WORK PROGRAM

Transcription

1 INDEPENDENT EVALUATION OFFICE (IEO) of the INTERNATIONAL MONETARY FUND THREE-YEAR WORK PROGRAM 1. The Status Report on the Independent Evaluation Office (which was put up on the IEO website for comment) described a two-stage process to develop the work program. In the first stage a large menu of potential topics was identified on the basis of IEO s internal assessment and suggestions received in the course of internal and external consultations. The large menu of topics is reproduced in Appendix I. 2. In the second stage the large menu was to be narrowed down to smaller list of topics based on feedback received from consultations with internal or external groups. This note describes the outcome of the second stage of the process. I. PROPOSED WORK PROGRAM 3. A two-stage process was adopted to develop the work program. In the first stage, a large menu of potential topics was identified on the basis of IEO s internal assessment and suggestions received in the course of internal and external consultations. These are listed in Appendix I. 4. In the second stage, the large menu was narrowed down to a subset of 15 topics consistent with IEO s proposed capacity. At present the IEO is working on the assumption of a steady state capacity of 4 studies, which implies about 10 studies could be undertaken over a three-year period. In case the steady state capacity is increased to 5, the total number of projects would increase to 11 or 12. To plan for about 10 to 12 studies over three years it was felt necessary to identify about 15 potential topics based on criteria such as giving priority to topics that have been the subject of controversy or criticisms, that have the greatest interest for the wide range of the membership, and where the learning potential is the greatest. 5. Inevitably, a number of potentially important topics had to be dropped to narrow the field. This is the unavoidable consequence of a deliberate strategy of concentrating on a smaller number of studies and emphasizing quality rather than quantity. There is some overlap between some of the topics. This too is inevitable and where the topics are sequenced over time relevant findings in the first study will inform and improve subsequent efforts. 6. The 15 topics selected on the basis of these criteria are set out in Box 1. Of these, the three topics for the first year are firm choices. The work program for subsequent years will be determined annually choosing from the remaining topics on the basis of priorities as assessed at the time. Entirely new projects may also need to be introduced to reflect new issues and priorities that may arise.

2 The topics listed in Box 1 present a mix of thematic studies focusing on a single issue across countries and evaluations of programs in individual countries. Each type of study has its advantages. Thematic studies are more likely to be of interest to a broad range of the membership and are also more useful for distilling lessons with general applicability. The thematic topics have been chosen on the basis of their immediate policy relevance to a wide range of the Fund s membership, with priority given to issues where external observers are most concerned about whether the Fund s approach is appropriate, or where there is a need to evaluate a major new initiative by the Fund (e.g. the PRGF and the FSSA process), after allowing sufficient time for internal experience to be gained. 8. Individual country studies suffer from the disadvantage that the learning aspect is more case specific. However, credible lessons on many difficult issues relevant for Fund policy can only be drawn on the basis of detailed examination of individual cases. Such studies can also generate useful inputs to some of the broader thematic questions and could help to stimulate more systematic internal assessments by Fund staff of particular country cases. FY 2002/2003 (Firm) Box 1. Proposed Work Program 1. Fiscal adjustment in IMF-supported programs 2. A group of three capital account crisis cases, i.e., Indonesia, Korea and Brazil 3. Repeat use of IMF resources FY 2004 (3 to 4 of the following and in FY to 5 of the rest) 1/ 4. PRGF/PRSP (to be undertaken jointly with the World Bank s OED) 2/ 5. The IMF s advice on financial sector restructuring after a crisis 6. Structural conditionality in Fund-supported programs. 7. The role of IMF surveillance in crisis prevention 8. The IMF s advice on exchange rate policy 9. FSAP/FSSA 10. IMF Technical assistance, 11. Private Sector Involvement (PSI) 12. The IMF s approach to capital account liberalization 13. The role of multilateral surveillance 14. Additional country case (possibly Argentina or Turkey) 15. Low-income country case 1/ The number of projects chosen in FY 2004 and FY 2005 will depend on the size of the steady state capacity of IEO that is finally determined. 2/ Since this project is in the OED s work program for FY 2004 it is expected to be included in the IEO program for that year. 9. A brief overview of the issues that would be addressed in each of the studies listed in Box 1 is provided in the following sections. Detailed terms of reference for each study will be

3 - 3 - prepared separately and, as indicated in Section III, interested parties will have an opportunity to comment on the terms of reference. These comments will be taken into account by IEO in determining the final TOR. A. Outline of the First-Year Program Fiscal adjustment in IMF-supported programs 10. Fiscal adjustment, especially in low-income countries, has long been the subject of criticism in large part because it involves severe trade-offs between stability and growth, or stability and social expenditures, which are often not adequately articulated or quantified and involve distributional issues that are highly sensitive politically. 11. Based on a sample of low- and middle-income countries (thus involving PRGF and non- PRGF countries) the study will examine the major features of fiscal program design, the nature of the dialogue between Fund staff and the country authorities and other groups, the extent of country ownership, and the results in terms of the efficiency, sustainability, and social impact of the fiscal adjustment. 12. Inter alia, the study will address the following issues: Were the stated objectives of the program successfully achieved? If not, what factors contributed to the failure? Where program objectives were met, was there improvement in growth, poverty reduction and key social indicators? To what extent have programs helped to achieve a more sustainable fiscal adjustment? Are fiscal deficits and sustainability defined appropriately? How were aid flows factored in? To what extent has adjustment encompassed the broader public sector? Was there room to change the mix between adjustment and financing? How was debt dynamics treated? What was the impact of debt relief if relevant? How did programs deal with shocks? How critical were structural reforms for achieving overall adjustment objectives? How were trade-offs considered, e.g. amount and speed of fiscal adjustment versus the social cost of adjustment; speed of deficit reduction versus inflation path; spending cuts versus tax increases; efficiency gains versus higher spending? How important were efforts to sharply reallocate the budget toward protecting the most vulnerable groups, particularly if they were adversely affected by the adjustment program or shocks? How did the negotiations of fiscal adjustment take account of constraints on implementation, explore alternatives and analyze the social impact of key measures? How were major differences and perceptions of trade-offs by government and civil society handled and how were programs and dialogue crafted so as to foster domestic support? What institutional changes were proposed to monitor the composition of spending and actual implementation?

4 - 4 - To what extent have programs contributed to building institutional capacity and fostering ownership? What has been the role of IMF technical assistance in these efforts and how could it be made more effective? How effective has Bank-Fund collaboration been in addressing public expenditure policy and management, and social safety net issues? Do Fund-supported programs over time show improvements reflecting lessons learnt? In the case of the low-income countries, has the PRGF process made a substantial difference to how these issues are addressed? How can the PRSP process of strengthening ownership and considering social impact be adopted to non-prgf countries? Group of capital account crisis cases (Brazil, Indonesia and Korea) 13. An evaluation of the role of the Fund in recent capital account crises is a high priority because of the systemic importance of these cases, and because there is considerable controversy among external observers about key elements of the strategy pursued. The proposed study would focus on three recent country cases to evaluate the effectiveness of the Fund s role in spotting vulnerabilities during the pre-crisis period, and the effectiveness of Fund-supported programs in resolving the crises. 1 The proposed country cases represent contrasting examples of the factors underlying the crisis and also of outcomes. They will be used to assess a range of issues that have been raised by critics of the Fund approach, including: What was the Fund s assessment of the countries policies prior to the crises? What was the nature of dialogue with the authorities? Was the nature of the crisis correctly diagnosed and was the macroeconomic adjustment strategy notably fiscal and monetary policy suited to the characteristics of the crisis? Why was the extent of capital outflows so underestimated? Since a restoration of confidence was central to the success of the programs, how could program design have dealt with the risk that confidence would not be quickly restored? How was ownership by government and civil society taken into account? How did the size or structure of official financing packages influence the restoration of confidence? What were the tradeoffs involved in alternative approaches to involving the private sector in halting capital outflows? Were the structural components of the program including with respect to the financial and corporate sectors helpful in crisis resolution? What was the extent and quality of coordination with the World Bank and other multilateral institutions? 1 The evaluation would take account of the synergies with the broader study on the role of surveillance in crisis prevention. An alternative approach would be to include an evaluation of a single country case (Indonesia) in the work program for FY2003 and a second country case in FY2004.

5 - 5 - Could the sharp initial declines in output (in the Indonesia and Korea cases) have been avoided and why was the impact on output in Brazil smaller? Could the adverse social impact of the crises have been mitigated? The study will seek to identify lessons from the diversity of the experience across the three countries. Repeat use of Fund resources 14. Some 25 countries have been indebted to the Fund for more than thirty years out of the last fifty. Sixteen countries have been under Fund-supported programs for twelve years or more out of the last eighteen. Such prolonged use risks turning the Fund into a source of long-term financing, in contradiction with the mandate set forth in its Articles of Agreement. Many of these countries have acute debt sustainability problems and most are now enrolled in the HIPC initiative. Drawing on 4 5 case studies, the project would aim to answer the following questions: What are the causes of this pattern of repeat use of Fund resources? Are there specific program-design or other weaknesses that might have contributed to the pattern, including as regards institutional capacity building, assessments of debt sustainability, institutional arrangements for official debt restructuring, and the overall availability of long-term financing? What have been the costs entailed by repeat use for the borrower, for the Fund, and for the rest of the membership? The project will also assess possible remedies whether in the design of lending facilities and programs, in the use of technical assistance, or in the division of labor with providers of long-term financing. Part of this assessment will include an analysis of the effects of the HIPC Initiative on prolonged users. B. Summary Outline of the Proposed Subsequent Evaluations Poverty Reduction and Growth Facility (PRGF) and Poverty Reduction Strategy Papers (PRSP) Building on previous internal and external reviews, and coming at a time when significant experience with the PRGF would have been accumulated, the evaluation will assess the extent to which PRGF arrangements differ from those under the Fund s previous concessional facilities. It will address issues related to program design, ownership, implementation, as well as progress in reducing the incidence of poverty and in growth performance. In addressing these issues, particular attention will be paid to the treatment of countries seeking debt relief under the enhanced HIPC Initiative, to the adequacy of Bank-Fund 2 The study of the PRSP process will be undertaken in collaboration with the World Bank s Operations Evaluation Department.

6 - 6 - collaboration mechanisms, and to issues related to institutional capacity building and the sustainability of reforms. 16. The following specific questions will be addressed: What should be the role of the Fund in low-income countries, especially those that have attained macroeconomic stability? Are the macroeconomic programs underlying PRGF arrangements clearly linked to the strategies for fostering growth and reducing poverty set out in the PRSPs? Has the approach to program negotiations been modified in line with the PRGF operational guidelines? How effective has Bank-Fund collaboration been in ensuring that programs include measures to protect the most vulnerable? To what extent has conditionality become more selective in PRGF arrangements (compared to ESAF arrangements)? To what extent has the degree of country ownership made a difference to program implementation and performance against program targets? Has good performance against program targets been associated with (i) improvements in poverty-reduction indicators; and/or (ii) level of donor support? To what extent are capacity weaknesses identified in programs being addressed over time? How effective has Fund technical assistance been in helping to build capacity in areas of the Fund s competence? To what extent is the PRGF having a measurable impact on growth and/or poverty reduction? The IMF s advice on financial sector crisis resolution and restructuring 17. A substantial proportion of the Fund s membership has experienced financial sector crises over the last two decades and such crises have been at the core of many of the recent capital account crisis cases. There has been considerable debate among external observers about the appropriateness of the Fund s policy advice in the resolution of these crises. The evaluation would focus on a panel of countries broadly representative of the types of financial sector crises the IMF has had to deal with in recent years (e.g. those involving large external liabilities and those involving primarily domestically-driven asset price bubbles). It would address the following questions: How was the nature of the financial sector crisis and the implications for the macroeconomic outlook diagnosed? Has the Fund approach to such crises evolved as experience has been gained?

7 - 7 - What were the common ingredients of the crisis resolution strategies embedded in Fund programs (e.g. bank closures, deposit guarantees, bank recapitalization, regulatory framework)? Did the strategies pursued adapt to country circumstances? Was the policy response successful in minimizing disruptions, both at the macroeconomic and at the microeconomic levels? How well did Bank-Fund collaboration work, including in the integration of debt workout and corporate restructuring strategies into financial sector resolution? Did policies adopted in Fund-supported programs improve the resilience of the financial sectors in a substantial way? Structural conditionality in Fund-supported programs 18. The recent streamlining of conditionality attached to Fund-supported programs has been an important element in the Fund s response to criticism of the large expansion of both the scope and the volume of structural conditionality over the last decade. The areas in which the expansion of conditionality has been most noticeable are the financial sector and privatization. The fiscal sector has always been and remains the single most important focus of structural conditionality, while conditions related to the trade regime have been traditionally important and remain so. 19. The project will focus on those four broad areas of structural conditionality and will assess the impact on the effectiveness of Fund-supported programs in achieving their core objectives. The methodology used will combine the analysis of a range of indicators across Fund-supported programs, both before and after the streamlining drive, with a more in-depth examination of a few post-streamlining case studies. 20. Specific issues to be investigated include: The rationale and overall effectiveness of structural conditionality in the four core areas as well as the potential costs and benefits of alternative approaches to conditionality. the extent to which the streamlining has resulted in a shift of conditionality towards sectors where the Fund has a clear comparative advantage over other institutions, and more generally how it has affected program design. how the measures associated with the streamlining (in particular prioritization of conditionality, enhanced collaboration between the IMF and the World Bank, and targeted provision of technical assistance) have had an impact on the quality (in terms of economic efficiency and poverty reduction) and sustainability of adjustments promoted in programs. whether the streamlining has resulted in a greater ownership of program objectives by country authorities and a lesser degree of social disruption.

8 - 8 - The role of IMF surveillance in crisis prevention 21. In the wake of the Asian and Russian crises of 1997/98, crisis prevention has become a major concern of both the international financial community and critics of the IMF. The Fund has launched a series of initiatives over the past two years to try and address those concerns, both in its internal work processes and in its relations with member countries. 22. The study will evaluate the Fund s record in anticipating and preventing balance of payments crises before and after these initiatives were launched. The analysis will be based on a panel of countries which experienced crisis, and some countries where crises were at risk (as identified by various vulnerability assessment approaches and independent judgment) but were avoided. The group of countries will be chosen to reflect crises with different origins, but with the primary focus on capital account crises. The evaluation will distinguish between the Fund s ability to anticipate the vulnerabilities that lead to crises (i.e. crisis prediction) and its ability to identify and encourage policy corrections (i.e. crisis prevention). The evaluation will seek to identify lessons for improving further the effectiveness of Fund surveillance in crisis prevention. 23. Specific issues to be investigated include: Has multilateral surveillance been constructively used to help anticipate and prevent crises? Were the implications of international developments for individual countries appropriately analyzed and incorporated into surveillance of individual countries? Did multilateral surveillance identify vulnerabilities associated with volatility of capital flows? Were contagion risks properly assessed? Has bilateral surveillance contributed to the anticipation of balance of payments crises? How does the track record of the Fund compare with that of private agencies? How have private capital markets used the increased information now made available by the IMF and has greater transparency led to a better pricing of risk? Did surveillance suggest sufficiently specific corrective steps? What factors affected the ability of surveillance to influence the policy response? Are certain types of vulnerabilities better diagnosed and corrected than others? In particular, how well did surveillance handle vulnerabilities related to public debt dynamics, balance-sheet mismatches, and sustainability of exchange rate pegs? To what extent is adequate data available to support vulnerability assessments? How have the reforms introduced in recent years altered the way in which Fund surveillance identifies and responds to potential vulnerabilities in member countries? What further changes could improve crisis prevention? Is all available information effectively incorporated into systems that can give early warning of potential vulnerabilities?

9 - 9 - The IMF s advice on exchange rate policy 24. There is considerable external criticism and misunderstanding about IMF advice on exchange rate policy, especially to emerging market economies. This evaluation would focus on the following issues, drawing inter alia on specific country case studies. Are there common guiding principles governing the IMF s policy advice on the exchange rate and are they implemented consistently in practice? 3 For example, the Fund s public position is that, in line with the Articles of Agreement, the Fund would abide by a member s preferred choice of exchange rate regime and advise on policies needed to support that choice. To what extent has this general approach been followed in practice both in surveillance and in program design? How valid is the bipolar view of feasible exchange rate regimes for countries with an open capital account? What position has the Fund taken on this issue? How effective was the Fund s policy advice on the strategy for exiting a pegged exchange rate regime, if the peg is inconsistent with underlying policies? Why have some exits been relatively successful, in terms of minimizing disruptions, and others not? How has the Fund influenced the timing of such exits and accompanying domestic policy commitments? Has the Fund s advice on exchange rate policy paid sufficient attention to the regional context? Financial Sector Assessment Program (FSAP) and Financial Sector Stability Assessments (FSSA) 25. The FSAP/FSSA is one of the major initiatives undertaken by the Fund in response to the crises of the late 1990s. It absorbs substantial Fund (and World Bank) resources and is expected to play an important role in bilateral surveillance of all member countries. By the end of FY 2003, approximately 80 assessments are expected to be completed. The evaluation would focus on the following areas: How well do FSAPs and FSSAs identify the major risk factors in a country s financial system and have they then proposed appropriate and feasible measures to address them? Are they using the best approach to diagnose vulnerabilities or would other methods be more productive? How well are TA needs assessed and implemented? Are recommendations articulated as a coherent blueprint for reform with proper regard to suitability, sequencing, and implementation capacity? How well are the assessments integrated into ongoing surveillance? 3 See, for example, the Chairman s summing up of the Executive Board discussions of Exchange Rate Regimes in an Increasingly Integrated World Economy (September 21, 1999 and November 15, 1999).

10 Does the FSAP/FSSA achieve its goals in a cost effective manner? Do countries perceive the FSAP as adding value to the domestic policy development process? Has it deepened the quality and coverage of discussions with authorities on financial sector issues? How has the private sector reacted to the new information published in FSSAs and how has it impacted their activities and risk perceptions? IMF technical assistance 26. Technical assistance (TA) is one of the major activities of the IMF. In FY 2002, it is anticipated to absorb $125 million, or 18 percent of the total administrative budget. In 1999, a comprehensive internal review of TA identified a number of shortcomings in the management of those resources, and gave a mixed assessment of the overall effectiveness of the Fund s TA efforts. 4 In response to those findings, a series of initiatives were taken since late 2000 to realign TA with the Fund s overall policy priorities and to standardize TA management across provider departments. 27. The project would focus on assessing to what extent these initiatives have provided adequate remedies to the identified shortfalls? Based in part on a number of case studies, the project would evaluate the following issues: The framework used to allocate TA resources across countries and subjects, and the process for ensuring cooperation and setting priorities with other TA providers. The effectiveness of TA, especially in building institutional capacity and fostering sustainable reforms. How is absorptive capacity taken into account and how effectively is the policy advice tailored to the circumstances of each country? How is transfer of knowledge emphasized and what measures are taken to make TA self-limiting? The interactions between TA and other major Fund activities, such as surveillance and use of Fund resources. The project will also assess the Fund s internal TA evaluation procedures, including the outcome of ongoing efforts to establish a common best practice evaluation methodology for all TA-providing departments. Private sector involvement (PSI) in crisis resolution 28. The need to involve the private sector effectively in the resolution of financial crises has been much discussed and it clearly impacts on decisions regarding the appropriate limits on access to Fund resources. There are wide differences of views on how to handle this issue, including within the official community. Although policy in the area is still evolving, it is likely 4 During the course of the IEO s consultations on the work program, a substantial number of observers within and outside governments also expressed the view that IMF TA programs, while generally providing sound policy advice, were not especially effective in building institutional capacity.

11 to build to a more definite shape over the next two years and an evaluation of country cases could therefore be undertaken in FY 2005 to inform the broader debate. Specific questions to be addressed would include: How are private market creditors being involved in the resolution of financial crises and are principles in this respect being applied consistently? Is debt sustainability/solvency analyzed in a consistent manner? Is there a tendency to be over-optimistic about the expected return to normalcy, e.g. expected reductions in risk premia? What has been the catalytic role of Fund-supported programs and official financing packages in mobilizing private capital (in terms of both volume and terms of financing)? Why in many cases were private flows less than assumed in the program? How effective have been the various concerted efforts, short of a standstill, to secure private sector involvement? What has been the outcome in terms of burden-sharing among creditors? Are further institutional changes required to facilitate PSI? What lessons do the early cases suggest about the impact of concerted PSI on the real economy on the resumption of voluntary flows, and on the availability of financing to other countries? The IMF s approach to capital account liberalization 29. This study will address issues on which there is significant disagreement and which are important for a broad range of Fund members. The evaluation would assess the Fund s policy advice on the scope, pace and composition of capital account liberalization. Drawing, inter alia, on a number of country case studies, it would address questions such as: What has been the IMF s advice regarding capital account convertibility? How have policies towards the capital account changed in recent years in the light of experience with capital account crises? Has the Fund s policy advice on capital account liberalization been consistent across countries with respect to the interaction with macroeconomic policy and the sequencing of other reforms, especially vis-à-vis the financial sector? What has been the effect of the proposed framework for taking account of trade-offs between strengthening intermediation and minimizing vulnerabilities and how is it being implemented in practice? 5 Is multilateral surveillance giving sufficient attention to possible measures on the supply side to reduce the volatility of capital flows? 5 Capital Account Liberalization and Financial Sector Stability Considerations for Sequencing (SM/01/186, June 25, 2001).

12 Given that occasional crises are inevitable, what role is there for temporary capital account restrictions as part of a crisis response? The role of multilateral surveillance 30. Multilateral surveillance is a core activity of the IMF that aims to provide a basis for analyzing the forces driving the world economy, identifying global vulnerabilities, and advising on appropriate policies, especially in the most systemically important countries, to avoid suboptimal outcomes. Potential issues to be addressed would include the following: The role of the World Economic Outlook and International Capital Markets Report exercises in forecasting prospects for the global economy and identifying global vulnerabilities. The effectiveness of the interaction between multilateral surveillance and bilateral surveillance of the major industrial countries. What is the value added and how could it be improved? 6 The effectiveness of the IMF s inputs into the deliberations of various international groups such as the G-7 and various regional groupings (such as APEC and the Manila Group). What has been the impact of the increased emphasis on capital market issues in multilateral surveillance and the enhanced interactions with private financial market participants? II. COMMUNICATION FRAMEWORK FOR EVALUATION PROJECTS 31. The terms of reference of the IEO imply that it must address multiple audiences: Fund staff and management to enhance learning culture ; those interested in the Fund s activities to enhance its external credibility ; the public at large in all member countries to promote greater understanding of the work of the Fund ; and the Executive Board to strengthen its institutional governance and oversight responsibilities. This has implications for the extent and manner of consultation with different interested groups and dissemination of its output. 32. Consultation at the stage of determining the terms of reference for individual projects is extremely important for credibility and transparency although independence also requires that the final decision must necessarily rest with IEO. It is proposed to adopt the following procedures for consultation at various stages. 6 The 1999 External Evaluation of IMF Surveillance identified an inadequate cross-fertilization between multilateral and bilateral surveillance as a problem and made a number of recommendations for improvements. The study could also assess the impact of these recommendations.

13 Issues papers and/or draft terms of reference for each study, identifying the questions to be addressed and also, to the extent practicable, the methodology, would be prepared and comments sought from the Executive Directors, staff/management, and member country governments (especially in the case of evaluations involving individual countries). The issues papers/draft terms of reference would also be posted on IEO s website to elicit comments from external observers. The final terms of reference, as determined by IEO after these consultations, would also be posted on the website. The IEO evaluation team would conduct its own research and would interact with concerned parties inside and outside the Fund. For evaluations involving individual countries, it is expected that consultations would be held with the authorities and with others. The evaluation team would also be open to receive inputs on substantive issues covered by the terms of reference from interested parties wishing to make submissions. IEO s website will be used to receive written comments. The IEO will interact extensively with Fund staff in the course of each study. Draft reports will be submitted to management and (where appropriate) the country authorities, for their comments. Draft reports sent to management will also be sent to the Chairman of the Evaluation Group for information. The final reports will then be circulated to the Executive Board, along with comments of management and country authorities (where relevant). To enhance the Fund s ability to learn from evaluation, each report would identify specific lessons and recommendations for the future. The recommendations should be sufficiently specific to allow management, and subsequently the Executive Board, to indicate whether they agree (in which case they may wish, at their discretion, to indicate specific actions they plan to take) or disagree, with an explanation of the reason for disagreement. 33. The IEO s terms of reference state that there is a strong presumption that reports will be published promptly (within the constraints imposed by the need to respect the confidentiality of information provided to the Fund by members) unless in exceptional circumstances the Executive Board were to decide otherwise. 34. Once an evaluation report is made public, external stakeholders would be given an opportunity (within a specified period) to make formal comments on the report. As part of IEO s outreach strategy, more easily readable summaries for a broader public will also be prepared, and an outreach conference will be organized to discuss each report after publication. 35. The IEO Annual Report will set out the main conclusions and recommendations of evaluation reports completed during the year and will provide an opportunity to assess the progress made in implementing specific recommendations and in tackling issues identified in earlier evaluations. It will also include a summary of suggestions for evaluation topics received from within and outside the Fund, along with a discussion of priorities set by IEO and a summary of Executive Board discussions of the whole program.

14 -21- APPENDIX I Initial Broad Menu of Potential Topics for Evaluation by IEO The items listed below represent a menu of possible subjects which were identified in the first stage of developing the country program. Surveillance Surveillance is a core activity of the IMF and takes up an estimated 40 percent of staff resources. It has gained in importance in recent years in the context of the emphasis on crisis prevention. The following are some possible studies in this area: 1. The IMF's role and effectiveness in crisis prevention based on an examination of recent Article IV reports including all the recent capital account crisis cases. Are early warning procedures effective, and do they influence the Fund s advice and countries policies? 2. Review of Financial System Stability Assessments (FSSA). What lessons can be learned from the experience of the first two years? Are they identifying the key vulnerabilities and proposing remedies in a cost-effective manner? 3. Follow-up of the 1999 external review of surveillance. Specifically, the review could focus on how the recommendations agreed to by the Executive Board have been implemented. 4. Effectiveness of IMF surveillance of industrial countries. Is the scope of surveillance appropriate and what is its value added? 5. The IMF s role in multilateral surveillance, including the World Economic Outlook and the International Capital Markets Report exercises. 6. Role and effectiveness of regional surveillance (e.g., European Community, other regional groupings). 7. The IMF s approach to liberalization of the capital account. Possible topics include whether the Fund s policy advice on the pace of capital account liberalization, and its sequencing with other reforms especially vis-à-vis the financial sector has been appropriate and consistent across countries. How has policy changed in the light of experience with capital account crises? 8. The IMF s work on standards and codes of good practice (in collaboration with other agencies). Is the approach effective in building institutional capacity and reducing vulnerability in member countries?

15 IMF-supported programs and related issues The IEO could undertake three different types of studies to learn from actual experience with IMF-supported programs. Review Individual Country Programs A sample of programs from the past few years could be chosen for review. The past three to four years would yield a potential universe of about 70 programs. Questions to be addressed would include: was surveillance effective in identifying vulnerabilities prior to the onset of the crisis; was the program design appropriate; was the negotiating process conducive to country ownership; were the key objectives of the program achieved and, if not, why not; and what are the lessons to be learned; and how effectively is post-program surveillance being carried out? The countries chosen should reflect the diversity of country/crisis types as indicated below: 1. Capital account crises (e.g., Argentina, Brazil, Ecuador, Indonesia, Korea, Thailand, Turkey). Some stakeholders have suggested that IEO should evaluate all programs where exceptionally large access to Fund resources is involved. 2. Low-income/highly indebted cases (possible cases include Bolivia, Burkina Faso, Cameroon, Honduras, Mozambique, Nicaragua, Niger, Senegal, Tanzania, Uganda, Vietnam, Zambia) 3. Transition countries (Bulgaria, Hungary, Poland, Russia, Ukraine, other FSU countries). 4. Selected countries that have had repeat programs could be chosen (e.g., Bolivia, Madagascar, Philippines, Zambia). Are there problems with program design that contribute to such repeat usage? Review of broad policy issues cutting across programs: This category covers thematic studies focusing on particular policy issues and what can be learned about these issues from cross-country experience. The following are some possibilities: 1. Exchange rate policies in IMF-supported programs and the Fund s policy advice on exchange rates as part of surveillance. Has the Fund s policy advice on exchange rate regimes and associated policies been consistent across countries? Has the design of exit strategies from exchange rates pegs been appropriate? Have the contractionary consequences of devaluations, stemming from their balance sheet consequences, been underestimated in program design? Has financial program design adapted effectively to inflation targeting regimes? 2. Fiscal adjustment in IMF-supported programs. Does fiscal adjustment take sufficient account of longer-term goals (e.g., for growth, poverty reduction) and is it implemented in a sustainable manner? Has the potential, immediate adverse impact of programs on particular vulnerable groups been adequately assessed and taken into account in program design?

16 Policies toward financial sector stability and financial sector restructuring. In addition to the effectiveness of IMF policy advice in helping member countries avoid financial crises (also discussed above under surveillance), possible issues could include whether the design of financial sector restructuring packages has been appropriate, how the fiscal consequences have been handled, and how the macroeconomic consequences of corporate restructuring were taken into account in program design. 4. Debt reduction and debt sustainability issues. Possible issues (either in the context of the HIPC cases or more generally) could include whether the approach to assessing the sustainability or unsustainability of debt positions (external or public debt) has been appropriate and consistent across countries; whether debt sustainability analyses in surveillance reports have been adequate; what use has been made of the sustainability analysis; and what can be learned from best practice approaches. 5. Has IMF support of a country s program had a positive catalytic effect, in terms of generating additional external financing flows within a specific timeframe? Are there objective measures of this catalytic effect? What factors influence the impact on market credibility? 6. The nature and effectiveness of conditionality and issues involving the ownership of national/imf-supported programs. Also, IMF policy toward structural conditionality has been modified recently; a review of the impact of the new policy could be undertaken at the end of the first two-year experience (that is, in FY2004). 7. Why do many IMF-supported programs remain uncompleted and what difference does it make? Are there particular aspects of program design (e.g. optimism of projections, extent of conditionality) that have a strong influence on the probability of completion? Do outcomes depend on the extent to which programs are completed and what lessons can be learned from uncompleted programs? 8. Conditionality with respect to trade policies. Has the IMF adopted a consistent approach to trade policies in the design of conditionality? Is the approach consistent between the surveillance stage and the program stage? 9. Private sector involvement in crisis resolution. This is a growing area of concern in which policy is still evolving but there are lessons to be learnt from experience thus far. Does the existing experience suggest that some forms of PSI are likely to be more successful than others? Many of the questions involved are also closely related to the size of the IMF financial support, including through the Supplemental Reserve Facility (SRF) for countries undergoing crises that are centered primarily in the capital account. 10. Experience with privatization in IMF-supported programs. Possible questions to be addressed could include: was the sequencing with regard to implementation of regulatory frameworks appropriate? How realistic was the timeframe for privatization? What was the impact on prices for services and investment in the privatized sectors? And what was the social impact?

17 Review of experience with particular lending facilities and related issues 1. The role of the Poverty Reduction and Growth Facility (PRGF) and the need to evolve special procedures to overcome problems in achieving stated objectives in countries with PRGF-supported programs has been much discussed. Although an internal review is currently under-way, there is scope for an independent review beginning perhaps in late FY2003. The review could address such issues as: Have all of the lessons from the ESAF reviews and best practices on particular policy issues been incorporated into PRGF adjustment strategies? Has the increased emphasis on country ownership resulted in real changes in the approach to negotiations? Has program design effectively incorporated the analysis of the social impacts of major reforms? Has program design and monitoring improved the targeting of spending in key sectors relevant to growth and poverty reduction? Have resources been effectively channeled to social sectors? 2. An external review of the Enhanced Heavily Indebted Poor Country (HIPC) Initiative would be important once a critical mass of countries reach their completion points. 3. The Contingent Credit Line (CCL) would also be a candidate for evaluation, but only after there is sufficient concrete operational experience with country cases. 4. The strategy vis-à-vis member country arrears to the IMF. 5. The IMF s role in countries emerging from conflicts. Technical assistance and training Technical assistance is extremely important for capacity building, especially in lowincome countries. More recently, it is also viewed as critical for crisis prevention in the more advanced emerging market economies. The following possible evaluations could be explored: 1. A follow up of the 1999 internal review of technical assistance. This could include an assessment of whether there are effective and consistent internal systems for evaluating technical assistance advice and for setting priorities. 2. Does technical assistance help improve national ownership of Fund supported programs? Has it been effective in improving program implementation or in enhancing crisis prevention? 3. Assessment of the effectiveness of technical assistance in such areas such as: Tax policy and revenue administration Expenditure policy and expenditure management Banking supervision/financial stability Debt and external reserve management

18 Internal IMF Processes and Governance The IMF s effectiveness of the Fund in pursuing its objectives depends clearly on the procedures it adopts both for operational work and Executive Board supervision. The following studies seem relevant: 1. Are the IMF s internal review systems adequate? For example, are there adequate mechanisms for early internal re-assessments of the adequacy of program design? 2. Is World Bank-IMF collaboration effective? How can it be improved, given their distinct operational approaches and objectives? Is the division of labor/degree of overlap between the two institutions appropriate? 3. Do staff papers on country programs contain the necessary information and analysis for the Board to make an informed judgment on the probability of success? Research Research is an important part of the IMF s work but since the Research Department is currently being restructured this need not be a high priority area for review. A following-up on the recommendations of the 1999 External Evaluation of Research Activities could be considered for the longer term.