Pakistan: Earthquake-Displaced People Livelihood Restoration Program

Transcription

1 Validation Report Reference Number: PVR-281 Project Number: Loan Number: 2335 December 2013 Pakistan: Earthquake-Displaced People Livelihood Restoration Program Independent Evaluation Department

2 ABBREVIATIONS ADB AJK Asian Development Bank Azad Jammu and Kashmir EAL emergency assistance loan ERRA Earthquake Reconstruction and Rehabilitation Authority GDP GVAP gross domestic product Gender and Vulnerability Action Plan M&E monitoring and evaluation MOF Ministry of Finance NGO nongovernment organization NWFP North-West Frontier Province PCR program completion report RCC reinforced cement concrete TA technical assistance NOTE In this report, $ refers to US dollars. Key Words asian development bank, pakistan, earthquake, emergency assistance, livelihood restoration, performance evaluation, rehabilitation expenditures, project validation The guidelines formally adopted by the Independent Evaluation Department (IED) on avoiding conflict of interest in its independent evaluations were observed in the preparation of this report. To the knowledge of IED management, there were no conflicts of interest of the persons preparing, reviewing, or approving this report. In preparing any evaluation report, or by making any designation of or reference to a particular territory or geographic area in this document, IED does not intend to make any judgments as to the legal or other status of any territory or area.

3 PROGRAM BASIC DATA Project Number: PCR Circulation Date: Sep 2011 Loan Number: 2335 PCR Validation Date: Dec 2013 Project Name: Earthquake-Displaced People Livelihood Restoration Program Country: Islamic Republic of Pakistan Approved ($ million) Actual ($ million) Sector: Multisector Total Project Costs: 1, , ADB Financing: ($ million) ADF: Loan: (SDR equivalent, million) Borrower: OCR: 0.00 Beneficiaries: Others: Cofinancier: Total Cofinancing: Approval Date: 27 Jun 2007 Effectiveness Date: 16 Jul Jul 2007 Signing Date: 4 Jul 2007 Closing Date: 31 Dec May 2008 Project Officers: C. C. Yu M. S. Shafi Location: ADB headquarters PRM From: Jun 2007 Oct 2007 To: Sep 2007 Dec 2008 Validator: I. Green, Consultant Peer Reviewer: T. Ueda, Principal Evaluation Specialist, IED1 Quality Reviewers: C. Kim, Principal Director: W. Kolkma, IED1 Evaluation Specialist, IED2 R. Vasudevan Evaluation Specialist, IED1 ADB = Asian Development Bank, ADF = Asian Development Fund, IED1 = Independent Evaluation Department, Division 1, IED2 = Independent Evaluation Department, Division 2, OCR = ordinary capital resources, PCR = project completion report, PRM = Pakistan Resident Mission, SDR = special drawing rights. A. Rationale I. PROJECT DESCRIPTION 1. The Earthquake-Displaced People Livelihood Restoration Program was the second loan from the Asian Development Bank (ADB) for earthquake reconstruction and rehabilitation in Pakistan following the devastation caused by the October 2005 earthquake. 1 Rural housing was to be provided to some 30,000 people who were living in tents, and to 3.5 million more living in temporary or semipermanent shelters. 2. Pakistan received a concessional emergency assistance loan (EAL), which was handled by the Economic Affairs Division, Ministry of Economic Affairs and Statistics, on a quickdisbursing basis. The loan was to finance the Earthquake Reconstruction and Rehabilitation Authority (ERRA) Housing Program for the reconstruction of houses in earthquake-affected areas of Khyber Pakhtunkhwa 2 and Azad Jammu and Kashmir (AJK) according to the seismicresistant construction standards of ERRA. The program was to ensure that most earthquake- 1 ADB Report and Recommendation of the President to the Board of Directors: Proposed Loan and Technical Assistance Grant to the Islamic Republic of Pakistan for the Earthquake-Displaced People Livelihood Restoration Program. Manila. 2 The government changed the name of the North-West Frontier Province to Khyber Pakhtunkhwa in 2010.

4 2 displaced people would be living in such houses by December 2008, and would have their livelihoods restored or improved. 3. The loan proceeds were designed to flow to the government s central budget with the government contributing funds to ERRA through the budget allocation process. The project completion report (PCR) noted that ADB lending modalities were compared to determine the best way to meet the urgent needs of housing reconstruction. 3 The EAL was selected as the most appropriate modality based on the following: (i) the tight time frame (the program was to be completed by May 2008); and (ii) the owner-driven implementation approach, which required providing money to many individual households for procuring needed materials and for building their own houses based on seismically compliant housing designs provided by ERRA. 4. An ADB project or sector financing modality would not have been able to meet this need in a timely and effective manner. Hence, the program combined the EAL modality with features of the programmatic financing approach, particularly the use of tranches with built-in performance indicators (mostly related to housing reconstruction progress) as conditions for loan release. As utilization and disbursements of the proposed loan were closely tied to the earthquake reconstruction and rehabilitation expenditures incurred by the government, the program approach was fully justified. 5. Through an attached technical assistance (TA) Capacity Building of Institutions Related to Earthquake Reconstruction and Rehabilitation Project the program was able to build the capacities of ERRA and other related institutions in seismic training, financial and strategic management, and environmental and social protection. B. Expected Impact 6. The intended impact of the program was the restoration and improvement of the livelihood of the earthquake-displaced people in AJK and North-West Frontier Province (NWFP). Performance indicators and targets for the associated impact included the (i) increase in gross domestic product (GDP) for NWFP from 5.6% in 2005 to 6.5% in 2013, and (ii) increase in the percentage of houses in NWFP with burnt bricks or blocks for walls from 49% in 2004 to 70% in The assumptions were that (i) the government and development agencies would allocate adequate resources for reconstruction in other sectors, and (ii) macroeconomic and security conditions would be favorable for economic development. Additional risk was that another earthquake may occur before C. Objectives or Expected Outcomes 7. The intended outcome was for most affected people to be living in seismically compliant houses with greater safety and security. Performance indicators and targets for associated outcomes were that, by the end of 2010, (i) at least 85% of the 3.5 million affected people have moved into seismically compliant houses; and (ii) of these people who have moved into seismically compliant houses, more than 23% of the households in NWFP and 31% in AJK will be headed by women. However, it was noted that another earthquake may occur. 3 ADB Completion Report: Earthquake-Displaced People Livelihood Restoration Program in Pakistan. Manila.

5 3 D. Components and Outputs 8. Two intended outputs were identified in the program design, as follows: (i) Seismically compliant houses completed without major delays; and (ii) Capacity built in seismic building techniques, financial management, monitoring and evaluation, protection of vulnerable groups, and environmental and social safeguards. For output 1, the performance indicator and target was that 85% of the 585,000 rural houses would be rebuilt and seismically compliant by June For output 2, that (i) at least 50,000 additional house owners have been trained by May 2008, including women and other vulnerable groups; and that (ii) trainings in other areas have been determined according to actual needs during implementation of the attached TA project. 9. It was expected that (i) the government would allocate enough funds from its budget for housing reconstruction, (ii) there was enough capacities for housing construction, (iii) seismically compliant standards were strictly enforced, (iv) alternative sites could be found for houses where their original sites were either lost due to landslides or are located in the red zone, and (iv) most of the affected people would use the compensation money for house construction and have enough savings to finance expansion beyond the minimum living areas. Risks were also identified, such as (i) fiduciary risks, e.g., corruption; and (ii) a major escalation of costs of construction materials due to increased demand. 10. The loan was to be released in two tranches. The first tranche of $200 million equivalent, to be released upon loan effectiveness, was to provide financing support on a retroactive basis for housing expenditures already incurred by the government. The second tranche (i.e., remaining amount in the loan account) was to be released within 6 months after the first tranche, subject to compliance with performance targets. The conditions for releasing the second tranche were that (i) ERRA had paid the third installment of the housing subsidy to 85% of the eligible households who received the second installment, and submission of financial monitoring reports and other documents as evidence of payment; (ii) an unqualified audit opinion on ERRA s financial statements for FY2007 had been submitted to ADB; (iii) training facilities have been provided by the government to commence the training program for households under the attached TA project; and (iv) all consultant contract awards for the Legal Assistance, Governance, and Institutional Building component under the Earthquake Emergency Assistance Project have been made. 4 E. Provision of Inputs 11. The Government of Pakistan requested ADB to provide a concessional loan of $400 million from ADB s Special Fund resources to finance the ongoing housing reconstruction program. At the same time, the government was in contact with a number of other development agencies for possible financing. Total housing reconstruction needs amounted to $1.38 billion. To realize such an undertaking in a short time, massive amounts of human resources and materials had to be mobilized. An aggressive approach had to be taken in the areas of training, planning, management, and monitoring. 12. The loan from ADB did not directly fund the procurement of supplies or services. With government funds, the Ministry of Finance (MOF) hired the National Engineering Services of Pakistan to prepare housing designs, develop compliance manuals, and train the inspection teams to ensure uniformity in the housing reconstruction standards and inspection regimes of the program. 4 ADB Report and Recommendation of the President to the Board of Directors: Proposed Loan to Pakistan for the Earthquake Emergency Assistance Project. Manila.

6 4 13. As part of its owner-driven approach, ERRA used the services of national and international nongovernment organizations (NGOs) to train a cadre of skilled individuals in earthquake-affected areas. Except for two NGOs hired under the TA project, for which ERRA used single-source selection based on ADB guidelines, all partner organizations were hired directly through donors. ADB hired individual consultants in consultation with ERRA, either directly or through firms, for the two remaining components related to financial management and safeguards. 14. The attached TA project addressed the weaknesses identified during inception. However, since the loan was designed without the benefit of a detailed preparatory technical assistance, some gaps in training (for the inspection teams) and monitoring were not identified or addressed in the design of the loan. Similarly, the urgency and short duration of the program did not allow these weaknesses to be picked up early enough for changes to be made during implementation. F. Implementation Arrangements 15. ERRA paid housing subsidies to the targeted affected people through its two provincial or state branches the Provincial Earthquake Reconstruction and Rehabilitation Agency for NWFP, and the State Earthquake Reconstruction and Rehabilitation Agency for AJK. All transactions were conducted through bank accounts, with no cash changing hands. Both provincial and state agencies conducted checking and verification to ensure that the list of affected people was up-to-date and correct. Feedback indicated that this approach minimized potential leaks during transactions, and had been effective in getting the funds to the targeted affected people. 16. Individual households carried out housing reconstruction based on standards and designs specified by ERRA. Subsidy schemes were developed based on the market replacement value of a typical house (39 square meters), irrespective of the size of the original houses. For completely destroyed houses, payments to households were made in four installments based on benchmarks of construction progress. For partially damaged houses, payments were made in two installments. For houses with minor damage, owners received one payment. Payments were made based on progress reports by field inspectors provided by the army and NGOs. 17. Building on an ongoing TA loan that is attached to the Earthquake Emergency Assistance Project, 5 another TA loan Capacity Building for Institutions Related to Earthquake Reconstruction and Rehabilitation was attached to the program. This TA project was estimated to cost the equivalent of $2.5 million. ERRA was the executing agency although the TA project was to provide additional support for all institutions related to the earthquake reconstruction and rehabilitation program. This support included (i) community-based training on seismic standards and construction monitoring techniques, (ii) financial and strategic management, and (iii) environmental and social protection (emphasizing gender and vulnerability aspects of social protection). Special attention was given to capacity building at the provincial or state and district levels. This TA project was completed on 30 June ADB Technical Assistance to the Islamic Republic of Pakistan for the Capacity Building for the Office of the Auditor General. Manila.

7 5 II. EVALUATION OF PERFORMANCE AND RATINGS A. Relevance of Design and Formulation 18. The program was rated relevant in the PCR. The intended impact and outcome were to restore or improve the livelihoods and increase the proportion of seismically compliant housing for earthquake-displaced people. Considering the overall extent of the housing reconstruction (more than 3.5 million people were to be served), the owner-driven implementation of the program was strongly relevant. The program also supported capacity building to ensure a consistent inspection regime and to train the home owners in seismically compliant building standards and techniques. The inspection regime, the composition of the inspection teams, and the training of homeowners were all designed according to international best practices, which were adapted to local conditions. 19. However, there were some design weaknesses. The overall approach taken to housing reconstruction had never before been applied on such a huge scale. Mainly due to the large number of affected home owners, weaknesses in the design and implementation of the reconstruction program emerged during implementation. Deficiencies included (i) weak impact and outcome monitoring, (ii) training in seismic compliance that focused on homeowners though they were not always the ones who actually rebuilt the houses, (iii) delays in micro-zoning and hazard risk mapping, (iv) inability of the inspection teams to respond on time to the many forms of noncompliance, and (v) insufficiency of funds that prevented homeowners from making the recommended repairs to achieve compliance. To address these weaknesses, ERRA made many changes in the training modules, increased the range of house design options, improved the compliance catalog to cover more scenarios, and expanded the grievance redressing mechanisms. However, the speed of response was slow for this short-duration program. Local consultants designed the input, output, and impact monitoring programs, and staff directly hired by ERRA operated the monitoring programs. Unfortunately, by the time the whole system became operational, a major part of the housing reconstruction program had already been completed. Considering both the strong relevance of the program in meeting the urgent demand and some weaknesses in project design, this validation also rates the project relevant. B. Effectiveness in Achieving Project Outcomes 20. The PCR recommended a rating of partially effective. Levels of effectiveness have varied throughout the program. At impact level, it was premature for the PCR to consider the achievement of the performance target of increased GDP for Khyber Pakhtunkhwa. However, in the performance indicator associated with houses with burnt bricks or blocks for walls in Khyber Pakhtunkhwa, based on the ADB Housing Survey of 2010, 6 the number of people who used burnt bricks or blocks for construction of houses increased from the pre-earthquake level of 49% to 92% of the houses using reinforced masonry, which is 22% above the 70% figures envisaged at inception. 21. At the output level, 92.1% of completely damaged rural houses affected by the earthquake have been rebuilt. Of a total of 351,234 completely damaged houses, 323,396 have been rebuilt. The remaining 31,930 houses are in the no work started (7.6%) and under process (0.3%) reconstruction category. 6 ADB Housing Survey in the Earthquake-Affected Districts of Azad Jammu and Kashmir and Khyber Pakhtunkhwa. Islamabad.

8 6 22. However, at the outcome level, of the 2.91 million people in the ERRA program, only 1.46 million or 50% have moved into seismically compliant houses, compared with 85% targeted at inception and 77% during implementation. The compliance level increases to 69% if only completely damaged houses are taken as the universe. The rest of the houses may or may not be compliant. Of the 319,304 houses that have been reconstructed, 55,000 (16%) belong to households headed by females 21,388 (7%) in Khyber Pakhtunkhwa and 33,612 (11%) in AJK. These figures are less than the targeted proportions at inception. 23. The ERRA rural housing reconstruction program trained a total of 783,314 people (32% in technical aspects and 68% in social aspects). TA consultants conducted training on initial safeguard screening, preparation of screening checklists (due diligence), land acquisition resettlement planning, and environmental impact assessment. Due to the off-target results at outcome level and the failure to reach vulnerable groups, this validation agrees with the PCR rating of partially effective. 24. At midterm review, it was noted that not all vulnerable groups (i.e., widows, elderly, handicapped, landless, orphans) were receiving support. It would have been useful for the PCR to have provided further information in this aspect since such a survey was undertaken, a monitoring and reporting system created, but status of compliance of the related covenant was still assessed as partly complied with. Considering the cited weaknesses in delivered outcomes, this validation also rates the project less than effective. C. Efficiency of Resource Use in Achieving Outcomes and Outputs 25. The PCR recommended a rating of efficient and this validation agrees. The time taken by ERRA to carry out inspections and release payments was satisfactory. This included the time taken by ERRA s assistance and inspection teams to inspect the damaged houses, collate information from the field provided by the National Database and Registration Authority, and release payments to the beneficiaries personal accounts. On average, for 83% of the respondents to the 2010 housing survey of ADB, less than 3 months elapsed between the time a house owner completed the plinth and the plinth was inspected by the assistance and inspection team. Similarly, for 73% of the survey respondents, it took less than 3 months after the inspection for the third payment to be released. 26. No major holdups occurred in meeting delivery targets. The single exception was a 6- month delay in the release of the second tranche of the loan because a large number of homeowners had not yet started rebuilding their houses as many of them preferred to resettle in other locations. However, this delay was minor, considering that ERRA was taking an ownerdriven approach and was serving a population of 2.91 million. The housing reconstruction started on 16 July 2007, and the final tranche was released on 5 May 2008, well before the program closing date of 31 December 2008 provided in Article III of the loan agreement. D. Preliminary Assessment of Sustainability 27. The PCR rated the program s achievements very likely to be sustainable. Sustainability was measured according to the following criteria: (i) the diversity of seismically compliant house designs to better fit the needs of the local population, (ii) the capacity of the stakeholders to apply the designs, and (iii) the level of adoption. 28. ERRA was able to use local construction practices and introduce changes to make them seismically safe. All house design options of ERRA were based on the skills and materials that were either available or made available in these regions through the program. ERRA and various NGOs

9 7 conducted extensive training and prepared manuals in the national language. Finally, the design options chosen were based on the use of reinforced cement concrete (RCC), timber, and bhatar traditional house design options popular in these regions, but modified to ensure seismic resistance. 29. The PCR also noted that the program s sustainability and achievements should also be gauged by the continuing influence of the ideas spawned by the program. Private house construction has, in fact, been shifting gradually toward the application of ERRA standards. The 2009 ERRA report assessing the social impact of the program found an increase in seismicresistant RCC construction in earthquake-affected areas. As of 2009, 53% of the houses had been built with safer RCC construction methods (instead of simpler RCC construction), compared to 19% of pre-earthquake houses. 30. Overall, 72% of houses built since the earthquake without any subsidies have used seismic-resistant construction techniques, with the rate in Khyber Pakhtunkhwa being slightly higher than in AJK. Most homeowners who built their houses after the earthquake without help from the program were aware of ERRA s seismic-resistant construction techniques before they started building. 31. While the use of ERRA construction standards increased, a decline in unsafe house construction generally based on the use of mud or heavy sections of timber, or a combination of the two was also noted. Mud construction has seen the most dramatic decrease from 35% before the earthquake to only 8% afterwards. This change is attributed to the enhanced awareness and skills development in the communities, and the acceptance by local authorities of earthquake-resistant construction and design. As referred in para. 64 of the PCR, the obstacle to adoption of seismically compliant construction techniques has not been lack of know-how or its rationale, but affordability factors. And that new funding is needed to be allocated to public information campaigns to increase compliance levels. While sustainability measures are generally in place, there were lower-than-target outcomes delivered (paras ) and increased compliance rates were to be dependent on fresh allocations of funds. Considering all these assessments together, this validation rates the project likely sustainable. 7 E. Impact 32. The overall impact of the program was rated positive by the PCR and this validation recommends a rating of significant, although at the time of the PCR s preparation, it was difficult to assess its effect on the GDP growth in Khyber Pakhtunkhwa as security and insurgency issues in other parts of the province must also be taken into account. 33. The number of people who now use burnt brick or blocks to build houses using reinforced masonry increased from the pre-earthquake level of 49% to 92% 22% better than the 70% envisaged at appraisal. An additional benefit from this change is the better health and hygiene of the population now living in these permanent structures. 34. The program helped restore and improve the livelihood of the earthquake-displaced people in AJK and Khyber Pakhtunkhwa. Construction-related businesses and the availability of skilled and semiskilled construction labor have increased greatly. New construction methods have allowed the growth of new businesses, such as mechanized block production. The building boom has had a ripple effect, providing jobs to people from areas outside the earthquake-affected regions. 7 The concerned operations department expressed that it would like to see the PCR s rating maintained.

10 8 35. The PCR noted that the owner-driven approach has fostered self-confidence and selfreliance of the communities that participated in the program. Due to a specially targeted campaign that was part of the program, over 55,000 households with female heads (owning 16% of all houses reconstructed by ERRA), and many individuals from vulnerable groups now have land titles, own houses, and enjoy higher status in their communities. Most people in the earthquake-affected regions now have access to and knowledge of the formal banking system, as a result of their having opened more than 600,000 bank accounts to receive subsidies from the program. Villages are now more organized and socially cohesive, and they are more familiar with community-based approaches to solving problems all because of the program s creation of village reconstruction committees. III. OTHER PERFORMANCE ASSESSMENTS A. Performance of the Borrower and Executing Agency 36. The performance of MOF as executing agency and of ERRA as implementing agency was rated by the PCR satisfactory and this validation agrees with this rating. ERRA and the Economic Affairs Division of MOF held regular portfolio performance and donor coordination meetings to discuss outputs and coordinate responses. The funds released by ADB to MOF were made readily available to ERRA. There was a minor delay (about 6 months) in the release of the second tranche of the loan as ERRA had not yet fulfilled one of the conditions the payment of the third installment of the housing subsidy to 85% of eligible households. This late compliance by ERRA was mainly because many homeowners have not yet started rebuilding their houses even after receiving the second installment of their subsidies, which was beyond ERRA s control. 37. ERRA satisfactorily met the loan conditions for the release of the second tranche. These related to progress in reconstruction, acceptable audit reports, annual financial management reports, training programs, and contract award for establishing legal aid centers. ERRA regularly produced reports on the progress and delays in implementation, and responded quickly when its monitoring system detected problems. B. Performance of the Asian Development Bank 38. The PCR assessed the performance of ADB partly satisfactory and this validation agrees with this rating (less than satisfactory). ADB closely monitored the progress of the loan throughout its implementation, and resident mission personnel worked regularly in the field. The individual consultants and NGOs working under ADB s capacity building TA were promptly hired and sent out into the field. Various donors supported the housing reconstruction, and coordination among them was well established, as was the support from organizations assisting with monitoring and financial management. 39. The support of ADB concentrated on training homeowners in seismically compliant reconstruction, improving fiduciary risk management, establishing a fully transparent and functional financial management information system, strengthening internal controls, safeguarding capacity, and ensuring regular and adequate reporting. ADB also conducted a survey to analyze the effectiveness of the various implementation processes. It disbursed the loan efficiently, and responded immediately, by making a minor change, to relax one of the second-tranche conditions to accommodate the 34,234 homeowners who did not start reconstruction even after repeated notices and deadline issued by ERRA.

11 9 40. Although ADB administered and supervised effectively most of the activities, not enough attention was given to outcome and impact monitoring, a key implementation activity that was to be supported by the TA project. ERRA took on this function, but by the time it started reporting on monitoring outcomes, the housing reconstruction was nearing completion. Moreover, data collection and analysis for the housing reconstruction was slow. If this problem were noted earlier, the TA project could have remedied it in time. The failure of ADB to respond effectively in this case was a missed opportunity to correct major compliance gaps in housing reconstruction. C. Others 41. The Gender and Vulnerability Action Plan (GVAP) noted that women have been particularly challenged after the disaster since their roles have changed drastically from caregivers and contributors to the family income in a conservative society to household heads and main bread earners for families where the earning male household head has died or suffered from permanent physical disabilities (footnote 1, Appendix 6). It was recognized in the GVAP that widows and households with female heads needed outside support specifically in establishing appropriate living arrangements and sources of livelihood within the community that enable them to enjoy protection and to rebuild their family s safety net on a new ground. 42. The program aimed to ensure that women will receive the same assistance and compensation as men, including training support from the TA project on seismic building techniques and designs. The PCR included an update of the GVAP (PCR, Appendix 4). This update recorded very substantial equality achievements that were not conveyed in the main text and findings of the PCR. The midterm review noted that vulnerable groups included the widows, elderly, handicapped, landless, and orphans and it was not clear if these groups were receiving adequate support. It would have been useful for the PCR to have assessed performance in this area. 43. The program included an Initial Environmental Examination (footnote 1, Appendix 7), which proposed an Environmental Monitoring and Management Plan for Housing Reconstruction with recommended mitigation measures. Based on the ADB 2010 housing survey, 85% of household debris is still disposed of in the open, and reuse is low. Moreover, although the program discouraged timber houses, at higher altitudes there were no alternatives. Among owners of timber houses (7% of the sample), 28% had cut down an average of 9.5 trees. The 2010 housing survey of ADB found a 30% increase in the use of flush systems in reconstructed houses, a more than 30% increase in the use of septic tanks and soakage pits for the disposal of excreta, and a corresponding 20% decrease in the open disposal of excreta. Environmental management appeared to have a low profile in program design, with no performance measures in the design and monitoring framework (DMF), for example. The PCR could have provided more emphasis on the implementation of the Environmental Monitoring and Management Plan for Housing Reconstruction. IV. OVERALL ASSESSMENT, LESSONS, AND RECOMMENDATIONS A. Overall Assessment and Ratings 44. The program is rated successful by the PCR (see table). Its approach was only partly effective in reaching the intended outcome and outputs, but the program was efficient and is very likely to be sustainable. Overall, the impact of the program on the lives and livelihood of the affected population had been positive.

12 The housing reconstruction was generally implemented as conceived except for houses that suffered minor or negligible damage. The undertaking had built-in flexibility, making it possible to add many new construction options to accommodate local skills and materials. Similarly, improvements were made in the training methods, compliance manuals, and outreach mechanisms to overcome shortcomings identified during implementation. However, there were major limitations short time frame, large geographic area, large number of affected home owners, and unanticipated challenges posed by the owner-driven approach. There was often not enough time to develop and adopt changes, and although the print and electronic media were used extensively, accessibility problems due to terrain and distance meant that physical contact was not always possible. 46. The DMF was a weakness of the program. There was a major attempt to develop and implement an extensive monitoring and evaluation (M&E) system, but it took much more time than anticipated. While sustainability measures were generally in place, there were lower-thantarget outcomes delivered for the project to be sustained. Overall Ratings Criteria PCR IED Review Reason for Disagreement and/or Comments Relevance Relevant Relevant Effectiveness in achieving outcome Partially effective Less than effective Efficiency in achieving Efficient Efficient outcome and outputs Preliminary assessment of sustainability Very (Most) likely Likely While sustainability measures were generally in place, there were lowerthan-target outcomes delivered for the project to be sustained (para. 31). Overall Assessment Successful Successful Borrower and executing Satisfactory Satisfactory agency Performance of ADB Partly satisfactory Less than satisfactory Impact Positive Significant Refer to paras Quality of PCR Satisfactory Refer to para. 51. ADB = Asian Development Bank, IED = Independent Evaluation Department, PCR = project completion report. Note: From May 2012, IED views the PCR's rating terminology of "partly" or "less" as equivalent to "less than" and uses this terminology for its own rating categories to improve clarity. Source: ADB Independent Evaluation Department. B. Lessons 47. This validation agrees with the lessons identified in the PCR. The validation stresses that rigorous and fast-paced monitoring and evaluation, which is well suited to the implementation arrangements, is critical in urgently needed and large programs like this one, hence, it needs emphasis in the design of such programs. C. Recommendations for Follow-Up 48. This validation agrees with the recommendations made in the PCR.

13 11 V. OTHER CONSIDERATIONS AND FOLLOW-UP A. Monitoring and Evaluation Design, Implementation, and Utilization 49. The PCR noted in para. 18, that a fully functional M&E system was established with support from the ADB loan and TA funds, and from other agencies such as the United Kingdom Department for International Development (DfID), the World Bank, and the United Nations. M&E reports, covering all activities managed or coordinated by ERRA and including special case studies on topics such as gender and vulnerability, have been generated regularly. However it is noted elsewhere in the PCR that one weakness of ERRA was its monitoring and evaluation of outcomes despite reports from its own validation teams of noncompliance with seismic standards, the authority failed to appreciate the full extent of noncompliance by participating homeowners and could not address the noncompliance efficiently. 50. It is also noted that although ADB effectively administered and supervised most of the activities assigned to it, not enough attention was given to outcome and impact monitoring. This function was to be performed by a third party to be hired by the Department of International Development. Repeated attempts to outsource this activity to international agencies failed because of concerns about security. As an alternative arrangement, ERRA took on this function (para. 40). B. Comments on Project Completion Report Quality 51. The quality of the PCR is satisfactory. It has shortcomings, such as the lack of discussion on social safeguards, especially on points raised in the ADB midterm review mission concerning vulnerable groups and their participation in the program. However, considering the high-risk condition in the area that made it difficult for project staff to supervise, monitor, and record project outcomes, the project is commendable as it was able to conduct the housing survey, which chronicled the achievements and shortfalls of rehabilitating 580,000 damaged housing units. The PCR made use of the survey findings to present a detailed account of implementing such emergency loan assistance by ADB and other partners. C. Data Sources for Validation 52. Apart from the program completion report, the report and recommendation of the President, and back-to-office reports were used for this validation. D. Recommendation for Independent Evaluation Department Follow-Up 53. The PCR provided useful lessons for the management of natural disasters involving large areas and large numbers of affected people. Further follow up through a Program Performance and Evaluation Report is not recommended, but the disaster management lessons learned could be disseminated to relevant governments and ADB offices.