Samoa: Small Business Development Project

Transcription

1 Validation Report Reference Number: PCV: SAM Project Number: Loan Number: 1785-SAM(SF) January 2011 Samoa: Small Business Development Project Independent Evaluation Department

2 ABBREVIATIONS ADB Asian Development Bank BDS business development service CBS Central Bank of Samoa DBS Development Bank of Samoa EA executing agency FSPL Financial Sector Program Loan IA implementing agency IED Independent Evaluation Department M&E monitoring and evaluation MOF Ministry of Finance MSEs micro- and small enterprise OCR Ordinary Capital Resources PAHQ Pacific Operations Division PARD Pacific Regional Department PCR project completion report PMU project management unit PPSA Personal Property and Securities Act PSOD Private Sector Operations Department RRP report and recommendation of the President SBDF Small Business Development Fund SBDP Small Business Development Project SBEC Small Business Enterprise Centre SBLGS Small Business Loans Guarantee Scheme SDR special drawing rights SPSO Pacific Subregional Office in Suva, Fiji ST Samoan tala TA technical assistance VCF venture capital fund WIBDI Women in Business Development Incorporated WIBF Women in Business Foundation NOTE In this report, $ refers to US dollars. Key Words asian development bank, adb, borrower, executing agency, private sector, project, micro- and small enterprises, poverty reduction, guarantees, business advisory services, venture capital, legal and regulatory reform, consultants, performance, relevance, outputs, outcomes, effectiveness, efficiency, sustainability, impact, lessons, recommendations

3 Director Team leader Team members H. Hettige, Independent Evaluation Division 2 (IED2), Independent Evaluation Department (IED) K. Thukral, Senior Evaluation Specialist, IED2, IED R. Lumain, Senior Evaluation Officer, IED2, IED R. Perez, Senior Evaluation Assistant, IED2, IED In preparing any evaluation report, or by making any designation of or reference to a particular territory or geographic area in this document, the Independent Evaluation Department does not intend to make any judgments as to the legal or other status of any territory or area.

4 PROJECT COMPLETION REPORT VALIDATION A. Basic Project Data PCR Validation Date: January 2011 Project Number: Approved Actual Loan Number: 1785-SAM Project Name: Small Business Total Project Costs Development Project ($ million): Country: Independent State of Samoa Loan ($ million) (SDR million equivalent): Equity ($ million): 0.40 Sector: Small and Medium Total Cofinancing ADB Financing ($ million): Enterprise Development ($ million): ADF: $3.5 (SF) Borrower ($ million): OCR: $0.4 a Beneficiaries ($ million): Cofinanciers: Others ($ million): Approval Date: 21 Nov 2000 Effectiveness Date: 4 Jun Jun 2001 Signing Date: 6 Mar 2001 Closing Date: 31 Aug Sep 2008 Project Officers: Validator: Quality Control Reviewer/Peer Reviewer: Name: W. Wicklein A. Vatucawaqa M. Melei S. Thalakada, Consultant K. Thukral, IED2 R. Lumain, IED2 Location: PAHQ PAHQ/SPRM/SPSO SPSO Director: From (yr) Hemamala S. Hettige To (yr) ADB = Asian Development Bank, ADF = Asian Development Fund, IED2 = Independent Evaluation Division 2, OCR = ordinary capital resources, PAHQ = Pacific Operations Division, PCR = project completion report, SAM = Samoa, SDR = special drawing rights, SPRM = South Pacific Regional Mission, SPSO = Pacific Subregional Office in Suva, Fiji. a Equity investment in the venture capital fund. B. Project Description (summarized from the report and recommendation of the President [RRP]) 1 (i) Rationale: More than 80% of all businesses in Samoa had five or fewer employees. Most of the businesses did not have adequate skill levels to create, manage, and increase the size of their enterprises. They lacked equity and access to bank borrowings to provide a sustainable financing basis for their development. Furthermore, customary communal ownership (which covers at least 80% of land) restricted the availability of acceptable securities to access financial services. Although chattels were used extensively for security, commercial banks no longer placed reliance on such security because of enforceability problems and the state of the relevant registries. Since 1996, the Government had embarked on a process of economic reform and liberalization, which recognized the private sector as the engine of growth. The reform process had resulted in an environment that was conducive to economic growth. After focusing on major macroeconomic reforms, the Government developed strategies to broaden the spread of benefits of economic growth. The Asian Development Bank (ADB) supported the reform efforts through the Financial Sector Program Loan (FSPL). 2 A key component of the reform strategy was the development of infrastructure of the village economy, which centered on micro- and small enterprises (MSEs). Improving MSE access to credit supported by improved access to services provided by business development service centers was expected to contribute to growth in employment and income-earning opportunities in this sector. 1 ADB Report and Recommendation of the President to the Board of Directors: Proposed Loan and Technical Assistance Grant to Samoa for the Small Business Development Project and Proposed Equity in a Venture Capital Fund. Manila (31 October, approved on 21 November). 2 ADB Report and Recommendation of the President to the Board of Directors: Proposed Loan and Technical Assistance Grant to Samoa for the Financial Sector Program. Manila (Loan 1608, for $7.5 million, approved on 19 February).

5 2 (ii) Impacts: The Small Business Development Project (SBDP) was expected to contribute to poverty reduction through improving access of MSEs to credit, and business development services thus facilitating income-generating activities. During the 5 years of project implementation, the project was expected to create an additional 1,400 direct income and employment opportunities, benefiting at least 5,600 people. The specific target group of the project included rural households with residences outside urban Apia. There was no specific poverty or geographic area that required priority support. The most important target groups in the project were small entrepreneurs, both male and female, who wished to expand or consolidate existing businesses and required additional financial resources for that purpose. The project facilitated diversification of the financial sector in Samoa, logically extending policy dialogue initiated under the earlier FSPL (Loan 1608-SAM). Also, by improving the legal and regulatory environment pertaining to entrepreneurship development, the project was expected to facilitate private sector-led economic growth. (iii) Objectives or Expected Outcomes: The objective of the project was to generate income and employment opportunities mainly in rural areas through expanding the MSE sector by improving access to credit and to business training and advisory services. At least 180 new MSEs were to be started and at least 400 existing enterprises were to be diversified through the guarantee scheme. (iv) Components and/or Outputs: The project included five components (a) expand the Small Business Loans Guarantee Scheme (SBLGS) utilizing the existing management framework of the Small Business Enterprise Centre (SBEC); (b) support a pilot microfinance facility on-lent through the Development Bank of Samoa (DBS) to approved participating organizations; (c) establish a Small Business Development Fund (SBDF) to help MSEs with additional business advisory and training services from approved business development service (BDS) providers; (d) improve the policy and legislative environment for small business development by supporting the Government s ongoing efforts to improve the legal and regulatory environment for the corporate sector, particularly in those areas that impact the MSE sector; and (v) subscribe up to the equivalent of 25% of issued and paid-up capital in a Samoan venture capital fund (VCF) up to a maximum of $400,000, and targeting MSEs. C. Evaluation of Design and Implementation (project completion report [PCR] 3 assessment and validation) (i) Relevance of Design and Formulation: The project was consistent with government policies and plans. It promoted private sector-led economic growth; more equitable distribution of economic benefits nationally, in particularly rural areas and the village economies; and employment creation through village and urban-based enterprises (RRP, paras ). Likewise, it was in line with ADB new poverty reduction strategy (R issued on 19 October 1999) that stated that enterprise development would be one way to enhance poverty reduction. ADB had been conducting policy dialogue with the government on financial sector reforms and development over about the last two and half decades (RRP, paras ). This had resulted in the establishment of the DBS and giving five credit lines and six technical assistance (TA) grants for its operations and institutional strengthening, respectively; five TAs for privatization and/or liquidation of 25 state-owned enterprises over the last 15 years; and the FSPL (Loan 1608-SAM). This project was intended to build on those efforts. The project was formulated by an ADB TA approved in September It took into account lessons learned (RRP, paras ) that related, inter alia, to the need for legal and regulatory reforms to precede or at least accompany direct project interventions, and the need for innovative approaches to improve the quality of financial services (example, business advisory and training services). The project was also coordinated well with the assistance being provided by other development partners (RRP, paras ). Nonetheless, this validation considers that the project design was complex for the nascent development stage of the Samoan financial sector. (ii) Project Outputs: The project was to be implemented over a 5-year period to be completed on 28 February 2006 and ADB loan utilization ending 30 June The actual outputs against the five outputs envisaged at appraisal were as follows: 3 ADB Completion Report: Small Business Development Project in Samoa. Manila. 4 ADB Technical Assistance to Samoa for Small Business Development. Manila (for $150,000, approved on 24 September).

6 3 (a) Expand the small business loans guarantee scheme: The appraisal envisaged providing 592 guarantees totaling 3.9 million Samoan tala (ST) issued by SBEC by the end of year 5 on 31 August At that date, 421 guarantees (71% in number) had been issued for ST4.4 million (113% by amount). As a result of extending the loan closing date to 30 September 2008 through three extensions and several reallocations of funds into this component, 853 guarantees had been issued for ST11.4 million covering loans totaling ST15.8 million by about the end of As at that date, a total of 504 guarantees (59%) had been issued to new businesses, exceeding the target of 180 or 21% of the total guarantees issued. At the extended project completion date, the SBDP exceeded its appraisal targets for the number of guarantees issued (853 vs. 592), the total value of guarantees issued (ST11.4 million vs. ST3.9 million), and the percentage share going to new businesses (59% vs. 30%). At project completion, 38% of the loans in number and 30% by value were in arrears, and the arrears represented 7.5% of the loan balances at that date. At the end of 9 months since project completion i.e., by about 30 June 2009, the arrears had worsened, with a 50% increase in the number of loans in arrears. Although the guarantee scheme did not appear to be all that successful judging by the high loan arrears data, the outstanding guarantees appeared to be secured, with deposits equivalent to 100.6% of the value of guarantees outstanding as at that date (PCR, paras. 9 10). (b) Support a pilot microfinance facility: The project provided $300,000 to establish a revolving fund in the Central Bank of Samoa (CBS) to be managed by DBS as a microfinance facility. During implementation, all funds were transferred to DBS as a loan. The loan was converted into a revolving fund in DBS at project completion. Although outputs were delayed substantially (93% of the loans given in the last year of the project), the project exceeded its target (minimum of 250 loans) by providing 681 microfinance loans with an aggregate value of ST875,550. At the time of the PCR, there were no arrears on 647 loans initiated by the Ministry of Women, Community and Social Development in the last year of the project; while 16 of the 34 loans (47%) given by the Women in Business Development Incorporated (WIBDI) were in arrears. Only WIBDI participated in the project as against at least two organizations envisaged at appraisal. This component exceeded the appraisal targets by lending $392,000 for 681 microfinance loans but, fell short of the number of participating organizations. The other setbacks were the delay in the component coming into operation and some loan arrears (PCR, paras ). (c) Enhance business advisory and training services: ADB was to provide an initial investment of $160,000 and the Government $40,000 to establish the SBDF. This sum was to be invested to provide income to purchase support services from participating BDS providers. ADB and Government were to provide a further $200,000 per year. Instead of establishing a fund, the original allocation was placed in an imprest account, through which the project management unit (PMU) made payments for selected BDS provided by SBEC and WIBDI. The total withdrawals under the SBDF came to $97,342, which was below expectations, being only 12% of the expected allocation of $800,000. In total, the component provided support for 305 business training courses involving 5,079 existing or intending small business operators (PCR, para.16). (d) Improve the policy and legislative environment for small business development: The amendments made to the Company Law 2001 by the associated TA were now fully operational (PCR, para. 18). The SBDP supported the preparation of a comprehensive Personal Property and Securities Act (PPSA) to cover all chattels including all personal property and securities, and to establish a registry of all personal property. Work on the PPSA was still ongoing on the loan closing date (PCR, para. 19). Measures were under preparation to expedite procedures relating to debt recovery and dispute resolution (in terms of streamlining court procedures, reassigning responsibility for some tasks from judges to court administrators, and enabling small claims to be determined in a lower court) to provide lenders with more confidence in assisting small business operators (PCR, para. 20). The RRP (para. 83) envisaged that the project would help to establish a credit reference facility (a credit bureau) on which considerable preparatory work was done under the associated TA, but it was later decided not to proceed with the Government considering options to reinstate a credit bureau using a regional facility

7 4 (PCR, para. 21). Lastly, through the associated TA 5 and follow-up TA approved in 2005, 6 the SBDP supported government efforts to make it easier to lease customary land; work on this matter was still continuing with further possible ADB TA (PCR, para. 22). (e) Make an equity investment up to $400,000 in a Samoan VCF to fund mainly MSEs: The VCF was established in 2005 after considerable delay. Largely due to lack of venture capital opportunities in the country, and financial viability doubts, the fund and the fund management company were terminated when the shareholders decided in February 2007 not to proceed with the VCF (PCR, para. 23). (iii) Project Cost, Disbursements, Borrower Contribution, and Conformance to Schedule (as relevant to project performance): The total project cost was estimated at $4.464 million equivalent, consisting of $2.604 million (58%) equivalent in foreign exchange costs and $1.860 million equivalent (42%) in local currency costs. It was to be financed by an Asian Development Fund loan of $3.50 million and a government contribution of $0.964 million equivalent. The actual project cost came to $4.798 million, financed by an ADB loan of $3.263 million and government funds of $1.535 million (Table 1). Table 1: Total Project Costs ($ million) Item Appraisal Actual Ratio (Actual/Appraisal) Small Business Loans Guarantee Scheme Small Business Development Fund Pilot Microfinance Facility Project Management Unit a Other Consultants Equipment and Vehicles Interest Unallocated Total Project Cost a Includes $148,380 for customary land expenses (PCR, Appendix 3). Source: ADB project completion report for Loan 1785-SAM. There was a reallocation of funds between components, as against the appraisal estimates, due to lack of demand for some components during the project implementation period. For example, the SBLGS took up $3.161 million as against the appraisal estimate of $1.500 million (111% increase) while the SBDF used only $0.129 million against the appraisal estimate of $1.000 million (87% decrease). The pilot microfinance facility used $0.397 million against the appraisal estimate of $0.301 million or a 32% increase. Likewise, actual cost for consultants/pmu office was $0.654 million against an appraisal estimate of $0.565 million or a 16% increase. At loan closing, an undisbursed loan balance of $0.237 million was cancelled. Implementation was delayed by about 2 years due to the delayed start of the project, lack of demand for some components, need to reallocate funds between them to ensure greater ADB loan utilization, and need for increased government funding for some components, all these taking time for negotiation and implementation. (PCR, para. 26) The PCR stated that the disbursement schedule in the RRP was unrealistic given the pioneering nature of the project (PCR, para. 8). (iv) Implementation Arrangements, Conditions and Covenants, Related Technical Assistance, Procurement, and Consultant Performance: Implementing arrangements generally followed those proposed at appraisal (RRP, paras ). The PCR deemed them complex and presented major challenges to all the agencies involved. According to the PCR, the covenants were relevant and adequate, and the majority of the covenants were fully complied with during implementation. Compliance with some covenants was delayed, while the covenant relating to the establishment of security over personal property and reform of land law was yet to be complied with (PCR, paras ). TA was 5 ADB Proposed Technical Assistance to Samoa for Capacity Building of Financial and Business Advisory Intermediaries. Manila (31 October, approved on 21 November). 6 ADB Technical Assistance to Samoa for Promoting Economic Use of Customary Land. Manila (approved on 22 December).

8 5 approved together with the project for capacity enhancement of the financial and business advisory intermediaries and for supporting ongoing efforts to improve the legal and regulatory environment for the private sector. The technical assistance completion report rated the TA as partly successful because of continuing capacity issues and number of policy components still needing completion (PCR, para. 36). 7 Selection of consultants was in accordance with ADB s Guidelines on the Use of Consultants, and no significant problems were encountered in their selection (PCR, paras ). The PCR judged the performance of the project manager only and stated that his performance was satisfactory and generally in accordance with the terms of reference (PCR, para. 40). (v) Performance of the Borrower and Executing Agency: The PCR assessed the performance of the borrower and the executing agency (EA) as "satisfactory." According to the PCR, The delay in initiating the project was partly due to slow establishment of the project management unit by the EA, but the complexity of the project was also a contributing factor (PCR, para. 41). Regarding the implementation arrangements, the PCR, as pointed out earlier, deemed them complex and presented major challenges to all the agencies involved (PCR, para. 34). While, agreeing with these statements in the PCR, this validation is of the view that at least in part, some delays and other problems may also be attributed to the fact that the borrower and EA had previously agreed with ADB to implement this complex and pioneering project without due consideration for the nascent development stage of the Samoan economy, its institutions, and their staff competencies. They had opportunities to reformulate the project during the project formulation and negotiation stages, to make it more appropriate for the Samoan circumstances. But apparently this does not appear to have happened. Hence, this validation assessed the borrower and EA performance as partly satisfactory. (vi) Performance of the Asian Development Bank: The PCR rated ADB performance as "satisfactory". This validation found it difficult to support that assessment, being of the view that, as formulated by ADB and taking into consideration the Samoan development context as well as its institutional and staff competencies, the project had upfront design flaws and appeared to have been overburdened with too many and too diverse components (loans, guarantees, business development efforts, venture capital, and institutional and policy reform). Under the circumstances, project implementation became difficult and time consuming, resulting in the need to drop an inappropriate component (VCF), underutilize another component (SBDF), reallocate funds between components and overutilize another component (SBLGS) and in delay in implementing the policy component. All these issues caused considerable implementation delays (by 2 years including three extensions). Despite these inherent difficulties, ADB project implementation supervision appeared to have continued adequately to ensure almost full ADB loan utilization, albeit with considerable implementation delays, and with some components not achieving their envisaged outputs and outcomes. On balance, this validation assessed ADB performance as partly satisfactory. Alternative options would have been to consider some components as standalone projects with a program loan modality to pursue the institutional and policy reform component, and for the Private Sector Operations Department (PSOD) to have considered the VCF component and the loan guarantee facility component under more rigorous due diligence. D. Evaluation of Performance (PCR assessment and validation) (i) Relevance: The PCR rated the project as "relevant" (PCR, para. 43). This validation, however, was of the view that the project s relevance declined during implementation and at completion. Design flaws made project implementation difficult, resulting in nonutilization of the VCF component and underutilization of the SBDF, and hence their appraisal-envisaged outputs and outcomes were not achieved. The project ultimately ended up more as a loan guarantee scheme, with the SBLGS taking up as much as 80% of the ADB loan as against the appraisal-envisaged 43%. The relevance of the SBLGS was also questionable, based on its high loan arrears (30% in value) and loan foreclosures (6%) situations. The only redeeming feature appeared (at loan closure) to be the success of the microfinance facility, though with some loan arrears, but it accounted for a relatively small allocation at appraisal (9%) and final usage (12%) of the total ADB loan proceeds. The policy reform component activities were 7 ADB Technical Assistance Completion Report: Capacity Building of Financial and Business Advisory Intermediaries. Manila (TA 3549, for $500,000, approved on 21 November 2000).

9 6 delayed and still continuing at project completion. Taking all these factors into account, this validation assessed the project as partly relevant. (ii) Effectiveness in Achieving Outcomes: The PCR rated the project as "less effective" (PCR, para. 44). The envisaged outcome of the project was the creation of additional income and employment opportunities within the MSE sector for all Samoans with an emphasis on those in rural areas. This outcome was to be achieved through the successful implementation of the five project components as described in Table 2. Based on PCR data, this validation concurs with the PCR assessment and finds the SBDP to have been less effective in achieving its outcome. Table 2: Assessing Achievement of Outcome and Outputs Outcome/Outputs Assessment Remarks Outcome: MSE growth through improved availability of financial and nonfinancial business development services. Less effective Some of the components do not appear to have been implemented effectively and sustain the outputs achieved. Thus, their contributions to outcomes such as creation of additional incomes and employment opportunities do not appear to have been fully achieved. Outputs: (i) Expand Small Business Loan Guarantee Scheme (ii) Small Business Development Fund (iii) Establish Pilot Microfinance Facility (iv) Policy and Infrastructure Development Effective, but with risks on long-term viability Less effective Effective, although loan intervention was relatively small in size Effective; reform process still ongoing (v) Establish the Samoa Ineffective Venture Capital Fund Source: ADB project completion report for Loan 1785-SAM. Though utilized exceeding the appraisal target allocation, the effectiveness of the SBLGS component in contributing to income and employment creation appeared questionable in view of the high loan arrears and loan foreclosures. The SBDF component was largely underutilized. The relatively small share of loan utilization (actual about 12% of ADB loan proceeds as against the 9% envisaged at appraisal) for the microfinance facility suggested that it could not have contributed much to creation of additional incomes and employment. The microfinance facility also encountered high loan arrears (WIBDI component had loan arrears of 47% by number of loans). The adoption of the institutional development and policy reforms were yet to be fully realized, and their successful enforcement would take a much longer time than envisaged and would require additional funding support according to the PCR. The VCF component was not utilized. (iii) Efficiency in Achieving Outcomes and Outputs: The PCR rated the project as "less efficient" (PCR, para. 45). Project implementation was delayed by about 2 years due to reasons given earlier (Section C[iii]). With regard to the five loan components, some were either not utilized (VCF) or underutilized (SBDF); the component that was overutilized (SBLGS) had high arrears and foreclosures; the institutional and policy reform component was still under implementation; and the implementation of the microfinance facility component picked up only in its last year of operation. Because of these implementation delays, underutilization, and poor quality of utilization (SBLGS), the appraisal-envisaged outcomes and outputs were either not fully achieved or took that much longer than anticipated. As a consequence, the outcomes and outputs achieved under the SBDP did not appear to be commensurate with the resources expended by the Government and ADB in processing and implementing the project. Government and ADB resources expended on the VCF did not yield any outcomes, as it was wound up. Overall, the PCR validation analyses support a less efficient rating. (iv) Preliminary Assessment of Sustainability: The PCR judged the sustainability of the project to be less likely (PCR, para. 46). The VCF and SBDF were wound up. According to the PCR, there were good prospects for the SBLGS and microfinance facility, as there was demand for their services, but this validation would like to add a caveat that their loan arrears would need to be kept within prudent limits for their sustainability. The policy reform efforts were still continuing, and their sustainability depended upon the

10 7 Government s continuing commitment to reform and availability of further funding for their adoption and enforcement. This validation assessed the sustainability of the project, and rated it less likely. (v) Impact (both intended and unintended): According to the PCR, the SBDP had generated "significant" and beneficial impacts (PCR, para. 47) in terms of contributing to income and livelihood development projects in rural areas while being conscious of preventing adverse environmental impact; encouraging family and new entrepreneurs, particularly those undertaken by women; and supporting some innovative projects such as an ecotourism project involving the protection of the green turtle. Since the institutional and policy reform component could not be fully implemented, its impact was not realized during the project period. However, measures being taken, if followed up for proper adoption and enforcement, should contribute to positive impacts in the medium term (example, alienation of customary lands and making them available to be pledged as security for borrowings). The SBDP supported the establishment of a large number of MSE operators as outlined earlier. However, this validation was of the view that the real impact of the SBDP for the country s economic development was questionable, as a large number of loans extended or guaranteed through the SBDP had loan arrears that suggested uneconomic and nonviable financial operations. Therefore, to ensure their greater economic impact, there is a need to strengthen policies and procedures for loan guarantee operations, microfinance lending, business development services, and procedures for project monitoring, and tighter arrears control. Overall, this validation rates the impacts of the SBDP as being moderate. E. Overall Assessment, Lessons, and Recommendations (validation of PCR assessment) (i) Overall Assessment: The PCR assessed the project as "successful" on an overall basis. The project had upfront design flaws. It was overburdened with too many and diverse project components in the context of the Samoan absorption and implementation capacity. Also, some components, for example the VCF, appeared too sophisticated as a financing instrument for promotion of MSE operations, and inappropriate in the Samoan development context (lack of viable venture capital opportunities and venture capital expertise), while some components, for example the SBDF, appeared inappropriately structured to implement a viable business advisory service. The SBLGS, though well utilized, was beset with large loan arrears, which brought its financial viability into question. The policy component was yet to be fully adopted and enforced. The microfinance facility showed good prospects for success but seemed underfocused, with a relatively small loan allocation at appraisal (9%) and low utilization of 12% of the ADB loan at closing. All these problems led to implementation difficulties, the need for reallocation of funds among the various project components, and less than appraisal-envisaged realization of outcomes and outputs. In making the overall assessment, the PCR arrived at a total score of 1.63 points (PCR, Appendix 7). This rating however, incorporates a significant rating for project impacts. Following an exchange of s with the concerned regional department in February 2010, the Independent Evaluation Department (IED) was made to understand that the intent of the PCR was to say that an application of the rating criteria as per the PPER rating guidelines produced a rating of "partly successful," but using evaluator judgment (provided for in the guidelines), the rating was adjusted to "successful." However, based on this PCR validation analysis, this validation rated the overall performance of the project as partly successful. (ii) Lessons: The major lessons that may be learned from the project relate to improving the prospects for successful project implementation, for yielding the desired outcomes and outputs, and for maintaining sustainability in the medium term. In particular, (a) the focus and selection of project components should take into consideration the developmental context of the country and its absorptive capacity; (b) in respect of the components to be included in a project, rigorous due diligence should be carried out on the various proposed components, to ascertain their viability and increase the likelihood of their implementation in practice by obtaining firm upfront commitments from the parties that champion their promotion; (c) the components should be better prepared and formulated to facilitate their successful implementation. The latter does not appear to have been done, for example, in the case of the VCF. The RRP (para. 84 and Appendix 4) was more of a description of what the proposed VCF would look like. Many loose ends, including critical issues such as shareholder contributions, choice of the fund manager and key staff, and tax liability, were left to be sorted out during implementation (RRP, Appendix 4, para. 22) which took time

11 8 for resolution, and the VCF was closed without doing any business after determining that it was unlikely to be financially viable (PCR, paras.33 and 52). (iii) Recommendations: These are as follows (a) The SBDP had some components (e.g., the SBLGS and microfinance facility) that appeared to have good prospects for success. ADB (Pacific Department [PARD] and PSOD) should promote them further, accompanied by steps taken, in parallel, to strengthen competence in proper evaluation of guarantee and microfinance operations, and provision of supporting BDSs. (b) Action had been initiated on a number of important policy initiatives that should be followed up by ADB (PARD) with appropriate assistance for their early adoption and enforcement. The PCR made a number of recommendations (paras ) that are supported by this validation. F. Monitoring and Evaluation (M&E) Design, Implementation, and Utilization (PCR assessment and validation): To help with M&E, a project framework was attached to the RRP, Appendix 1. The PMU was to submit to ADB, through the Government s Treasury Department (later renamed the Ministry of Finance), monthly and quarterly reports on various project activities (RRP, para. 107). The PMU was also required to submit to ADB certified copies of the audited accounts and financial statements within 6 months after the end of each fiscal year as well as a PCR prepared 3 months after project completion (RRP, para. 108). The requirement that the PMU develop an operating project performance monitoring system was only partly complied with, as the project data base was no longer operational. ADB fielded 11 project supervision missions, which included an inception mission and 2 country portfolio review missions. These missions appeared to have performed well, considering the difficult and challenging issues that came up for resolution during project implementation arising from the project design flaws, complex implementation arrangements put in place, and lack of a project data base. G. Other (e.g., safeguards, including governance and anticorruption; fiduciary aspects; government assessment of the project, as applicable) (PCR assessment and validation): The annual audit report included the audit of the imprest account and statement of expenditures procedures. The annual audited financial statements were submitted to ADB, albeit with delay. No irregularities were reported by the PCR. Procurement followed ADB guidelines. The PCR did not mention whether the Government submitted to ADB its own PCR, as required after project completion. H. Ratings PCR IED Review Reason for Disagreement/Comments Relevance: Relevant Partly Relevant For the nascent development stage of the Samoan financial sector, the project design was complex. Only the microfinance facility component appears to have been utilized well, but it was a small part of the project, and the project ultimately ended up more as a loan guarantee scheme with the SBLGS taking up about 80% of the ADB loan. See sections C(ii) and D(i) for further details. Effectiveness in Less Effective Less Effective Achieving Outcome: Efficiency in Less Efficient Less Efficient Achieving Outcome and Outputs: Preliminary Assessment of Less Likely Less Likely Sustainability: Borrower and EA: Satisfactory Partly Satisfactory The Borrower and the EA should have contributed more during project formulation and negotiation to make the project more in line with the nascent development stage of their economy, institutions, and staff competencies (i.e., more in line with Samoan circumstances).

12 9 Performance of ADB: Satisfactory Partly Satisfactory The project had upfront design flaws and appeared to have been overburdened with too many and diverse components, which made project implementation difficult and time consuming, resulting in the need to drop an inappropriate component (VCF), underutilize another component (SBDF), reallocate funds among components, overutilize another component (SBLGS), and delay the implementation of the policy component. Knowing the DBS capacity and other sector limitations, ADB could have suggested a less complex project design. Impact: Significant Moderate Although, as per the PCR, the SBDP had generated significant and beneficial impacts at loan closing (in September 2008), the real impact of a large number of livelihood development projects on the country s economic development was questionable, as a large number of them had loan arrears, suggesting uneconomic and nonviable financial operations. In particular, the SBLGS, which took up as much as 80% of the ADB loan against the appraisal-envisaged 43%, had high loan arrears (30% in value) and loan foreclosures (6%) situations in September 2008 (i.e. when the ADB loan was closed). Overall Assessment: Quality of PCR: Successful Partly Successful Satisfactory According to the four-category evaluation criteria of the present ADB guidelines, the SBDP is rated partly successful. Nonetheless, the PCR judged the project to be successful. I. Comments on PCR Quality: The PCR is generally comprehensive and well prepared. It appears to be an objective presentation on most issues. At the same time, there is room for improvement in terms of clarity and consistency. The PCR wording on overall assessment could have been clearer and consistent. For example, this validation notes an inconsistency in Appendix 7 regarding the rating in words as "partly successful" and the rating score of Besides, the PCR did not follow PPER guidelines regarding evaluation criteria used for overall assessment. Another area for clarification relates to project cost data in the Basic Data section and Appendixes 3 and 4, which show two different estimates of total project costs. This could have been reconciled or better explained in the text. The PCR analysis on the status of guaranteed loans issued could have analyzed the reasons for and/or implications of the observed trend in loan arrears and foreclosures (Appendix 2 on project outputs). This could have elucidated whether long-term viability risks are in part due to startup losses (with prospects for improved profitability) or indicate the need to address issues on debtor awareness of their obligations for repayment or any other reason. J. Recommendation for IED Followup: As the validation has downgraded the ratings for the project Relevance criteria, performance of the Borrower and the EA, performance of ADB, Impact, and the Overall Assessment, it is requested that a PPER be prepared. K. Data Sources for Validation: Data sources included the RRP, PCR, and the back-to-office reports of factfinding/appraisal and review missions, and other documents.

13 REGIONAL DEPARTMENT S RESPONSE TO THE PROJECT COMPLETION REPORT VALIDATION REPORT On 15 December 2010, the Director of the Independent Evaluation Division 2 of the Independent Evaluation Department (IED), received the following comments from the Pacific Subregional Office in Suva, Fiji of the Pacific Department. We have reviewed the Draft PCR Validation Report of Loan 1785-SAM circulated to us on 25 November We appreciate that our views on ratings of the project impacts have been suitably reflected in the final version of the PCR Validation Report, although the validation rating remains unchanged as moderate. We understand the rationale for the variance between IED s validation ratings and PCR ratings on the project s overall performance, performance of the borrower and executing agency, and ADB s performance, and take note of IED s ratings of partly successful, partly satisfactory and partly satisfactory, respectively. We have no further comments on this validation report.