Terms of Reference Independent Completion Report (ICR) for ADB Infrastructure Technical Assistance Background Program overview Timor-Leste has critical infrastructure needs given the widespread destruction of infrastructure during the period of independence and the 2006 crisis. While the Government of Timor-Leste (GoTL) is committed to scaling up infrastructure in line with its Strategic Development Plan, its Ministry of Infrastructure (MoI) has limited capacity in its budget execution and management of infrastructure activities. The Infrastructure Technical Assistance (ITA) program (total value of around US$18 million) is managed by the Asian Development Bank (ADB) and co-financed by Australia. 1 The ITA was approved by ADB in June 2007 and was scheduled to finish in June 2011. The main Consultant (SKM) mobilized in January 2008 and is now contracted until 31 March 2012 (although one adviser will remain until the end of 2012). Priority project management support will be provided for a short period following completion of the main Consultant s contract. A scholarship program within the ITA commenced in January 2010 and will run until June 2013 to allow students to finish their studies. The ITA is aimed at enabling MoI to create and upgrade infrastructure assets in accordance with the medium term national development plan and annual targets. It has supported a range of directorates and secretariats within MoI. The technical assistance (TA) provides long-term experts in procurement, project management, capacity development, institutional improvement, and monitoring and evaluation to assist MoI with implementation of infrastructure development projects in transport, information and communication, water and power sectors. In the process the MoI staff received on the job training to develop and enhance their capacity to improve their performance. In addition, the TA provides short-term experts to respond to the specific needs of MoI in implementation of the annual capital development plans, and supports a scholarship program for Ministry staff. The TA s two major components are: (1) implementation of capital development programs and execution of their budgets, and (2) building of project management capacities. The objective of Component 1 is to assist in the execution of annual capital development program through its direct involvement in the agency s management of infrastructure projects. It had been recognised that a major cause of project implementation delays was the inadequacy of the workforce to effectively manage the volume of annual capital development projects. The TA was to assume the following 1 ADB. 2007. Timor-Leste Infrastructure Project Management (Cofinanced by the Government of Australia. Project Number 39151. Manila (May). 1
project management functions: i) the preparation and procurement of contracts for consulting services, goods and works already identified in the government s annual capital development work plans and, ii) the implementation, management and supervision of contracts already procured. The objective of Component 2 is to build the project management capacities of the agency with particular attention given to contract administration. Key institutional processes, systems and policies, standards and individual competencies were to be identified and strengthened to support the agency s capacity to procure and administer project contracts. A staged capacity building approach was adopted for this component. The expected impact of the ITA is enhanced opportunities for economic participation, and improved access to basic social services for all East Timorese. The expected outcome is infrastructure assets created and upgraded in line with GoTL annual capital development plans, including transport, communications, urban development, power, and water supply and sanitation. A mid-term review, which was finalised in March 2011, determined the final technical inputs to be provided by the consultant. AusAID made its final payment in May 2011. Policy and context for Australia s involvement in the infrastructure sector in Timor-Leste Australia s support in infrastructure in Timor-Leste is consistent with the Australia Timor-Leste Country Strategy 2009-2014 objective of increasing employment by improving infrastructure, including through labour intensive initiatives. The recently signed Timor-Leste Australia Strategic Planning Agreement for Development aligns Australia s support to Timor-Leste s Strategic Development Plan 2011-2030. Primary support to the SDP s infrastructure objectives are in the areas of roads and water and sanitation. Australia s current engagement in the infrastructure sector is centred on two major initiatives in the rural water supply and rural road sectors. Roads for Development (R4D), which builds upon the recently completed multi-donor TIM-Works program, is a $30 million rural roads program that began implementation in March 2012. R4D is being implemented through the Directorate for Roads, Bridges and Flood Control (DRBFC) in the Ministry of Infrastructure with technical assistance from the International Labour Organization (ILO) and aims to support GoTL to effectively plan, budget and manage rural road works using labour based methods, as appropriate. R4D s main thrust is to develop and institutionalize adequate capacities and instruments in the public sector in particular within the Directorate of Roads, Bridges and Flood Control of the Secretariat of State for Public Works under the Ministry of Infrastructure that will enable GoTL to effectively and equitably plan, 2
budget and implement investments in rural road construction, rehabilitation and maintenance. A holistic capacity building strategy will be followed that will focus on strengthening capacities in the public sector. This will be done by supporting DRBFC in establishing functional management and technical capacities and operational tools, in supporting policy/strategy dialogue and development, by providing leading coordination support and by supporting the development of a performance culture and knowledge management capability. Capacity building will be fully integrated within the institutional structure of DRBFC and ILO R4D specialists will work in-line with DRBFC staff. Considering the uncertainty of the direction and pace of the decentralisation process and engagement levels of DRBFC, R4D s design incorporates sufficient flexibility to ensure that its interventions can respond to the actual situation on the ground. Capacity development will follow AusAID s staged approach and will take place on the back of substantial AusAID and GoTL capital investments for rural road works. The other major initiative is the Rural Water Supply and Sanitation Project (RWSSP or BESIK), a $40 million program working with the National Directorate of Water Supply and Sanitation of MoI. The program is also assiting the Ministry of Health, particularly to promote improved sanitation and hygiene. The program assists the government to plan, budget, design, procure and supervise the construction and rehabilitation of rural water systems, as well as assist communities with operation and maintenance. The program is about to enter a second phase in the latte half of the year. In line with the broader objective of consolidating support in fewer and larger initiatives, Australia s ongoing support in infrastructure will be centred on these two initiatives as stipulated in the Strategic Partnership Agreement. No further support to the ADB is envisaged at this stage, while it is important to gather lessons learnt from this program to inform AusAID s continued engagement with the Ministry of Infrastructure particularly with DRBFC. Preliminary TA Performance Report The main Consultant (SKM) is preparing a final report modelled on the approach of the ADB s standard project completion report. A first draft has been provided to ADB and AusAID and commented on by ADB. A second draft is to be available to ADB and AusAID during April 2012. Drawing on the main Consultant s final project report and other inputs, the ADB will prepare a preliminary TA performance report which follows the approach of ADB s TA performance evaluation report. 2 The preliminary TA performance report will pay particular attention to the observations and suggestions provided in the ADB s paper on capacity building, 3 and the Ministry of Finance s draft policy on technical assistance. 2 ADB. 2006. Guidelines for Preparing Performance Evaluation Reports for Public Sector Operations. Operations Evaluation Department. Manila (January). 3 ADB. 2011. Closing Capacity Gaps in Timor-Leste. Pacific Studies Series. Draft. Dili (December). 3
The preliminary TA performance report is being prepared now in order to (i) provide a resource document for the evaluation (ii) guide the allocation of the TA s final resources, (iii) inform the preparation of a follow-on TA, Infrastructure Management, and (iv) inform dialogue on infrastructure and capacity building with the government and stakeholders, and (v) inform the provision of other support by ADB through investment projects and TA. Purpose of the evaluation AusAID requires an independent completion report (ICR) of ITA because Australia s contribution is over $3 million in value. No independent evaluation has been undertaken to date for this program. As the TA nears completion, ADB s Pacific Department will discuss with ADB s Independent Evaluation Department the potential for an independent review of the TA. 4 The primary purpose of the proposed evaluation is program improvement. Lessons learnt from the ITA program will inform ongoing and future programs in the infrastructure sector in Timor-Leste that involve capacity development, particularly R4D and, to a lesser extent, BESIK. The secondary purpose of the evaluation is to meet AusAID s accountability requirements, by evaluating the program s achievements more broadly using AusAID and ADB evaluation criteria as a guide (addressing, where appropriate, relevance, effectiveness, efficiency, sustainability, gender equality, monitoring and evaluation and analysis and learning). In accordance with AusAID policy, the evaluation will be published on AusAID s external website. Scope of the evaluation The primary users of the AusAID report will be: 1. AusAID Timor-Leste infrastructure team in Dili and Canberra 2. AusAID Timor-Leste broader program teams in Dili and Canberra 3. ADB Secondary users of the report include: 1. Partners in Timor-Leste, including the Ministry of Infrastructure 2. The International Labour Organisation team responsible for the Roads for Development program 3. AusAID s Infrastructure Thematic Group 4 The Independent Evaluation Department has autonomy in the selection of TAs to review, and only a small number are reviewed each year (for a discussion of how the Department sets its workplan, see http://adb.org/site/evaluation/ied/work-program). 4
The evaluation will be completed using the AusAID Independent Completion Report template which is consistent with AusAID and ADB evaluation standards. Evaluation questions The evaluation team should address the following questions as part of the review: Program improvement 1. How well was the ITA program managed and how was the performance of TA specialists? What was the level of government commitment to the ITA? How can future programs ensure that use of technical advisers in Timor-Leste is better managed, more effective and has better ownership by the government? 2. To what extent are various ITA products (for example, the unit cost databases for roads and buildings, standards for roads and bridges, and the Project Monitoring and Control System) being utilised and of value to MoI? 3. How successful was the staged capacity building approach? As R4D is also adopting this approach with DRBFC, what lessons learnt should be passed on to ensure that it is utilised effectively and to maximise sustainability? 4. What are the specific capacity issues in DRBFC revealed by this program that AusAID and the ILO will need to continue to address and consider as they work to implement R4D? 5. How sustainable are the program outcomes? What factors increased or reduced the sustainability of this program, and how? To what extent has the program led to ongoing benefits for GoTL? 6. How effective was the ADB s and AusAID s management of the program? What contribution did the ADB add in its management of SKM in this program, and were there any challenges or difficulties? The evaluation team will undertake a ten day visit to Timor-Leste for consultations (interviews, focus groups) with national and international stakeholders, including ADB, SKM and MoI. There will also be at least one field trip to visit infrastructure projects implemented under this program. Management decisions The ICR will provide analysis and recommendations that will inform the following management decisions, to be made by the AusAID Timor-Leste management team: 1. Does AusAID and the ADB need to take any actions or make any changes in its approach to technical assistance in the Ministry of Infrastructure? 2. Does AusAID need to take any actions to shape and influence the implementation of capacity development in R4D or BESIK at an operational level (eg through the Program Steering Committee) as a result of findings from this review? Key stakeholders to meet 5
The following list of possible key stakeholders to meet is provided as a recommendation. The evaluation team will confirm persons and organisations they wishes to meet in the evaluation plan. The Minister of Infrastructure Managers and staff of the Ministry of Infrastructure Advisers currently or formerly working with the ITA program that are available in Timor-Leste during the mission ADB, AusAID and SKM personnel in country The former ITA Team Leader Sally Bannah (based in Brisbane, Australia either in person or by teleconference) and/or other former project staff. Key documents to review Australia Timor-Leste Country Strategy 2009-2014 ADB design documents Inception report Annual reports Mid term review documents Final report Roads for Development (R4D) Design Summary and Implementation Document AusAID Indonesia and East Timor (IET) Monitoring and Evaluation Standards Main consultant s (SKM) draft final report. Preliminary TA performance report prepared by ADB. Evaluation team The evaluation team will comprise: a) Team Leader Capacity Development Specialist with expertise in Monitoring and Evaluation; b) Infrastructure specialist with transport sector expertise Representatives from ADB and from AusAID s Timor-Leste infrastructure team will also join the mission for some meetings and visits. Reporting requirements The evaluation team will review relevant documents including ITA design documents and reports and prepare an evaluation plan to be submitted to AusAID and ADB prior to the mission. The evaluation plan should outline the methods and timeframe the evaluators will use to meet the objectives and scope in these ToRs, including: An evaluation design that describes a logical model for assessing the activity, including an evaluation matrix; A process for information collection and analysis; 6
Any substantive challenges to achieving the evaluation objectives that will have to be addressed. The matrix will cover the key evaluation questions, and also draw on the broader AusAID evaluation criteria as a guide where deemed appropriate. The team leader will be responsible for the following written outputs, which he/she may delegate to team members as agreed: a) Evaluation Plan to be developed in consultation with AusAID and ADB and submitted in final copy by 1 June b) Initial Aide Memoire (3-4 pages) to be submitted by 22 June outlining the key findings of the evaluation. c) Draft Independent Completion Report (a maximum of 20 pages excluding attachments) by 4 July d) Final Independent Completion Report (a maximum of 25 pages excluding attachments) within one week of receiving AusAID and ADB comments. The evaluation plan and report will be appraised against the AusAID Indonesia and East Timor (IET) Monitoring and Evaluation Standards which will be provided to the evaluation team. Review requirements AusAID will seek ADB input on the draft report and internal views from relevant sector experts and country staff for consideration for the final report. Revisions to the report may be required, and would be negotiated as appropriate. Evaluation timeline Dates Activities Location Maximum number of days Team Leader Infrastructure Specialist 30 May 1 June Pre-mission desk review and preparation of evaluation plan (includes teleconference briefing) Home 3.5 3 13 22 June For team leader, includes half day Brisbane meeting with Sally Bannah, former ITA Team Leader. In-country mission, including Initial briefing with AusAID Consultations (individual and group) with stakeholders as per Dili 10 + travel days 10 + travel days 7
evaluation plan Teleconference with Sally Bannah Field visits Prepare and present an Aide Memoire to AusAID (AusAID will propose a draft schedule for the incountry component and discuss it with the evaluation team as part of the Evaluation Plan preparation process) By 4 July Draft ICR report Home 3 3 Within a week of receiving comments from ADB and AusAID Amendments to first draft report. Submit final report 1 1 8