PERFORMANCE MONITORING PLAN (PMP) Third Revision

Transcription

1 USAID INDONESIA URBAN WATER SANITATION AND HYGIENE PERFORMANCE MONITORING PLAN (PMP) Third Revision MAY 2014 This document was produced for review for USAID/Indonesia by the Indonesia Urban Water, Sanitation and Hygiene (IUWASH) project, implemented by DAI, in accordance with ADS Chapter (e) 05/05/2009 Revision.

2 Cover Photo A monitoring activity which was conducted for new water supply system of Negeri Passo in Ambon, Maluku Province. USAID IUWASH and the Drinking Water Working Unit of Maluku Province are committed to building a clean water supply system in Negeri Passo. IUWASH have provided technical assistance to develop its detailed engineering design (DED). The new clean water supply system in Negeri Passo will obtain water from Waiheru 3 spring (as shown in this picture), which the Public Works Office built in 2011 with a capacity of 20 liters/second and Wai Mahu 2 spring with 35 liters/second capacity. The water from the two springs will flow into a 500 cubic meter water tank by gravity and is expected to fulfill the need for water in Teluk Ambon Baguala subdistrict, including Negeri Passo neighborhood until Photo Credit: USAID IUWASH SSEI

3 PERFORMANCE MONITORING PLAN (PMP) Third Revision Program Title: Assistance Objective (AO): Sponsoring USAID Office: Contract Number: Contractor: USAID Indonesia Urban Water Sanitation and Hygiene DAI Project Number: AO Improved Management of Natural Resources, under (IR) 3 Increased Access to Water and Sanitation USAID/Indonesia Office of Environment AID-497-C DAI Date of Publication: May 2014 Author: DAI This document was produced for review for USAID/Indonesia by the Indonesia Urban Water, Sanitation and Hygiene (IUWASH) project, implemented by DAI, in accordance with ADS Chapter (e) 05/05/2009 Revision. II

4

5 TABLE OF CONTENTS 1 INTRODUCTION IUWASH PMP: OBJECTIVES TYPES OF INDICATORS PROGRAMMATIC ATTRIBUTION IUWASH PMP INDICATORS HIGH-LEVEL RESULT (HR): MDG OUTCOMES COMPONENT I (MD): DEMAND MOBILIZATION FOR IMPROVED WATER AND SANITATION SERVICES COMPONENT II (IC): CAPACITY IMPROVEMENTS TO PROVIDE SUSTAINABLE SAFE WATER AND SANITATION SERVICE COMPONENT III (EE): CREATE AN ENABLING ENVIRONMENT SUPPORTING EQUITABLE WATER AND SANITATION SERVICES PERFORMANCE MONITORING SYSTEM QUANTITATIVE PERFORMANCE MONITORING QUALITATIVE PERFORMANCE MONITORING QUALITY ASSURANCE REPORTING SYSTEM OF IUWASH OUTCOME INDICATORS PARTNER COLLABORATION IN IUWASH MONITORING ACTIVITY PERFORMANCE INDICATOR REFERENCE SHEETS ANNEXES ANNEX 1. ESTIMATED COST ALLOCATED FOR IUWASH MONITORING AND EVALUATION ANNEX 2. PARAMETERS OF PDAM PERFORMANCE INDEX ANNEX 3. PARAMETERS ASSOCIATED WITH THE CREDITWORTHINESS LADDER ANNEX 4. WATSAN CAPITAL PROJECT DEVELOPMENT FLOW CHART REFERENCES... 77

6

7 1 INTRODUCTION The USAID Indonesia Urban Water, Sanitation and Hygiene program (IUWASH), Contract No. AID-497-C , is a five-year program funded by the United States Agency for International Development (USAID). Under this award, DAI provides technical support to the Government of Indonesia (GOI) in the achievement of the nation s MDG target on water and sanitation targets in the water and sanitation sector. More specifically, the USAID IUWASH Project (the Project ) works with both national and local government agencies to expand access to water and sanitation services to Indonesia s urban poor, a programmatic priority of USAID/Indonesia. In addition to government agencies, DAI works with relevant stakeholders of the water and sanitation sectors, including the private sector, universities, non-governmental organizations (NGOs) and community groups. According to the Assistance Objective (AO) Improved Management of Natural Resources as set forth in USAID Indonesia s Country Strategy for , the implementation of IUWASH contributes to the Intermediate Result (IR) 3 Increased access to water and sanitation with an emphasis on reaching the urban poor. The Project s approach addresses the following three areas: 1. Increasing demand for safe drinking water access and improved sanitation; 2. Improving public and private service provider capacity to meet increased demand 3. Improving governance and finances for an enabling environment that supports equitable access to safe drinking water and improved sanitation in urban areas These three areas are expected to contribute to the comprehensive partnership between the US and Indonesian governments aimed at strengthening the health of urban populations through the improvement of urban water and sanitation services. This bilateral partnership advances cooperation in a wide range of issues to sustain the environment, promote prosperity, strengthen democracy, enhance understanding and ensure security. Local ownership is at the center of this partnership, and the inherent focus of IUWASH is to build the capacity of water sector service providers, local government owners and civil society organizations. 1.1 IUWASH PMP: OBJECTIVES In accordance with Section C.9.3 of the above referenced contract, this document provides a detailed description of the IUWASH Performance Monitoring Plan (PMP). The PMP is a performance management tool for planning, managing and documenting how performance data are collected and used and, more broadly, tracking progress over the course of implementation. Specifically, the IUWASH PMP serves to: Define specific performance indicators for each SO and IR, determine baselines and set targets; Plan and manage the Annual Report data collection process to meet quality standards; Incorporate relevant data collection requirements into activities and obligation agreements; Plan potential related evaluative work to supplement Annual Report indicator data; Estimate costs related to data collection and plan how these will be financed (See Annex 1 for more detailed); USAID INDONESIA URBAN WATER SANITATION AND HYGIENE 1

8 Support reliable data collection by documenting the frequency and schedule of data collection and assigning responsibilities. Communicate expectations to partner institutions responsible for producing the outputs intended to cause measurable changes in performance. IUWASH is committed to provide monitoring information to USAID/Indonesia and its partners in accordance with the requirements and guidelines outlined in USAID's ADS Chapters (as well as USAID OP Guidelines). IUWASH strives to ensure that the PMP adequately measures impact of work related to project financing from various earmarks. All indicators are presented in a uniform format. Each Performance Indicator Reference Sheet describes its indicator in detail and includes definitions, data sources and frequency of data collection and reporting. The March 2014 version of the IUWASH PMP represents the third iteration of this report. The first was submitted on 5 April 2011 after close collaboration with USAID to define the terms of the PMP. 1 The second was then executed on 15 May 2012, with the updates primarily consistency of refinements to the indicator definitions and data collection processes based upon one year of implementation. The present version provides for contractual changes enacted in late 2013, at which time USAID expanded the IUWASH scope of work and increased the Project s funding. 1.2 TYPES OF INDICATORS Section C of the IUWASH Contract sets forth the targeted outcomes for each of the three technical components. These outcomes then provide the basis for the definition of the Project s indicators. Notably, several USAID IUWASH indicators contribute to the standard indicators of foreign assistance of the US Government. These standard indicators will jointly measure both what is being accomplished with U.S. Government foreign assistance funds and the collective impact of foreign and host-government efforts to advance the country development. These standard indicators are divided into the following three categories, as follows: 1. Strategic level indicators. These indicators capture the impact of foreign and hostgovernment efforts at the objective level. The USAID IUWASH Project contributes to two Strategic Objectives: Investing in People and Governing Justly and Democratically. 2. Program area level indicators. These indicators measure country performance within subsectors of the functional objectives. Under the Investing in People Strategic Objective, for example, there are several sub-sectors such as Health, Education, and Social and Economic Services and Protection for Vulnerable Populations. The USAID IUWASH Project contributes mainly to the Health subsector under Investing in People; and Good Governance under the Governing Justly and Democratically. 3. Element level indicators. These indicators primarily measure outputs that are directly attributable to U.S. Governments programs, projects and activities. The USAID IUWASH Project contributes mainly to the Water Supply and Sanitation Services program element under Investing in People and the Local Government and Decentralization element under Governing Justly and Democratically. Figure 1 below describes the linkages between the Standard Indicators of the foreign assistance framework and USAID IUWASH indicators. 1 While this exceeded the 30 day requirement set forth in the IUWASH contract, written COR approval was obtained for this delay in order to allow sufficient time to complete the first year work plan. USAID INDONESIA URBAN WATER SANITATION AND HYGIENE 2

9 USAID Standard Indicators of Foreign Assistance Strategic Objective 3. Investing in People 2. Governing Justly and Democratically Contributed to Program Area 3.1. Health 2.2. Good Governance Contributed to IUWASH results and deliverables indicators Water Supply & Sanitation Element Local Government & Decentralization Contributed to Whenever possible, the definitions for IUWASH indicators were adapted from standard indicator definitions used by international organizations and USAID for water supply, sanitation and hygiene programs. The relevant documents from which the Project s definitions were derived are listed under the references. Where such commonly accepted standards do not exist, the proposed indicators closely follow the description of the Project s outcomes, tasks and sub-tasks. Importantly, when developing the PMP indicators and the precise definitions, IUWASH incorporated the characteristics of good performance indicators as described in the relevant USAID guidelines: Direct: An indicator should closely track the result it is intended to measure. When direct indicators cannot be used because of costs or other factors, a reasonable proxy indicator may be used. Objective: Objective indicators are operationally precise and uni-dimensional. They should be unambiguous about what is being measured and what data are being collected. Useful for Management: Indicators should be useful for management purposes at relevant levels of decision making Practical: An indicator is practical if data can be obtained in a timely way and at reasonable cost. Attributable to IUWASH: Performance indicators should measure change that is clearly and reasonably attributable, at least in part, to the efforts of IUWASH and therefore USAID. That is, indicators should credibly reflect the actual performance of the IUWASH Statement of Work. Timely: Performance data should be available when they are needed to make decisions. USAID INDONESIA URBAN WATER SANITATION AND HYGIENE 3

10 Adequate: Taken as a group, a performance indicator and its companion indicators should be the minimum necessary to ensure that progress toward the given results is sufficiently captured. Table 1 below summarizes the linkages of IUWASH indicators and USAID s Standard Indicators for Foreign Assistance. The information in this table will be used to track the Project s contribution to the achievement of development in Indonesia by USAID. The progress associated with each indicator will be reported on an annual basis. Table 1: IUWASH Indicators contributing to USAID Standard Indicators for Foreign Assistance. IUWASH Results and Indicators Deliverable High Level Results (HR): MDGs Outcomes Result HR-1: Increased Indicator HR-1: access to safe water Number of people in supply system urban areas gaining access to improved water supply as a result of US Government Result HR-2: Increased access to improved sanitation facilities assistance Indicator HR-2: Number of people in urban areas gaining access to improved sanitation facilities as a result of US Government assistance IUWASH Component 1: Demand Mobilization for Improved Water and Sanitation Services (MD) USAID Operational Plan Objective Program/ Indicators Area/Element IIP Water Supply and Sanitation IIP Water Supply and Sanitation Indicator : Number of people in target areas with firsttime access to improved drinking water supply as a result of USG assistance. Indicator : Number of people in target areas with firsttime access to improved sanitation facilities as a result of USG assistance. USAID s IR 1: Demand Mobilized for Water Supply and Demand Mobilized for Sanitation Outcome MD.1 Civil society groups and/or government cadres implementing programs to mobilize improved access to safe drinking water and adequate sanitation Indicator MD-1: Number of civil society groups and/or government cadres implementing programs to mobilize improved access to safe drinking water and adequate sanitation GJD Local Government and Decentralization Indicator : Number of local Non- Governmental and public sector associations supported with USG assistance. USAID INDONESIA URBAN WATER SANITATION AND HYGIENE 4

11 IUWASH Component 2: Capacity Improvements to provide sustainable safe water and Sanitation Service (IC) USAID s IR 2: Improved Public/Private Capacity to Provide Sustainable Safe Water Services and Expanded Access to Santation Outcome IC.1 PDAMs with improved technical, financial and management performance Outcome IC.4. Local government institutions implementing necessary climate change adaptation measures, based on preliminary raw water sources vulnerability assessment Indicator IC-1: Number of PDAMs with improved technical, financial and management performance Indicator IC-4: Number of local government institutions implementing necessary climate change adaptation measures, based on preliminary raw water sources vulnerability assessment GJD Local Government and Decentralization IIP Water Supply and Sanitation EG Clean Productive Environment Indicator : Number of sub-national government entities receiving USG assistance to improve their performance. Indicator : Number of stakeholders implementing riskreducing practices/ actions to improve resilience to climate change as a result of USG assistance Indicator : Number of stakeholders with increased capacity to adapt to the impacts of climate variability and change as a result of USG assistance IUWASH Component 3: Create an Enabling Environment Supporting Equitable Water and Sanitation Services (EE) USAID s IR 3: Strengthen the Governance and Financing Enabling Environment for More Equitable Expansion of Water and Sanitation Outcome EE.1. Participating local governments that put greater priority on safe drinking water and sanitation through supportive local policies and budget allocation increases Indicator EE-1: Number of participating local governments that put greater priority on safe drinking water and sanitation through supportive local policies and budget allocation increases GJD Local Government and Decentralization Indicator : Number of local mechanisms supported with USG assistance for citizens to engage their sub-national government. Outcome EE.5. Local Governments adopt new or improved mechanisms for citizens to engage local government in water and sanitation Indicator EE-5: Number of Local Governments adopting new or improved mechanisms for citizens to engage local government in water and sanitation GJD Local Government and Decentralization Indicator : Number of local mechanisms supported with USG assistance for citizens to engage their sub-national government. 1.3 PROGRAMMATIC ATTRIBUTION Finally, despite the development of a rigorous performance plan and extensive data collection efforts, it is important to note that direct attribution of the Project s impact is an inherently complex issue. First and foremost, numerous organizations including government, NGOs and donors are active in the water, sanitation, and hygiene (WASH) sector throughout the country, USAID INDONESIA URBAN WATER SANITATION AND HYGIENE 5

12 with many working in the same districts, municipalities, communities as those in which IUWASH will be active. It is often impossible to disaggregate the impact of other actors with those of the Project itself. Indeed, a hallmark of the Project s approach is to build upon and leverage the resources of other actors to maximize the impact of the USG s investment in the sector. Such partnerships may take multiple forms, with the most common being that IUWASH provides soft side capacity building to complement an infrastructure investment by a government or another donor. In addition, when conducting capacity-building activities, it can be difficult to discern and quantify how the performance of a local partner (such as a PDAM) would have differed in the absence of the Project s interventions. Nonetheless, the indicators described herein were developed to relate as much as possible to specific field activities. Where indicators are process-based, we have sought to delineate and track the processes that correspond to IUWASH assistance activities. USAID INDONESIA URBAN WATER SANITATION AND HYGIENE 6

13 2 IUWASH PMP INDICATORS 2.1 HIGH-LEVEL RESULT (HR): MDG OUTCOMES RESULT RESULT HR-1: People gaining access to safe water supply system as result of US Government assistance RESULT HR-2: People gaining access to improved sanitation facilities as a result of US Government assistance RESULT HR-3: Per unit water cost paid by poor in targeted communities decreases by at least 20% through more participatory, transparent and financially enabled services RESULT HR-4: Participant Training report INDICATOR Number of people in urban areas that gain access to improved water supply as a result of US Government assistance Number of people in urban areas that gain access to improved sanitation facilities as a result of US Government assistance Percent of targeted communities that experience a decrease in unit water costs by at least 20% as a result of more participatory, transparent, accountable and financially enabled services Number of people participated in IUWASH training type of activities UNIT OF MEASUREMENT Number and percent change Number and percent change BASELINE TARGET 0 2,000, ,000 Percent change 0 20 Number 0 75,000 USAID INDONESIA URBAN WATER SANITATION AND HYGIENE 7

14 2.2 COMPONENT I (MD): DEMAND MOBILIZATION FOR IMPROVED WATER AND SANITATION SERVICES DELIVERABLE OUTCOME MD-1: Households willing to pay for sanitation improvements OUTCOME MD-2: Civil society groups and/or government cadres implementing programs to mobilize improved access to safe drinking water and adequate sanitation OUTCOME MD-3: Civil society groups that report on PDAM operations or performance OUTCOME MD-4: Households that have adopted improved health and hygiene practices OUTCOME MD-5: Exposure for Master Trainers to new urban promotion strategy to support participatory planning activities including triggering exercises and Behavior Change INDICATOR Number of households willing to pay for sanitation improvements Number of civil society groups and/or government cadres implementing programs to mobilize improved access to safe drinking water and adequate sanitation Number of civil society groups that report on PDAM operations or performance Percent of households that have adopted improved health and hygiene practices Number of master trainers exposed to new urban promotion strategy to support participatory planning activities, such as triggering exercises, and behavior change programming developed UNIT OF MEASUREMENT BASELINE TARGET Number 0 40,000 Number Number 0 20 Percent change 0 20 Number USAID INDONESIA URBAN WATER SANITATION AND HYGIENE 8

15 2.3 COMPONENT II (IC): CAPACITY IMPROVEMENTS TO PROVIDE SUSTAINABLE SAFE WATER AND SANITATION SERVICE DELIVERABLE OUTCOME IC-1: PDAMs with improved technical, financial and management performance OUTCOME IC-2: PDAMs in default of old debts are assisted in restructuring their outstanding debts OUTCOME IC-3: PDAMs with improved creditworthiness OUTCOME IC-4: Local government implementing necessary climate change adaptation measures, based on preliminary raw water sources vulnerability assessment OUTCOME IC-5: Local governments implementing integrated sanitation and hygiene interventions that reflect their citywide sanitation strategic plans OUTCOME IC-6: Small and medium businesses providing affordable construction and sanitation facility management services OUTCOME IC-7: Poor residents in targeted communities that report greater satisfaction with water and sanitation services OUTCOME IC-8: Establishment of Sanitation Management INDICATOR Number of PDAMs with improved technical, financial and management performance Number of PDAMs in default of old debts that are assisted in restructuring their outstanding debts Number of PDAMs with improved creditworthiness Number of local government implementing necessary climate change adaptation measures, based on preliminary raw water sources vulnerability assessment Number of local governments implementing integrated sanitation and hygiene interventions that reflect their citywide sanitation strategic plans Number of small and medium businesses providing affordable construction and sanitation facility management services Percentage (%) of poor residents in targeted communities that report greater satisfaction with water and sanitation services Number of Sanitation Management Units established with supporting policies, UNIT OF MEASUREMENT BASELINE TARGET Number 0 50 Number 0 20 Number 0 20 Number 0 20 Number 0 20 Number 0 30 Percent change 0 20 Number 0 10 USAID INDONESIA URBAN WATER SANITATION AND HYGIENE 9

16 DELIVERABLE Unit OUTCOME IC-9: Adoption of Sustainable Urban Sanitation Framework by Government Institutions OUTCOME IC-10: Increased Capacity of Stakeholders to Adapt to the Impacts of Climate Variability on Water Supply infrastructure INDICATOR budgets and personnel Number of Sustainable Urban Sanitation Framework adopted by the government institutions as a key part of national sanitation programming policy Number of stakeholders from central government agencies, universities and NGOs, as well as those from local government institutions among current IUWASH sites, that will have increased capacity to adapt to the impacts of climate variability and change as a result of USG assistance UNIT OF MEASUREMENT BASELINE TARGET Number 0 1 Number USAID INDONESIA URBAN WATER SANITATION AND HYGIENE 10

17 2.4 COMPONENT III (EE): CREATE AN ENABLING ENVIRONMENT SUPPORTING EQUITABLE WATER AND SANITATION SERVICES DELIVERABLE OUTCOME EE-1: Participating local governments that put greater priority on safe drinking water and sanitation through supportive local policies and budget allocation increases INDICATOR Number of participating local governments that put greater priority on safe drinking water and sanitation through supportive local policies and budget allocation increases UNIT OF MEASUREMENT BASELINE TARGET Number 0 50 OUTCOME EE-2: PDAMs or local government obtain access to long-term funding for water or sanitation investment plans OUTCOME EE-3: Percent increase (%) in financial resources accessed by service providers from public and private sources for expansion of improved water and sanitation services OUTCOME EE-4: Low-income households accessing micro-finance for household improvements in water and sanitation OUTCOME EE-5: Local Governments adopt new or improved mechanisms for citizens to engage local government in water and sanitation OUTCOME EE-6: Development of New or Improved Regulations to Facilitate Access to Capital Financing in Water Sector Number of PDAMs or local governments that obtain access to long-term funding for water or sanitation capital expenditure investment plans Percentage change (%) in financial resources accessed by service providers from public and private sources for expansion of improved water and sanitation services Number of lowincome households accessing microfinance for household improvements in water and sanitation Number of Local Governments that adopt new or improved mechanisms for citizens to engage local government in water and sanitation Number of new or improved regulations to facilitate access to capital financing in the water sector Number 0 15 Percent change 0 10 Number 0 40,000 Number 0 20 Number 0 1 USAID INDONESIA URBAN WATER SANITATION AND HYGIENE 11

18 DELIVERABLE OUTCOME EE-7: Central Government Officials Exposed to Alternative Financing Options INDICATOR Number of central government officials exposed to alternative financing options in the water sector in the US or other applicable context UNIT OF MEASUREMENT BASELINE TARGET Number 0 12 USAID INDONESIA URBAN WATER SANITATION AND HYGIENE 12

19 3 PERFORMANCE MONITORING SYSTEM A Performance Monitoring System is necessary to collect and present data and results for program indicators in a timely and reliable manner. The IUWASH Performance Monitoring System consists of five activities: 1. Quantitative Performance Monitoring 2. Qualitative and Impact of Performance Monitoring 3. Quality assurance 4. Reporting system of IUWASH outcome indicators 5. Partners Collaboration in IUWASH Monitoring Activity Each of these components is described in greater detail below. 3.1 QUANTITATIVE PERFORMANCE MONITORING IUWASH Quantitative Performance Monitoring consists of tracking and reporting numeric results against the indicators defined in Section 2.4. The best quantitative monitoring systems combine appropriate indicators, cost-effective data collection systems, rigorous analysis and efficient reporting procedures to provide a representative picture of program performance and specific achievements. Toward this end, each quantitative indicator will be described in detail using a Performance Indicator Reference Sheet (PIRS) that contains the following information: a. Indicator Description: Each performance indicator will have a precise definition, which will include interpretation of specific terms that may be open to interpretation. The description will also include the applicable unit of measure as well as the extent of data aggregation, if any; b. Plan for Data Acquisition: This section describes the complete methodology for the collection of the requisite data, including data sources (e.g. government database, government report, structure interview, focus group discussion, household survey, etc.), steps to collect the data, the individual(s) responsible for the data collection, the frequency and timing of data acquisition (i.e. monthly, quarterly, annually, etc.), and the storage location of hard data; c. Data Quality Issues: Data quality is integral to an effective monitoring plan, and thus it is important to identify potential data weaknesses and identify mitigation strategies; d. Plan for Data Analysis, Review and Reporting: This section discusses the necessary steps to turn raw data into finalized PMP values, including the types of analysis necessary (e.g. statistical analysis and the assumptions utilized), the review and verification process, and, finally, how and when the data will be reported to USAID. According to the IUWASH Statement of Work, PMP reporting will be carried out principally through the Quarterly and Annually Progress Report; e. Baseline/Target Notes: The baseline provides the point of reference for each indicator, and, where applicable, this section describes the methodology for determining the baseline value; and f. Performance Indicator Values. The final section presents the cumulative achievement targets for the indicator for each project year. USAID INDONESIA URBAN WATER SANITATION AND HYGIENE 13

20 3.2 QUALITATIVE PERFORMANCE MONITORING To provide a more complete picture of the Project s impact, IUWASH will utilize qualitative performance monitoring techniques to supplement the hard numbers presented for each respective indicator. Qualitative impact assessment will focus on IUWASH s broader results defined by USAID and outlined in the Results Framework and support the causal relationship between IUWASH activities and the impact achieved with the intended beneficiaries. More broadly, qualitative performance monitoring seeks to tell the IUWASH story in a manner that is accessible and tangible. Qualitative performance monitoring will take the following forms: Success Stories. IUWASH success stories will be presented in the IUWASH Newsletter, website or as cover stories of progress reports. These stories will highlight the significant impact of IUWASH interventions through the eyes of the beneficiaries. The success stories will focus on human-interest stories that show immediate impact of IUWASH activities in people s lives. The success stories will also be a part of IUWASH regular reporting such as Quarterly and Annual Progress Report. Before and After Story. IUWASH Before and After Story will emphasize the voice and perspective of IUWASH beneficiaries in different aspects of the program. This tool will compare two pictures that show conditions before and after IUWASH interventions. These pictures will be accompanied by photo captions to describe the changes made. The stories will be collected from several IUWASH target locations. 3.3 QUALITY ASSURANCE The progress presented in IUWASH quarterly and annual performance reports is only as reliable as the raw data that underlies these results. Thus, it is important that the Project proactively address the quality of this raw data. While there is no way to guarantee the accuracy of all indicator data for a project as large and complex as IUWASH particularly when the project itself does very little primary data collection there are, nonetheless, steps that can be taken to minimize collection errors and improve the integrity of the data. Specific quality assurance strategies employed by IUWASH are as follows: PMP Database. IUWASH houses PMP datasets and indicator results within DAI s proprietary Technical and Administrative Management Information System (TAMIS), allowing the team to track progress in a transparent and accessible manner. The TAMIS is a cost-effective online management information system to collect data for project reporting and communications. TAMIS allows for real-time exchange of a wide range of information for collaboration, project planning, management reporting, and quality improvement. Importantly, it allows users in dispersed geographic locations to share the same set of data, bolstering efficiency and transparency. Once data is loaded into the TAMIS, standardized summary views are set up to allow Project staff to easily monitor progress. Where possible, soft copies of the data sources can be directly stored under each applicable indicator. Monitoring and Evaluation Advisor and Assistant. The IUWASH Project employs a Monitoring and Evaluation (M& E) Advisor as well as an M&E Assistant to oversee the collection, review and analysis of indicator data, The Advisor and Assistant collaborate with the IUWASH regional teams to conduct regular training for data collection, quality assurance and validation, including working with TAMIS on the activities above. Further, USAID INDONESIA URBAN WATER SANITATION AND HYGIENE 14

21 the Advisor works closely with the technical and regional staff to review the results of each quarterly and annual analysis prior to disseminating the results in IUWASH reports. Standardized PMP Intake Forms. IUWASH has developed standardized PMP intake forms to record results and transfer to TAMIS. The usage of standardized forms and data acquisition processes helps to minimize entry errors. Regular Field Visits and Report Reviews. The M&E Advisor and other senior management staff including COP, DCOP, Advisors, National and Regional Coordinators regularly travel to the field to monitor Project activities, assessing on the ground the accuracy of PMP quantitative and qualitative information where appropriate. Field visits are conducted to observe and document general activities per component as well as per integrated activity. Data Quality Assessment. USAID regulations require that a mission representative conducts a data quality assessment once every three years. IUWASH plans to assist the mission in conducting a data quality assessment (DQA) approximately mid-way through Project implementation. The DQA activity at least will be implemented once during the course of the IUWASH Program. For more detail see the reference sheet on the section Time of DQA. 3.4 REPORTING SYSTEM OF IUWASH OUTCOME INDICATORS The reporting system of IUWASH outcome indicators is part of the IUWASH regular reporting system. The progress achievement of each IUWASH indicator will be presented as part of the IUWASH components progress report. This data presentation will show the progress made by each IUWASH outcome indicator in different timeframes. This approach will be a traditional and common approach for reporting of IUWASH outcome indicators. 3.5 PARTNER COLLABORATION IN IUWASH MONITORING ACTIVITY The Monitoring and Evaluation Advisor will work with all National and Regional teams to increase the participation of IUWASH partners in monitoring, especially the local government partners. IUWASH will use, for example, the PDAM Performance Index and the Creditworthiness Ladder as entry points to involve other relevant government partners to increase their capacity on measuring the performance of PDAM. IUWASH plans to provide capacity building for local government institutions in tying adaptive management techniques to M&E results. USAID INDONESIA URBAN WATER SANITATION AND HYGIENE 15

22 4 PERFORMANCE INDICATOR REFERENCE SHEETS To support a standardized, transparent approach to data collection and analysis for performance monitoring under IUWASH, Section 4 of the PMP consists of a performance indicator reference sheet (PIRS) for each of the Project s 21 indicators.. As noted previously, the PIRS provides general information and implementation guidelines for the IUWASH team for each indicator, including the name of the indicator, linkage with the USAID standard indicators, precise definition, unit of measurement and disaggregation, plan for data acquisition, potential data quality issues, plan for data analysis, review and reporting, and the baseline and target numbers. Each PIRS that contributes to a USAID Standard Indicator includes the applicable code in the top-right corner of the sheet. The background of the PIRS is also in light gray. The program year stated in the reference sheets below represents the financial year of USAID, which runs from October 1 through September 30. The IUWASH Project started in March 2011, which was the middle of the 2011 USAID financial year. To synchronize the IUWASH project period and the USAID financial year, each reference sheet has 6 columns such that the first and last years of the IUWASH program consist of 6 months. USAID INDONESIA URBAN WATER SANITATION AND HYGIENE 16

23 IIP IIP Performance Indicator Reference Sheet Result HR-1: Increased Access to Safe Water Supply System Assistance Objective: Improved Management of Natural Resources Intermediate Result: IR3 Increased Access to Safe Drinking Water and Adequate Sanitation Deliverable: People gaining access to improved water supply and sanitation services as a result of US Government assistance Indicator: Number of people in urban area gaining access to improved water supply as a result of US Government assistance DESCRIPTION Precise Definition(s): People considered under this outcome are both poor and non-poor living in urban and peri-urban areas who will receive piped water access from a PDAM. Every PDAM has an obligation to service all residents within their service area, including those recognized as poor by the local government (i.e. fulfill the definition of Masyarakat Berpenghasilan Rendah and therefore generally qualify for a subsidized tariff.) Improved water supply is defined as clean water that comes from an improved water source through piping system. The improved water source under this outcome will focus on PDAM connections direct to individual households, master meter connections (i.e. urban communal systems), and public water facilities. There are two ways to count the number of people gaining access to an improved water supply service. These are as follows: 1. Number of new connections for all PDAMs within IUWASH clusters, provided that those PDAMs demonstrate a significant improvement in service quality as indicated by at least a 20 percent increase over baseline in respective Performance Index scores. Please see Indicator IC-1, which explains the functionality of the Performance Index and the types of technical assistance activities implemented by the Project. PDAM connections under this criterion include the house connections to the Master Meter systems. 2. Number of committed additional connections which are made through IUWASH support for capital financing programs. The counting of these connections will be done at the end of period of IUWASH implementation, in The committed additional connections will only be counted when the capital expenditure financing is secured but the actual construction of these connections will not be completed by the end of IUWASH. IUWASH will count these connections as additional achievements of the HR.1 outcome, by using the standard formula used by the Ministry of Public Works: each 1 liter/second committed additional water supply will benefit 75 households (or 375 people). Raw data for this outcome is generally provided in terms of number of households. To determine the number of people, IUWASH uses the assumption used by the Government of Indonesia: 1) Five people per connection for domestic and commercial connections, and 2) 20 people per connection for social connections. IUWASH will use the total number of new connections to determine the increase in percentage of households receiving improved access to safe water supply. To calculate this percentage, IUWASH will divide the increased number of PDAM connections receiving improved access to safe water as the numerator and the total number of households in the PDAM s coverage area based on National Socioeconomic Survey (Survei Sosial Ekonomi Nasional/SUSENAS) as the denominator. The total number of households in PDAM s coverage area is different for city (kota) and district (kabupaten) settings. IUWASH follows the assumption used by the GOI that says the total population of a city (Kota) is considered as an urban population. While for districts (Kabupaten) the urban population is 60% of the total population. In another words, the service area of the Kabupaten is confined to 60% of the population according to national government standards. The remaining 40% is generally assumed to be directly from local springs, rivers, or wells. All the systems developed with IUWASH support under this outcome should follow the IUWASH environmental compliance as approved by USAID (IEE and EMMP). The development of water supply facilities will include the facilities developed by the IUWASH partners/stakeholders through grant or public private partnership schemes. Unit of Measure: Number of people and percent change of household Disaggregated by: Gender (men and women); geographic region. USAID INDONESIA URBAN WATER SANITATION AND HYGIENE 17

24 PLAN FOR DATA ACQUISITION Data Source(s): PDAM Semi-annual and Annual reports. The PDAM Annual report is an audited report (Laporan Audit Kinerja). IUWASH couldn t specify a chapter or subchapter of the report as the data reference because it often varies from one utility to another Staff Responsible for Collecting Data: IUWASH Regional Urban Water Supply Specialists (UWS) Data Collection Method: Data of new PDAM connections compiled from regions, verified and upload to IUWASH TAMIS by National Coordinator of Water Supply Program. Frequency/Timing of Data Acquisition: Semi-annual and annual data report of TAMIS entries Storage of Data: Soft copies of connection data are attached in the TAMIS under PMP Module/HR Indicators/HR-1 as well as on the IUWASH server at //Jakarta/Common drive/pmp. Hard copies are maintained in the locked PMP file drawer in the office of the Monitoring and Evaluation Advisor. DATA QUALITY ISSUES Known Data Limitations and Significance (if any): For PDAM connections, IUWASH relies on datasets furnished by the PDAM itself. IUWASH does not independently determine the accuracy of these reports, an action which is done instead by other GOI agency (BPK/National Audit Agency). The audit done by this agency is accountable enough so the data obtained for this report are generally considered to be accurate and reliable. Data availability for reporting sometimes depends on the readiness of the data by IUWASH partners, which can be delayed by not collecting data in time. In terms of timing, IUWASH will sometimes utilize unaudited data in its reports when the audited data is not yet available. We always specify this, however, and then update any changes after the final audited data is issued. Date of Most Recent Data Assessment: 14 February 2012 PLAN FOR DATA ANALYSIS, REVIEW, & REPORTING Data Analysis: No specific analysis applied for the reporting of this outcome. Presentation of Data: Counting number of new PDAM connections in IUWASH sites and number of households gaining new access from non-pdam system in IUWASH sites. The PDAM connections divided by three categories such as domestic, commercial and social. The number of domestic and commercial connections are transferred into number of people by using assumption of 1 connection is multiplied by 5 people and the social connection is transferred into number of people using assumption of 1 connection is multiplied by 20 people. This assumption is based on previous surveys done by PDAM. IUWASH assesses the existing connection at the time the report is made. The existing numbers are then compared with the baseline number to get the number of new connections. Over the time, IUWASH will divide the achievement per year based on this calculation. The counting based on gender disagregation is based on the National Statistical Data on the comparison of the male and female population which is 3 compare to 2, respectively. The results of this counting will be presented in tabular data on the number PDAM connections and households per city with a final number of people gaining new access to improved water supply facility. This Table will be provided by narrative as needed to highlight important issues on the data presented. Review of Data: M&E Advisor prepares the Table as stated above to show the progress of this outcome and submits it to DCOP/WATSAN Technical Advisor for final review before reporting Reporting of Data: IUWASH Semi Annual Report (submitted at the end of March to USAID) and Annual Report (submitted at the end of September to USAID) BASELINE AND TARGETS Notes on Baselines/Targets: The baseline figure for IUWASH is 0 as since this indicator only counts new connections made by PDAMs from from two pre-determined start dates (for Phase one December 2011 and for Phase 2 December 2012).For each participating PDAM the actual number of connnections at that date were used as baseline for each PDAM. IUWASH will count new connections of PDAM partners under HR-1 after they achieve an increase of a minimum 20% of their PDAM Performance Index (described under Outcome IC-1) The achievement under this outcome begins by year 2 and increases cumulative in subsequent years. CHANGES TO PERFORMANCE INDICATOR Changes to Indicator: The main change under this outcome includes consideration of the poor under this outcome. IUWASH plans to include an analysis of IUWASH poor beneficiaries, using PDAM data, that get access to the improved water supply. USAID INDONESIA URBAN WATER SANITATION AND HYGIENE 18

25 PERFORMANCE INDICATOR OUTPUT TARGET People gain access to improved water supply BASELINE 0 RESULT DATA Target (people) Target (% household) Estimated Baseline (hh) Actual PROGRAM YEAR FY11 FY12 FY13 FY14 FY15 FY16 TOTAL ,000 4% 1,3 million 500,000 4% 2,5 million 1,100,000 9% 2,5 million 1,700,000 14% 2,5 million 2,000,000 16% 2,5 million 2,000,000 16% THIS SHEET LAST UPDATED ON: 5/26/14 USAID INDONESIA URBAN WATER SANITATION AND HYGIENE 19

26 Performance Indicator Reference Sheet Result HR-2: Increased Access to Improved Sanitation Facilities Assistance Objective: Improved Management of Natural Resources Intermediate Result: IR3 Increased Access to Safe Drinking Water and Adequate Sanitation Deliverable: People gaining access to improved water supply and sanitation services as a result of US Government assistance Indicator: Number of people in urban area gaining access to improved sanitation facilities as a result of US Government assistance DESCRIPTION Precise Definition(s): People considered under this outcome are those who live in urban- and peri-urban areas who gain access and benefit from improved sanitation facilities for the first time. Urban and peri-urban areas are defined as areas that are at populated by at least 150 people/hectare. This definition only includes populated areas and not open land or agricultural land. The Improved sanitation facilities are defined as sanitation facilities that use proper technology to ensure both privacy and hygienic conditions. The types of qualifying sanitation facilities counted by IUWASH under this definition are as follows:: 1. Improved individual (household) latrines 2. Community-based sanitation facilities (household connections to communal septic tank) 3. Public toilet/shared sanitation facilities (community) 4. Piped connection to the centralized sewer system (city wide and small) IUWASH will use the total number of people gaining access to improved sanitation services to determine also the percentage increase of households obtaining access to improved sanitation services. To calculate this, IUWASH will compare the achievement made (numerator) with the total existing households in the specific IUWASH sites (denominator), using the following definition for the numerator and denominator under this outcome: Numerator: number of households with access to or benefiting from improved sanitation services with support from IUWASH Project as explained above. Denominator: For the types of improved systems #1 to #3, the denominator is the total number of households in the specific sites (RT or combined several RTs) level. For the city sewerage system (option #4), the denominator is total potential customers situated along the piped sewerage system which could be directly connected to the sewerage network. The data collected by IUWASH in form of number of household. To determine number of people, IUWASH uses the assumption of 5 individuals per household as the standard used by the Government of Indonesia. This outcome has a connection with MD-1 Outcome (willingness to pay) because the number of people of the households counted under this outcome should have the commitment to pay for these improved sanitation facilities as well. So, the access to improved sanitation facilities will be counted under this outcome and their real commitment in the form of financing either investment in or operation of the improved sanitation facilities will be counted under MD-1 Outcome. All the systems developed with IUWASH supported under this outcome should follow the IUWASH environmental compliance as approved by USAID (IEE and EMMP). The development of sanitation facilities will include the facilities developed by the IUWASH partners/stakeholders through grant or public/private partnership schemes. Also, during in the IUWASH period, there may be alternative financing schemes supporting the development of improved sanitation facilities but the improved facilities may be realized in the period after the IUWASH period of performance. IUWASH will include the planned (but not yet realized) beneficiaries in the final calculation in the last year of IUWASH (2016) as long as the financial commitment has been made by local government or other donors, Unit of Measure: Number of people and percent change of household Disaggregated by: Gender (men and women); geographic region. IIP IIP USAID INDONESIA URBAN WATER SANITATION AND HYGIENE 20

27 PLAN FOR DATA ACQUISITION Data Source(s): The sources of data depend on the sanitation facilities constructed. 1) For the individual system, the source is the sanitarian report to Puskesmas (Health Post) or District Health Office or other key IUWASH partner on household construct improved individual toilet; 2) For communal sanitation system, the source is report of LG institutions and IUWASH grantee manage communal sanitation programs; 3) For sewerage system, the source is report of LG institutions managing the city or small sewerage system. In the case that key IUWASH partners don t have the intake form, IUWASH will provide its own intake form to be filled in by key partners. Staff Responsible for Data Collection: Depending on the system constructed such as for individual system, data is collected by the IUWASH BCC Specialist in each region; for communal system the data is collected by either the BCC Specialist position and/or the Urban Sanitation Specialist (USS) position in each region; for sewerage system the data is collected by the USS position in each region. Data Collection Method: Data of households getting access to improved sanitation facilities collected by staff responsible and submitted to M&E Assistant. After verification by the M&E Assistant, she will upload to TAMIS and the M&E Advisor will conduct final verification. Frequency/Timing of Data Acquisition: Semi-annual and annual data report of TAMIS entries Storage of Data: Soft copies of data on household getting increased access to improved sanitation system are attached in the TAMIS under PMP Module/HR Indicators/HR-2 as well as on the IUWASH server at //Jakarta/Common drive/pmp. Hard copies are maintained in the locked PMP file drawer in the office of the Monitoring and Evaluation Advisor. DATA QUALITY ISSUES Known Data Limitations and Significance (if any): The data source for reporting this indicator will depend upon the process of the development of sanitation facilities. In this regard, the counting of this deliverable will be conducted after the facilities are completely developed. After a certain period, if there are additional beneficiaries of sanitation facilities, it will be added later. This usually happens for communal sanitation system. IUWASH will count and initially report on the number of households that connect immediately following construction. Often, the number of households that connect immediately is less than the system s full capacity. Thus, if additional households come online in the months following completion, the communitybased organization (CBO) will inform IWUASH as such, and IUWASH will conduct a site visit to verify additional household connections before report the additional data. Date of Most Recent Quality Assessment: 14 February 2012 PLAN FOR DATA ANALYSIS, REVIEW, & REPORTING Data Analysis: No specific analysis applied for the reporting of this outcome. Presentation of Data: Simple counting on number of households getting access to improved sanitation facilities. The number is disaggregated by different type of systems developed. The number of households is then converted into an estimate of the number of people (1 household will be multiplied by 5 people as the standard of the average number of people per household based on the National Statistical Data). The counting based on gender disaggregation is based on the National Statistical Data on the comparison of male and female population which is 3 compared to 2, respectively. The results of this counting will be presented in tabular data on number of households per city with a final number of people getting new access to improved sanitation facility. This Table will be provided by narrative as needed to highlight important issues on the data presented. Review of Data: M&E Advisor to prepare the table as stated above to show the progress of this outcome and submit to DCOP/WATSAN Technical Advisor and Senior Sanitation Advisor for final review before reporting Reporting of Data: IUWASH Semi Annual (submitted in end of March to USAID) and Annual report (submitted at end of September to USAID) BASELINE AND TARGETS Notes on Baselines/Targets: The baseline number is 0 since this indicator is counted only for additional people benefiting from new improved sanitation facilities developed by IUWASH. The beginning phase of the program is used as preliminary and preparation stages. The target number begins by year 2 and increases in subsequent years. IUWASH has no contractual target for the percent household increased access to safe water supply; however, IUWASH will track the percent increase of access to safe water supply based on actual achievements made in every reporting period. CHANGES TO PERFORMANCE INDICATOR Changes to Indicator: The main change under this outcome is only in increasing the target after the IUWASH contract modification #8. USAID INDONESIA URBAN WATER SANITATION AND HYGIENE 21