Going Digital: Use of Technology in Monitoring and Evaluation

Transcription

1 Going Digital: Use of Technology in Monitoring and Evaluation Urmy Shukla & Sree Sen CLEAR South Asia at J-PAL South Asia at IFMR Second Development Marketplace 2013 Convening Workshop April 24 th, 2014

2 Presentation Overview Introduction M&E Data Challenges and Scope for Using Technology Using Digital (Mobile) Tools for M&E When to Use Digital Data Collection Types of Mobile Data collection Examples

3 INTRODUCTION

4 Abdul Latif Jameel Poverty Action Lab (J-PAL) Research affiliate network of over 92 professors in 34 universities; 491 evaluations in 56 countries across 7 global offices

5 J-PAL South Asia: A Brief History J-PAL South Asia office set up in 2007 at the Institute for Financial Management and Research, Chennai Over 90 completed and ongoing projects across South Asia: education, finance & microfinance, environment & energy, health, political economy & governance, labor markets and agriculture Scientific Directors: Esther Duflo (MIT), Iqbal Dhaliwal (Ex-IAS 1996 batch)

6 CLEAR Initiative CLEAR: Regional Centers for Learning on Evaluation and Results Global initiative coordinated by the World Bank Independent Evaluation Group The CLEAR South Asia Regional Centre has been hosted by J-PAL South Asia at IFMR since May 2011 Established a partnership with the Centre for Economic Research (CERP) in Pakistan and works closely with Innovations for Poverty Action (IPA) in Bangladesh CLEAR s Strategic Vision: Building evaluation skills of practitioners, program implementers, and donors Strengthening M&E systems and practices for strategic clients Building a culture of evidence based decision-making and strengthening demand for evaluations

7 CLEAR South Asia Mandate CLEAR South Asia strategy includes a focus on the use of cutting-edge technology for effective collection and utilization of M&E In collaboration with Fieldata.Org, CLEAR South Asia aims to promote the use of M&E technology, such as mobile phones and PDAs, for more real-time aggregation and effective utilization of M&E.

8 Context E-society The fundamental objective of all e-society initiatives is to make Information, Communication & Technology (ICT) more inclusive i.e. ALL members of society, especially those that are socially and economically marginalized, should gain access to and benefit from the knowledge, power and opportunities brought about by new ICTs Emphasis on the use of technology for more effective collection and utilization of M&E

9 About Fieldata.org Fieldata.Org is a mobile-&-web portal for NGOs to raise funds, by offering real-time monitoring, and objective data for evaluation. Mission is to improve transparency & decision-making in development organizations and government agencies, by empowering them with technology-tools for better monitoring, sharing, and application of data. Want organizations and donors to objectively answer: Which projects should resources be allocated? How well do projects utilize these resources?

10 About Fieldata.org

11 Challenges and Scope for Using Technology M&E DATA

12 Programme Theory Log Frame Inputs/Program Activities Outputs Intermediate outcomes Goal What we do as a part of the program - deliver, teach, offer loans, etc. What are the resources used funds, staff, equipment, curriculum, all materials. Tangible products or services produced as a result of the activities - usually can be counted. Reporting: On expenditures, activities, coverage (targets) Monitoring : Of use of inputs/activities and process Random spot checks of intermediate outcomes: Annual surveys Short-term behavioral changes that result from the outputs - preventive health habits, usage of tablets. Long-term changes that result from outcomes the result of the program. Evaluation: To assess long-term outcomes and impact through studies

13 M&E Framework Use of Data Reporting On Inputs and Outputs (Achievement of Targets) Monitoring Of Processes and Implementation (Doing things right) Evaluation Of Outcomes and Impact (Doing the right thing) Management and Decision Making (MIS) Using relevant and timely information for decision making (reporting and monitoring for mid term correction; evaluation for planning and scale up) ALL OF THE ABOVE DEPEND ON THE AVAILABILITY OF RELIABLE, ACCURATE AND TIMELY DATA

14 Problems in Data Collection and Management What do the following mean? Data reliability (will we get the same data, when collected again?) Data validity (Are we measuring what we say we are measuring?) Data integrity (Is the data free of manipulation?) Data accuracy/precision (Is the data measuring the indicator accurately?) Data timeliness (Are you getting the data in time?) Data security/confidentiality (Loss of data / loss of privacy) Can technology help us address the above issues? How?

15 Why use technology to collect data? USING DIGITAL (MOBILE) TOOLS FOR M&E

16 Digital Tools for Data Collection

17 Digital Data Collection Device Data Software Data Transmission Data Visualization Mobile Devices Open Data Kit, Other softwares Real time Dashboards, Reports, summary Tables, Graphs Laptops, Computers Visual Basic, Java, SQL, etc. Online / Offline Excel/.csv files, customized MIS reports and dashboards GPS Arch GIS Real time Maps, Coordinates Smart Cards and hand-held devices Software Online / Offline Summary tables / customized reports Biometric devices (online) Software Online / Offline Summary tables / customized reports

18 Why use Mobile Technology in Evaluations Improving transparency & accountability in development organizations and government agencies, though technology-enabled M&E for better monitoring, sharing, and application of data. Enabling organizations, donors and citizens to use M&E data for real-time decision-making, better implementation and delivery of projects and services

19 Mobile Technology Options

20 How is Mobile Technology Used

21 Why Mobile Data Collection? Real-time data from the point of collection Built-in logical flow and validation checks improves data quality Ability to collect new types of data Location (GIS), media (pictures, audio) Cost effective over time- involves one-time hardware costs and ongoing maintenance. No paper, printing costs Easy to manage and analyze large amounts of data Reduces intermediate levels of data transmission

22 Why Digital Technology for Routine Monitoring Paper Reports Using Mobile Phones Delay in reporting Multiple levels of reporting Information flow is one way and not actionable Bulky hard copies of reports Errors in entry, needs additional scrutiny Almost instantaneous reporting Direct reporting Information flow is both ways and actionable Web-enabled reports Field-level checks on quality incorporated

23 Why Digital Technology for Evaluations / Periodic Monitoring Paper Surveys Using Mobile Phones Printing, tracking forms is tedious for large surveys, changes costly Deployed remotely, tracked in real time, changes easy on the field Many errors in data collection, entry, manual scrutiny Limited errors due to built in logic flows and validation, Single entry Effective monitoring of data quality is complicated and laborious Real time tracking, time/date/gps features improve monitoring Requires additional hardware devices for non-text data such as gps, pictures, audio etc. Single device for gps, audio, pictures. Easy to integrate, can be used for verification

24 Potential Challenges with Digital Data Collection Limited flexibility on field: Standardized formats for data collection; upfront development of software tools upfront Local language compatibility: Being developed for all languages Typing may be slower: learning may take time (scribbling on paper is faster) Lack of connectivity: on field limits real-time transmission of data

25 Examples TYPES OF MOBILE DATA COLLECTION

26 Data Quality & Real-time Data for Monitoring Purposes Quick & Easy Set up: Improving: Data quality Speed Transparency Accessibility Flexibility

27 Mobile-based Monitoring and Evaluation in Action: MFI agents entering weekly loan repayments for instant tabulation Community health workers feeding back information on beneficiaries for automatic identification of high risk cases Auditors collecting survey, observational, photographic and GIS data on infrastructure in slums.

28 Use of Mobile-based Technology in M&E REPORTING/MIS Routine (Real-time) reporting Weekly loan repayment information of MFI clients reporting by field staff MONITORING Ongoing program monitoring Beneficiary information collected and sent by health workers (for disease surveillance, delivery of benefits etc.) Occasional (Surprise) checks Spot checks by supervisors to monitor attendance and performance of staff EVALUATION Survey data Household survey data to assess impact of <<xx>> program

29 Use of Mobile Technology as key intervention in Programs Information/Messages SMS reminders to beneficiaries about important health activities (treatment compliance) Implementation Tools Videos and online teaching aids used in schools for regular or remedial education programs Biometric, GIS information Record beneficiary information for easier tracking and follow up. Complementarities with other programs by same organization

30 Case Study: Delhi Voters Project Project description and objectives An RCT evaluating whether providing information to government officials and slum dwellers can lead to higher accountability and thereby improved service delivery. Interventions evaluated: The effect of providing information on spending and quality of public services delivered by elected officials during election sensitive periods

31 Role of mobile-based data collection The intervention Field-based audits of public services in slums using mobile-based technology Surveys and observations Photographs GIS indicators Send report cards to elected representatives highlighting the quality and access to public services in their area

32 The Implementation Started by auditing GARBAGE FACILITIES 1) Dumpsters 2) Bins 3) Informal Points TOILETS Public Urinals Open Public Toilet Complex

33 Audit Report Cards

34 Going a step further with Mobile Based GIS software MOBILE GIS! Accessibility to public services How far is the nearest toilet? How far is the nearest formal point of disposing garbage? Spread Analysis Does cleanliness of a toilet affect household health?

35 WHEN TO USE DIGITAL DATA COLLECTION?

36 When to Use Digital Technology? Mobile Phones / Tablets (Smart phones) For periodic surveys, mapping, supervision, electronic databases SMS/IVRS based (Simple phones) Routine monitoring for delivery of key services GPS / Biometric Devices Mapping and data verification at a micro level (HH, beneficiary) Smart (Biometric) Cards Direct payment to beneficiaries upon verification, create digital databases

37 Cost Effectiveness Paper No one-time hardware cost Ongoing costs of printing, transporting and storing paper questionnaires Data-entry operations take significant time and resources training Longer time-frame before data is available for analysis Mobile Initial one-time cost of devices Additional costs for maintenance (batteries, replacement etc) Ongoing data-plan costs, and service-provider costs Real-time access to data to monitor quality and progress Environmentally friendly

38 THANK YOU