The BLC approach to assessing Management Information Systems for civil society organization monitoring and evaluation

Transcription

1 CONFERENCE PAPER SAMEA CONFERENCE - September 2013 The BLC approach to assessing Management Information Systems for civil society organization monitoring and evaluation Authors: Jerusha Govender, Pule Setshotloa, Jabulani Mavudze 1 1 Affliliation: Management Sciences for Health (MSH), Building Local Capacity for Delivery of HIV Services in Southern Africa Project (BLC) Background The core components of a well-functioning organization are work climate, management systems (structures, processes, and procedures), and the capacity to respond to change (MSH, 2005). Monitoring and evaluation (M&E) are key factors necessary to support these components, particularly in a resource-limited environment. Actionable information is data you can use in decision making and take action [the information system] must be based on useful indicators produced in a simple format that is on time for the planning or reporting cycle. (MSH, 2010). Literature on theories related to data for decision making present a variety of forms to use data generated from monitoring and evaluation. A comprehensive summary of the theories in Jones et al (2009) describes the use of data for decision making in the forms of ritual or symbolic use, instrumental, conceptual, facilitating improvements, and generating knowledge, 1

2 among others. There is limited evidence that any of the theorized uses are a reality for civil society organizations (CSOs), except perhaps for rendering judgments, a use described by Patton (1975). CSOs often have limited capacity, including insufficient knowledge on tools and approaches, to develop or enhance their M&E systems. CSO systems tend to be donor-focused, without a broader understanding of the relationship between M&E and organizational effectiveness. As Chaplowe (2007) states, M&E priorities can push CSOs into a crisis of conformity, where risk is avoided to look good, resulting in short-term projects that focus on discrete outcomes. Little effort may be made to adapt appropriate systems or make use of innovative means to monitor program progress. In contrast, data in the private sector is seen as a corporate resource, and integrated database development is a priority. These systems protect data and ensure data security, integrity, and reliability for generation of simple reporting and use in decision making (Ricardo, 2010). Information systems for data for decision making Using a systems theory approach, one can identify many reasons for the challenges CSOs face in using data for decision making, and why these differ in the corporate sector. The people within the organization and the existing systems and structures are all contributing factors. Knowledge and tools are also key areas as they relate to the people and systems. When comparing CSOs to the corporate sector, the differences often manifest in systems: corporations operate in an environment with significant resources, and investments are made in information systems which support staff by reducing their need for advanced knowledge in data analysis making the data more easily accessible for decisions. The associated costs and the process of developing such systems may appear as hurdles to CSOs, but in fact there are affordable and simple options and approaches available. Information management systems are developed to manage large-scale databases and reporting. The most appropriate system is based on the type of data collected, how the data is used or accessed, and the number and type of users. A simple paper-based filing system can work, but with the growing complexity BLC staff presenting on the database 2

3 of M&E needs, more advanced electronic and webbased systems, such as district health information systems, are increasingly used to ensure secure and efficient management of information. Such systems become a source of easily accessible routine data for organizational purposes. In an age of rapid technological advancement, business intelligence and mobile tools are being developed as information systems for program management purposes. A business intelligence tool is application software used to retrieve, analyze, and report data which has been captured and stored. These systems were once expensive but are now easily accessible and adaptable for CSOs. This provides an opportunity for CSOs to customize M&E systems, easily access data for decision making, and expand beyond monitoring to maintain aggregate data for evaluation. The BLC Monitoring and Evaluation Challenge As part of BLC s reporting obligations to USAID and PEPFAR, an efficient mechanism to provide feedback on progress is central to the structure of the project. It is also important that the reported information generates actionable data used to focus, align, and mobilize staff to meet targets in creative ways and make course corrections as needed. The project s monitoring and evaluation is based on a results framework which details the processes followed by BLC as well as civil society organization, intergovernmental, and government partners to demonstrate how the project inputs lead to intended outputs and outcomes, and contribute to health impact. The BLC M&E system is currently paper-based. Activities are recorded in four tools which provide records for 47 indicators in nine geographical and thematic areas (Angola, Botswana, Lesotho, Namibia, South Africa, Swaziland, HIV Prevention and Migration, Regional Civil Society Organization Capacity Building, and Southern African Development Community). Progress on meeting the indicators, listed in a Performance Management Plan, is reported to the donor on a quarterly basis. The time taken to receive the records and narrative from each area ranges from two to twelve weeks. This limits time for quality assurance, and the data cannot be used for day-to-day management purposes. As a result, the system only allows for reactive rather than proactive use of the information. The reactive nature of the system also limits timely identification of successes, promising practices, and lessons learned that can be shared internally and externally. BLC s current paper-based M&E system poses challenges to reporting and restricts staff to use data for decision making and project management, despite the creation of M&E tools and processes to overcome challenges. These challenges prevent BLC from operating at an optimal level of efficiency and include: Increased time required to access data A time lag between data collection, consolidation, and action Provision of data which relates to indicators only Limited integration of quantitative information, narrative, and success stories Loss of system integrity with increased complexity of the project To address these challenges, maximize the use of available technology, and improve efficiency and effectiveness, BLC developed a management information system which monitors all BLC activities. This central system will be used by all BLC staff, teams, and units to manage activities by recording and viewing data in comparison to targets. It will generate information required for donor reporting and allow for simple management reports. In addition, BLC envisions that the system will record and submit administrative or financial documentation to reduce the number of other electronic programs used by teams and add value to utilizing the system. This more proactive system will move BLC from simply monitoring to evaluating its work. BLC developed a database which addressed its needs 3

4 Methodology Developing any system can be complex and timeconsuming. However, it is important to use a comprehensive approach in developing systems that people will use for decision making. BLC undertook a participatory and detail-oriented process to identify a management information system for effective project management, monitoring, evaluation, and reporting. This process provides a guide for other CSOs who need to develop or strengthen their M&E system to improve use for decision making. This does not require extensive technical knowledge; instead, it supplies direction on the necessary skills and information. The available plans and reference documents for information systems are technical and complex. One simple, key factor which the references highlight is the role of people. No management system can be strengthened without inclusion of people in the process (MSH, 2010). Ricardo (2011) and CompuMentor (2001) emphasize the importance of including key personnel in planning an information system. However, they do not indicate the review of communities of practice or discuss tapping into resources available to CSOs. Taking these factors in account, BLC s approach to developing its management information system was structured into five key steps: STEP 1: Define general requirements based on organization/project needs and budgetary constraints A similar approach has been used for conceptual data model and logical model development (Ricardo, 2011). BLC used a conceptual background based on work plans and performance management plans relating to the project results framework and donor indicators. BLC considered user and access restrictions, roles and responsibilities, validations, data capture needs (including types and frequency of data captured), reporting needs (including types of reports based on donor and management needs), security and cost of database, generation of user-friendly outputs, and ease of branding. STEP 2: Consult all project area teams to define project management requirements The success of an M&E system is dependent on the analysis of the user environment in order to understand current systems and needs: present and possible future needs (Ricardo, 2011). Monitoring and evaluation, communications, grants, and finance teams were consulted along with project area teams, program associates, and project directors. These staff provided input on current data and reporting flow and needs, as well as the desired results of the data. STEP 3: Conduct an investigation to understand the technical environment and internal information technology (IT) infrastructure capacity BLC conducted an investigation to understand its technical environment and options for developing the system. This included internet searches (keywords: databases, business intelligence tools, M&E platforms, project management), accessing technical insight from online communities of practice, and consultations with developers, MSH s IT departments, and teams who will use the platform. STEP 4: Test system options based on the criteria identified in steps 1-3 Systems of interest were selected and tested based on criteria developed from the consultations and requirements identified in steps 1-3. These include 22 aspects listed below: 1. Administer people 2. Hierarchical users 3. Hierarchical datasets 4. Update activity status in form 5. Relational database 6. Validate/approval levels of data captured 7. View data as graphics 8. View comparison to targets 9. Generate reports 10. Export data 11. Offline functionality 12. Web-based 13. Development time estimate 14. Technical support 15. Links to work plan 16. Upload documents 17. Alerts on data captured for approval 18. Function 19. Record/upload stories 20. Branding 21. Compatibility 22. Value add A detailed assessment is contained in Table 1. A rating of 0 to 3 stars was used to assess each aspect for each system. STEP 5: Create a management information system development and implementation plan Based on the assessment and keeping in mind BLC s desire for a more proactive system allowing for increased evaluation, BLC identified the most appropriate system. This system encompasses BLC s broad monitoring and reporting requirements, the technical aspects, and most importantly the use of data for management. BLC then created a database 4

5 development and implementation plan. The plan included a background to the M&E challenge described, a consolidation of the requirements, insight from consultations, a system review, recommendations, and a Gantt chart planning the development. Results Six web-based management information systems were identified and tested. BLC selected DHIS Version as its management information system based on its ability to address BLC s core needs and recommendations from other MSH projects. DHIS provides a dashboard for each of BLC s nine areas providing directors with a broad overview and team leads with detailed information on their specific program areas. BLC has strengthened its IT infrastructure and technical support, as well as senior input and approval, to ensure successful implementation of the system. Culture, or people factors, were addressed through an intensive consultation process. BLC teams were primarily interested in an M&E system that could address their specific project management needs. They emphasized the need to monitor partner deliverables and deadlines for effective day-to-day management. Directors cited the need to identify long-term needs and the importance of the system s adaptability. Conclusions and Recommendations BLC s approach to developing a management information system can be replicated by other CSOs and provides a detailed critique of system options. The results show that effective open source systems which meet reporting requirements and program management needs are available. This is particularly important for efficient and effective management and using data for decision making in resource-limited settings. Management information systems can be implemented in many CSOs to improve local and national service delivery and maximize development outcomes through supporting evidence-based decision-making. 1 District Health Information Software, Version 2.11 (DHIS 2) is open source software which can be configured for organization specific needs to capture, aggregate, and generate reports and dashboards to used data in decision making. (More information is accessible via: Screenshot of the BLC Dashboard 5

6 Table 1. Critique of Database Options 6

7 Launched in 2010, the USAID-funded Building Local Capacity for Delivery of HIV Services in Southern Africa Project (BLC) strengthens government, parastatal, and civil society entities to effectively address the challenges of the HIV and AIDS epidemic. Throughout the Southern Africa region and with specific activities in six countries, BLC provides technical assistance in organizational development, including leadership, management, and governance in three key program areas: 1) care and support for orphans and vulnerable children; 2) HIV prevention; and 3) community-based care. BLC is working with Global Fund Principal Recipients in Angola, Namibia, South Africa, and Swaziland, as well as the Southern African Development Community. References: 1. Chaplowe, Scott G. and Engo-Tjéga, Ruth Bamela Civil Society Organizations and Evaluation: Lessons from Africa. Evaluation 13: CompuMentor Database Planning Guide. CompuMentor. San Francisco 3. Jones, N., Jones, H., Steer, L., and Datta, Ajoy Improving impact evaluation production and use. Overseas Development Institute, London. 4. Management Sciences for Health Health Systems in Action: An ehandbook for Leaders and Managers. Cambridge, MA: Management Sciences for Health. Available online at resource-center/health-systems-in-action.cfm 5. Management Sciences for Health Managers Who Lead: A Handbook for Improving Health Services. 3rd ed. Cambridge, MA: Management Sciences for Health. 6. Ricardo, C Databases Illuminated: Chapter 2: Database Planning and Database Architecture. Jones & Bartlett Publishers, Sudbury. This publication is made possible by the generous support of the United States Agency for International Development (USAID) under the Leader with Associates Cooperative Agreement GPO-A The contents are the responsibility of The Building Local Capacity for Delivery of HIV Services in Southern Africa Project and do not necessarily reflect the views of USAID or the United States Government. Access BLC publications online: Building Local Capacity Project (Regional Office) Ditsela Place 1204 Park Street (Cnr Park and Jan Shoba Streets) Hatfield, Pretoria, South Africa Tel: ; Fax: blcsouthernafrica@msh.org; 7