MONITORING & REPORTING MECHANISM INFORMATION MANAGEMENT SYSTEM

MONITORING & REPORTING MECHANISM INFORMATION MANAGEMENT SYSTEM FINAL REPORT 1 Chris Robertson December 2009

CONTENTS 1 EXECUTIVE SUMMARY 6 1.1 Introduction 1.2 Mapping of Current MRM Information Systems 1.3 Other Information Systems and Databases 1.4 Needs Assessment 1.5 Proposals for the Task Force Information System 1.6 Proposals for the Global Information System 1.7 Monitoring the Impact of the MRM 1.8 The Information needs of MRM Practitioners 1.9 Recommendations and Next Steps 2 INTRODUCTION 13 2.1 Terms of Reference 2.2 Concepts and Definitions 2.3 Overall Approach to the Consultancy 2.4 Work Plan 2.5 Constraints 2.6 Structure of the Report 3 MAPPING OF CURRENT MRM INFORMATION SYSTEMS 18 3.1 Questionnaire Responses 3.2 Sudan 3.3 DRC 3.4 Sri Lanka 3.5 Nepal 3.6 Somalia 3.7 Common Issues at the Country Level 3.8 Global Information Management 4 OTHER INFORMATION SYSTEMS & DATABASES 37 4.1 OHCHR Global Human Rights Database 4.2 OHCHR Special Procedures Database 4.3 ICRC Database 4.4 HURIDOCS Human Rights Database 4.5 UNHCR/UNFPA GBV Monitoring System 4.6 IMSMA Mine Action Database 4.7 OCHA Access Monitoring & Reporting Framework 4.8 Inter Agency Child Protection Database 4.9 DevInfo 4.10 MRM Task Force Feedback 4.11 Common Issues 5 NEEDS ASSESSMENT 48 5.1 MRM Task Forces 5.2 Global Partners 2

5.3 Overview 6 PROPOSALS FOR A COUNTRY BASED INFORMATION SYSTEM 54 6.1 The Rationale for a Standardised Information System 6.2 System Objectives 6.3 System Design 6.4 The Interface with Other Information Systems & Databases 6.5 Ensuring Security 6.6 Challenges and Practical Issues 7 PROPOSALS FOR A GLOBAL INFORMATION SYSTEM 68 8 MONITORING THE IMPACT OF THE MRM 8.1 National Impact Monitoring 8.2 Global Impact Monitoring 9 THE INFORMATION NEEDS OF PRACTIONERS 73 10 RECOMMENDATIONS AND NEXT STEPS 76 10.1 Recommendations 10.2 Implementation Next Steps LIST OF APPENDICES 1 Terms of Reference 2 Schedule of Visits & Consultees 3 Field Questionnaire (Blank Form) 4 Summary Analysis of Field Questionnaire Responses 5 Global Questionnaire (Blank Form) 6 Summary Analysis of Global Questionnaire Responses 7 OCHA Access Monitoring & Reporting Framework 8 Data Requirements Framework a) Explanatory Notes b) Data Requirements Format c) Confidential Data Sheets d) Data Requirements Example 1 Recruitment & Abduction e) Data Requirements Example 2 School Attack 9 Perpetrator Reports a) Commentary b) Graphs and Tables 10 Risk Assessment Framework a) Security Guidelines b) Risk Assessment Template c) Risk Mitigation Matrix 3

11 Database Specification a) Database Specification b) Data Entry Menus and Screens 4

ABBREVIATIONS CAFAAG DPKO IACPD IACPIS GIS ICRC INSMA IRC MRM MRMTF OSRSG CAAC OCHA OHCHR SCWG UNFPA UNHCR Unicef Children Associated with Armed Forces and Groups Department of Peacekeeping Operations Inter Agency Child Protection Database Inter Agency Child Protection Information System Geographical Information System (Mapping Software) International Committee of the Red Cross International Management System for Mine Action International Rescue Committee Monitoring and Reporting Mechanism Monitoring and Reporting Mechanism Task Force Office of the Special Representative of the Secretary General for Children Affected by Armed Conflict Office for the Coordination of Humanitarian Affairs Office of the High Commissioner for Human Rights Security Council Working Group United Nations Population Fund United Nations High Commission for Refugees United Nations Children s Fund ACKNOWLEDGEMENTS Thanks are due to everyone who contributed their time, knowledge and ideas to this piece of work, especially all those who managed to find time in their heavy schedules to complete my mammoth questionnaire, the Task Forces that hosted my field visits and gave unstintingly of their time, and Stephane, Lara and colleagues at Unicef in New York who supported my work throughout and patiently answered my endless questions. 5

1. EXECUTIVE SUMMARY The consultancy was commissioned by Unicef on behalf of the Monitoring and Reporting Mechanism (MRM) for the six grave violations against children in situations of armed conflict It was designed to assess the information systems and databases currently being used at the country level to support the MRM and to make recommendations for developing and rolling out a more standardised system. This involved a mapping of current practice and a needs assessment at both country and global levels, together with the development of a detailed specification for any new system required. The main focus was on the monitoring of the incidents themselves, but the review also touched on other aspects of information management, including impact monitoring and the information needs of MRM practitioners. 1.1. Introduction (Section 2) The overall approach taken to the consultancy and the methodology adopted are both outlined in the Introduction. The analytical framework focused on the way in which the information system helps support the strategic objectives of the MRM and its partners, what data is recorded and how the system performs against a range of indicators. The methodology included a review of available documentation, meetings with global partners in New York and Geneva, field visits to Sudan, DRC and Sri Lanka and detailed questionnaires that were completed by country Task Forces and global partners. An effort is also made in the Introduction to clarify some of the key concepts and definitions involved, including the differences between an information system and a database, quantitative and qualitative data and monitoring as opposed to case management. 1.2. Mapping of Current MRM Information Systems (Section 3) The mapping of the information systems that are currently used to support the MRM was based upon a very detailed questionnaire that was completed by 16 countries and three field visits that were made to DRC, Sudan and Sri Lanka. Additional information from previous work undertaken for Nepal and Somalia has also been included in the report. It is clear that most Task Forces have invested heavily in information management, but that despite some examples of good practice, many are frustrated with their systems failure to deliver what they need. The main issues that emerged from the mapping were: Approaches to Monitoring There is considerable diversity in the way monitoring itself takes place, due not only to the nature of the conflicts themselves, but also Task Force capacity and the lack of detailed guidance on certain key issues (e.g. how the minimum dataset should be implemented). This is reflected in the range of data being recorded and the way in which it is collected. Of particular importance are the differences between real time and retrospective recording, child specific and event monitoring and centralised and decentralised verification. Whether focal agencies are used for each violation and how the MRM interfaces with other child protection programmes has also been crucial to the design of information management systems. System Development Very few Task Forces have access to specialist information management expertise and as a result system development tends to lack a strategic focus, is often data led and gives insufficient priority to objectives and outputs. Resources are always limited and there is a reliance on IT staff from MRMTF members, who have many other competing priorities. Regular 6

staff turnover can also create difficulties. It can often take a very long time to develop a fully operational system and this can be very frustrating when responding to an emergency. There are also several instances of systems being abandoned in favour of a fresh start, either because the first system fails to meet expectations or because new staff want something different. System Architecture & Implementation Most Task Forces use both manual and electronic systems. Data collection forms vary enormously in format and complexity. Some use just one form while others have one for each violation. Some make considerable use of multiple choice questions as well as open ended ones, but other forms are very unstructured. It is clear that many questions are routinely left blank, so data collection aspirations are often unrealistic and data quality is quite variable. More Task Forces make use of a database than a spreadsheet. MS Access is the most common software being used. Some databases are completely centralised with all the data being entered in the capital by the UN agency hosting the system, while others are highly decentralised and are used to manage the documentation and verification process in he field. The data manipulation capacity of systems varies considerably, but in general their reporting capabilities are quite limited. System Security Everyone recognises that security is crucial, but few countries have implemented a full risk assessment or developed a security plan for their information system. The result is a mixture of good practice alongside serious weaknesses. Manual systems are generally more vulnerable than electronic ones, especially in the field and where data has to be moved from place to place. Some Task Forces have addressed this by entering data directly into their electronic system. Others have decided to exclude personal data that can be used to identify individual children from the main information system. No security system is foolproof. Human error and a lack of vigilance is just as important as the technical measures, so the lack of security training and security audits is a concern. At the global level the information system is primarily focused on the annual reports and horizontal notes submitted by each country and the minutes of SCWG deliberations. While there are guidelines for these reports, the format varies from country to country and there is little systematic analysis of the caseload. The diversity of practice at the country level and the lack of a common dataset or agreed data definitions mean that it is impossible to undertake any global analysis of the data. 1.3. Other Information Systems and Databases (Section 4) The review also looked at other information systems that are currently being used to support human rights monitoring and child protection programming in emergencies, both from the perspective of transferring data into the MRM and to see if they can provide insights into the challenges that the MRM might face in developing its own information system. The review encompassed a wide range of systems, most of which used a database of some kind. A number of common issues emerged: There is a very clear distinction between systems that focus on monitoring and those that are designed to support case management as they require very different data structures and functionality. Very few, if any, systems attempt to fully integrate these two different functions. The majority of the systems examined were being used for decision support. They provide a convenient mechanism for storing, accessing and manipulating data about individual cases. Quantitative analysis is allocated a relatively low priority by most systems. A number of organisations have used their information systems and, in particular, their databases as a means of ensuring consistency in the implementation of their monitoring procedures across their many different country programmes. 7

Most organisations highlighted the importance of having a robust data structure, focusing on clear one to one or one to many relationships and explicit data definitions that are rigorously enforced. Many highlighted the considerable challenges involved in trying to balance the drive for consistency and standardisation with the demand and need for local flexibility and customisation. The solutions to this dilemma varied depending on the overarching purpose of the system. Apart from the UNHCR/UNFPA GBV system, none of the systems attempted to consolidate their analysis at the global level. There are several reasons for this reluctance. Many agencies believe that the diversity of the situations being monitored and the variable monitoring capacity in each country make such comparisons unhelpful and potentially misleading. Some agencies allow significant local customisation of their systems, making global level comparisons impossible. Many also believe that the data belongs to the country level offices and that they are the ones that are best placed to make decisions about its use. Protecting confidentiality was a priority for all those focused on human rights monitoring and most tried to ensure this by separating personal data from incident details, often by excluding the former from their main database. All the organisations emphasised the importance of allocating adequate resources for the implementation as well as the development of the system and indicated that these could be substantial. Most of the more impressive systems had been developed over a period of years and had undergone significant upgrades. 1.4. Needs Assessment (Section 5) Respondents at the country and global levels were asked whether they favoured the creation of a more standardised approach to information management and, if so, what form it should take. The majority of country based Task Force respondents favoured a more standardised approach and 77% supported the development of a common database. The main advantages of this approach were seen as access to specialist information and data management expertise, resource savings and the opportunity to make comparisons between different countries. The main concerns revolved around the degree of diversity at the country level and whether this could be accommodated effectively. The top priorities for any new system were identified as: Flexibility the ability to customise the system to meet local needs Strong reporting and analytical capabilities Simplicity and user friendliness Strong system security Some people also wanted to reduce the use of, or even eliminate, paper forms When asked which strategic objectives people wanted the system to support, most Task Force respondents included advocacy, programme planning, immediate interventions, referrals and the management of the MRM itself, in addition to the core function of preparing the formal reports for the Security Council. There was a greater diversity of views among global partners about the priorities for information management at the country level. This involved issues of principle as well operational concerns. The former reflected the differing organisational mandates of the partners. At one end of the spectrum, the OSRSG CAAC were primarily concerned about the information needs of the formal 8

reporting to the Security Council and believed that these were best met by having access to quality data about a limited number of fully verified case studies that could be used to highlight key issues. At the other end of the spectrum, partners with wider child protection mandates wanted the MRM information system to play a broader role and were keen to see more comparative analysis and trends that could be used to support advocacy, programme planning and other objectives. The operational concerns related to the value of any quantitative analysis and the cost effectiveness of using a database. The OSRG CAAC felt that the MRM caseload was too small and unrepresentative to be used in any quantitative analysis. While many of the other respondents recognised this problem, they still saw a benefit in being able to analyse the data, especially if it could be combined with other contextual information (e.g. non verified incidents and the proposed macro monitoring). Most partners recognised that much of the database development that has taken place at the country level has been problematic, but drew different conclusions from it. The OSRSG CAAC felt that a database was unnecessary. They would prefer to receive descriptive reports and think the development of a standardised database would be a waste of valuable resources. However many other partners feel that a database is essential for managing and analysing the caseload and think that developing one standardised version could help overcome some of the problems experienced to date. These differences of view were also reflected in respondents views about the information requirements at the global level, although data security and reconciling the objectives of different partners were seen as additional challenges. Some of those who wanted to make comparisons between countries and plot trends favoured having a unified database system at the global level, while others were happy to rely on reports provided by the country based Task Forces. 1.5. Proposals for a Country based Information System (Section 6) It is clear that, despite the expenditure of significant resources and some limited successes, the current situation at the country level is generally unsatisfactory. A more standardised approach could help improve things, but only if the diversity of views outlined above can be resolved and the system can be made flexible enough to accommodate the needs of all the countries involved. The differing objectives of the global partners are not unexpected and are consistent with their individual mandates, but they are not as mutually exclusive as might initially appear to be the case. There is also a strong case for trying to develop a system with sufficient flexibility to meet their differing objectives and in so doing to ensure the maximum buy in and participation from all stakeholders in the MRM itself. There is actually more overlap in what people want out of the information system than is often recognised. For instance, those who are drafting reports for submission to New York need to be able to analyse the caseload and identify which cases are most significant and merit inclusion, while those who are looking at advocacy and programme planning are likely to want access to case study material and qualitative data as well as quantitative analysis. There is no reason why the information system cannot provide quick and easy access to both case studies and quantitative analysis, but this needs to be based on a robust and consistently applied mechanism for recording the verification status of cases in which everyone can have confidence. This will require greater standardisation in how the minimum dataset and common data definitions are operationalised, including those relating to case documentation and verification. These ideas are explained in Section 6 and Appendix 8. The use of a more standardised system will also provide an opportunity to improve the consistency with which documentation and verification takes place in the field and consequently reduce the tension that sometimes arises between New York and some individual Task Forces. 9

In theory this data could be collected, stored and manipulated manually using paper forms and files, but only if the numbers remain fairly small and the monitoring system very simple. Once the numbers rise, as has happened in most countries implementing the MRM, and more agencies become involved in the monitoring process, some form of electronic system becomes inevitable. The volume of data and the complexity of data relationships mean that a database is the most appropriate solution. The problems experienced with the current MRM databases are primarily due to the way in which they have been developed and do not imply a fundamental weakness in the technology or that it is unsuited to the task. In addition to making it possible to manage large numbers of cases, a database can offer real advantages in terms of data security. It can also play a role in enforcing documentation and verification standards, while at the same time allowing Task Forces to monitor other non MRMverified violations if they want without undermining the quality of the core MRM. The unrepresentative caseload does mean that analytical reports can be problematic and need to be used with care and always in context, but can still be of value as shown by the example in Appendix 9. Analytical reports can also be used in combination with other sources of information, including the proposed Macro Monitoring, to create a more holistic picture. Section 6 and Appendices 8 and 11 contain detailed proposals for the information system and database. They are based upon a data structure and definitions that would need to apply in every country, but would still allow for significant local customisation of the user interface, include a facility for adding user defined fields and provide for flexible report generation. To ensure maximum protection for all the individuals involved, it is proposed that no personal data (names, addresses, etc) should be included in the database itself. This sensitive data that could be used to identify individuals should be held in a separate location, but with the ability to cross reference it with the database using shared ID codes. The proposed database is designed to accommodate different monitoring arrangements and implementation models (centralised and decentralised, standalone, networked or web based) and includes comprehensive security provisions. Section 6 also explores the relationship between the proposed MRM database and the IACPD and concludes, for a mixture of technical and non technical reasons, that this should be based on having maximum compatibility of data formats and, where needed, automated data exchange, rather than full integration. The proposed database contains robust security features, but protecting confidentiality is about much more than technology. All Task Forces should undertake a comprehensive risk assessment and prepare a full security plan for their information system, including provision for appropriate training and security audits. Detailed guidance is contained in Appendix 10. Section 6 concludes with a discussion of some of the practical issues that will be faced in putting the proposals into effect, including the choice of software and the need for phased implementation, and contains an outline of the next steps required to take things forward. 1.6. Proposals for the Global Information System (Section 7) If a standardised database is implemented at the Task Force level, it will be possible to consolidate the data globally to provide the comparisons and trends requested by most partners. The challenges in doing so will be not be technical, but about how useful any analytical reports will be, how security will be ensured and how the ownership of data and the resulting reports will be handled. Two main technical options are identified. The first involves the export of standard reports from each Task Force database and their consolidation globally. This could be automated and would provide a 10

range of basic reports without the need to transfer any case specific data to New York, but no further investigation or customised analysis would be possible. The second option would involve the consolidation of all or part of the Task Force databases in New York, which would provide unlimited opportunities for analysis. Data security would be a bigger risk, but this could be manageable if, as proposed, personal data is excluded from the databases at the country level. A global back up would also guard against data loss during any serious unrest at the country level, especially if this resulted in the evacuation of staff and/or the withdrawal of key agencies. The diversity of factors impacting on the data from each country would make interpreting the analysis more difficult, so it is more likely to be of value internally within the MRM as a catalyst for further investigation and research. Once again context will be crucial and the analysis will be more useful when viewed in combination with other sources of information. 1.7. Monitoring the Impact of the MRM (Section 8) Although not the main focus of the consultancy, monitoring the impact of the MRM was a concern for many consultees. Impact monitoring is notoriously difficult, especially for something like the MRM, where there are many other factors that can have a much more significant impact on the level of violations. Hence it is necessary to identify surrogate indicators which are more sensitive to the MRM. Apart from the two narrowly defined indicators that are focused on the children whose cases have been recorded by the MRM itself, the report also proposes the use of a simple matrix to provide a more structured approach to monitoring the impact of advocacy initiatives, Action Plans negotiated with parties to the conflict and the outcomes of annual reports and the resulting recommendations from the SCWG. 1.8. The Information Needs of MRM Practitioners (Section 9) The consultancy also looked at the information needs of those involved in implementing the MRM and reviewed the MRM website CAACNET. Apart from their wish to see the MRM Guidelines finalised, improved access to key documents, opportunities for networking and improved bilateral contacts were practitioners highest priorities. A serious lack of resources means that CAACNET is currently unable to meet these needs. The user interface is uninspiring, it is very difficult to search for documents, many of which are now out of date and there are no opportunities for inactive use. The report includes detailed suggestions for a replacement website with a full range of resources and opportunities for much greater interaction between users (blogs, discussion topics, etc.). However for this to be implemented additional resources will be required, most critically so someone is able to allocate time to managing and supporting the site on a regular basis. 1.9. Recommendations & Next Steps (Section 10) The final section of the report contains a full list of the main recommendations, which are reproduced below for ease of reference. More detail of these proposals is included in the appropriate sections of the report and in the individual appendices. Section 10 also contains details of the practical steps required to implement the recommendations. a) Minimum Dataset 11

The MRM should seek to operationalise the minimum dataset using mandatory and nonmandatory fields that can be supplemented locally with additional user defined fields and used in either a manual or electronic format. For this to be successful standardised data definitions and options must be adopted. (Section 6 and Appendix 8) b) Treatment of Personal Data The MRM should move towards a system which keeps confidential personal details about all the individuals involved in the monitoring process separate from the main data about incidents and violations. (Section 6) c) Manual or Electronic Systems The MRM should, to the extent possible, minimise the use of manual (paper) forms and maximise the use of suitably protected electronic systems for the storage and manipulation of data. (Section 6) d) Standardised Database The MRM should develop a standardised database that can be used by Task Forces at the country level. It should be based upon a robust data structure, clear data definitions and a mix of mandatory and non mandatory data fields, with sufficient flexibility to cope with the diverse approaches to monitoring adopted in different countries. (Section 6 and Appendices 8, 9 & 11) e) System Security All MRM Task Forces should undertake a full Risk Assessment and use it to develop a Security Plan for their information system. This should be revisited on a regular basis and updated to take account of changing circumstances. (Section 6 & Appendix 10) f) Global Analysis Any global analysis of MRM data should initially be based on a small number of simple Excel proformas, but once the standardised MRM database is in place at Task Force level, it should be used to consolidate non confidential data. This should be undertaken by one agency on behalf of all the partners with clear protocols in place for the disclosure of reports into the public domain. (Section 7) g) Impact Monitoring The MRM should consider utilising a more structured approach to impact monitoring utilising a matrix of the kind described in the report and place greater emphasis on communicating the results to its stakeholders, including those in the field who have contributed to the collection of data. (Section 8) h) Website Development The MRM should consider the possibility of establishing a new upgraded website for MRM practitioners with an emphasis on providing users with opportunities to engage interactively with each other, BUT only if adequate resources can be provided for its ongoing support and management. (Section 9) 12

2.1. Terms of Reference 2. INTRODUCTION The consultancy was commissioned by UNICEF s Child Protection Section in New York on behalf of the wider partnership of agencies working on the Monitoring and Reporting Mechanism (MRM) for the six grave violations against children in situations of armed conflict that was established in response to Security Council Resolution 1612 in 2005. The MRM is currently being implemented or in the process of being established in 17 countries. In July 2007, after examining the lessons learnt from the first seven pilot countries, the Global MRM Steering Committee agreed to conduct a comprehensive study to assess the existing databases and information management systems being used to support the MRM and to make recommendations for developing and rolling out a global MRM Information System and in particular to assess the need for and viability of a standardised global database system. The consultancy was finally commissioned in September 2008. The full terms of reference are contained in Appendix 1, but the main requirements were as follows: 1. Conducting a comprehensive database and information management needs analysis for the standardised MRM database and information management system at the global and country levels of those countries of those countries implementing the MRM to clearly define: a) The minimum data requirements and formats b) The case management requirements and how the two datasets need to interface. 2. Executing a comprehensive global mapping and assessment of all existing initiatives of database and information management (DBIM), including inter agency systems used by UN and NGO agencies and relevant government agencies, in the fields of: a) Monitoring and reporting mechanism (MRM) on grave violations against children, b) Human rights monitoring; and c) Child protection case management and follow up (separated children, sexual violence, CAAFAG, etc) 3. Distinguish between aspects of MRM data management requirements that need to be developed as part of global systems, vs MRM data management that are better developed at country level. 4. Assessment of hardware and software capacity available at regional, national and field levels with UN and other NGO partners involved in the MRM. 5. Based on the analysis of them aping and needs, recommend the most efficient way to develop an MRM database and information management system template for both the national and global level and identifying all of its components (software, security measures, fields, etc) with two requirements documents, one for a simple MRM system and the other for a full case management system. Recommendations should also consider how this global system can link into and communicate with existing databases in the field. 6. To develop a final report with the findings, recommendations and with a detailed work plan identifying resources (financial, material, human) and timeline needed to develop and roll out the MRM DBIM. 13

2.2. Concepts and Definitions The Terms of Reference include a number of key concepts that are of fundamental importance to the consultancy. Hence it is important at the outset to clarify our understanding of these terms and their relevance to the MRM. 2.2.1. Information Management & Databases The most fundamental distinction is the one between an information management system and a database. Information management is the more strategic term and describes the way in which organisations use information to fulfil their objectives. Information comes in many different forms and can be organised in different ways depending on the aims of the system. It can also be managed using many different techniques and tools. A database is just one of these tools. By using the power of a computer it is possible store, organise, access and manipulate large quantities of data very quickly and easily. However all these tasks can also be carried out using a manual system with paper files, card indexes, etc. or even an oral system of the kind common in pre literate societies (e.g. the tradition of griots and praise singers in West Africa). It is important that the choice of tools is based upon the needs of the information system (the outputs required or the questions that it needs to answer), rather than just opting for the highest tech solution. This is particularly important in the case of a database, which can often be a very resource intensive solution, both to develop and operate. The most common risk is that database design will be driven by the availability of data rather than the information system s objectives. As a consequence it can end up containing the wrong data organised in inappropriate ways, making it difficult to generate the outputs required and leading to frustration for all those involved. 2.2.2. Quantitative and Qualitative Data The second key distinction is between qualitative and quantitative data and the way in which they can be used. Quantitative data is information that can be counted or measured and is used to define something. It is usually concerned with attributes, characteristics and properties. Qualitative data on the other hand is not normally countable or measurable and is used to describe something. Qualitative data can be a richer source of information, providing a more holistic picture, but is much more difficult to manipulate or use in making comparisons or identifying trends. There is always a risk that the two types of data will be seen as mutually exclusive alternatives or that one is intrinsically more valuable than the other (e.g. qualitative data is more useful than simple numbers that can over simplify the situation). In reality they are complementary and most information and monitoring systems make use of both and often try to quantify some of this soft qualitative data by creating indicators for things like attitudes, competencies, outcomes, and impact, using carefully defined categories and definitions. 2.2.3. Monitoring & Case Management The third key distinction is the difference between monitoring and case management. The Terms of Reference are not explicit, but it is often assumed that monitoring relates to the MRM, while case management is about response. This however would be misleading as the crucial difference is a more generic one. Monitoring is concerned with gathering data in order to describe and analyse something, 14

while case management involves using the information system to manage a process of some kind. The provision of support and services to children who have experienced a violation could certainly fall into the second category, but case management could equally be about the way in which an incident is documented and verified (i.e. collecting data about the different stages of the process and making it possible to manage the system by which this happens). The two systems can contain some common data, but they are fundamentally different in the way in which they process the data and the outputs that they deliver. The former is primarily concerned with describing a situation, either statistically or in the form of case studies, and is largely static. The latter is more dynamic with an emphasis on providing operational support to field staff and enabling managers to keep a check on the progress of work so that resources can be redirected or corrective action taken to ensure that targets are met. 2.3. Overall Approach to the Consultancy The Terms of Reference for the consultancy are extremely detailed, but essentially focus on the development of information systems at the country and global levels, with a requirement to map existing provision, assess needs and develop recommendations that will meet these needs given the available resources and constraints impacting on the MRM. There is an inevitable overlap between the mapping of existing information systems and the needs assessment, so the methodology has sought, wherever possible, to integrate the two processes and minimise the level of duplication. This is particularly important when looking at the basis for planning and evaluating the systems and a common framework has been developed utilising three key questions: How does the information system support the strategic objectives of the MRM? What key data should be recorded in the system How does the information system perform against a range of key indicators? These can be understood as follows: 2.3.1. Support for Strategic Objectives The framework identifies eight strategic objectives to which the information system might be expected to contribute. These range from ones that everyone would recognise and support (i.e. the preparation of the Annual Reports and Horizontal Notes) to ones that may be more contentious (e.g. Referrals, Response and Managing the MRM itself). These can be summarised as follows: Strategic Objective Annual Reports & Global Horizontal Notes Advocacy & Action Planning Impunity & Accountability Immediate Interventions Programme Planning Referrals Details Providing access to caseload profiles and detailed individual case studies that can be used in drafting reports for New York Providing information that can be used to help plan and implement advocacy initiatives, either locally or nationally, develop Action Plans with the various armed forces/groups and assist in monitoring the implementation of these Plans Providing information that can be used in negotiations with armed groups about specific incidents and/or in efforts to hold individual perpetrators to account. Providing information about individual incidents that can be used to initiate immediate interventions designed to address the violations involved (e.g. to secure a child s release) and to monitor their success. Providing a profile of the caseload than can help in planning programmes to prevent and respond to violations. Providing the information that is required by the agencies that provide long term support to children who have experienced violations and ensuring that referrals take 15

Strategic Objective Long Term Response Managing the MRM 2.3.2. Data Requirements Details place in a timely manner. Providing information that can help these agencies deliver and manage the provision of this longer term support to children. Providing the information needed to assess how effectively the MRM is being implemented and how it might be strengthened. The second part of the framework focuses on the ten types of data that might be included in the system. The first three are concerned with the process by which the case is documented and verified, the next four with the details of the incident itself and the final three with the actions that may be taken in response to it. These can be summarised as follows: Type of Data Initial Source of the Allegation Documentation of the Allegation Verification Details Incident Summary Child(ren) s Details Perpetrator(s) Details Violation Specific Details Details of Immediate Interventions Referral Details Follow up/reintegration Details 2.3.3. System Performance Details Date and source of the alert, whether an UN Monitor was an eye witness, etc. Dates and names/categories of informants, sources of documentary/physical evidence, reliability of evidence, etc. Status of case, date of verification, etc Date and location of incident, description of what occurred, etc Name, DOB/age, address, ethnic/religious affiliation, family situation, etc. Name/group, state/non state, armed/unarmed, etc. Different data fields for each violation e.g. type/size of school, type of attack, impact on school, etc. Date, type of Intervention, outcome, etc. Date of referral, agency involved, purpose, etc. Detailed data about the actions undertaken by the agency providing longer term response. The third part of the framework comprises nine indicators against which the performance of the system can be assessed and which can also be used to establish the priorities for any new system. The first five are concerned with various aspects of the data recorded in the system, while the next three are focused on the way in which the data is used to meet the needs of users and the final one focuses on cost effectiveness. They can be summarised as follows: Performance Data Quality Data Reliability Data Comprehensiveness Data Consistency Data Coverage Timeliness Relevance Accessibility Value for Money Details The extent to which the data collected is in the required format (i.e. one which can be used in any automated analysis) The extent to which the data accurately reflects/describes the reality of the case. The extent to which the data required is regularly collected (i.e. how many data gaps exist) The extent to which data is reported in a consistent manner for each case (e.g. is it always a data of birth, or is sometimes an age or even an age range). The extent to which the caseload reported is representative of the actual profile of incidents (e.g. by area and violation). The extent to which the system provides users with access to data and reports at the times when they are needed and the extent to which any reports contain up to date data. The extent to which the outputs of the system (reports) are relevant to and meet the needs of users (e.g. content, format, etc). The extent to which the data about individual cases and/or statistical reports (both standard and bespoke) are accessible/available to those who need them. The extent to which the benefits gained from the information system justify the cost involved, taking into account the opportunity costs involved (i.e. what else the resources could have been used for). 16

The wording of the Terms of Reference makes it very clear that the main focus of the consultancy should be on the management and use of data relating to the incidents that the MRM is designed to monitor. At first sight this may seem fairly obvious, but the information management requirements of the MRM are actually much wider than just incident monitoring. It is also important to consider the way in which the impact of the MRM itself can be monitored and how the information requirements of the MRM partners and Task Force members can best be met. It is not possible for all these diverse requirements to be accommodated in one database or manual data system, so where appropriate these issues are discussed separately in the report. 2.4. Work Plan The evaluation began with a brief visit to New York in September 2008 to agree the scope of the work with UNICEF and hold preliminary meetings with other partners at the global level. This was followed by desk research and field visits to three countries: Sudan, DRC and Sri Lanka to undertake more detailed field research. These visits took place in November 2008 and January 2009 and were followed by another series of consultations with global partners, this time in Geneva, in June 2009. On the basis of the desk research and the outcome of the various field visits, two detailed questionnaires were developed, one targeted at the country level MRM Task Forces and one at the global level partners. These were designed to provide a comprehensive picture of all existing information management systems and to enable all interested parties to contribute towards the needs assessment. A second visit was made to New York in July 2009 for a further round of meetings with global partners, to present and discuss the preliminary findings of the consultancy and agree the format of the final report. 2.5. Constraints The Terms of Reference for the consultancy were very ambitious and fulfilling them has been dependent on the availability of a large amount of detailed documentation and the collaboration of a wide range of partners, both globally and from the individual MRM Task Forces. While many individuals have committed a considerable amount of time and effort to supporting the consultancy, for many, the MRM is just one of their many responsibilities. As a result timescales have often slipped, but this has been a small price to pay for the exceptionally high level of participation in the process. Of more significance is that very few countries have fully documented their information systems. Most have been developed incrementally in response to emerging needs, often by staff who were not information management experts and so obtaining all the data required for the consultancy has been a considerable challenge. The implementation of the consultancy has taken considerably longer than originally anticipated. While this has partly been due to the time required to obtain feedback from the field and set up visits, the consultant also had to undertake the work alongside another unrelated piece of work and then experienced a prolonged period of ill health at the beginning of 2009 that led to a delay of several months. 2.6. Structure of Report The main report is split into 10 substantive sections, including the Executive Summary. 17

This introduction is followed by the mapping of the existing information systems, both those used by MRM Task Forces (Section 3) and by other organisations (Section 4). This is followed by the needs assessment based upon the field and global questionnaires and the outcome of various face to face meetings with MRM partners at the global level (Section 5). The next part of the report contains a synthesis of all this information and includes proposals for what should happen at the country/task Force level (Section 6) and globally (Section7). The final two substantive sections cover other information management issues: impact monitoring (Section 8) and the information requirements of MRM practitioners in the field and suggestions for website development (Section 9). The report concludes with a summary of the main recommendations and an indication of what needs to be done to implement them (Section 10). The report is supplemented by 11 appendices. In addition to the detailed Terms of Reference and a schedule of visits and meetings (Appendices 1 & 2), these include the two questionnaires used in the consultancy and a summary of the responses (Appendices 3 6), details of the OCHA access monitoring and reporting framework (Appendix 7) and four appendices containing the detailed specifications for the proposed information system and database, including the minimum data set, the suggested format for analytical reports, the security guidelines and the database specification itself (Appendices 8 11). 3. MAPPING OF CURRENT MRM INFORMATION SYSTEMS Two approaches have been taken to mapping the information systems and databases that are currently being used to support the MRM at the country level: A detailed questionnaire that was sent to every MRM Task Force Field visits that were made to 3 countries (Sudan, DRC and Sri Lanka). Additional information about Nepal and Somalia based upon other work undertaken by the consultant prior to this contract has also been included. 3.1. Questionnaire Responses The questionnaire was designed to map the existing information systems and seek the views of each MRM Task Force on the potential for developing a more standardised approach. There was inevitably some overlap between these two issues, but this section of the report focuses exclusively on the mapping. A response was received from 16 countries. This provided an excellent basis for the mapping, although, some respondents were only able to complete part of the form. This was inevitable given that some countries are still in the process of setting up the MRM and others lacked the necessary information or the time to cover everything. Some of the responses represented the views of the Task Force as a whole, while others were limited to those of the person completing the form. A full analysis of the responses is contained in Appendix 4. The consultant had to summarise some of the comments and interpret others that were unclear, so if there are any errors these are the responsibility of the consultant. The quantitative analysis of the factual questions was relatively straightforward, but it is clear that different respondents approached the subjective questions in very different ways, so it was often not possible to make direct comparisons. This was particularly true where the effectiveness of existing 18

information systems was being assessed. Perhaps inevitably there was a tendency for those who had played a major role in creating the systems to be on average more positive than those who had not been! 3.1.1. MRM Background The first section of the questionnaire asked for background information about the MRM itself, the membership and structure of the Task Force and the way in which monitoring and reporting is organised, as this will have a major impact on the design and operation of the information system. The MRM Task Forces have between 6 and 12 members, with most between 8 and 10. Some are exclusively or primarily made up of UN agencies (e.g. Cote d Ivoire and Sudan), while others have more significant participation from NGO s and local Human Rights bodies (e.g. Nepal where non UN agencies make up over 60% of the Task Force). Sri Lanka is the only country where government linked organisations have participated in the Task Force. The MRM Task Force is normally co chaired by Unicef together with a representative of the UN Resident Co ordinator, SRSG or OHCHR. Of more significance to the consultancy, responsibility for information management and, in particular, for hosting any database was shared among 5 different UN agencies: UNICEF (6), DPKO (3), OHCHR (1), UNHCR (1) and ILO (1). There is considerable variation in the structure of the Task Forces and the way in which responsibilities are allocated. 6 countries have adopted a Lead or Focal Agency approach with responsibility for each violation being given to different agencies. In practice the situation is often less clear cut, with some partners much more active than others and in some countries it appears that most of the work is done by the two or three core agencies. However the Lead Agency approach can make it more difficult to create an integrated information system, especially if each organisation uses its own in house systems to record data. 3 countries have, or are in the process of establishing, a decentralised structure with regional Task Forces and this will have additional implications for information management. The monitoring process itself involves three distinct stages receiving the initial alert, documenting the case and finally verifying it. The complexity of the information system will be dependent on the number of agencies involved in each stage and how this process is organised. The responses indicate considerable diversity with between 2 and 10 agencies involved in receiving the initial alerts, between 1 and 7 in documenting the incidents and between 1 and 5 agencies with responsibility for making verification decisions. This means that some countries have a highly centralised system with just one agency having responsibility for all three tasks, while in others there are multiple agencies involved in each of the processes. The questionnaire also attempted to quantify the caseload being handled by each MRM Task Force, but many of the responses were incomplete and some were contradictory. This may have been because the information system was unable to provide this kind of quantitative data, but the wording of one of the questions may also have been at fault. However the answers that were provided indicated that the number of cases varied enormously from just a few hundred to many thousands. Clearly it is extremely difficult to manage a caseload of many thousands without some kind of electronic system in place, but at the other end of the spectrum it will be hard to justify the cost of a database for just a few hundred records. In some countries, it is possible to collect detailed data about each child that has experienced one of the four individual violation (i.e. abduction, killing and maiming, recruitment and sexual violence), while in others, access and security problems mean that only very limited incident data can be collected (e.g. just the number of children involved). Irrespective of the consequences of the diversity 19