CCVRI Help Desk Request & Response

Transcription

1 CCVRI Help Desk Request & Response Beneficiary Feedback in FCAS Finance, Performance and Impact Department, UK Michelle Spearing, Tim Midgley, Isabella Jean Date: November 2013

2 Help Desk Request & Responses for the CCVRI Improving Measurement in DFID Crime, Conflict & Violence Programming This document is a Help Desk Response to a request by a DFID country office to the Help Desk developed under the Conflict, Crime, and Violence Results Initiative (CCVRI). The full set of Help Desk Responses are intended to support DFID country offices and their partners to develop better measures of programme results in difficult conflict and fragile environments. DFID recognises the need to focus on the results of its work in developing countries. To this end, DFID strives to account better for our efforts on behalf of UK taxpayers, offering clarity regarding the value and impact of our work. The Results Initiative operates under the assumption that we will achieve our development objectives with our national partners more effectively if we generate collectively a clear picture of the progress being made. Within DFID, the Conflict Humanitarian and Security Department has established a partnership with a consortium of leading organisations in the fields of conflict, security and justice to develop more effective approaches to the use of data in the design, implementation and evaluation of programmes that contribute to reducing conflict, crime and violence. In addition to producing these Help Desk Responses, the consortium will issue a series of Practice Products that address a variety of substantive areas to improve measurement of results in local contexts. The Help Desk can be accessed by contacting helpdesk@smallarmssurvey.org. The views expressed in this Help Desk Response are the sole opinions of the authors and do not necessarily reflect the opinions of all consortia partners. This Help Desk Response does not reflect an official DFID position. Members of the consortium 2

3 Help Desk Request Background: Beneficiary feedback is about listening to and responding to the beneficiaries of programmes. DFID is aiming to put in ways of ensuring that they and their implementing partners (those they fund) are able to hear from beneficiaries. DFID aims to clarify expectations for staff and our partners, addressing the questions of when and how to do beneficiary feedback across types of programmes, aid instruments and contexts. It is important to look at how to do beneficiary feedback in fragile and conflict affected states (FCAS) because increasingly DFID programmes are operating in these contexts and additional factors may need to be taken into account when designing and implementing beneficiary feedback mechanisms in FCAS contexts. FCAS countries sit on a spectrum from those in active conflict to those where political and civil rights are more fragile. They cover a range of very different political, social, economic, and conflict- related contexts and vulnerabilities. DFID s hypothesis is that the challenges across these states (and into developmental contexts) are the same (see DFID How to Note on Beneficiary Participation) but that the base or starting point is different, so what can be achieved is likely to differ. Factors that are likely to be important considerations in FCAS are, for example, risk management, state building, political space and risks associated with participation, and elite capture. These are important considerations in any context but could be even more important in FCAS, for example, not undermining accountability between a nascent state and its citizens. Specific request The request is to produce a short think piece (3-5 pages) that addresses the question: how best to do beneficiary feedback in fragile and conflict affected states? This should draw on evidence and expert opinion to explore the following areas: What are the impacts and value (e.g. opportunities to measure/verify results) of beneficiary feedback in FCAS? What does the evidence tell us? How have other agencies attempted to do this in FCAS? What are the key considerations for designing and implementing beneficiary feedback processes in FCAS, building on the ones in the HTN on beneficiary monitoring and adding considerations of specific relevance to FCAS, and identifying a series of questions programme teams and implementers can ask to help design the process What skills and capabilities do staff need to do this e.g. conflict sensitivity, power analysis? Working with others this could be a challenge or a skill or an expectation of our partners. There are some specific challenges we face working with others in FCAS as our own ability to get out, understand the context etc. is fairly limited so how do we set up beneficiary feedback processes in such a way that we have security we are hearing from real beneficiaries, and that opportunities have not been captured by elites or the views heard do not contain How do we ensure rigour and reliability of the process and evidence; and manage risks? Are there key experts on beneficiary feedback in FCAS who could offer support? What key resources exist to guide beneficiary feedback mechanisms in FCAS? Project Deliverables A 3-5 page think- piece for sharing internally within DFID to spark debate around beneficiary feedback in FCAS and inform content on DFID s intranet to support DFID staff in this area. 3

4 Help Desk Response Why gather and respond to beneficiary feedback in fragile and conflict affected states (FCAS)? The reasons for establishing beneficiary feedback systems in most international organizations fall under four overlapping categories. They aim to improve: 1) accountability; 2) effectiveness/program quality; 3) satisfaction of donor and media demands and; 4) build trust and positive relations with communities 1. Programmes operating in FCAS, whether directly intended to address conflict or not, face challenges under each category which can undermine their ability to achieve impact whilst also potentially contributing to unintended negative impacts. However, appropriate beneficiary feedback systems can also help address the specific challenges of programming in FCAS, thereby contributing to better and more sustainable programme impacts. Beneficiary feedback can also be an important component of conflict sensitive approaches which should be applied to all programming in FCAS. Conflict sensitivity is a deliberate approach to minimise negative impacts and maximise positive impacts of an organisation s presence and activities on broader conflict in that operating context. Its underlying premise is that any intervention in a conflict context becomes part of that context and inevitably has an impact on conflict dynamics. These can be both positive (reducing conflict or tensions) and negative (exacerbating conflicts or tensions). Approaches range from a minimalist concept aimed at limiting harm, to a maximalist concept which explicitly seeks to address conflict drivers. A conflict sensitive approach requires: 1) Analysis of conflict; 2) Analysis of the organisation s presence and operations and how they interact with conflict; 3) Adaptation in order to minimise negative impacts and maximise positive impacts on conflict; 4) Monitoring impacts on conflict, on- going review and adaptation if necessary. Conflict sensitive approaches also ensure that programmes are prepared for and able to adapt to conflict if and when it flares up, thereby safeguarding development gains, and contributing to staff security and well- being. Accountability to affected populations Insecurity often restricts access for donors and implementing partners to communities in FCAS, making it difficult to develop and sustain positive engagement with communities, achieve accountability and respond to the needs and opinions of affected populations. Beneficiary feedback systems intentionally tailored to the context can help programmes to elicit useful, representative and accurate feedback from affected communities without strong presence on the ground. Using community voices to inform planning and implementation can help ensure that programmes are relevant to genuine community needs and perspectives, are appropriate to the context and are able to adapt as circumstances change or as their understanding of the local context improves. Boosting Conflict Sensitivity: Beneficiary feedback mechanisms are a useful way to build a programme s conflict sensitivity, essential for accountability in FCAS. By actively seeking feedback on interventions in relation to the conflict context, an organisation is better placed to capture and respond to unintended impacts of programmes on inter- group and intra- group relations and avoid doing harm. As the starting point for a conflict sensitive approach, context analysis needs to include the voices of local people who are better placed to identify indicators of conflict or peace in their communities. Community members can also engage in monitoring activities to ensure that ongoing implementation is responsive to changes in the local context and to flag unintended impacts that the programme may have on conflict. When monitoring and evaluating the success of efforts towards conflict sensitivity, seeking in- depth, qualitative beneficiary feedback can help establish causality by articulating how specific actions led to specific outcomes whether intended or unintended 2. In FCAS, beneficiary feedback mechanisms can also help ensure that programmes maximise potential peacebuilding impacts, for example, by fostering a culture of participation and accountability across different groups. Feedback systems also give agencies channels through which to communicate the reasons for programming decisions and adaptations back to communities and other stakeholders, which can help address tensions. Conflict sensitive approaches based on feedback from across communities can help ensure that programmes do not exacerbate underlying tensions by bolstering elites, entrenching existing power structures or undermining local 1 CDA Collaborative Learning Projects, Feedback Mechanisms in International Assistance Organizations. August Monitoring and Evaluating Conflict Sensitivity: Methodological challenges and practical solutions. CCVRI Practice Product. CARE and CDA

5 capacities and initiatives. The Conflict Sensitivity Consortium 3 reported that in situations where elites control decision making or community decision- making processes are co- opted by political or armed groups, creating safe spaces for communities to discuss and take collective decisions in an open and transparent manner has contributed to reducing the negative influence of conflict actors and to reducing tensions 4. Providing opportunities for informal and individual feedback, for example during monitoring visits, as well as formal and group feedback can help to validate data collected from formal consultations and mechanisms as it provides a channel for unstructured communication. This may help identify instances where marginalised voices have been suppressed within a community. Furthermore, inclusive feedback systems can and should incorporate voice of elite or powerful groups as well as direct beneficiaries. Without this there can be a high risk of disruption to activities as well as security risks for staff, partners and project participants. For example, CARE s work in Nepal was sometimes disrupted by Maoist groups but engaging these groups at local levels helped CARE staff understand that the reasons for disruption lay in lack of clarity and transparency about project aims and activities which led to suspicion as well as failure to take into account Maoist activities (for example a training was blocked because the timing clashed with a Maoist meeting that community members were expected to attend). Applying a Do No Harm approach to conflict sensitivity helped overcome many of these barriers to programming 5. Strengthening the Social Contract: Improving the accountability and responsiveness of legitimate duty bearers to the needs of local people should be a core objective of all DFID funded development interventions in FCAS. 6 A key challenge for development actors in many FCAS is building or strengthening the social contract between citizens and the state. Effective feedback loops are one means for enhancing social accountability, and should aim to influence government institutions to be more responsive and accountable to communities and citizens. In FCAS, communities often have little experience or expectation of holding duty bearers to account for services or resource allocations. Modelling of accountability through aid programmes can empower communities who have had little or no say in how priorities are set and programmes implemented. 7 This can contribute towards a gradual culture shift towards participation, inclusion and an expectation of transparency and accountability, all of which are likely to be lacking in FCAS. Likewise, donor agencies can use available leverage to feed community perspectives, voices and choices into coordination and planning meetings with government officials modelling their commitment to accountability and responsiveness. Such support for local capacities for positive change can lay foundations for governance- focused peacebuilding and complement statebuilding processes although it is important to be deliberate in these efforts and avoid undermining the emergence of a social contract between state and citizens. It should not be assumed, however, that governments in FCAS never listen to citizens or document feedback and efforts should be made not to undermine existing local accountability mechanisms. In Sri Lanka, a bureaucratic state affected by conflict and social divisions, provincial government offices record requests and feedback received from communities. An example from 2007 recorded by the Listening Project 8 noted that local people walked for miles to file requests for improvements in public transportation among other issues at provincial level but UNDP and some bilateral donors who wanted to fund improvements in local transportation worked only with the national level Ministry of Transportation, never asking if there was data from the provinces. Ministries at the capital have fallen into a particular rhythm when working with external donors don t ask, don t tell. Similar examples were documented in the Kenya Listening Project report. In these cases, donors and implementing agencies can take greater steps to support nascent accountability mechanisms and work with the government to improve feedback collection, analysis and response. 3 The Conflict Sensitivity Consortium was made up of 35 agencies working in 4 countries to gather and consolidate best practise in integrating conflict sensitivity into programming. The consortium was funded by DFID from How To Guide. Conflict Sensitivity Consortium 2012, Available at: to- guide- to- conflict- sensitivity 5 A Review of CARE Nepal s Use of the Do No Harm Framework, CARE and CDA Available at: Review- of- CARE- Nepals- Use- of- the- Do- No- Harm- Framework- Full- Report.pdf 6 DFID (2012) Building Peaceful States and Societies: A DFID Practice Paper, p6. peaceful- states- and- societies.pdf 7 Featherstone, A. (2013) Improving Impact: Do Accountability Mechanisms Deliver Results? A joint Christian Aid, Save the Children, Humanitarian Accountability Partnership report

6 Improved Effectiveness, Ownership and Sustainability FCAS are characterised by complex social dynamics and hidden alliances and tensions. It is often difficult to identify social and political fracture lines during initial assessments, and key conflict dynamics may only come to light once programmes have become established within a community, and a certain level of trust and familiarity has been achieved. FCAS contexts are dynamic and can change rapidly. Interventions are likely to need to adapt in order to stay relevant to the turbulent conflict context. Additionally, as noted above, interventions may have unforeseen impacts on conflict, whilst communities in FCAS often have little experience of participating in the design or implementation of development initiatives. Adaptability and responsiveness to feedback from affected communities is therefore essential to achieving effective and conflict sensitive programming in FCAS. Beneficiary feedback mechanisms can improve the quality and effectiveness of programming by ensuring that implementation is based on on- going community dialogue and context monitoring. It can also provide necessary real- time feedback on the nature and relevance of the outcomes of the program while helping to identify opportunities for programme adaptation and evolution 9. Feedback mechanisms should be seen as a central component of programmes. Beneficiary participation at all stages of the programme cycle should contribute to increased sustainability of impacts through improving the relevance of programming to evolving circumstances, minimising unintended impacts that may prevent core goals being achieved, obtaining a higher level of community empowerment and ownership and ensuring that programme design and implementation is locally appropriate. For example, involving beneficiaries in designing monitoring systems and establishing indicators can increase positive development outcomes, as seen in comparison of standardised scorecards against participatory scorecards in education in rural Uganda 10. The risks of diversion of programme resources and corruption are often greater in FCAS compared to other contexts, exacerbated by the lack of strong NGO and donor presence at implementation sites. Diversion presents both a challenge to an organisation s neutrality, but can open it up to legal threats from donor governments where aid ends up in the hands of proscribed groups. Strong anti- corruption, and anti- diversion measures are therefore critical for organisations to remain operational for long in FCAS. Beneficiary feedback can help to reduce corruption, by providing opportunities for anonymous reporting of financial irregularities, misconduct by staff or community based leadership and increasing transparency within programme management. Furthermore, by helping to ensure conflict sensitivity, beneficiary feedback can improve cost- effectiveness as it is generally better value for money to prevent and respond to conflict before it escalates and before it disrupts programming. Respond to demands from donors and media There is often a high degree of public and media interest in how aid is delivered in FCAS, particularly humanitarian aid. The media often report cases of aid diversion, misspent funds and conflict resources. At the same time, private givers increasingly demand more information about how donations are spent and what impact they have. Beneficiary feedback mechanisms can provide material to satisfy this legitimate interest, by gathering a range of perspectives on aid effectiveness, ensuring transparency and providing examples of effective (or deficient) programming, which can help satisfy media scrutiny of aid in FCAS. In humanitarian programming, beneficiary feedback and complaints mechanisms have become the norm, driven in part by the high degree of public scrutiny and media interest. Beneficiary feedback helps humanitarian agencies convey the human dimensions of the humanitarian imperative, demonstrate efforts to avoid the potentially huge unintended impacts of their work on FCAS contexts and mitigate potential backlash against humanitarian aid. Initiatives such as SPHERE and the Humanitarian Accountability Partnership (HAP) have driven the efforts to strengthen the two- way communication and feedback loops in humanitarian response. Donors and their partner agencies are under increasing pressure to demonstrate results. Beneficiary feedback provides additional and critical data by which to verify results of programming. In FCAS, where access to project areas may be limited, this data is an important mechanism by which to demonstrate impact where conventional 9 Monitoring and Evaluating Conflict Sensitivity: Methodological challenges and practical solutions. CCVRI Practice paper, CARE and CDA we- know/impact- and- results/management- and- evaluation- in- ugandan- primary- schools- impact- evaluation/ 6

7 monitoring systems may not be feasible. Innovations in beneficiary feedback systems that capture data remotely can overcome some of the barriers to demonstrating results in these circumstances. Beneficiary feedback is also particularly good for capturing qualitative data that can provide a more complete picture of change processes and the impact they have on people s lives than quantitative indicators. In FCAS contexts, there is less likely to be consensus on the best ways to achieve change and address development challenges. Qualitative data captured through beneficiary feedback systems can add to the knowledge base around change processes, additional factors that need to be taken in to account in FCAS contexts and areas where further research and learning is needed. Improve Trust, Relationships and Security for Staff and Partners Perception has a considerable impact on the quality of operations as well as on safety of both national and international staff in the field, and beneficiaries. In FCAS, communities are likely to have a high level of initial mistrust in external actors, particularly where the identities of military and humanitarian actors have become blurred. This can make it harder to engage with community members, to gain accurate or full contextual information to inform programming or to forge cooperative relationships necessary for implementation. Furthermore, security threats may be high for staff and partners, often leading to greater separation between implementers and the communities they serve. Beneficiary feedback systems that empower communities and foster two- way communication may help to increase acceptance, support and cooperation by the community of programme staff and local partners (who are often the direct implementers closest to the communities in high- security settings). This is likely to increase the ability of staff and partners to implement programmes efficiently and effectively as well as improving their security 11. Staff members able to establish a close and supportive relationship with communities are more likely to have prior warning of potential violence or to be protected by communities in the event of upsurges in violence. Providing complaints mechanisms can also alert staff to discontent amongst affected populations before it escalates. Examples of why and how beneficiary feedback systems have been implemented in FCAS? Below are illustrative examples to show how and why beneficiary feedback systems can add value at each stage of the programme cycle in FCAS. These span a range of programming areas, including those focussed on addressing conflict directly as well as those primarily focused on other sectors. These also indicate innovations in obtaining and using beneficiary feedback and some additional factors taken into consideration when implementing beneficiary feedback systems in FCAS. Many of these examples involve participation by community action groups at all stages of programming in FCAS. Utilizing community action groups in conflict analysis, planning and design of responses, negotiation with security providers, implementation, monitoring progress and evaluating results can ensure greater accountability, effectiveness, and sustainability through real local ownership and establishment of a channel for two- way communication between affected communities and implementing agencies. Participatory Analysis Including local people in context analysis is key to ensuring that the findings and recommendations are realistic and locally owned, particularly important and difficult to achieve in FCAS. When local actors conduct context analysis, they are more likely to use it to inform better planning and practise within their own context. The inclusion of multiple perspectives can help to ensure that voices of those with and without power or influence are heard. This can allow analysts to challenge the dominant discourses (or accepted wisdom ) about that context. At the same time, participatory context analysis, when approached with an open mind, can challenge and transform the analysts own perceptions about that context 12. Additionally, engagement of beneficiaries in conflict analysis processes can add depth to understanding of the conflict context and the dynamics of conflict experienced on the ground. As noted above, this can help ensure that programmes are well designed to either address drivers of conflict or ensure conflict sensitivity. As part of the People s Peacemaking Perspectives Programme (PPP), Saferworld and Conciliation- Resources conducted 11 Through the Eyes of Others: How people in crisis perceive humanitarian aid, MSF See Robert Chambers, Whose reality Counts, 1997, and Participation for Development: Why is it a good time to be alive,

8 participatory conflict analyses in 17 countries. Another approach, Making Sense of Turbulent Contexts (MSTC), pioneered by World Vision brings together a broad range of stakeholders in country (including possible beneficiaries through partner agencies and local civil society) to analyse drivers of conflict and identify key strategic needs. An analysis of PPP and 50 MSTC analysis processes found that participatory analysis resulted in greater ownership of findings by local people, and more realistic and sustainable recommendations 13. In these cases participatory analysis was a first step in engaging affected communities throughout the programme cycle. Implementation Beneficiary feedback systems can by an integral component in achieving project goals in programming in FCAS. Where programming is aimed at building accountability, good governance and a social contract between state and citizens, for example, instituting community monitoring and feedback systems are a means to achieve core goals. In Brazil, a pilot programme has developed a phone app which is allowing police to record their movements and film their interactions with citizens in the favelas of Rio when on patrol. The film is transmitted to a central command and can promote accountability 14. In Haiti and Nepal, CARE has used Community Score Cards to generate dialogue between communities and service providers to improve the quality of the service. In areas where government is either weak, absent or active in conflict a great deal of work has to be done to communicate the intentions of the activity clearly to all sides, and to train staff in appropriate mediation and facilitation techniques. In these examples, establishing the feedback mechanism is a programme activity in itself. Citizen Voice and Action 15 has been applied by WV to enable citizens to hold their government service providers to account for quality of service provision. It has been used extensively in more stable contexts but is now being piloted in South Sudan and DRC. In DRC, International Alert s Integrating Peacebuilding Approach (IPA) 16 aims to foster peace and stability by creating spaces and opportunities for all groups in targeted communities in a given context to participate in a combination of reconciliation and recovery activities, combining peacebuilding approaches with the community- driven development (CDD) model of the World Bank. The peacebuilding approach brings potential for greater impact by building community level resilience and cohesion, which may exert a positive influence on the higher levels of post- conflict processes and interventions. Emphasising participatory planning and accountability, CDD mobilises and empowers local community groups, including local government, by giving them control over planning decisions and investment resources. In so doing it contributes to social and governance outcomes, while building infrastructure assets. The focus on local management of resources and decision making in CDD programmes signifies a shift in existing power arrangements, creating opportunities for poor and marginalised groups to gain voice and control over their own development. The IPA emphasises building linkages between actors at all levels of post- conflict society, beginning with community level actors and it promotes conditions for cooperative relationships across entire post- conflict communities. Lessons from International Rescue Committee and CARE International s Community Driven Reconstruction (CDR) programme in DRC (2007), noted that CDR can serve as a valuable community component in national governance promotion programmes (such as the National Solidarity Programme in Afghanistan) but can equally threaten the vested interests of local authorities if relationships are not carefully fostered. It is important to promote citizens understanding of the need for active participation and transparent, accountable, decision making at local levels as well as social cohesion - the willingness to work together both between people and between people and their institutions. Local level (horizontal) accountability between the population, state actors and the private sector is promoted. Favourable conditions for sustained community support can be created where community development plans are drafted and promoted by community stakeholders together. Where programmes in FCAS explicitly aim to increase security, address emergent violence or respond to indicators of fragility then beneficiary feedback can provide essential information on evolving situations as a basis 13 World Vision, 2013, Bridging the Participation Gap: developing macro level conflict analysis through local perspectives. the- participation- gap.pdf voice- and- action 16 alert.org/resources/publications/ending- deadlock 8

9 for real time responses. The Kenyan National Steering Committee on Peacebuilding and Conflict Management (NSC) 17, implemented a conflict early warning system which combines media monitoring and a SMS- based reporting system (Ushahidi) allowing local committees to respond to emerging violence. This allowed affected people to engage in debates around hate speech and to alert the NSC when situations evolved and new responses were needed 18. In this example beneficiary engagement and use of media are integral components of the programming design. In Nepal the Freedom from Fear Index monitoring system was established by community groups and built around outcomes that they themselves determine. Progress against the Freedom from Fear Index is monitored and information displayed publicly to prompt discussions amongst the broader community. In this example, feedback from communities is used to generate discussions around security issues, an outcome in itself. In Georgia, Saferworld has supported community- based conflict trends monitoring, using a local SMS network that allows local partners and communities to access up to date information about conflict trends and incidents in the area and to link them up with security providers. The information is communicated back to communities and local government and used to inform activities 19 within security and justice programmes. Monitoring Saferworld has used tracker surveys to capture public perceptions on issues of conflict, security and justice in a range of contexts where public confidence in security and justice are poor, including in Somalia. This allows perceptions to be used as a measure of results in conflict, security and justice programming but can also be applied to broader governance programming in FCAS where changed perceptions can be a measure of impact or to support conflict sensitivity. They have the benefit of being able to measure intangible results and can be valuable source of additional data for triangulation in FCAS, where official data sources may be limited. Tracking responses to the same set of questions over time is valuable for identifying how conditions are changing and perceptions and experiences changing due to an intervention. Whilst challenging, it is possible to do regular surveys of public perceptions on conflict, security and justice issues in fragile contexts 20. Issues of access and inclusion and adaptations to help solicit opinions on sensitive subjects in a manner appropriate to the context all need to be taken into account when designing perception surveys in FCAS. Also important to the success of perception surveys are timing, sequencing and phrasing of questions, and stakeholder participation in design of the research instruments 21. The OECD- DAC checklist is particularly useful in considering these issues 22. Perception data should be disaggregated according to different stakeholder groups identified through initial conflict analysis in order to understand how programme impacts are viewed across different stakeholder groups. In Burundi, CARE used a system of community identification and monitoring of conflict indicators. Qualitative research methods were used, particularly oral testimonies, to understand how individuals in affected community experienced conflict. Following this, community members joined a workshop where they defined indicators of conflict at community level which could be monitored to determine the positive or negative impacts of CARE programmes on conflict. Engaging community members in the design on the monitoring system and definition of indicators ensured support at community level and ensured that the system was asking the right questions to determine impact. Beneficiary engagement in monitoring can overcome the weak field presence common in FCAS as well as achieving a sense of ownership and strong relationships with communities. In Sierra Leone, CARE partner Future in our Hands introduced community diaries 23 where there was previously no structured channel for communities to express concerns or views during implementation. After training, project participants documented sensitive issues or tensions arising during implementation. These were reviewed during monthly meetings between project participants, partners and staff and used to support collective decision making of changes to the project. This helped to track project impacts and identify unintended consequences using a mechanism that was easy for the community to understand and was conducive to ongoing two- way 17 Made up of the Government of Kenya, local and international civil society and UN agencies 18 cc 19 Understanding and responding to security needs in conflict- affected areas - p resource/ Perception surveys in fragile and conflict- affected states, GSDRC for DFID, Checklist to commission, design and run perception surveys, OECD- DAC, Conflict Sensitivity How to Guide, CARE et al..p19 9

10 communication and decision making. At the same time it did not require constant presence by staff or partners. Greater trust and openness by project participants was reported and changes made to activities helped to reduce tensions and increased the effectiveness of the intervention. World Vision s in Darfur, operates a Community Help Desk (CHD) in all camps where World Vision distributes food assistance, with focal points selected by a committee in each IDP camp to solicit and document feedback from camp residents. CHD focal points are provided training and use logbooks to record requests, complaints and feedback provided by aid recipients during food distributions. In addition there are suggestion boxes in all supplemental feeding sites, and informal processes for soliciting feedback during community meetings and post- distribution monitoring visits. 24 Evaluation Saferworld has supported use of community security action plans in Central Asia whereby community groups work with local authorities to develop and implement, monitor and evaluate action plans. They participate in a structured evaluation process which includes assessment of outcomes, development of a sensitivity index, relationships between actors, contribution of each actor to the outcome, and an evidence box. The Most Significant Change methodology is based on a wide selection of community members participating and sharing their personal reflections on the greatest impact of the programming. This is then prioritised by the evaluator and passed on. Importantly, the testimonies are shared back with the community, along with the decision on which was deemed most significant by the evaluator. Save the Children Norway documented the use of Most Significant Change tools to capture the impact of programmes focused on children affected by armed conflict in a range of FCAS, including Northern Uganda and Guatemala ALNAP- CDA Collaborative Learning Projects. (2013). We Are Committed to Listen to You: World Vision Experience of Humanitarian Feedback Mechanisms in Sudan Save the Children Norway, 2008, A KIT OF TOOLS FOR PARTICIPATORY RESEARCH AND EVALUATION WITH CHILDREN, YOUNG PEOPLE AND ADULTS of- tools.pdf 10

11 Key considerations for designing and implementing beneficiary feedback processes in FCAS In FCAS, a number of key considerations need to be taken into account when designing and implementing beneficiary feedback systems. These include: Ensuring conflict sensitivity within beneficiary feedback systems themselves In FCAS, a beneficiary feedback system could itself exacerbate divisions in society or heighten grievances and mistrust of authorities if not well designed. The beneficiary feedback system itself therefore must be designed and implemented in a conflict sensitive manner. It is critical that the design is informed by a solid understanding of the various divisions and power dynamics in the community/country, and how these affect access to, and distribution of resources. It is therefore essential that the process of establishing a beneficiary feedback system is informed by conflict analysis and that conflict analysis is used to identify who needs to be included and how and when to engage beneficiaries as well as non- beneficiaries, who are stakeholders if not directly gaining from a programme. It is for example important that the views and perceptions of both those with power (elites) as well as those without power (marginalised groups) are actively sought out and included in a balanced manner. The process used to develop a conflict analysis can also help to build links and communication between donors, government and/ or implementing agencies and local communities, for example through the use of participatory conflict analysis. In situations of heightened mistrust and polarization, gathering and recording information and sharing it with other entities (as in referral of feedback within the cluster system or through shared feedback practices with government authorities) may be viewed with suspicion by segments or majority of the population. Clear communication on the aims and processes of a beneficiary feedback system is particularly important to counter this. Furthermore, people s anonymity and confidentially need to be protected and attention needs to be paid to how feedback channels are perceived, who uses them for what purposes and what alternative (often informal) ways are being used for communicating feedback outside formally established channels. In FCAS, the context can change rapidly, For example, key people in communities and government can move positions with little notice, activities can be disrupted by unforeseen events, upsurges in conflict or natural hazards can result in suspension of activities or displacement etc. Context analysis (including conflict analysis) should be regularly updated. Scenario and contingency planning is also important. Input from affected populations is needed at all stages. The recent ALNAP- CDA Case Study concluded that in Darfur, as in many other humanitarian settings, cultural appropriateness goes beyond mere recognition of existing cultural norms, gender roles and social protocol. Protracted displacement has uprooted villages and communities and has impacted the traditional leadership structures that are increasingly being challenged by young armed males who also exert their influence over aid distribution and decision- making processes in the camps. Both camp residents and aid workers routinely highlighted the importance of understanding the Sudanese and Darfur context, the dominant social norms and the shifting power dynamics and how these must be considered when designing an appropriate feedback mechanism 26. Broad inclusion: In FCAS it is particularly important to recognise the affiliations people have to different groups and to understand the way that aid may benefit, or be perceived to benefit, one group over another. It is therefore important to seek feedback across affected populations, from beneficiaries and non- beneficiary stakeholders to ascertain how aid may be impacting relationships between groups in conflict (whether positive or negative). It is important to ensure that the voices of both those with and without power are included in design and implementation. For example, it can be important to include both direct beneficiaries and those who are not selected as aid recipients or project participants, identification of key activities and setting of monitoring indicators. There should be clear criteria for selection of participants of projects and in the feedback processes. 26 We are committed to listen to you: World Vision s Experience with humanitarian feedback mechanisms in Darfur. CDA and ALNAP Case Study Available at: Are- Committed- to- Listen- to- You.pdf 11

12 Ideally, potential beneficiaries and other stakeholders should be involved in defining these criteria. Where there are restrictions on how aid can be spent, and which groups are targeted, these should be clearly communicated to local communities, both direct beneficiaries and non- beneficiary stakeholders Using community feedback tools means you will almost certainly come into contact with actors who are, in part at the very least, responsible for some of the violence you are trying to address. Dangers include conferring legitimacy on such actors if you are seen to be working with them openly. However, this risk must be balanced with the reality that peacebuilding requires building difficult relationships and bridging between otherwise estranged sections a community. Ensuring broad participation of groups can help to ensure that voices are representative of the community perspectives. Being able to distinguish between voices of intended beneficiaries and voices of elites or dominant groups is crucial. A robust conflict analysis should help identify some of these actors. Furthermore, as programmes and staff develop deeper understanding of the various social and political dynamics within a community, they are likely to develop a deeper understanding of these dynamics, and become better able to manage them. CDA s recent desk review on beneficiary feedback mechanisms noted ubiquitous and real concerns with elite capture and challenges with ensuring inclusiveness when designing transparency and feedback mechanisms 27. Document review and interviews with practitioners indicate that even agencies that have put in place a variety of feedback processes continue to struggle with ensuring active participation of marginalized people due to cultural, political and security constraints related to physical access and level of transparency. Ignoring political context has been recognized as a weakness of participatory M&E, which is often seen to assume that beneficiaries are free agents, operating without a biased opinion or unencumbered by personal agendas, merely missing a regular opportunity to provide feedback. In particular, minority groups, youth and women may not feel able to respond openly just because they are provided with a feedback mechanism. Creative ways to build trust and empower different groups to respond are needed to solicit meaningful feedback. An inclusive process that is based on a though understanding of local agendas and power dynamics requires a level of presence and/or close and trusting working relationship with a local partner organization that is supported and encouraged to relay both positive and negative developments and feedback. Unfortunately, in the context of funder- recipient dynamic, this is not always possible. As CDA s case studies on application of Do No Harm demonstrate, some local implementing partners and national staff of international NGOs are afraid to report negative developments up the chain of decision making due to fear that both funding and programs will be stopped and their jobs will be eliminated (DNH Case Study in Afghanistan and Nepal). Beneficiary Feedback Systems and New Technology Use of new technologies and media in FCAS demonstrates the need to conceive beneficiary feedback systems holistically, taking into account both the actual mechanisms used to gather data and the analysis and design needed to frame research questions and approaches as well as analysis and interpretation of data collected. Strong analysis is needed to inform the design of the feedback system and ensure that it asks the right questions, in the right places, of the right people and in an appropriate manner to elicit open responses. This applies regardless of whether the mechanism is a phone hotline, SMS response, use of mobile apps or more simple low- tech scorecards. Furthermore, sufficient time must be allocated to the interpretation of the feedback obtained, so that it can be used to develop a better understanding of the complexities within the community and programme. New technologies have the advantage in FCAS of allowing real time collection and communication of data in difficult circumstances and the possibility for a rapid response. It can also reduce the risk of data loss in insecure situations if data is transferred electronically. A large number of NGOs and organisations have been investing in using new technologies to improve feedback mechanisms in FCAS. However, there are limiting factors in the usefulness of new technologies including: Use of technology can raise suspicions in communities due to fears that electronic media could be monitored and result in a backlash against dissenting voices or simply result in respondents locations and personal details being accessible to authorities. Access to communications technology can be deliberately blocked. 27 CDA/ITAD Desk Study for DFID s Beneficiary Feedback Pilot Programme. (Available upon request) 12

13 Lack of access to mobile phones, computers and other media in some communities and societies due to poverty and cultural constraints (such as women in tribal areas of Pakistan not being allowed to use mobile phones). This can be particularly limiting in FCAS. Statistics sometimes fail to recognize multiple or family ownership of resources like mobile phones as well as the gaps where phone ownership is low. There is high potential for falsified information given the impersonal nature of data collection, which can undermine the assumption that ICT lowers information asymmetries There can be a danger of reductionism, if open- ended questions are not used. Gaining and processing qualitative feedback through new technology is challenging. However, qualitative feedback is particularly important in FCAS. Civil society efforts such as those utilizing Ushahidi software can only be effective if given adequate institutional support. The process cannot be sustained entirely by volunteer energy. Unless there is greater clarity regarding the kind of feedback being solicited, there is risk of the system being overwhelmed with messages. Danger that the technology platform is seen as the solution to beneficiary feedback challenges rather than a tool to be used within beneficiary systems which also require other types of community engagement and trust building to ensure participation and to validate data and avoid elite capture. The human dimension should not be neglected. Feedback mechanisms should be a core component of programming, built in from the outset in a structured, accessible and appropriate manner. Feedback mechanisms for beneficiaries and the broader community are needed at all stages of the programme cycle and indeed formal beneficiary feedback mechanisms may be more effective if designed on the basis of initial community consultation to reflect both impact indicators identified by communities (the data being sought) and appropriate mechanisms for collecting this data in the social, cultural and conflict context of that community (the way in which data is sought). Regular, simple and cost- effective participatory monitoring processes such as feedback forms or scheduled semi- structured interviews or group consultations can ensure the planned theories of change and program strategy continue to be relevant and applicable throughout the programme duration. Informal ways to gather feedback should complement structured methods, both to allow validation though different data sources and to ensure that both solicited and unsolicited feedback can be captured and used. For example, suggestion boxes, SMS feedback, or permanent community forums for dialogue between stakeholders are all important to permit grievances and adaptations as necessary. Combinations of formal and informal mechanisms for capturing data should be identified from the outset, taking into account cultural norms (such as oral feedback over written) and conflict or security concerns (such as allowing for minority voices to be heard without fear). 13

14 Staff skills and institutional capacities to enhance beneficiary feedback systems in FCAS Beneficiary feedback systems cannot work effectively without skilled staff and appropriate institutional structures and processes to support them. Conducting focus group discussions and key informant interviews in a FCAS context requires additional research and facilitation skills. For example, to help participants discuss sensitive subjects related to conflict, overcome fears around confidentiality, and anticipate and manage the impacts of trauma on participants abilities to engage. Specific qualitative research methodologies, such as oral testimony, can be useful. Ability to conduct conflict, power and stakeholder mapping sessions and capacity to obtain and analyse information through these lenses is essential. Facilitators should be aware of and trained to manage potential disputes or conflicts between participants which are common when discussing conflict. Facilitators also need to demonstrate a commitment to and nuanced understanding of participatory processes. Participatory tools need to be carefully facilitated to ensure that they capture the true views of participants. They require the expert leading the process to leave behind the role of chief analyst, and instead facilitate a process that draws out rich contextual knowledge of insiders. In FCAS, this becomes particularly important. Staff members soliciting feedback from communities are not neutral in the context. They require a capacity for self- reflection and awareness of their own identity and how they are perceived by the affected populations. With local partners and local field staff likely to be soliciting beneficiary feedback, it is important that they are aware of their own position in relation to conflict. Where international staff are involved, it is important to understand how outsiders are perceived and what signals are communicated that may be misinterpreted with regards to alignment with particular power interests. Staff members require in- depth knowledge of the community to ensure the right mix of participants from different parts of the community and to understand when groups should be consulted separately due to power dynamics that may limit marginal voices or prohibit free discussion. This challenge is even greater in FCAS, due to the contested nature of the issues being discussed. This can be particularly difficult in contexts with limited access. Failure to ensure the right balance can lead to inconclusive or prejudiced findings or exacerbate conflict by resulting in unbalanced recommendations. An ability to build relationships based on trust and openness is essential. Without trust, feedback will not reflect the true feelings of the beneficiary community. This trust emanates from a belief by the community that service providers actually have the capacity and sensitivity to address the problems they might share as well as respecting confidentiality A flexible mind- set in staff and flexible structures and institutional processes are needed to allow adaptations in programming based on accumulated feedback and other supporting evidence. Without this, feedback mechanisms can be dismissed as useless if people who have provided input on program quality continue to see no response, no action/ no significant modification. Where new technologies are used, staff members need confidence in both the technology itself and the supporting research and analytical skills that ensure the technology is useful. There is an ever- present gap in institutional capacity to analyse and make the decisions on the basis of data obtained. Staff members need support to make sense of qualitative data that arrives through regular monitoring and feedback processes, to aggregate it, track trends, report it in an appropriate manner for decision- makers to be able to digest and utilize the feedback. This helps close the feedback loop rather than remains with a series of anecdotes or complaints for the record or for learning purposes but does not lead to changes in programming. It is helpful for institutions to model the culture of feedback internally. Are senior managers and expats truly open to feedback from field level staff and national staff? Do they solicit it regularly and are they open and responsive to unsolicited feedback? Do they demonstrate accountability and transparency in decision- making internally? National staff members are from FCAS contexts too and may be informed by the same expectations as community members. There has to be a matching institutional environment for them to replicate. 14

15 Working with others in beneficiary feedback systems in FCAS: Opportunities and challenges Donors and implementing agencies are more likely to need to cooperate with others in FCAS as the ability of any single agency to obtain adequate information to underpin programming or ensure security is reduced. Peacebuilding in- particular requires collaboration. First, because the scale of the endeavour requires partnerships and coordination in order to have impact at any significant scope and scale. Second, because peacebuilding requires a multi- dimensional approach beyond the reach of a single entity. Third, because solutions to societal problems must be applied by people and organisations within conflict- affected societies to define and achieve goals in partnership with external supporters 28. Collaboration, coordination and coherence between initiatives are encouraged by key peacebuilding frameworks and approaches 29 and can extend to beneficiary feedback where this is undertaken jointly or where findings are shared. Furthermore, recognising the multiplicity of different efforts and initiatives with intertwined activities and impacts that may add- up to peace requires specific enquiry into the relative contribution of single initiatives. In- depth, qualitative data gained from beneficiary feed- back can help to define the contributions of specific activities relative to others in a dynamic context. Whilst partnerships are necessary, adequate due diligence needs to be conducted before entering in to partnerships and the linkages of any partners with politics and power should be monitored. Political dynamics within conflict contexts are fluid and often fast moving, for example, leaders with political influence often lose it quickly and civil society leaders can easily become embroiled in political dynamics. Ongoing dialogue with partners and monitoring of their political engagement is necessary. At the same time, maintaining an independent profile and transparency through clearly communicating programme goals to both affected communities and powerful stakeholders can help mitigate these risks. Forging collaborative relationships with a range of community, civil society and government stakeholders can also dissipate these risks. A deep knowledge of the politics within civil society organisations as well as political parties and agencies can help navigate potential problems. This is a significant challenge that requires constant monitoring. There are undoubtedly repercussions when empowering certain members within a community, and mitigating them depends on planning a well- balanced working group from the outset that reflects the wide variety of needs within any community. The question of who determines what success looks like, and consequently who holds the power to shape interventions to achieve the desired change is particularly critical in conflict affected contexts. It is important to have a deep knowledge of the potential biases and political economy within civil society, especially if relying on local partners to facilitate and manage these processes. It must be noted this is not an easy process. An intervention can itself exacerbate differences and rivalries between groups if not managed well. In addition to efforts to mobilise and engage different groups by identifying the marginalised, it is necessary take an empowering approach to enable those people not just to attend a meeting or use a feedback mechanism but to recognise, articulate and communicate their own needs and experiences as distinct from those of elite groups. 28 Programming framework for International Alert: Design, monitoring and evaluation. International Alert on- peace- practice/ 15

16 How to ensure rigour and reliability of the process and evidence and manage risks Ensuring rigorous process and robust data is a particular challenge in FCAS. Limited direct access to communities, complex political and social dynamics and weak or corrupted formal and informal institutions make it particularly challenging for development agencies to directly monitor community engagement processes. The risk of processes being captured or manipulated to benefit certain groups is very significant in FCAS. This can result in data that does not accurately reflect needs and priorities across different groups in society and particularly the poor and marginalised, which in turn can exacerbate tensions, and contribute to increased conflict within or between communities. In order to mitigate against these risks, an number of approaches can be applied: Do not rely on a single feedback channel: Ensuring a diversity of communication channels between local people and programmes helps to broaden the range of people engaging with the programme. It is also important not to undermine existing ways in which people communicate their views. Structured and unstructured feedback channels should be used to capture both solicited and unsolicited feedback. Develop confidential channels: These can allow local people to feedback without fear of backlash or reprisal. These can be particularly important for identifying corruption or abuse by actors closely associated with the programme, such as staff or community leaders. Be sensitive to how promoting feedback may be perceived by communities: A requirement to gather data such as phone numbers may be viewed with extreme suspicion in contexts where phones mean drones 30, for example in Pakistan. Use of staff from outside the community may be a problem if they are identified with different groups. Conversely, using staff or researchers from within a community may lead to them being viewed with suspicion. Negative perceptions of beneficiary feedback processes can reduce their usefulness if it limits respondents openness and can potentially put NGO staff at extra risk in sensitive areas. Maintain an independent profile and ensure adequate communication about aims of data collection: It is important that programmes are not seen to be taking sides with any particular party to a conflict, or be seen to be allying entirely with a particular partner. Clear and open communication with communities about the aims and processes involved in collecting feedback can avoid misunderstandings. Ensure adequate time and resources are set aside to analyse data gathered from communities, and assess trends over time: Data gathering is only one part of the system. Significant human resources must also be put into analysing data and making decisions to respond on the basis of new analysis. Ensure data is triangulated and validated: Procedures for verification of sensitive feedback and linking to other monitoring processes for triangulation/additional data gathering is essential. Feedback arriving sporadically or anecdotally, for example, can make a compelling case for programmatic changes when aggregated over time and supported by additional evidence from monitoring reports or specially commissioned surveys 31. It is also crucially important to focus on closing the loop. It is important that communities feel that their feedback is effective and results in action. Failure to respond to community feedback can result in community disengagement and perceptions of programme activities as being externally driven. In FCAS this can have serious implications. At best it may result in apathy towards programme activities and staff, at worst it can provoke anger, hostility and violence. This has proven to be a significant challenge for implementing agencies. It is significantly harder to respond to feedback than it is to gather it and attention has focussed more on gathering feedback rather than ensuring individual and organisational capacities to respond to it 32. In addition, it is important to seek ways to solicit and utilize feedback for changes beyond minor modifications at project implementation level. What is sorely missing in aid programming are examples of effective feedback utilization to inform strategies and policies that have significant impact on how on- going and future programmes and projects will be designed and implemented. Aid agencies, those implementing and those funding aid efforts in FCAS, require better systems for aggregating feedback and making sense of it as it relates to broader decisions taking place on how to best support positive change in these societies. 30 Study of the Impact of Counter Terrorism Measures on Principled Humanitarian Action, Kate Mackintosh and Patrick Duplat Available at: 31 See ALNAP- CDA Sudan Case Study, ALNAP- CDA Pakistan Case Study 32 See Owen Barder 16

17 Experts on beneficiary feedback in FCAS Madeline Church, Head of Organisational Development Unit, Saferworld Catherine Flew, Organisational Development Unit, Saferworld Isabella Jean, Director of Evaluation and Learning, CDA Dayna Brown, Director of the Listening Program, CDA Michelle Spearing, Conflict Team Leader, CARE International UK Lukas van Trier, Governance and Accountability Advisor, CARE International UK Angela Kellett, Evidence and Accountability Manager, World Vision UK Madara Hettiarachchi, Associate Director Humanitarian Accountability, World Vision International Selected resources to guide beneficiary feedback mechanisms in FCAS Monitoring and Evaluating Conflict Sensitivity: Methodological challenges and practical solutions. CCVRI Practice Product. CARE and CDA, 2013 (Tool 9: Feedback Mechanisms) How to Guide to Conflict Sensitivity, The Conflict Sensitivity Consortium Guidance for designing, monitoring and evaluating peacebuilding projects: using theories of change. CARE International UK, 2012 CIDA s Experience in Arbitration, Mediation, Negotiation and Reconciliation. Notes for the Paris Meeting of OECD DAC Task Force Working Group on Conflict Prevention, February Post conflict programming. Prendergast, J. Life and Peace Institute 1997 Complaints Mechanism Handbook. A comprehensive guide and handbook on how to establish Complaint Mechanisms in humanitarian projects. Danish Refugee Council. ALNAP- CDA Case Studies and Synthesis Report on Effective Humanitarian Feedback Mechanisms. (Forthcoming) In the Eyes of Others: How People in Crises Perceive Humanitarian Aid, MSF 2012 Humanitarian Accountability Partnership - Views Amidst Violence: can perception surveys improve aid in fragile states? Parks, T Available at Survey research in conflict and post- conflict societies, Eck, K. in Understanding Peace Research, Routledge, 2011 Perception surveys in fragile and conflict- affected states, GSDRC for DFID,