Meta-evaluation and Synthesis of the 2010 Pakistan Floods Response by SHO participants

Transcription

1 Meta-evaluation and Synthesis of the 2010 Pakistan Floods Response by SHO participants A Synthesis of Conclusions Report phase 2 Verona Groverman Joke Hartmans 5 December 2012

2 Content Acronyms Introduction Background on the emergency situation Synthesis: background and process Synthesis of the main conclusions of the evaluations of responses to the flood disaster in Pakistan Relevance/ appropriateness Effectiveness and impact Efficiency Coverage Connectedness: Coherence Standards Cross-cutting themes: Gender and Vulnerable groups Strengthening of responses Cooperation Lessons Learnt Concluding remarks Annex 1: SHO Terms of Reference Meta-evaluation and synthesis of the 2010 Floods in Pakistan Annex 2: List of evaluation reports and other literature used Annex 3: Overview of provinces with districts in which SHO participants supported interventions and the focus or sector of support

3 Acronyms AJK ALNAP CBO CoC DRM DRR HAP IASC NDMA NFI NGO KPK PHF RTE SHO UN WASH Azad Jammu and Kashmir Active Learning Network for Accountability and Performance in Humanitarian Action Community-Based Organisation Code of Conduct (for the International Red Cross and Red Crescent Movement and Non- Governmental Organisations in Disaster Relief) Disaster Risk Management Disaster Risk Reduction Humanitarian Accountability Partnership (Standard in Accountability and Quality Management) Inter-Agency Standing Committee (international inter-agency coordination forum for humanitarian assistance involving key UN and non-un humanitarian partners) National Disaster Management Agency Non-Food Items Non-Governmental Organisation Khyber Pakhtunkhwa Pakistan Humanitarian Forum Real Time Evaluation Samenwerkende Hulporganisaties (Cooperating Aid Organisations) United Nations Water, Sanitation and Hygiene promotion 3

4 1. Introduction In July and August 2010 heavy monsoon rains swept over Pakistan, eventually causing catastrophic floods in about one third of its geographical area. More than 18 million men, women and children, i.e. 10% of the population, suffered from the effects of these floods which caused the death of almost two thousand people and damaged buildings, infrastructure and livelihood assets. SHO raised just over 27,5 million euro for this emergency: 25,5 million in donations from the Dutch public, complemented with a contribution of 2 million from the Dutch government. i The participating SHO members, Cordaid (Mensen in Nood), ICCO & Kerk in Actie, The Netherlands Red Cross, Oxfam Novib, Save the Children, Stichting Vluchteling, Tear, UNICEF Nederland en World Vision, spent these funds on housing, medical care, food, water, sanitation, education and livelihood opportunities. ii In 2011 and 2012 each of the participating SHO members commissioned independent evaluators to evaluate their responses. Building on the conclusions of these evaluations, the present report summarizes and synthesizes the answers on key questions related to twelve issues, in accordance with the Terms of Reference attached in Annex 1. A first set of six issues concerns evaluation criteria, namely, appropriateness/relevance, efficiency, effectiveness and impact, coverage, connectedness and coherence. The second set of six issues regards the compliance with internationally agreed humanitarian standards (particularly CoC and Sphere), cross-cutting themes (gender, vulnerable groups, strengthening of local responses), and organisational issues (cooperation and coordination, lessons learnt from previous disasters). Chapter 4 of this report presents the summary of combined conclusions that answer the key questions related to the twelve issues. The synthesis concerns the second phase of a three-phased meta-evaluation. It builds on phase 1, which is a quality assessment of the evaluation reports mentioned above. Phase 3 of the metaevaluation is meant for learning purposes of the SHO members and will build on the outcomes of the previous phases. For proper understanding of the information on the twelve issues presented in chapter 4, the report first briefly describes the emergency context in Pakistan during 2010 and 2011 (chapter 2). This is followed by a description of the synthesis background and process, which includes relevant information on the quality assessment process (chapter 3). In the final chapter 5 we, two independent consultants commissioned by SHO to conduct the meta- evaluation and synthesis, Verona Groverman and Joke Hartmans, give some concluding remarks. 2. Background on the emergency situation Mid July 2010 Pakistan experienced heavy monsoon rains that lasted for more than eight weeks, causing floods that evolved from normal flash floods into a massive disaster affecting large parts of the country. The floodwater waves washed down from north to south as the Indus River extended to about forty times its usual size and - at one point - submerged a fifth of the country s land mass. The floods directly and/or indirectly affected 78 of Pakistan s 121 districts, devastating and submerging entire villages, washing away houses, health and education facilities, roads, bridges, water supply and sanitation infrastructure as well as livestock and agricultural lands. Synthesis of conclusions on the 2010 Pakistan floods response by SHO members 4

5 The Pakistan 2010 floods are considered amongst one of the major disasters of the 21 st century due to the disaster s widespread geographical scale and distribution (from the Himalayan Plateau to the Arabian Sea), the unprecedented caseload of affected population and its economic impact. iii According to the Pakistan s National Disaster Management Authority (NDMA), the 2010 floods constitute the country s largest disaster iv as over 18 million persons, almost 10% of the country s population, were affected. Despite the scale of the disaster, casualties remained relatively low: 1,985 people were killed and around 3,000 injured. Initially, the provinces of Balochistan and Khyber Pakhtunkhwa (KPK) were flooded. In mid-august, as flood waters flowed south, Punjab and Sindh provinces experienced resultant widespread flooding. The impact of the flood was very diverse in each province due to the changing nature of the disaster, the different levels of preparedness and the access to individual and common resources. Waters in Balochistan and KPK receded within days; this took several weeks in Punjab and months in some areas of Sindh. Sindh was the worst affected province, as the Indus River did not find an outlet due to the flat topography of this area. Where water receded rapidly most of the displaced population was able to return during the months of August and September. However, in January 2011 four districts of Sindh and one district in Punjab were still submerged. Over 75 percent of people affected were in Sindh and Punjab: Sindh had the highest number of persons affected (7.2 million), followed by Punjab with 6 million people and KPK with 3.8 million people. Areas in Sindh province were confronted with longer term displacements and situations where most vulnerable parts of the population remained under severe difficulties from recovering. Their livelihoods could not be re established as quickly as in other affected areas due to lack of access to land. The emergency was further compounded by pre existing chronic poverty, inequality, limited access to basic services, inadequate governance, fragmentation of the state, military dominance and the superposition of conflict and displacement in KPK and Balochistan provinces. Nationwide, the floods washed out years of achievements through developmental efforts; the country suffered extensive damage to crops and physical infrastructure. The total economic impact of the floods is estimated to be around $10 billion. The impact of the floods has worsened chronic poverty and inequality, especially among the most vulnerable parts of the Pakistani population. v Some of the main reasons behind the already existing poverty levels and inequalities are of a structural nature and related to restricted access to social services and land due to a feudal system of landownership. vi Recovery of livelihood in Sindh, for instance, was impeded by economic debts to landlords. Patriarchal socio-cultural structures and religious fundamentalism hindered the effectiveness of the relief effort and the recovery capacity, particularly, that of women and minorities. Relief work was also hampered by the difficult logistical terrain, the destruction of infrastructure, government bureaucratic hurdles, lack of effective coordination and the threat of terrorist attacks. These factors made this emergency response one of the most difficult ones in recent times. vii Synthesis of conclusions on the 2010 Pakistan floods response by SHO members 5

6 Synthesis of conclusions on the 2010 Pakistan floods response by SHO members 6

7 3. Synthesis: background and process Prior to making this synthesis, we assessed the quality of sixteen evaluation reports provided by the nine SHO participants mentioned above viii. An assessment grid was used with eight evaluation criteria, namely, meeting needs (i.e. demands in the Terms of Reference), appropriate design, reliable data, sound analysis, impartial conclusions, useful recommendations and clear report. ix The synthesis draws on the conclusions provided in nine evaluation reports which proved to be of satisfactory quality. Annex 2 gives the list of these reports. All of the nine selected reports scored positive on impartiality of conclusions. These reports evaluated the interventions of six of the nine SHO members that responded to the 2010 Pakistan floods emergency. x In the search for information on the twelve issues to be addressed in the synthesis, we faced some challenges. First, in six of nine reports, the conclusions were intertwined with the findings, mostly in a clear sub-section but at times as statements in the main text of findings. This complicated the extraction of the conclusions. xi Second, due to the different foci of evaluations, in many cases the evaluations conclusions did not separately address the issues regarding the second set of issues (standards, gender, vulnerable groups, strengthening of local responses, and so on). It required a search in the full evaluation reports to get sufficient information about the issues concerned. These two factors made the process of bringing the conclusions together rather time consuming. Another, methodological issue concerns the extent to which information in different evaluations can be summarized. First, although the Terms of References of the evaluations showed overlap, each evaluation had its specific emphasis. Second, the evaluation reports showed variety in nature and focus which required caution in comparing the information provided on the twelve issues. More in particular, the role of the commissioning organisation in the delivery of the response was diverse: some agencies were directly implementing activities, whereas others worked with local or international implementing partners or the national branch of the international umbrella organisation. As regards the focus, some evaluations merely reflected on programme interventions, others also took organisational and institutional issues into account, and one evaluation focussed mainly on institutional aspects of the emergency. Lastly, a hindering factor for the whole exercise of synthesising conclusions of the evaluation reports was the limited time available (six days including reporting). The synthesis has, therefore, the nature of a summary of main conclusions on the twelve issues. It does not include an analysis of the information brought forward in the reports. In only a few explicitly indicated cases we have added our thoughts about an issue. Still, we expect that the outcome will give a good input for phase 3 of the meta-evaluation which is about learning. Synthesis of conclusions on the 2010 Pakistan floods response by SHO members 7

8 4. Synthesis of the main conclusions of the evaluations of responses to the flood disaster in Pakistan In this chapter twelve issues are addressed which, together, will provide an overview of the main outcomes of the responses of the different SHO participants included in this synthesis. Each of the twelve issues answers key questions as per the Terms of Reference of the Meta-evaluation and Synthesis. The sections start with the main answer to the key questions based on our findings, which is elaborated in the text that follows. We have put the references to the reports in endnotes (to be found at the very end of this synthesis report). We made an exception for one report, i.e. the Inter- Agency RTE. This Real Time Evaluation does not concern an evaluation of one specific SHO participant but gives an overall assessment of the combined emergency responses (UN and NGO) nationwide. It, therefore, provides interesting lessons about the relief efforts in general. The full titles of the reports can be found in Annex 2. We like to stress that the main conclusions of the nine evaluations formed the basis of the description. For some issues though we had to rely on the findings presented because most reports did not draw conclusions on the issue. This holds for connectedness, coherence, gender and vulnerable groups, strengthening of responses and cooperation. We also like to underline that the evaluations concerned six of the nine SHO members participating in the Pakistan floods response (see Annex 2). Their evaluations mostly included various projects in a number of districts, implemented by different local or international partners of SHO participants or national branches of international umbrella organisations or international member organisations of international networks to which two SHO participants belong. 4.1 Relevance/ appropriateness: Was the response tailored to local needs, enhancing ownership and accountability? xii All evaluations addressed the relevance and appropriateness of the interventions. These issues are presented and discussed at length in the findings sections of the reports, which we will not repeat in this report. The evaluators concluded for both the relief and recovery phase that the overall flood response was relevant and appropriate. Additional to our elaboration below we like to add that various reports pointed at factors which contributed to the relevance, such as the selection of interventions, the selection of targeted districts (see section 4.4), flexibility to address the needs in different phases of the emergency (more in section 4.5), the involvement of local communities in the programme, and, last but not least, emphasized by one evaluation xiii, the focus on gender and vulnerability (see further in section 4.8). Higher levels of ownership and accountability were found in those interventions which purposely included the involvement of local communities. As regards the relief phase, people were said to be satisfied with the food packages and NFI packages provided. However, two evaluations added critical comments xiv. The first comment concerned the relevance and appropriateness of the standard roll out of (four) clusters which the Government of Pakistan, donors and implementing partners unanimously questioned as follows: the clusters operated independently from contextual realities and to a large extent also to the phases of the operation (more about the issue of cooperation in section 4.10). The second comment regarded a few items in the NFI packages that were less appropriate and proper, such as blankets, knifes and Synthesis of conclusions on the 2010 Pakistan floods response by SHO members 8

9 some expired goods. Another comment was about the need for cash. It appeared that there was a high demand for cash in the communities, which was not provided by the agency due to past negative experiences. According to the evaluators cash would have allowed the agency to respond faster and cheaper and more in line with people s specific needs in many though obviously not all places. The evaluations focusing on relief and recovery concerned interventions implemented by local or international partners of SHO participants and by a SHO participant itself. xv These interventions were based on the immediate needs of affected people and resulted in high community satisfaction. According to these reports the most crucial needs addressed were related to food security and livelihood, WASH and health. Distribution of NFIs and rehabilitation of shelter and water systems also responded to the urgent needs of the victims. One evaluation critically commented on the provision of shelters. xvi Firstly, the organisation concerned had overestimated the need for houses to assist the urgent cases. Secondly, although the quality of the shelter was generally good, the culturally appropriateness of the design left to be desired: the construction of the latrines not directly adjacent to the house did not ensure the safety and privacy for women, and the provision of a kitchen was insufficient. According to another evaluation xvii lands rights represent a key constraint both for permanent shelter as many people returning home find themselves without having a place to plant or to build a house. The same applied to livelihood restoration. Looking at the level of beneficiary appreciation for different types of interventions, livelihood services were rated among the highest because people felt that they increased their self-reliance, helped them stand on their own feet and allowed them to obtain the cash needed for many of their other needs. This conclusion is in line with the evaluation of the livelihood programme (recovery phase) xviii in which seed and fertiliser distribution, tube-well and generator repair, and land clearance were well in line with the expressed recovery needs and priorities of the affected people. Interestingly, not all responses included livelihood support. The extent to which accountability was enhanced in the local communities appeared to depend, among others, on the degree of their involvement in the intervention. This is evidenced by different findings in three evaluation reports. xix One evaluation of the relief phase that concluded a low level of accountability, found that the agency did not consult people at all about their needs at the village level. The people received the same general package given across all provinces. They were also not informed about the exact items they would receive when tokens were given as other organisations did. Other evaluators concluded that the implementing agencies addressed accountability concerns, though there were still large percentages of people who reported not being consulted, informed about the exact programme support planned or knowing the complaint mechanisms. Positive evidence comes from a third evaluation report. It concluded that the involvement and active participation of the affected communities from the very early stage of the response highly contributed to better planning and response and the project s success as a whole. It also pointed to the importance of the use of data from other secondary sources and information generated by other humanitarian actors, to ensure relevance and accountability during project implementation, especially concerning NFI distributions, health and hygiene awareness-raising and sanitation interventions. Synthesis of conclusions on the 2010 Pakistan floods response by SHO members 9

10 4.2 Effectiveness and impact: Was the response effectively implemented? Generally speaking, the evaluation reports concluded that the response in the relief and recovery phases was effectively implemented and had a positive impact, but with clear variations between locations and sectors. Overall, objectives and coverage targets had been achieved, in many cases exceeding the planned targets. The reports presented and discussed many findings that underlie the conclusions on effectiveness and impact, which we will not repeat in this report. Factors that contributed to the interventions effectiveness and impact were, among others, prior presence of implementing agencies in Pakistan, the decision to work through local partners, organisations experience in responding to previous emergencies, the ability to mobilize staff, teams, funds and equipment internationally, strong appeal success, and, for some agencies, the large size of the volunteer base. Various factors hindered effectiveness and impact, such as the security situation, bureaucratic hurdles, delays in appeal funds release, high volunteer turnover and coordination issues. xx The latter will be dealt with in section 4.10 on Cooperation. According to various evaluations xxi the interventions contribution was significant in operational terms, i.e. meeting immediate needs related to food, non-food items, drinking water, health, sanitation, shelter and livelihoods. More specifically on the latter, seed and fertiliser distribution and land clearance scored (very) well, while tube-well and generator repair scored average on effectiveness. xxii As mentioned earlier, effectiveness and impact varied between locations and sectors. According to one evaluation focusing on the relief phase, not in all communities the livelihoods and water needs were met, while another evaluation of the same phase gave examples of less met needs in the WASH and health sector. xxiii Also, on timeliness evaluators found different situations. For one programme considerable delays in delivery were reported for some districts, while in other programmes the beneficiaries received help in the relief phase very quickly, but the transition to early recovery was slow, and started almost a year after the floods. xxiv Findings in one evaluation led to the conclusion that the interventions contribution was less successful in making a strategic contribution in some areas, such as addressing pre-disaster vulnerabilities and enhancing local capacities. xxv This brings us to the impact of the interventions. In terms of impact on the people affected, various evaluators of the relief phase xxvi noticed an increased food intake of women, men and children which positively affected their ability to engage in repair and income activities. The tools supplied as NFIs greatly helped in repairs and livelihoods activities. Food aid and NFIs distribution also reduced the need to take loans and to sell animals for food purchases and replacing household items that were lost. Additionally, providing NFIs reduced the burden on non-beneficiary households that were hosting them and using their scarce resources to meet their immediate needs. People mentioned a sharp decline in child diseases due to the preventive and curative health care provided and as a result of improved health and hygiene practices. The overall environmental situation of the target villages also improved as a result of waste management and sanitation facilities, which contributed to a reduction in cases of malaria and other flood related diseases. One report concluded that women felt empowered due to their participation in different project activities and their involvement in awareness-raising sessions and decision making. According to us, such a statement is quite remarkable after just a few months of support. An evaluation of full period of support found that women got confidence and started getting organized themselves in groups with the help of partner organisations. xxvii (more about women in section 4.8). Synthesis of conclusions on the 2010 Pakistan floods response by SHO members 10

11 Evaluations of the combined relief and recovery phase xxviii concluded the following about the interventions impact: demand and need for WASH related programmes had increased; good hygiene practices were noticed; the capacity of local and affected people were enhanced in terms of humanitarian work; food security had improved; the scope and outreach of partner organisations were enhanced, to mention some of the conclusions. These reports also added that by the time of the evaluations two main sectors still needed more support, namely, livelihood (livestock) and shelter, and that communities would have preferred more cash support, especially in the recovery phase. These examples of impacts were mostly noticed during field visits of the evaluators. We felt it interesting that one evaluation xxix stated that it was difficult to verify the actual impact on meeting basic needs and the restoration of livelihood options, as there were no quantifiable monitoring systems measuring impact. In general, we did not find much information on monitoring and evaluation systems in the evaluation reports (see also section 4.8). 4.3 Efficiency: Was the response efficiently implemented? Five of the nine evaluation reports included, very briefly and without much reflection, the issue of efficiency and cost-effectiveness. Moreover, they addressed it in entirely different ways. Therefore, a concluding answer to the question Was the response efficiently implemented? cannot be given. Instead, we summarize the conclusions of the various reports below, which at the same time show the variety in focus of the different reports: - About one programme the conclusion was that it was implemented efficiently and also costeffective, and that it reflected a rational distribution of programmatic and operational costs. The evaluators had calculated the cost per person which they assessed as cost effective. For another agency the ratio of relief services and administrative costs had been calculated which compared favourably with other agencies, concluded the evaluator. xxx - Another evaluation focused on different types of interventions and concluded that the seed and fertiliser distribution and land clearance scored (very) well and tube-well and generator repair average on efficiency using a number of criteria. xxxi - On the availability of (financial) data we found two observations. xxxii In one case, the agency had not provided detailed budgets to the evaluator which hindered an analysis of efficiency in various sectors or provinces. In another case, the evaluators found all the necessary documents and data on program activities in digital form. They felt the program s implementation schedules, the project logical framework, and monitoring and evaluation mechanisms and tools efficient and concluded that they presented an evident case of good practice. - In two evaluation reports cost efficiency was related to the costs of materials. xxxiii For the WASH sector it was found that indigenous resources like materials and local labour had not been fully utilized. Thus, it was concluded that such utilization would have favoured cost efficiency. Regarding infrastructure an evaluation team made efforts to compare costs but did not draw conclusions on cost-effectiveness. Synthesis of conclusions on the 2010 Pakistan floods response by SHO members 11

12 4.4 Coverage: Did the response reach major population groups facing life-threatening risk wherever they were? Was there appropriate geographical coverage within the affected region? xxxiv When looking at coverage, it is relevant to distinguish between people who face life-threatening risk but who are likely to recover relatively quickly due to resources and position and people who hardly have adequate coping strategies. Most of the SHO participants focused their interventions on the latter, the poor and most vulnerable groups, which generally refers to vulnerable in terms of resources and position. Section 4.8 further elaborates on vulnerable groups and, also, on needs assessment. In terms of geographical coverage, the conclusions of the nine evaluations were clear: the interventions of the SHO participants took place in those areas that were most affected and, generally, they reached the major population groups facing life-threatening risk. Where targeting of beneficiaries took place in consultation with the community the most deserving were well reached. Those agencies that worked more through government officials and local authorities, however, covered most vulnerable as well as low vulnerability regions and families. When looking at coverage in terms of people at risk, the evaluations of responses of SHO participants - implemented directly or through local or international partners - showed that the flood responses reached the most affected people both in the relief and in the recovery phase. xxxv Evidently, due to the sheer scale of the flooding the organisations adopted a targeting mechanism to reach those who were at greatest need. xxxvi The role of local village committees, at times especially set up under the intervention, appeared to be of crucial importance in identifying the people mostly at risk. The evaluations of responses of international umbrella organisations and their national branches, which looked at the relief phase, were very critical of the coverage issue. xxxvii The evaluators spoke of considerable variations in the vulnerability of the locations targeted and cases where aid mainly reached people that were locally well positioned and/or aligned to political parties. Interference, primarily at local levels from politicians, landlords or tribal leaders played a negative role in reaching the most vulnerable people. Access issues in conflict affected areas, such as Khyber Pakhtunkhwa (KPK) and parts of Balochistan proved another factor that influenced the coverage of assistance. In terms of geographical coverage, most responses evaluated took place in provinces and districts where the impact of the flooding was severe or extreme: Sindh with the highest number of people affected (7,2 million) and largest area covered by flood, followed by Punjab (6 million affected people) and Khyber Pakhtunkhwa (KPK, 3,8 million affected people). In Annex 3 a table can be found which shows the districts in which the SHO participants supported interventions, either directly or through local or international implementing partners or the national branch of the international umbrella organisation. 4.5 Connectedness: Did the response ensure that activities of a shortterm emergency nature are carried out in a context that takes longterm and interconnected problems into account? Immediately after the floods, many people fled and lived dispersed in camps where emergency relief was provided. After the first weeks (in KPK) or months (in Sindh) the camps had largely been dismantled and the majority of the displaced people had returned home to their villages where relief Synthesis of conclusions on the 2010 Pakistan floods response by SHO members 12

13 and recovery needs persisted. During the first phase of the emergency all the organisations focussed on immediate lifesaving activities. Some of these relief activities, particularly in the WASH, health, and in the NFI/ shelter sectors had a longer term effect (see below). There is not always a clear demarcation between the relief and the recovery phase, however, over time and in accordance with the changing needs of the affected population the activities of the SHO participants increasingly focused on early recovery. With regard to the connectedness between relief and early recovery activities, the evaluation reports demonstrated that most organisations made a smooth transition from one phase to the other, although some delays in the implementation of essential recovery activities were also reported. Most organisations purposely continued working in the same communities, some even in the same sectors -particularly WASH and NFI/ Shelter - while others started activities in new sectors -particularly livelihoods. Setting up community committees and other CBOs, such as farmer groups, was generally seen as an important instrument to ensure sustainability of the interventions and a smooth transition towards recovery and development, as will be elaborated below. Various activities carried out during the relief phase did create some long-term impact as we have presented under section 4.2 Effectiveness and impact. To add some more examples, three evaluations reported that activities in the WASH sector xxxviii, such as hygiene promotion, had improved personal hygiene practices in the long run. Beneficiaries - women in particular - also stressed that the sanitation facilities installed had brought a positive change in the overall living conditions of the communities, notably in the health and hygiene areas. One report clearly stated that in the affected districts the situation in terms of safe drinking water and sanitation was poor even before the floods occurred. Activities to improve these facilities, contributed to strengthening of the resilience of the targeted communities xxxix. The SHO participants used different approaches with regard to shelter. One organisation that focussed on large scale relief provided affected communities with tarpaulin or tents for temporary shelter as part of the NFI kit. xl Others were involved in the construction of more durable shelters out of corrugated iron sheeting or local materials. Unfortunately, the programme using local building materials encountered delays due to complex procurement procedures. xli It should be added that, as one evaluator stated: even the tents and tarpaulins - and other NFI items, such as tools, mosquito nets, kitchen items - may last much longer than the 6 months life expectancy on paper because people in poor communities are generally able to stretch the durability of these goods considerably through careful use and repair. xlii The longer term effect of preventive and curative health care activities is evidenced by the wish of communities that these services could continue. Where one SHO participant successfully managed to increase governmental engagement for the sustainability of programme interventions another SHO participant involved in the health sector experienced significant challenges in trying to ensure sustainability of its interventions. The transfer of the health facilities developed by the SHO participant to the Ministry of Health was not feasible due to structural organisational weaknesses of those national structures. The organisation eventually secured continuity of its work by passing the project on to another international medical organisation xliii With regard to the connectedness between relief and early recovery the Inter-Agency RTE established that most of the international responses focused on relief rather than on recovery activities and that Synthesis of conclusions on the 2010 Pakistan floods response by SHO members 13

14 strategies related to recovery and rehabilitation were not timely and carefully planned for by most clusters, because the clusters were to a large extent operating independently from the phases of the operation. In addition, recovery had been dealt with from a cluster perspective rather than in a more integrated fashion. Most of the evaluation reports gave a more positive impression of the connectedness between the two phases. Most organisations built their early recovery interventions on their relief activities xliv. One report stated that the organisation wanted to move into the recovery phase with the same communities, because the relief interventions had provided a solid foundation for the recovery phase by cultivating a strong relationship between the organisation, its local partners and the communities and by creating a high degree of trust. xlv The early recovery phase of another organisation targeted some of the same areas where the relief activities had been conducted. The evaluator continued stating that the organisation will benefit from its prior knowledge about those communities even though the high turnover of volunteer staff of the national organisation will reduce some of this connectedness. xlvi Another report stated that there was a time gap in moving from relief towards the early recovery phase and that the delay proved especially critical for timebound activities like livelihoods, because delays in seed distribution caused problems in timely plantation by communities and slowed down self-sufficiency in food production xlvii. Some SHO participants that initially focused on life-saving sectors moved almost (exclusively) to livelihoods and reconstruction activities. One organisation, for instance, started to focus on providing cash for agricultural inputs and shelter construction during the early recovery phase. The evaluator assumed that the reconstruction activities would benefit from the tools given during the relief phase. The same report also stated that the food aid given during the relief phase had given people more physical strength and economic resources to engage in shelter construction and agricultural work. Other organisations continued working in the same sectors but adjusted their activities. xlviii An organisation that worked in the WASH sector, for instance, moved from the distribution of hygiene kits towards activities which facilitated recovery, such as repair of irrigation and water infrastructure at the household and community levels. xlix Another evaluation advised about the recovery phase that, with the coming of winter and the start of a new cropping season, the emerging priority should focus on winter NFIs, shelter and livelihood assistance, e.g. repair of irrigation channels, seeds, fertilizer, water for crops and livestock. Other needs identified to support recovery were construction and repair of water supply infrastructure and latrines. Furthermore, NFIs were still deemed relevant because displaced people were returning and had the same basic needs for NFIs and shelter. l Several evaluation reports praised the strategy of continuing to work in the same communities. One evaluator stated that the strong relationship developed with targeted communities during the relief phase will provide a strong platform for humanitarian actors to build upon during the early recovery phase and later for development interventions. li Another report established that some opportunities for early recovery and long term interventions were created by involving women from the affected communities in relief activities, which gave them confidence and helped them organise themselves. lii A third report stated that, for future sustainability and development, the organisation had trained community groups at the village level and encouraged them to relate their development needs with the relevant local government structures. liii Yet another evaluator expected that the large participation of the beneficiaries in the rehabilitation works in the shelter and WASH sector would result in a feeling of ownership, because the communities themselves will have to maintain the Synthesis of conclusions on the 2010 Pakistan floods response by SHO members 14

15 systems. liv Finally, the report concerning a livelihood project lv established that the organisation developed a strong and innovative approach to recovery by utilising a one-time relief distribution to create long-term development structures, such as farmer groups and a revolving CBO loan fund. Accounts from beneficiaries and surveys indicated that both food security and the restoration of livelihood options of the farmer households had significantly improved and that activities had been in line with beneficiaries expressed recovery needs and priorities. Beneficiaries, male and female, voiced an interest to pay back their loans. They, furthermore, saw the farmer groups as a means to increase control over their lives and to lessen dependency on money lenders. 4.6 Coherence: Did the response take into account humanitarian and human rights considerations (assess security, developmental, trade and military policies, as well as humanitarian policies)? The majority of the evaluation reports did not specifically look at coherence or observance of fundamental humanitarian and human rights principles separate from checking adherence to the Red Cross and NGO Code of Conduct. Four of the evaluations explicitly mentioned coherence, three of which linked coherence to coordination and solely discussed whether relief efforts had been appropriately and sufficiently coordinated. This may not be surprising for the following two reasons: according to ALNAP coherence is often confused with coordination lvi and ALNAP also states that coherence may be less relevant for evaluating single-agency or single-project interventions lvii. The issue of coordination and co-operation is discussed under section In our elaboration below we looked at the humanitarian and human-rights considerations, to which the question on coherence refers. Basically, these are the first four fundamental humanitarian principles of the Red Cross/NGO Code of Conduct (CoC), i.e. 1) humanity, meaning the centrality of saving human lives and alleviating suffering wherever it is found; 2) impartiality, meaning the implementation of actions solely on the basis of need, without discrimination between or within affected populations; 3) neutrality, meaning that humanitarian action must not favour any side in an armed conflict or other dispute where such action is carried out; and 4) independence, meaning the autonomy of humanitarian objectives from the political, economic, military or other objectives that any actor may hold with regard to areas where humanitarian action is being implemented. lviii As will be set out in section 4.7, all evaluation reports confirmed a certain general level of adherence to the CoC. However, the reports also mentioned some challenges that were encountered, particularly with regard to impartiality and neutrality, but also to some extent with regard to humanity and independence, as we elaborate below. Regarding the humanitarian principle humanity, all the SHO participants provided life-saving assistance, trying to make sure that the first basic needs of the affected households within their reach were met with regard to physical safety, food, health, WASH, NFI and emergency shelter (the relevance and effectiveness of this assistance is addressed in section 4.1 and 4.2 respectively). In one case, the local implementing partners of an SHO participant were directly involved in the evacuation of affected people to safer areas. They supported the local authorities and enabled the Pakistani army to better fulfil their rescuing task by lending them their boats. lix One evaluation report stated that due to the large scale of the disaster, and because of limited presence and capacity, organisations had been confronted with challenges trying to deliver an independent needs based response. Initially, the response was poorly prioritised, because there Synthesis of conclusions on the 2010 Pakistan floods response by SHO members 15

16 was an excessive rush to start without deciding on how and where to respond. lx All evaluations reported that the relief effort had been hampered by the destruction of infrastructure. Roads and bridges had been washed away by the floods, which created problems to access the affected communities. In some instances, people had to travel far to receive food and NFI assistance, because implementing partners could not reach their villages. Therefore, several SHO participants supported activities to rehabilitate physical infrastructure. lxi Some evaluators remarked that the relief response was late in reaching people due to a combination of slow donor allocations, complex or overlystringent procurement policies and staffing issues. lxii The evaluation reports give us the impression that most SHO participants and their implementing partners made serious efforts to adhere to the principle of impartiality. However, this was a challenge during the general confusion in the early stages of the large scale flood emergency and within the cultural and political context of Pakistan. The latter relates to the issue of interference from politicians, landlords or tribal leaders, which has been discussed under section 4.4. lxiii According to the evaluations, most organisations carried out independent needs-assessments, involving the affected communities. lxiv In how far this community involvement was representative, i.e. included representatives from all different segments of the population (young, old, men, women, minorities, people with disabilities, etc.) varied. The SHO participants which had strong longstanding partnership relations with local organisations and provided their local partners with continuous guidance and capacity support did particularly well in securing impartiality. lxv Yet, several evaluation reports mention that vulnerable groups had not been reached. lxvi For more information on these groups we refer to section 4.8. Relief efforts in the provinces bordering Afghanistan were complicated by the prevalent political and security situation, the presence of Taliban - both from Afghanistan and from Pakistan and terrorism threats. In KPK and parts of Balochistan, the impact of the flood was compounded by conflict and insurgency; the situation in this part of the country is considered a complex emergency lxvii. The security situation for international agencies in Pakistan and their ability to do their work unhindered worsened after 1 st of May 2011 when the US attacked the residence of Osama Bin-Laden in Abbottabad and subsequently killed him. This complex emergency situation in KPK and parts of Baluchistan created extra challenges to international aid organisations, including SHO participants, to adhere to the fundamental humanitarian principles of neutrality. One SHO participant experienced operational difficulties as a result of the ensuing security situation: in KPK international staff needed security clearance from the government, a Non Objection Certificate lxviii, to permit them to travel to Kohat and Shangla districts; they were not allowed to stay overnight in these districts. lxix Referring to civil-military co-operation, the Inter-Agency RTE explicitly mentioned that the humanitarian space had been compromised, especially in KPK and Balochistan. Where civilian assets were insufficient and physical access constrained, military assets were sometimes used during the rescue operation and the early stages of the emergency. The RTE stated that civil-military cooperation in emergencies is a sensitive matter and recalled that the UN had previously determined that military assets should not be used in areas such as Balochistan and KPK, where the government or regional actors are party to the conflict. Although a set of country specific guidelines for civil-military interventions had been adapted in early 2010, donors pressurized aid agencies to use a NATO air bridge during the relief phase. The evaluators praised the UN Humanitarian Coordinator, who stood up to this pressure. lxx Synthesis of conclusions on the 2010 Pakistan floods response by SHO members 16

17 One evaluation report mentioned that the agency undertook advocacy through the national NGO coordinating body Pakistan Humanitarian Forum (PHF) on several issues, including concerns related to use of humanitarian space for counter-terrorism by the CIA. lxxi This indicates that the organisations that implemented the programme had encountered challenges with regard to maintaining their independence. In other evaluation reports no reference was found to attempts by foreign donors to interfere with the humanitarian aid efforts. 4.7 Standards: Did the response adhere to the Code of Conduct and the Sphere Standards? All the evaluation reports referred to the Code of Conduct for the International Red Cross and Red Crescent Movement and Non-Governmental Organisations in Disaster Relief (CoC) lxxii and/or Sphere Standards lxxiii, however, not all discussed or substantiated their conclusions with regard to adherence to these standards. Perhaps this is due to the fact that it is well known that all SHO participants have signed the CoC and committed themselves to adhere to the Sphere Standards. In some cases issues relevant to adherence to the standards were included in reports sections on, for instance, accountability, coordination and partnership. The reports give us the impression that overall adherence to CoC and the Sphere Core Standards was adequate. Due to the enormous scale of the disaster, adherence to the Sphere Minimum Standards for the different sectors was posing some challenges, particularly those with regard to shelter and NFI. All SHO participants appeared to have made serious efforts to fulfil the standards. Several of them provided specific training on the standards as part of their quality and accountability training for staff and partners and some put posters with core standards and principles up in their offices. lxxiv Yet, despite having been briefed about them, some of the local partner organisations were not familiar with widely used international standards and principles such as Sphere and CoC. lxxv Code of Conduct (CoC) Interestingly, one of the evaluation reports stated that the strong adherence to most of the principles in the COC was one of the strengths of the response, thereby implicitly confirming the importance of these principles lxxvi. The first four principles of the CoC, the humanitarian imperative, impartiality, neutrality and independence have been discussed in section 4.6 on Coherence. Most reports specifically paid attention to the remaining principles, particularly, building disaster response on local capacities, participation of affected communities, reducing future vulnerabilities and accountability, predominantly, towards beneficiaries. To a lesser extent consideration is given to respecting local culture and custom. Obviously, some of these issues are strongly interrelated; however we try to present them separately. Where it comes to building on local capacities (i.e. human and material resources), the evaluation reports show that most SHO participants worked with local partner organisations or their national branch in the case of an umbrella organisation to implement their response programmes. lxxvii Most local partners recruited their staff locally. lxxviii Two of the SHO participants included in this evaluation implemented (parts of) their programmes directly. In one of these cases, the evaluation report mentioned that local staff was employed and materials were purchased locally, but no efforts were made to identify local NGOs having the capacity to assist in the implementation of the projects. lxxix Most evaluations reported that NFI and building materials were purchased from local sources. In Synthesis of conclusions on the 2010 Pakistan floods response by SHO members 17

18 some instances this caused delays, particularly, because the local market could not instantly fulfil the enormous demand for building materials. Several reports also mentioned the benefits of local procurement, such as people already being familiar with certain types of water filters. lxxx With regard to participation of affected communities, the evaluations established different levels of involvement of local communities in planning, implementation and monitoring of response. Some of the implementing agencies/ partners ensured representation of people from all segments of local communities in all stages of the project management cycle whereas others only involved the leadership in some stages (see also section 4.8). Concerning the first category, one report found that the participation of affected people, including women, in the identification and selection of beneficiaries and the involvement of beneficiaries in distribution and purchasing processes contributed to creating trust among the communities. lxxxi Another report, however, concluded that people s participation was low, information provision insufficient and complaint mechanisms absent with the result that the agency s programmes caused conflicts and grievances in 8 of the 18 places visited. lxxxii The evaluation reports found that implementing agencies/ partners ability to contribute to reducing future vulnerabilities differed. To some extent this depended on whether they were primarily involved in emergency relief or also focussed on recovery, but it also depended on the size of their operation. One report established that insufficient attention could be paid to developing the capacities of communities due to time constraints and the scale of operation during the relief phase. lxxxiii Particularly, organisations implementing (early) recovery programmes appeared to contribute to some extent to reinforcing disaster resilience of affected communities. This will be discussed in further detail in section 4.9 on Strengthening of responses. The evaluation reports showed remarkable differences between implementing agencies/ partners in their ability to translate the principle of accountability towards beneficiaries to local realities. Most organisations tried to address accountability concerns, however with varying success as has already been outlined in the paragraph on accountability in section 4.1 Relevance/Appropriateness. One report established that although accountability concerns had been addressed to some extent, there were still large percentages of people not consulted or informed about the planned programme support or knowing the complaint mechanisms, because local partners often did not understand the overall objectives and values of accountability principles and took it as a routine activity. This suggests, according to the evaluators, the need for far greater investment in developing a local approach to accountability. lxxxiv Prove that this can be done is given by another report stating that the SHO organisation had a clear position on ensuring the implementation of standards and accountability mechanisms in humanitarian assistance, which led local partners to be proactive in designing and implementing accountability towards beneficiaries and adapting strategies to create awareness among the affected population about the relief packages, and how and where to lodge complaints. lxxxv The concept of accountability towards beneficiaries is further elaborated on and specified in the Humanitarian Accountability Partnership Standard in Accountability and Quality Management (HAP). Four evaluations specifically refer to the HAP Standard and mention is made of implementing partners that acquired HAP certification. Synthesis of conclusions on the 2010 Pakistan floods response by SHO members 18

19 Reference to respect for local culture and customs is particularly made in connection to the shelter and NFI and the health sector, and is mainly related to the sheltered position of women. In the shelter sector, the positioning of latrines was an important issue as was the placing of a kitchen and the inclusion of certain items in the NFI kits; and in the health sector special lady visitors were trained and employed. This is described in detail in section 4.8 Women and Vulnerable groups. Sphere Standards Four evaluation reports discussed in more detail how adherence to Sphere was established, whereas the other reports only mention that the response provided was in accordance with Sphere Standards. One evaluation checked all the sectors against the Core Standards but did not reflect on the Minimum Standards per sector. lxxxvi The other three evaluations also focused on adherence to the Minimum Standards. Two of these stated that information was not always readily available but they had made serious efforts to check on observance of the Standards for the different sectors. lxxxvii All three reports mentioned that it appeared difficult to put the standards into practice. This was partly, but not exclusively, due to the enormous scale of the disaster as a result of which resources were spread thin. In addition, there were other factors that constituted difficulties in meeting the Minimum Standards. One SHO participant, for example, made a serious effort to implement the Standard requiring that houses should be disaster resilient by taking measures to reduce erosion on the shelter sites and making efforts to build new shelters on alternative sites to reduce the risks of future flooding. However, to build real safely one would have to look for completely new locations, which was not easy because beneficiaries were attached to their properties and obtaining new land for construction was a real challenge. lxxxviii The Inter-Agency RTE concluded that the experiences from the Pakistani floods clearly showed the need to adapt the response to the context and conditions on the ground, and have international standards serve as guidelines for what should be achieved. Some clusters managed to agree on adapted standards using Sphere as a guiding objective. One example is shelter. Many NGOs reported difficulties in implementing the standards due to their relatively high costs and consequently decided to go for cheaper and less permanent solutions, often inadequately prepared for future floods. Eventually, the shelter cluster applied a one room standard policy for permanent shelter. lxxxix The Inter-Agency RTE also revealed that the situation was further complicated because the Government of Pakistan decided on standards other than those agreed to internationally and concluded that in a country like Pakistan standards, principles and guidelines should be negotiated in due time with relevant government entities and institutions, and not only be dealt with once the emergency strikes. xc Synthesis of conclusions on the 2010 Pakistan floods response by SHO members 19

20 4.8 Cross-cutting themes: Gender and Vulnerable groups xci Gender: Was gender considered in the agencies emergency assessments? Did relief provision include special components for women, men, girls and boys and, if so, were these systematically monitored? Vulnerable groups: Were the special needs of acute vulnerable groups (e.g. children/ elderly/people with disabilities, etc.) considered in the agencies emergency assessments and were they consulted in the same way as other groups? Did relief provision include special components for them and if so were these appropriate and systematically monitored? Overall, the evaluated programmes paid attention to most vulnerable groups, but in very different degrees. It concerned women, more particularly widows and other female-headed households, and children, and to a much smaller extent, elderly and people with disabilities, while even less consideration was given to religious, ethnic and tribal minorities. None of the interventions evaluated included special components for women or specific vulnerable groups, although the health programmes mostly targeted women. Needs of women and other vulnerable groups were usually identified during the needs assessments. The assessments and, consequently, interventions carried out by the national branches of international umbrella organisations showed considerable gaps in terms of participation of vulnerable groups and addressing their needs, apart from some good exceptions. Many programmes implemented by local or international partners of SHO participants and by SHO participants themselves, however, integrated women s concerns in the different stages of the interventions and made efforts to increase their participation in consultations, activities and decision making. In a few programmes a systematic approach was used to that effect. In the evaluation reports of responses implemented by (partners of) SHO participants much less attention is paid to other vulnerable groups than women. It is not entirely clear to us whether this limited information is due to the lack of attention of the evaluators or to the programme itself. Most evaluators reported without providing much detail that vulnerable groups were considered in the response, more specifically in needs assessments. Children, people with disabilities and elderly people were mostly mentioned. As regards monitoring, the evaluators found that for all implementing agencies and partners monitoring from a gender or vulnerability perspective left to be desired. Although in different degrees, all evaluations paid attention to vulnerable groups being the most affected amongst those who suffered from the floods. In the reports, the term vulnerable usually refers to women, more particularly widows and other female-headed households, and children, elderly people and people with disabilities, as well as religious, ethnic and tribal minorities. These groups were considered more susceptible due to a lack of social services, and, especially in the case of (young) women, deprivation of basic human rights due to cultural and social norms, and the threat of exploitation by traffickers. As one evaluation pointed out, the majority of the women who reached to the camps, or lived in open areas particularly in Sindh and Punjab, had never experienced displacement in their lives. Similarly, in districts of KPK, for many women it was the first time they left their homes and had never been exposed to public life. They had to quickly cope with the situation either living in the camps with other communities, castes and social groups or live in isolation to bear the pain of disaster. xcii Synthesis of conclusions on the 2010 Pakistan floods response by SHO members 20