Using Qualitative Information For Impact Assessment: Introducing The Qualitative Impact Assessment Protocol (QUIP) By Katie Wright-Revolledo, INTRAC Introduction Qualitative methods can provide insight into the impact of services and products provided by NGOs. They are useful for uncovering sensitive and nuanced information and unexpected or indirect impacts that cannot be gleaned so easily using survey-based methods. Furthermore, they can reveal rich information about the social processes that lead to particular outcomes and expose the causal links between particular life-cycle events, thus helping practitioners to understand the different kinds of clients they serve, the characteristics of those most benefiting from the programme and the profiles of those who may be losing out from it. Despite the advantages, many practitioners are unfamiliar with these methods, and are unsure of how to analyse qualitative data in a rigorous way. This article is designed to encourage NGOs to use qualitative methods to assess the impact of their services, and introduces a protocol that has been developed at the University of Bath called the QUIP (qualitative impact assessment protocol), aimed at practitioners who want to reveal the reasons behind the diverse impact of their programmes. Why use qualitative methods? In the field of development practice there is a wealth of experience with impact assessment methods: good, bad and ugly. Analysing the wider literature on the impact of development programmes it can be observed that academics, consultants and practitioners have tended to use quantitative indicators more than qualitative indicators to assess the impact of their programmes and as a result, impact studies have tended to draw predominantly upon quantitative methods. To conduct impact studies, the main methodologies that are in theory available for use are: (i) Positivist or quantitative methods; (ii) Rigorous qualitative methods; and, (iii) Participatory methods. These methodologies are based on diverse epistemological traditions, have different methods of data collection and analysis and have different strengths and weaknesses, as summarised in Table 1. As Table 1 demonstrates, many of the social outcomes of development programmes (such as gender impacts) cannot be as easily measured using positivist (quantitative) approaches as they can using qualitative measures that expose the causal links between events, processes and outcomes and allow for the more indirect impacts of development programmes to be revealed (Wright 2003). 1
Table 1: Methods for impact assessment information Data collection method Data analysis Epistemology Potential strengths Potential weaknesses Positivist or quantitative methods Sample survey using a closed questionnaire. Interpretation of statistical tables and multiple regression analysis. Rigorous statistical inference and peer review. Rigour. Possibility of quantitative estimates of impact. More convincing to sceptical outsiders. High cost and time lags; restricted to measurable impact indicators. Reveals little about causation. Difficult to counter selection bias problems. Rigorous qualitative methods Quota sample of semistructured narrative interviews. Systematic scoring of types of impact from interview notes and transcripts. Rigorous qualitative analysis and peer review. Richness in detail and understanding of differential impact. Ability to pick up unexpected and unmeasurable impact. Not suitable for demonstrating that findings are representative of wider populations. Lack of clarity and consensus about how to achieve rigour. Source: Adapted from Copestake, Johnson and Wright (2004) Participatory methods Case study focus group discussion and semistructured games. Verbal and experiential learning among participants. Interpretative reports. Process transparency and expert judgement. More timely and costeffective with potential for shorter feedback loops from suppliers to users of information. Can be a positive learning tool for respondents too. Participants may hide important facts from peers as well as facilitators. Risk of response bias can make it hard to convince outsiders of the reliability of findings. Despite the advantages of using qualitative methods, knowledge and use of these methods has tended to remain in the academic domain (for example, much use is made of them by social anthropologists who may spend extensive periods conducting field research in a particular community). Notwithstanding the fact that these methods have most commonly been used by academics, practitioners have expressed the need for guidelines to allow them to monitor their social performance using qualitative methods in a cost-effective, timely and reliable way. This was expressed, for example, by managers of microfinance organisations as part of the Imp-Act programme 1, who were keen to understand the social impacts of their programmes but needed more clarity as to the kind of factors they needed to take into account when experimenting with qualitative methods. In order to respond to this need a protocol called QUIP, developed by The 1 Imp-Act is a partnership between 30 microfinance institutions (MFIs) across 20 countries and three universities based in the UK (University of Bath, Sheffield and the Institute of Development Studies, Sussex) whose purpose is to improve the impact of microfinance on poverty alleviation. The three main objectives are: (i) To develop credible and useful impact assessment systems which are based on the priorities of MFIs; (ii) To widen the scope of impact assessment in order to incorporate wider and more indirect impacts; (iii) To influence thinking and practice with regard to the role of microfinance in poverty alleviation. See www.imp-act.org for further information. 2
University of Bath, was devised (Copestake, Johnson and Wright 2004). It was originally designed for microfinance practitioners in order to help them get a better idea of the social impacts of their programmes. Yet, it can also be useful to practitioners who work outside this realm and are keen to experiment with qualitative methodologies. In particular it can help them assess the social and more indirect impacts of their programmes. What is the QUIP? The QUIP (qualitative impact assessment protocol) is a protocol or step-by-step guide as to how to realise and analyse qualitative data for impact assessment in such a way that the data generated is credible and produced in a timely and cost-effective way. It is designed to give practitioners a clearer idea of the different profiles of clients that they serve and to understand the impact that their programme is having on these different groups, in order to help them tailor services and products to maximise this impact (for example on poverty reduction). Collection, analysis and use of qualitative data requires considerable care if it is to be credible. But it can also be carried out in a timely and cost-effective way capable of contributing to development of new products, training of staff, strategic planning and resource mobilisation. The QUIP, described below, is intended to supplement the informal understanding of staff, and to complement other methods. Compared to quantitative sample surveys it can reveal unexpected outcomes and rich information on the causal chains linking service use and impact. 2 Compared to participatory methods, relying on group interviews, it can generate more detailed and rigorous evidence. It is also intended to be a flexible tool, capable of being adapted for different purposes and to different contexts. This article is designed to encourage development organisations to use qualitative methods to assess the impact of the services that they provide. As a step-by-step protocol, the QUIP aims to remove some of the mystery surrounding use of qualitative methods by social scientists. The QUIP is a protocol rather than a tool kit. As such, it is no substitute for the insights, skills and values of individual researchers and should not be used as a blueprint but rather as guidelines in terms of highlighting the kinds of factors that need to be taken into account when conducting qualitative research. It will often be useful to recruit people with qualitative research skills to use the QUIP, but with support and practice we think it should also be possible for the staff of many development organisations to use the QUIP themselves. The QUIP in ten steps 1. Deciding what data is needed Are there particular groups of clients you need to know more about? Are there informal or anecdotal ideas about impact that need to be checked? Does the effect on clients of changes in services provided need to be assessed? QUIP is particularly useful when impact is unclear, complex and important. Being clear about what you want to find out is crucial. Defining broad subject areas (see Table 2) also helps to ensure that time is not spent collecting information that is not needed. Table 2: Checklist of possible research objectives 1. Who are my clients? e.g. age, gender, ethnicity, business activities, education, experience. 2 In a quantitative sample one may be able to see a correlation between two variables but not understand the causal relationship between them. Qualitative methods are useful for understanding the causes that lead to particular outcomes for example illness and when users of NGO services decide to leave. 3
2. How and why is their demand for the services that we provide changing? 3. What are the current levels of satisfaction with the services they use? 4. What are the direct impacts of their services on clients? 5. What are the indirect impacts of their use of these services? e.g. shocks, demographic and health changes. e.g. what do they like and dislike compared to alternatives? how is their use of services changing? e.g. material impact, changes in knowledge and skills, changing attitudes, changes in the quality of personal relationships. e.g. on children within client s household; gender relations, community activity. 2. Deciding who should do the work Who carries out the work is likely to affect the quality and nature of the results. Are there microfinance institution staff willing and able to do the work? Do you need to hire independent researchers? Or can you use a combination of both? Personal skills required for those doing interviewing should include: Patience, awareness of the need to avoid prompting, and a capacity for creative listening ; (e.g. the ability to clarify points that are unclear and probe in a sensitive way). Awareness of the need to avoid leading questions, while at the same time being open to unexpected information that may merit further questioning. Appropriate language skills and cultural sensitivity (for example, using language in a way that shows the informant that the interviewer is on the same wavelength, so that the interviewer gains credibility and cannot be mislead by minimalist answers). Discipline in accurately recording and writing up interviews. It may be appropriate for interviewing do be done in pairs, with one person leading the discussion and the other taking responsibility for note-taking under specific themes and taperecording. 3. Identifying baseline information The principal reason for getting baseline information is for managers to get a clearer picture of the different profiles of their clients. For example, this may be to glean information on those who are leaving their programmes or to ascertain the needs of remaining clients. This will include basic data such as: age of respondents, gender, number of dependants, number of assets and so on. It is helpful to consider events in the past that provide a useful reference point so that impact can be linked to a precise period of service use. Ideally, there will be baseline survey information that can also be used for sample selection and subsequent analysis. 4. Selecting the sample for qualitative interviews This entails obtaining lists of the clients you most need to know more about (e.g. profiles of clients that are exiting the programme and reasons for this), deciding on sample sizes for each and selecting respondents for interview randomly from each list. 5. Designing a semi-structured interview schedule A semi-structured interview has a basic structure but allows for more flexibility than a standard survey. Constructing it entails selecting key interview themes, then framing generative, supplementary and closed questions for each. Potential users of the information should be consulted to ensure that issues of particular interest are covered. 4
6. Preparing for interviews This includes testing the schedule and training interviewers. Initial introductions, the framing and ordering of particular questions, as well as the way information is recorded all need to be practised and refined how we ask questions will affect the kind of information divulged. 7. Conducting interviews Issues here include selecting appropriate times and places to ensure respondents are not rushed, inconvenienced or inhibited by the presence of others. 8. Collating and interpreting narrative data Raw data is best typed up as individual case studies. These may incorporate tape-recorded quotations. A sub-sample (a smaller selection from the original sample) should be randomly checked for quality. Typed transcripts or summaries make it easy to rearrange data into themes. 9. Collating and interpreting statistical data Findings can also be summarised by developing a system for scoring impact. Scores can then be entered onto a spreadsheet and subjected to various forms of statistical analysis. 10. Using the data The final and crucial step is to make the most of data to inform decisionmaking within the organisation, and perhaps beyond. Conclusion This article has suggested that to date little attention has been paid to producing guidelines for using qualitative methods in development research specifically aimed at practitioners despite the advantages that the use of such methods can offer. It is suggested that one protocol designed to respond to this need is the QUIP. This is not a magic bullet or a tool kit but provides some background on the kinds of considerations that need to be taken into account when embarking on qualitative research for impact assessment. It thus serves as one attempt to aid practitioners in obtaining useful, credible and rigorously obtained qualitative information in a cost-effective way. More information on this protocol can be downloaded from the Imp-Act website: www.imp-act.org References Copestake, J.G., Johnson, S. and Wright, K. (2004) Impact assessment of microfinance: Towards a new protocol for collection and analysis of qualitative data, in Combining Qualitative and Quantitative Methods for Development Research, ITDG and CDS Swansea. Imp-Act (2004) QUIP: Understanding clients through in-depth interviews. Practice Note 2. www.imp-act.org Wright, K.E. (2003) Problems? What problems? We have none at all. Qualitative data collection for impact assessment: Getting the questions right, Journal of Microfinance 5 (1): 115 38. 5