Introduction to Outcome Mapping

Transcription

1 Introduction to Outcome Mapping Ben Ramalingam and Sana Gull

2 ...not everything that counts can be counted and not everything that can be counted, counts Albert Einstein

3 Learning Objectives 1. Understand the main Outcome Mapping concepts and tools 2. Apply Outcome Mapping tools and principles to different elements of the PAN Localization programme 3. Develop plans for taking OM forward in your projects

4 Agenda for Day 1 Outcome Mapping Overview Vision and Mission Boundary Partners Outcome Challenges Progress Markers Discussion Documentation

5 Introduction Project Planning, Monitoring and Evaluation using Outcome Mapping

6 Outcome Mapping

7 Conceptual Overview

8 Issue 1: Dilemma of influence High Influence Low Inputs Activities Outputs Outcomes Impacts Time Beneficiaries/Indigenous Actors Partners / Local actors Project Program / Program

9 Issue 2: Dilemma of attribution Community and Family Other NGOs Church odi DFID / USAID / etc Developing Country Govmts World Bank / Multilaterals Private sector Civil Society

10 What is Outcome Mapping? A methodology for planning, monitoring and evaluating that looks at both development results and internal performance helping a program to: design interventions in relation to the broader context and assess within its sphere of influence focus on changes in the behavior of partners (as outcomes) recognize various contributions to the achievement of results

11 Outcome Mapping focuses on carefully defined outcomes High Influence Changes in behaviours, relationships, activities and/or actions Low Inputs Activities Outputs Outcomes Impacts Time Partners / Local actors Beneficiaries/Indigenous Actors Program Program

12 Outcome Mapping focuses on boundary partners or stakeholders with whom a program works directly Community and Family Other NGOs Church odi DFID / USAID / etc Developing Country Govmts World Bank / Multilaterals Private sector Civil Society

13

14

15

16 Step 1 : Vision Statement Reflects the broad human, social, & environmental betterment in which the program is engaged and to which it is contributing

17 Example Vision Statement Local authorities, communities, and international organizations in developing countries in Africa recognize the value of HIV/AIDS intervention as an integral part of social & economic development. Municipal, regional, and national governments actively support HIV/AIDS prevention activities by formulating and implementing effective public health policies. Using research findings, they have developed a comprehensive public health strategy to slow down the infection rate. Formerly marginalized groups (e.g. women and youth) are organized into advocacy groups that can effectively formulate their needs to policy makers. All groups have access to reliable and relevant technical information about HIV/AIDS prevention and are able to make informed choices. In essence, there are healthier, happier, and wealthier communities.

18 Facilitation Question Imagine that in 3-5 years the program has been extremely successful. What changes will you have helped bring about? What are your partners doing differently? What have they achieved? In essence, what would total success look like?

19 Step 2: Mission Statement Describes how the program intends to: Operationalize its role in support of the vision by determining the areas in which it will work Support the achievement of outcomes by its boundary partners

20 VISION STATEMENT Mission The Mission is that bite of the Vision Statement on which the program is going to focus

21 Facilitation Questions How can the program best support or contribute to the achievement of the vision? What areas do you need to work in? What do you need to do?

22 Example Mission Statement In support of this vision and on behalf of its donors, the program will work in the areas of research, dissemination, capacity building, & coordination. It will contribute to the production, synthesis, & dissemination of research data, position papers, & other information that will sensitize local & international actors to HIV/AIDS prevention. The program will seek to expand the range of disciplines involved in HIV/AIDS research. It will enhance HIV/AIDS research capacity in order to produce credible information for local, national, & international policy-making & program development. It will promote an interest in HIV/AIDS research among new researchers by providing research fellowships, mentorship, & training opportunities. The program will contribute to the development of linkages between Northern & Southern researchers & encourage partnerships between research organizations, advocates, & decision makers. It will increase its visibility & credibility among the donor community & will convince them of the utility of supporting HIV/AIDS prevention.

23 Developing a Vision or Mission Warm-up general conversation Individual write up Discuss & Post Facilitator and/or participant draft Present & discuss Purge the jargon Verify with partners (if required) & Finalize

24 Developing a Vision with the Community in Nagaland

25 Suggestions and Tips! If time is a limiting factor you can carry out an survey in advance to create a draft vision & mission before a workshop starts. Use the draft version as a starting point for discussion in order to save time.

26 Exercise: Correct Errors in Vision & Mission Statements

27

28 Step 3: Boundary Partners Those individuals, groups, & organizations with whom the program interacts directly to effect change & with whom the program can anticipate some opportunities for influence.

29 Boundary partners are within the program`s sphere of influence The Real World Project or Program = Boundary Partners

30 Boundary Partners have boundary partners of their own Program Program s Boundary Partners Boundary Partners Boundary Partners

31 Typology of IDRC Boundary Partners IDRC Program Applied Research Community e.g. researchers, institutions, NGOs, networks System Gatekeepers / Dev`t Agents e.g. business, policy makers, NGOs, gov`t Beneficiaries e.g. communities, farmers, women, youth

32 Facilitation Questions In which individuals, groups, or organizations is your program trying to encourage change so that they can contribute to the vision? With whom will you work directly?

33 Suggestions and Tips! A program can choose its boundary partners because it wants to influence them or because they will influence others Example: a rural development NGO chooses to work directly with farmer organizations who in turn, lobby the Minister of Agriculture to effect change.

34 Exercise: Identify the Program`s Boundary Partners (training, content and technology)

35 Step 4: Outcome Challenges Outcome Challenge Statements: Describe the boundary partner`s contributions to the vision Describe the ideal behavioural changes Are about a single boundary partner

36 Step 4: Outcome Challenge Description of: The changed behaviours (relationships, activities, and/or actions) of a boundary partner How they could contribute ideally to the vision

37 Example Outcome Challenge Statement The program intends to see researchers in developing countries that have gained the expertise required to carry out HIV/AIDS research in a multidisciplinary fashion using quantitative, qualitative, and/or participatory methodologies. They are conducting research on the legislative, economic, social, and health aspects of HIV/AIDS with special emphasis on vulnerable groups including youth and women. They are providing credible evidence that allows research findings to be translated into effective HIV/AIDS policies and programs. They are skillfully disseminating research results in the appropriate format for the audience (e.g., to the scientific community, the general public, NGOs, government officials). They are successfully obtaining funding from multiple sources to conduct HIV/AIDS research. More experienced researchers are mentoring those new to the field and HIV/AIDS researchers globally are engaged in productive communications using various means especially electronic networks.

38 Phrasing of an Outcome Challenge The program intends to see [boundary partner] who [description of behaviours in the active present tense]

39 Facilitation Questions Ideally, in order to contribute to the vision, how would the boundary partner be behaving? What would your boundary partner be doing to contribute maximally to the vision?

40 Suggestions and Tips! Similarity of behaviours sought among boundary partners (policy makers and community members) in international model forests was unexpected by program staff

41 Outcome Challenge Checklist Outcome Challenge Statements: Describe behavioural changes About a single boundary partner Set out the ideal behavioural changes Describe the boundary partner`s contributions to the vision

42 Exercise Identify Outcome Challenge Statements

43 Step 5: Progress Markers Love to see PMs Like to see PMs Expect to see PMs

44 Step 5: Progress Markers A graduated set of statements describing a progression of changed behaviours in the boundary partner Changes in actions, activities, & relationships leading up to the ideal outcome challenge statement

45 Levels of Progress Markers The program sets out what it would: Expect to see its boundary partners doing? Like to see its boundary partners doing? Love to see its boundary partners doing?

46 How Many Progress Markers? Limit to a manageable amount of data to be collected: 4 expect to see 8 like to see 3 love to see

47 Why Graduated Progress Markers? Articulate the complexity of the change process Permit on-going assessment of partners progress (including unintended results) Encourages the program to think about how it can intentionally contribute to the most profound transformation possible Make mid-course corrections & improvement easier

48 Example Progress Markers The program Expects to See local communities: 1. Participating in regular model forest (MF) partnership meetings Establishing a structure for cooperation in the partnership Acquiring new skills for involvement in the MF Contributing resources necessary to get the MF operational (minimum)

49 Example Progress Markers The program would Like to See local communities: 5. Articulating a vision for the MF that is locally relevant 6. Promoting their involvement with the MF nationally 7. Expanding the partnership 8. Calling upon external experts when necessary 9. Requesting new opportunities for training 10.Disseminating concrete examples of benefits arising from MF activities 11.Seeking out new partners for the MF 12.Obtaining funding from different national sources

50 Example Progress Markers The program would Love to See local communities: 13. Helping other MFs establish themselves 14. Sharing lessons-learned internationally 15. Influencing national policy debates & formulation on resource use and management

51 Facilitation Questions «How can the program know the boundary partner is moving toward the outcome? What would they be doing?» «What milestones would be reached as the boundary partner moves towards their intended role in contributing to the vision?»

52 Suggestions and Tips! If a program has multiple boundary partners but the primary result to be achieved is changing their relationships with one another then it is possible to create a single outcome challenge & set of progress markers for the whole group

53 Progress Marker Checklist Each Progress Marker: Describes a changed behaviour by the boundary partner Can be monitored & observed As a set, Progress Markers: Are graduated from easier to more difficult to achieve changes in behaviour Describe the change process of a single boundary partner

54 X

55 Exercise Brainstorm expect, like and love to see progress markers

56 Guiding Questions for Open Discussion In your context: How would the ideas of Outcome Mapping resonate? Is a focus on behavioural change appropriate? How would you foster the imagination of your group t come up with a vision? Are you clear about your contributions to the vision? (i.e., your mission) Who are your boundary partners? How could you include them in the process of setting outcome challenges and progress markers?

57 Agenda for Day 2 Outcome Mapping Recap Strategy Maps Organizational Practices Monitoring and Evaluation Next Steps

58

59

60 Influencing Outcomes is not just about boundary partners Program Boundary Partner Program relevance & viability Org. Practices Program Delivery Strategies Program Results Progress Markers

61

62 6 Types of Strategies Strategy Causal Persuasive Supportive Aimed at the Boundary Parnter Direct Output Arouse New Skills/ Thinking Supporter who guides change over time Aimed at the Boundary Partner`s Environment Alter physical or regulatory environment Modify the information system Create / Strengthen a Peer Network

63 Facilitation Questions I-1 What will be done to produce an immediate output? I-2 What will be done to build capacity? I-3 How will sustained support, guidance, or mentoring be provided? E-1 E-2 E-3 What will be done to change the physical or policy environment? How will you use the media or publications? What networks/ relationships will be established or utilized?

64 Exercise Plot Activities on a Strategy Map

65

66 «The oft repeated and echoing question in Outcome Mapping «what or who needs to change?» raised us from being providers of development, achieving outputs, to actually believing we were agents of change.» Raj Verma, NEPED

67 Step 7: Organizational Practices The things that you do as an organization to: foster creativity & innovation seek the best ways to assist your partners maintain your niche

68 Why Organizational Practices? Important to how the program is going to function to effectively fulfill its mission Supporting change in its boundary partners requires that the program be able to change and adapt as well

69 Organizational Practices Prospecting for new ideas, opportunities, & resources 2. Seeking feedback from key informants 3. Obtaining the support of your next highest power 4. Assessing & (re)designing products, services, systems, and procedures

70 Organizational Practices Checking up on those already served to add value 6. Sharing your best wisdom with the world 7. Experimenting to remain innovative 8. Engaging in organizational reflection

71 Facilitation Question What do you have to be doing in order to remain effective?

72 Suggestions and Tips! This can be a difficult set of concepts to get across; frame it in terms of changes in behaviour internal to the organization Have the group think of the program as an organization in its own right

73 Exercise Identify the Organizational Practices

74 Guiding Questions for Application In your own context: How would the idea of planning beyond what can be achieved be accepted? (i.e. outcome challenge and progress markers) What kinds of strategies do you use causal, persuasive, supportive? How will you get your group to talk about your own organizational practices?

75 Stage 2: Outcome & Performance Monitoring

76 Outcome Mapping Monitoring Systematic collection of data A regular learning & improvement cycle Encourages the program to challenge itself Collects & organizes, does not analyze data

77

78 Step 8: Monitoring Priorities Tempting to gather information on a whole range of issues but this is not practical Base priorities on intended use of intended users: Use(s) for findings and from process Improve performance through learning Help meet accountability/reporting requirements Supply information for planned evaluation Inform publicity documents, communication activities, or case-study materials

79 Monitoring Plan Monitoring Priority Who will use Purpose When is it needed Who will collect How often Proposed Tool Strategies Strategy Journal Org l Practices Boundary Partner Outcomes Performance Journal Outcome Journal

80 Boundary Partner Program

81 Boundary Partner Program Outcome Challenge & Progress Markers

82 Strategies & Activities RAPID Programme Boundary Partner Program

83 Boundary Partner Program Organizational Practices

84 Boundary Partner Program Outcome Journal Strategy Journal Performance Journal

85

86 Step 9: Outcome Journal Uses progress markers to chart boundary partner`s progress Sets starting point or benchmark Captures details on changes in the boundary partner, contributing factors & actors, and sources of evidence

87 Outcome Journal Work Dating From/To: Contributors to Monitoring Update: Outcome Challenge: Low = Medium = High = Expect to see: Who? LMH 1 2 3

88 Set Values for Low, Medium & High Can capture either quantity or quality of change: Quantity (number of boundary partners) Quality (depth of change)

89 Baseline multiple individuals, groups or organizations in each boundary partner set identify each boundary partner s starting point in terms of progress markers boundary partners may be at different stages of development individual change process can be captured over time

90

91 Step 10: Strategy Journal Captures data on the strategies being employed to encourage change in the boundary partner Helps the program gauge its contributions to the achievement of outcomes and modify its actions accordingly Feeds into work planning & (modifications) or new activities

92 Strategy Journal WorkingDatingFrom/To: Contributors to Monitoring Update: Strategy to be Monitored: Effectiveness? (How did it help the boundary partner?) Outputs

93 Step 11: Performance Journal Records data on how the program is operating as an organization to fulfill its mission Includes information on the program s practices Feeds into work planning

94 Performance Journal WorkingDatingFrom/To: Contributors to Monitoring Update: Practice 1: Prospecting for New Ideas, Opportunities, & Resources Example or Indicators: Sources of Evidence: Lessons: Practice 2: Seeking Feedback from Key Informants Example or Indicators: Sources of Evidence: Lessons:

95 Exercise Identify your monitoring priorities

96 Stage 3: Evaluation Planning

97

98 Step 12: Evaluation Plan A short description of the main planning elements of an evaluation study For Whom? What? Where? How? By Whom? When? How Much?

99 Setting Evaluation Priorities Criteria with which to select include: Learning needs Reporting requirements Partners information needs

100 Guiding Principles for Evaluation Balancing learning & accountability Action oriented Each study is unique Process not only product Build capacity at all levels through participation

101 Exercise: Elements of an Evaluation Plan Evaluation Issue: Who Will Use Evaluation, How, & When? Questions to be Answered Info. Sources Evaluation Methods Who Will Conduct & Manage the Evaluation? Date (Start & Finish) Cost

102 Conclusion

103 Conclusions OM can help a program to be more strategic about the actors it targets, the changes it expects to see, and the means it employs to support and facilitate those changes. OM is not based on a cause-effect framework; it recognizes that multiple, nonlinear events lead to change. It sees interrelationships, not a snapshot. Attribution becomes contribution.

104 Conclusions Focus is on behavioural change monitors and evaluates whether a program has contributed to changes in behaviours in a way that would be logically consistent with supporting development changes in the future. Program must also learn and change, reconsider and adjust its goals, methods, interventions. Self-assessment, systematically collected.

105 Next Steps Outcome Mapping Action Plans Completed and shared with Sana / Ben and online Keep in touch online, share problems and ideas Share your thoughts via pan10@outcomemapping.ca

106

107

108 To learn more & to share your experiences