CMAM Forum Monitoring and Evaluation Plan and Tools. Version: January 31, 2013

Transcription

1 CMAM Forum Monitoring and Evaluation Plan and Tools Version: January 31, 2013 i

2 Table of Contents Abbreviations 1. Introduction 1 2. Objective 1 3. Conceptual Framework 1 4. Methods 3 5. Analysis and Reporting 4 6. Activities and Timeline 4 6. M&E Deliverables 5 7. Budget (Pending) 5 ANNEX 1. CMAM Forum Information Products and Services 6 ANNEX 2. CMAM Forum Logical Framework 7 ANNEX 3. Detailed Conceptual Framework for M&E 9 ANNEX 4. Potential CMAM Forum Audiences 10 ANNEX 5. CMAM Forum Operations Tools 11 ANNEX 6. Terms of Reference for the Kenya Country Case Study 11 ANNEX 7. Network Functions Approach 14 ANNEX 8. CMAM Forum Indicators 15 ANNEX 9. Questionnaires 19 iii ii

3 Abbreviations BA - Baseline Assessment CCS - Country Case Study CMAM - Community-based Management of Acute Malnutrition ECHO - European Commission - Humanitarian Aid and Civil Protection EE - End Evaluation FAQ - Frequently Asked Question FGD - Focus Group Discussion KII - Key Informant Interview MOH - Ministry of Health M&E - Monitoring and Evaluation MTR - Mid Term Review NFA - Network Functions Approach NGO - Non-Governmental Organisation OECD/DAC - Organisation for Economic Cooperation and Development/ Development Assistance Committee OT - Operations Tools QA - Quarterly Assessment SAM - Severe Acute Malnutrition SC - Steering Committee TB - Technical Brief TOR - Terms of Reference UN - United Nations UNICEF - United Nations Children s Fund USAID - United States Agency for International Development iii

4 1. Introduction Community-based Management of Acute Malnutrition (CMAM) has been adopted by over 60 countries to help combat acute malnutrition in children under 5 and reduce mortality in children under 5. This rapid expansion has occurred in less than a decade since CMAM s inception in 2002, and involves a wide range of actors in diverse emergency, development and transitional contexts. Many governments and other stakeholders face similar challenges in tailoring the CMAM approach to their contexts, while improving the quality of implementation, and strengthening the integration and scale-up of CMAM interventions. These challenges have escalated the demand for shared evidence, experiences and lessons learned. The lack of such information has made it difficult to achieve greater impact and leverage coordinated and effective management of acute malnutrition in a coordinated and effective manner. The CMAM Forum was created in 2011 to support scale-up of quality services for the management of acute malnutrition by facilitating the dissemination of technical guidance, evidence and learning interactively with the wider nutrition and health community. These CMAM Forum information products and services (Annex 1) are available at The CMAM Forum is managed by a steering committee of representatives of national ministries of health, United Nations (UN) agencies, non-governmental organisations (NGOs) and technical advisory bodies, with day-to-day work coordinated by two part-time co-facilitators. The CMAM Forum is currently funded for 2 years (January 2013 December 2014) 1. Annex 2 is the CMAM Forum logical framework, which was updated in December Objective The CMAM Forum has established a monitoring and evaluation (M&E) system to learn the most efficient and effective ways to facilitate the dissemination of information products and services adapted to needs and for use at global and country levels. The specific objectives of the M&E system are listed below: 1. Assess effectiveness (audience, reach, usefulness, use and impact) of CMAM Forum information products and services at the global and country level. 2. Assess global and country-specific needs and formulate recommendations to improve CMAM Forum effectiveness. 3. Assess CMAM Forum project progress and report on a timely basis to donors and steering committee. 3. Conceptual Framework A conceptual framework adapted from the USAID Guide to Monitoring and Evaluating Health Information Products and Services 2 guided the development of the CMAM Forum M&E plan. Figure 1 shows the conceptual framework with the major headings, highlighting the key categories for M&E (that will be adapted for CMAM Forum use). The framework shows how data from routine monitoring activities and occasional in-depth studies feed back into inputs, processes and outputs, and inform and improve CMAM information products and services, completing the loop. Annex 3 is a more detailed conceptual framework that groups inputs, processes, outputs, and initial, intermediate and long-term outcomes. These elements are defined below: Inputs are all resources put into the project. Processes are activities undertaken to achieve specific goals using inputs (how and how well an activity is carried out). 1 Funded by ECHO (2 years) with co-funding from UNICEF (1 year) 2 Sullivan T et al. (2007) Guide to Monitoring and Evaluating Health Information Products and Services. USAID, Washington DC, USA 1

5 Outputs are results of strategic dissemination of information products and services to the defined audiences. Audiences are individuals or entities who engage with the CMAM Forum. They can be grouped by employment, educational background or professional profile. Annex 4 lists categories of potential CMAM Forum audiences. Reach considers how and whether CMAM Forum information products or services get into the hands of intended audiences through 1) active distribution or push, 2) passive re-distribution or pull, 3) interactive uploading of information or requesting the creation of information and messaging, and 4) referral. Initial outcome: Usefulness is determined by audience satisfaction and perception of the quality (appropriateness, innovation, applicability and practicality) of information products and services. Intermediate outcomes: Use is determined by the adaptation of information products and services to inform policy, improve the quality of services and programmes, enhance education and training and promote research efforts; what is done with knowledge gained how knowledge is absorbed and applied to initiate or implement changes, and to what extent information products and services are used or adapted by members and users. Collaboration and capacity development look at how development and use of information products and services promote sharing of ideas, leveraging of resources, and jointly developing innovative solutions, and improve the sustainability of health services. Intended long-term outcome: Improved health outcomes as a result of exposure to information products and services. While it is difficult to measure the actual impact of information sharing, indicators under Use are seen as proxies for measuring the impact of information sharing on improved health outcomes. To maximise use and thus impact, target audiences need to be covered as thoroughly as possible. In that respect, both the number and role of members and users for participating in the CMAM Forum are important. Figure 1: Conceptual framework for monitoring and evaluating CMAM Forum information products and services Adapted from: Sullivan T et al. (2007) Guide to Monitoring and Evaluating Health Information Products and Services. USAID, Washington DC, USA 2

6 4. Methods Qualitative and quantitative data collection methods are used for routine monitoring, in-depth studies and evaluation at global and country levels. a. Routine monitoring: Operations tools updates (e.g., guides, lists, spreadsheets, and matrices) (Annex 5) will provide information on input and process indicators. Website statistics will include website usage information on output, audience and reach indicators (e.g., membership, online database population, dissemination, visits and downloads, and information requests). b. Occasional in-depth studies: Surveys will provide information on reach, usefulness and use indicators. Surveys can target all members (e.g., census for country surveys) or a sample of members (e.g., simple randomised, purposive or convenience sampling for global surveys). Surveys may include the following: - e-surveys (and/or m-surveys) of members and targeted users through internet, or mobile phone, - Key informant interviews (KII), focus group discussions (FGD), discussions in workshop-style meeting or other forms adapted to the contexts, - Mini surveys that appear as pop-up request on the website focusing on specific questions such as barriers to becoming a member, satisfaction with the website, or suggestions for improving the use of the website. Web search for information on citations will count the number of times the CMAM Forum or an own information product is referenced in other information products or during events. Country case studies (CCSs) will be conducted in four countries to assess the effectiveness of CMAM Forum, identify opportunities to improve its global and country-level effectiveness and allow comparisons among countries and contexts to generate more broadly relevant evidence. Criteria for selection of countries for the CCSs are: 1) expressed interest from a Ministry of Health (MOH) team and/or technical working group, 2) geographical representation (one East African, one West African and one Asian country, including at least one francophone country, receiving ECHO and UNICEF support) and 3) different stages of implementation (CMAM integration into the basic package of health services versus CMAM pilot stage). Terms of reference (TOR) developed for the CCS in Kenya (Annex 6) will be adapted for the other three CCSs. The CCS methods include one in-country visit (or remote contact in case of political insecurity), member and user surveys, stakeholder meetings at national and subnational levels, and website statistics. A repeat CCS will be organised towards the end of 2014, and progress will be documented. d. Mid-term review: An internal mid-term review (MTR) will be conducted by the co-facilitators and M&E advisers, applying the Network Functions Approach (NFA) 3 (Annex 7) to evaluate the effectiveness of the CMAM Forum and identify possibilities for improvement in addition to the routine monitoring and in-depth studies. c. End-evaluation: 3 CMAM Forum staff and a few steering committee members will be invited to participate in a participatory discussion to analyse current activities and how they relate to the specific objectives of the forum (e.g., priorities, time consumption, budget and effectiveness). 3

7 An external end-evaluation (EE) will look at relevance, appropriateness, connectedness, coherence, coverage, efficiency, effectiveness, and impact of the CMAM Forum, by applying the organization for Economic Co-operation and Development (OECD)/Development Assistance Committee (DAC) evaluation criteria 4, and also consider the routinely monitored and reported information. It will also look at elements of complementarity or overlap with other initiatives and highlight the value added. CMAM Forum indicators to measure progress against the current project proposal (as summarised in the logical framework) relate to inputs, processes, outputs and audiences. Indicators measuring the effectiveness of the CMAM Forum s outputs relate to website usage, audience, reach, usefulness and use (including collaboration and capacity development) and impact as proposed in the conceptual framework. Annex 8 is a complete list of CMAM Forum indicators. Tools have been developed and/or used to accompany the data collection, assist in the data analysis and report the findings, and include the following: - Updated logical framework for the CMAM Forum project - Conceptual Framework for M&E - Lists, matrices, spread and tally sheets for updating information on operations - Website statistics on website usage - Questionnaires for country-specific e-surveys (and/or m-surveys), KII and FGD, and workshops (Annex 9). 5. Analysis and Reporting Input, process, output, outcome and performance data from routine and in-depth studies are summarized as per the conceptual framework and reported in quarterly updates, and mid-term and evaluation reports, and summarized in occasional presentations during events. In-depth studies, such as the four country case studies, will also be reported on separately. 6. Activities and Timeline Activities 2013 Q3 Develop M&E plan and budget, including M&E tools Develop, update and disseminate operations tools (on the website for internal use or shared drive with closed access) Assess website statistics quarterly and summarise website usage in quarterly and annual reports Develop the CCS TOR Conduct two CCSs in each of four target countries targeting audiences at national and sub-national levels, including member e-survey and analysis of website statistics Conduct e-surveys of members at global level Conduct e-surveys (and/or m-surveys) of members at country level Conduct in-depth discussions through KII, FGD, workshops or meetings during country visits Conduct a mid-term review of the effectiveness of the CMAM Forum and identify areas for improvement Conduct an external end evaluation (TOR to be based on this M&E plan) Submit quarterly reports, and reports on in-depth studies, including slide presentations for dissemination 2013 Q Q Q Q Q4 4 Beck T. Guide to Evaluating Humanitarian Action Using the OECD/DAC Criteria: An ALNAP Guide for Humanitarian Agencies. London: Overseas Development Institute

8 The M&E activities will institute a system to monitor and evaluate the overall successes of the CMAM Forum and the detailed elements of the M&E cycle. Global surveys and CCSs will be conducted using data from website statistics, surveys, KII, FGD and workshops or meetings. Data on website usage, audience, reach, usefulness and use information products and services will feed back into the system for continuous quality improvement. The in-depth studies will explore the needs of audiences (members and users), at the global level and in the target countries. They will also allow assessment of constraints and barriers to use of the CMAM Forum and discussion of solutions for improvement including for users in remote locations. A complete project package will be available at the end of 2014 for hand-over to the future home of the CMAM Forum. 6. M&E Deliverables 1. M&E plan and tools 2. Revised CMAM Forum logical framework 3. CMAM Forum conceptual framework 4. Country case study reports 5. Updated operations tools and databases 6. Mid-term review report 7. End evaluation report 8. Updated workplans that include action points for improving the Forum from the recommendations of the in-depth studies 7. Budget A specific M&E budget (pending) may be developed to include M&E plan and tools development and CCSs for which currently only CMAM Forum staff time and travel has been budgeted to date. The budget will also include the cost of a mid-term review and final external evaluation. Annexes 1. CMAM Forum Information Products and Services 2. CMAM Forum Logical Framework 3. Detailed CMAM Forum Conceptual Framework 4. Potential CMAM Forum Audiences 5. CMAM Forum Operations Tools Updates 6. Terms of Reference for the Kenya Country Case Study 7. Network Functions Approach 8. CMAM Forum Indicators 9. Questionnaires for CMAM Forum e-surveys, m-surveys, KII, FGD and Workshops 5

9 ANNEX 1. CMAM Forum Information Products and Services INFORMATION PRODUCTS External information products: Audio-visual materials Case studies Education materials Guides, guidance, guidelines, manuals, job aids Policies and strategies Publications, open access Publications, not open access, link Reports Reviews and evaluations Training materials Own information products: Audio-visual materials Country Case Study reports and summary report FAQs Flier (and/or brochure) Highlights: summary of new documents and initiatives listed on home page Joint agency technical support Monthly resources updates Policy briefs Published CMAM Forum presentations in journals or websites Quarterly community update (forthcoming) Quarterly MAM update (forthcoming) Quarterly SAM update (forthcoming) Slide presentations CMAM Forum Technical briefs Updated list of ongoing research Updated list of research needs (forthcoming) INFORMATION SERVICES Own information services: Advocating for translation of partner key information into French Answering requests for targeted information access Collating ongoing research Collating research needs Linking members through a messaging system Identifying new information monthly Interacting with members contributing to information and information development; Offering portal home for relevant task forces Populating and maintaining interactive website with searchable online database Producing advocacy briefs in response to expressed needs Producing FAQs in response to expressed or identified needs Producing technical briefs in response to expressed needs Providing information (and technical) advocacy support to partners Sharing monthly resources updates to members Translating own information products into French (and Arabic)* External information services: Discussion fora links e-learning links Related websites of partners and/or initiatives links Meetings and conferences links 6

10 ANNEX 2. CMAM Forum Logical Framework Aim and objective Specific objectives Core Indicators Sources of verification Aim Improve the survival, health and nutrition of young children and other vulnerable populations through improved quality and coverage of CMAM. Objective Support the scale-up of quality services for the management of acute malnutrition by facilitating the dissemination of technical guidance, evidence and learning interactively with the wider nutrition and health community. i. Ensuring relevant, up-to-date CMAM information products and services are accessible for policy making and advocacy, management and technical advice, service delivery, training and education, and research and knowledge management based on expressed needs from users ii. Producing technical and policy summaries (current thinking on priority themes) on acute malnutrition iii. Synthesising learning on efficient and effective ways to facilitate the dissemination of CMAM-related information products and services through conducting global and country case studies in selected countries, including in remote areas iv. Supporting CMAM-related advocacy campaigns and initiatives with relevant information products and services i. Number of countries committed to scaling up nutrition that achieve improved geographic coverage of CMAM [target value 10] ii. % of active CMAM Forum members who rate their experience of using the site as valuable or very valuable (citing specific examples of how used) [target value 70%] iii. Number of site users counted per month [target value 750] iv. Number of members counted in 2013, and in 2014 v. Number of technical summaries produced by CMAM Forum and accessed by users, by category [target value 30] vi. Number of new resources collected, vetted and uploaded, and accessed by users [target value 1000] vii. Consensus reached among stakeholders and action plan developed for transitioning CMAM Forum to a longer term institutional home [Target value: 1 consensus] viii. Number of CCSs and/or lessons learned reports on CMAM Forum use produced [target value: 8] ix. Number of adaptations to CMAM Forum way of working made based on lessons learned [target value 100%] x. Number of technical and policy-oriented events to which CMAM Forum actively contributed [target value=20] xi. Number of partner advocacy campaigns used CMAM Forum resources [target value=4] xii. Number of relevant technical and advocacy websites or groups linking [target value=10] UNICEF SAM Update, SUN strategies and monitoring reports, WFP monitoring data, Coverage Monitoring Network reports, website statistics, e- Survey, discussion KII, FGD, workshop, CCSs, quarterly reports, event reports. Intermediate Results Activities Indicators (Examples) Sources of verification Intermediate Result 1: Relevant, up-to-date resources on acute malnutrition made accessible. Website statistics; Operations tools. 1. Manage and promote interactive up-to-date website with information products and services: sourcing, collating and vetting of key acute malnutrition information products and services 2. Disseminate CMAM Forum developed information products and services 3. Share monthly updates on initiatives and events related to acute malnutrition and publications (newsletter) 4. Provide a flexible information portal to host related task forces or country- CMAM Fora 1. Number of information products and services uploaded, own/external, by category 2. Number of information products downloaded, own/external, by category 3. Number of newsletters disseminated 4. Number of taskforces using the portal service 7

11 Intermediate Result 2: Technical summaries and updates related to acute malnutrition produced and translated in response to expressed needs. Intermediate Result 3: Information on ongoing research and research gaps listed contributing to advancing the acute malnutrition research agenda. Intermediate Result 4: Learning on the most efficient and effective ways to improve website usage, reach, usefulness and use of information products and services globally and in selected target countries (country case studies) synthesised and shared. Intermediate Result 5: Advocacy initiatives on common and context-specific factors influencing or affecting acute malnutrition, quality improvement and scale-up supported using information products and services. Intermediate Result 6: System of continuous quality improvement of information products and services developed and activated. 5. Produce technical summaries and updates 6. Translate key documents 7. Advocate for translation of partner key documents 8. Collect and update information on ongoing research 9. Collect and list research gaps identified through various sources, eg: research requests provided by members, gaps identified and collated during conferences, meetings, publications and reviews 10. Develop and instigate country case studies for assessing needs and improving website usage, audience, reach, usefulness and use of CMAM Forum products and services, including possibility to support CMAM practitioners in remote locations 11. Develop case study reports with recommendations for improved website usage, audience, reach, usefulness and in-country use of CMAM Forum products and services 12. Synthesize and apply learning from case studies and member surveys to improve website usage, audience, reach, usefulness and use of CMAM Forum products and services 13. Support advocacy initiatives (actors and campaigns) 14. Advocate for improving information access and translation of key information (including promotion of use of CMAM Forum) 15. Forge strong alliances with key actors and initiatives and avoid duplication (collaboration) 16. Convene a meeting with key stakeholders to determine the long-term home and management of the CMAM Forum 17. Update operations tools and make these available on website (closed access) 18. Develop and instigate a continuous quality improvement system to strengthen website usage, audience, reach, usefulness and use of information products and services 19. Monitor CMAM Forum activities and develop quarterly updates 20. Evaluate CMAM Forum activities: Mid Term Review and End Evaluation 5. Number and list of own information products produced, by category 6. Number of key information products translated 7. Number of partner information products facilitated (advocated for) to be translated into French 8. Updated list of ongoing research 9. Updated list of research gaps 10. CCS TOR developed 11. Number of CCS reports completed 12. Number of other occasional in-depth studies with recommendations translated into action points and adapted workplan 13. Number of advocacy campaigns actively supported with information 14. Number of external information products uploaded and translated into French (and other language) 15. Number of technical or policy events attended as participant, presenter (active) or facilitator (lead role) 16. Meeting to discuss home held 17. Existence of updated operations tools 18. Application and use of operations tools 19. Number of quarterly updates sent to SC members 20. Number of other occasional in-depth study reports completed (mid-term review, evaluation) Operations tools. Operations tools; Website statistics; e- Survey; KII, FGD and workshop. Operations tools (including adapted; Website statistics; workplan); e-survey; KII, FGD and workshop. Operations tools; Website statistics. Operations tools; Website statistics; e- Survey; KII, FGD and workshop. 8

12 ANNEX 3. Detailed Conceptual Framework for M&E Inputs Processes Outputs Audiences Reach Initial outcomes (Usefulness) Human resources: Co-facilitators, country liaison officer, technical advisor, support officers, steering committee members, website developer (insiders); Technical experts, donors, researchers, members (outsiders). Institutional resources: Literature (published and unpublished); Information and communication systems and technology; Financial resources; Administrative support; Operations tools. Products and services, and dissemination strategies developed: Collecting, organizing and making available key information; Helping out to find information; Identifying audience needs for knowledge; Producing and translating technical information in response to expressed needs; Advocating for translation of key information; Collecting and listing information on ongoing research, and research needs; Learning from information reach, usefulness and use in selected countries for improved knowledge management; Supporting networks and hosting fora Supporting advocacy and advocacy initiatives; Developing and maintaining continuous quality improvement system for knowledge management. Information products: Own products- Technical briefs, FAQs, case studies, flier, research needs and ongoing research overview, monthly resources updates, presentations, articles. External products- Case studies, education materials, guides and guidelines, job aids, manuals, publications, reports, slide presentations, tools and toolkits, training materials, videos. Information services: Own services- Population of website with key information collected and vetted, and categorized; Production of technical briefs and FAQs in response to expressed needs; Interaction with contributing to information; Translation or advocacy for translation of information; Monthly identification of new resources; Highlights on home page; Messaging system for members; Assistance for information requests access; Portal home for relevant task forces; Overview of ongoing research and research needs; Links to other services- Discussion fora; Other websites, fora and initiatives; Trainings, courses and e-learning. Professional profile Employment Educational background Primary user or Push: Products made available on website and actively shared. Secondary user or Pull: Products downloaded and uploaded. Interaction: Uploading of documents and links to services; Suggestions for technical summaries, Communicatio n between members. Referrals: Products or website link posted on other websites, shared or transmitted to colleagues; membership subscription assisted. User satisfaction: Content, presentation and delivery facilitate understanding, adoption, and adaptation. Information product and service quality: Quality products and services perceived as credible, reputable, authoritative, and reliable. Intermediate outcomes (Use) Environment informed: - Evidence-based information contributing to policy, guidelines and/or increased resources for acute malnutrition obtained; - Scale-up of proven approaches supported; - Public awareness of acute malnutrition increased; - Accurate media coverage obtained. Services, programmes and practice enhanced: - Evidence-based practices adopted; - Increased access and use facilitated; - Quality improvement supported. Education and training enhanced: - Evidence-based information informed and updated educational and training programmes, and encouraged collaborative learning initiatives (e.g., internships, mentoring and problem solving). Research enhanced: - Research agenda and methods guided by evidence, and priority need shared; - Generation of new evidence facilitated; - Speed of research-to-practice increased, and speed of practice-to research increased; - Research priorities shared and coordination enhanced. Collaboration and capacity development facilitated: - Availability and access to information increased; - Knowledge and experience sharing increased; - Communities and networks strengthened; - Collaboration and innovation increased; - Resources saved. Intended longterm outcomes CMAM quality and scale up improved: Quality of acute malnutrition services improved; Increased expansion (scaleup or geographical coverage) of acute malnutrition services enabled; Increased access and use (treatment coverage) of acute malnutrition services enabled; Preventive acute malnutrition behaviour improvements facilitated; Acute malnutrition outcome improvements supported. 9

13 ANNEX 4. Potential CMAM Forum Audiences Professional Profile: - Policy and decision makers, Advocacy (Policy and decision makers, advocacy journalists, donors) - Management, Technical Advice (Health and nutrition managers, supervisors, technical advisors) - Service Delivery (Health care providers) - Training, Education (Trainers and trainees, educators and students) - Research, Knowledge Management (Researchers, knowledge managers) Educational Background: - Medicine (Physicians, paediatricians, gastroenterologists) - Nutrition, Dietetics - Nursing - Social and community development - Other Employment Category: - Government - Academic, Training, Research or Technical Institutions - Donors - UN Agencies - NGO-international - NGO-national, Faith-based Organization (FBO), Community-based Organisation (CBO) - International Federation of the Red Cross and Red Crescent Societies (IFRC), National Red Cross (RC)/Red Crescent (RC) Society, International Committee of the Red Cross (ICRC) - Independent - Private Sector (for profit) - Student - Other 10

14 ANNEX 5. CMAM Forum Operations Tools Budget trackers Document for uploading permission Document on classification guidance Document on house style and branding Guidance on external information products and services selection tool for quality and relevance Intra-net or share drive repository of operations tools List of staff and consultants hired List of events attended List of helpdesk requests List of information products and services List of members mailing list List of research contacts and other sources searched for research information List of SC members mailing list List of websites and other sources searched for information M&E plan including questionnaires and tools Partner agency contracts Project proposals (UNICEF, ACF, ECHO) TOR Country Case Studies TOR SC and general membership TOR task forces to host TOR technical brief, frequently asked questions and policy briefs TOR working groups (WGs): WG Recruitment; WG Country Liaison; WG Advocacy TORs or JD of staff and consultants Website privacy policy Website terms and conditions document Workplan 11

15 ANNEX 6. Terms of Reference for the Kenya Country Case Study 5 Introduction Community-based Management of Acute Malnutrition (CMAM) 6 has been adopted by over 60 countries to help combat acute malnutrition in children under 5 and reduce childhood mortality. This rapid expansion has occurred in a decade but is not yet successful in covering global or country needs. Many governments and other stakeholders share comparable challenges in tailoring the approach to their context, and improving quality of implementation, integration and scale-up of services for the management of acute malnutrition. Whilst a wealth of lessons on evidence and good practices is being produced on a daily basis, knowledge gaps and barriers to access information obstruct consolidated learning. The CMAM Forum was created in 2011 to ensure relevant, up-to-date CMAM/IMAM information products and services are accessible to practitioners, policy makers and other key stakeholders that respond to their needs. Information products and services include: policy documents; peer-reviewed articles; reviews; conference and workshop reports; guidelines; case studies; training materials; advocacy publications; updated summary of ongoing research initiatives. In addition, the CMAM Forum produces summaries on priority themes and questions related to the prevention and management of acute malnutrition in Technical Briefs and Frequently Asked Questions (FAQs). The CMAM Forum also supports acute malnutrition-related advocacy campaigns and initiatives with providing relevant evidence, information and resources. See Annex 1 of TOR for a comprehensive list of CMAM Forum information products and services. As a way to synthesise learning of efficient and effective ways to disseminate and optimize use of acute malnutrition-related information, a series of case studies will be conducted in selected countries. In order to structure the Country Case Study assessments the generic Conceptual Framework for Health Information Products and Services will be used. This generic Conceptual Framework is drawn from the Guide to Monitoring and Evaluating Health Information Products and Services (USAID, 2007). The Conceptual Framework introduces the elements of reach, usefulness, use and audience for the assessment of the effectiveness of the forum (see Annex 2 of TOR). A further explanation for these terms and the related indicators for the monitoring and evaluation (M&E) of the CMAM Forum are presented in the Indicator Guide in Annex 3 of TOR that has been extracted from the same source (USAID, 2007). Justification With the rapid expansion of CMAM/IMAM services to support global and country needs, the demand for acute malnutrition-related resources have increased. High quality information products and services are essential for policy makers and advocates, health care managers and providers, educators and trainers, and researchers in improving quality and coverage of CMAM/IMAM. CMAM/IMAM information products and services must communicate on key issues /questions relating to acute malnutrition: what works and what does not, what is the evidence; what looks promising; what is sustainable. Making up-to-date information available to the people who need it in an accessible format will contribute to the improvement in quality of services and enhance scale-up. The Kenya case study will highlight how successfully the CMAM Forum products and services are having the intended reach, usefulness and use at the country-level. Key stakeholders in Kenya will be asked to express their satisfaction and provide an appraisal of the quality and relevance of the products and services, discuss current and potential use, and give suggestions to redefine the need and improve the role of the CMAM Forum for their country. The experience from the Kenya case study will be used to inform plans for other country case studies. Objectives 1. To assess how effectively CMAM Forum information products and services reach the intended audiences examine perceptions of usefulness, and the extent to which these are being used or adapted to inform policy, improve services and programmes, enhance education and training, and promote research. 2. To identify opportunities for the CMAM Forum to improve information products and services and enhance reach, usefulness and use at the country level. Strategy, Methods and tools The strategy is collaborative and participatory combining survey, focus group discussions (FGD), key informant interview (KII) and participatory discussions in a workshop. Hence, both qualitative and quantitative methods and tools will be used which are presented under planning, for each of the activities below. Methods and tools will be 5 Annexes to this TOR are not included as they overlap with information and annexes included in this M&E plan. 6 CMAM is also known as Integrated Management of Acute Malnutrition (IMAM) or Community-based Therapeutic Care (CTC) or French or Spanish equivalents. 12

16 further refined and developed along the way in collaboration with the CMAM Forum co-facilitators and the Country Liaison Working group (which includes two people from the Ministry of Health in Kenya). Planning and timeline A. Preparation (Aug 15 Sept 29, 2013) 1. Share draft and finalise Country Case Study TOR with Country Liaison Working Group 2. Start planning first country visit to Kenya with the CMAM Forum key contact persons 3. Prepare and launch Member Survey for members in Kenya only (see Annex 4 for questionnaire) 4. Access website statistics for website usage 5. Prepare data sheet to analyse survey results 6. Develop guide for FGD and KII 7. Analyse Kenya Member Survey data and summarize findings 8. Prepare slide presentation summarizing findings of member survey B. Country visit (Sept 30 Oct 10, 2013) National level: Nairobi 1. Meet CMAM Forum key contact persons in MOH division of nutrition and nutrition sector coordination secretariat to brief and refine programme of visit including: a. Complete list of people for Key Informant Interviews b. Identify possible locations and plan field visit c. Define workshop programme 2. Implement a half- day workshop with participants selected among IMAM key stakeholders and CMAM Forum members to assess current reach, usefulness and use of CMAM Forum products and services. A brainstorming session will be held to identify ways for further improving CMAM Forum information products and services (see Annex 4 of TOR for workshop outline). The invitation list will be developed with the 2 members of the Country Liaison Working Group from Kenya and the country liaison focal point. The workshop will also be used to further fine-tune the programme of the visit. 3. Conduct KII with key stakeholders who are CMAM Forum members to: a. Further assess reach, usefulness and use and identify opportunities for improvement at both country and global CMAM Forum level b. Develop timeline of CMAM Forum usefulness and use and identify during which phases of IMAM implementation the forum is more and less useful and used c. Identify If and how CMAM Forum products and services are supporting the day to day work d. Assess whether there are any unexpected outcomes from use of the CMAM Forum products and services e. Formulate recommendations for further improvements of the CMAM Forum interactive website and information products and services 4. Complete audience matrix to identify which (key) stakeholder groups are not members of the Forum 5. Conduct Key Informant Interviews (KIIs) with key stakeholders that are non-members to identify barriers and constraints to becoming member or users 6. Conduct Focus Group Discussion (FGDs) with nutrition and health staff (MOH/NGO) with experience in remote duty stations to discuss the potential role of the CMAM Forum in improving quality of IMAM services in remote or insecure areas where there is a limited or high turn-over of human resources, for example through improving confidence and enhancing job satisfaction. Discuss how this could be further encouraged or tested in Kenya and what kind of MOH/UNICEF/technical working group or CMAM Forum support this would require. 7. Debrief MOH division of nutrition and nutrition sector coordination secretariat country liaison focal points and discuss need and action points for follow up. Field level visit: Isiolo A CMAM partner will facilitate a 3 to 4-day field visit (2-days for field assessment, 2 days for travel) to a remote location where IMAM is implemented. Activities will include the following: 1. Conduct KIIs or FGD with health and nutrition managers and practitioners to assess current use, barriers, and opportunities for improving membership, reach, usefulness and use of the CMAM Forum products and services 2. Visit a CMAM site if feasible and observe service delivery and quality Expected Outputs (Deliverables) 1. Kenya CCS report, including recommendations for improving the CMAM Forum information products and services 2. Slide presentation of Kenya CCS results 3. Updated CCS questionnaires for future use Annexes 13

17 ANNEX 7. Network Functions Approach The Network Functions Approach is a new method to rethink and reshape the work of existing networks and communities of practice. 7 It will be used as an innovative method to evaluate the effectiveness of the CMAM Forum in an alternative way to identify possibilities for improvement. This approach was developed by ODI in 2008 and applied to the Global Network of Civil Society Organisations for Disaster Reduction and to assist ALNAP in the development of its 5-year strategy and is based on the six functions of networks or communities of practice. As a mid-term evaluation the CMAM Forum staff and a few relevant SC members will be requested to participate in a 2-day participatory workshop in order to analyse current activities and how they relate to the specific objectives of the CMAM Forum, in terms of, e.g., priorities, time consumption and budget as well as their effectiveness. The network s six functions are the following: Community building promotes and sustains the values and standards of a network of individuals or groups. Filtering allows the organised and productive use of information providing decision-makers with a valuable support service. Amplifying means taking a private or complex message and turning it into a public or more understandable message. Learning and facilitating. Networks help members undertake their activities more efficiently and effectively. Like facilitators at a workshop, this function enables network members to acquire new knowledge and develop practical abilities, and learn collectively. Investing and providing. Networks offer a channel to give members the resources they need to undertake their activities. Investor and provider networks act mostly to connect donors, thematic experts, and trainers with the members of a network. Networks may also invest in or provide resources to non-member third parties, for example by collaborating to provide resources to organisations outside the network. Convening. Networks bring together individuals and groups from different nationalities, disciplines and practices. Convening allows the development of more systematic and sustainable linkages between groups. However, due to the complexity of this function, attention needs to be paid to issues of audience demands and contexts, credibility, authority and communication. Steps for applying the Network Functions Approach: 1. Analyse the relevance of the network s vision and mission 2. Map existing and planned activities against the six functions 3. Identify the current and planned balance of effort across the six functions 4. For each function, identify how the network role is balanced between Agency or Support 5. Rate efficiency and effectiveness 6. Reflect on the vision and mission 7. Agree ideal functional focus and role 8. What is needed to move from the existing focus and role to the ideal focus and role 7 ODI Background Note April 2008: Strengthening humanitarian networks: Applying the network functions approach 14

18 ANNEX 8. CMAM Forum Indicators First, a list of core indicators is provided that are crucial to report on during the end evaluation to measure the effectiveness of the CMAM Forum. Next, a comprehensive list of indicators is provided that covers all aspects of the M&E framework. CMAM Forum information products and services include both external and own (CMAM Forum) produced products and services (Annex 3), which in fact lead to separate indicators but are merged here to facilitate presentation and comprehension. CORE INDICATORS 8 (with overall indicator code) Source Timing i. Number of countries committed to scaling up nutrition that achieve improved geographic coverage of CMAM [target value 10] ii. % of active CMAM Forum members who rate their experience of using the site as valuable or very valuable (citing specific examples of how used) [target value 70%](U8,U2) iii. Number of site users counted per month [target value 750] (A4) iv. Number of members counted in 2013, and in 2014 (A5,A6) v. Number of technical summaries produced by CMAM Forum and accessed by users, by category [target value 30](O1) vi. Number of new resources collected, vetted and uploaded, and accessed by users [target value 1000](O2) vii. Consensus reached among key stakeholders and action plan developed for transitioning CMAM Forum to a longer term institutional home [Target value: 1 consensus] viii. Number of CCSs and/or lessons learned reports on CMAM Forum use produced [target value: 8](O3,O5) ix. Number of adaptations to CMAM Forum way of working made based on lessons learned (e.g., midterm review, CCSs, SC advice) [target value 100%](O24) x. Number of technical and policy-oriented events to which CMAM Forum actively contributed [target value=20](o17) xi. Number of partner advocacy campaigns used CMAM Forum resources [target value=4](us5) xii. Number of relevant technical and advocacy websites or groups linked [target value=10] (R10) UNICEF SAM Update, SUN strategies and monitoring reports, WFP monitoring data, Coverage Monitoring Network reports. Website statistics, e- Survey, discussion KII, FGD, workshop, CCSs, quarterly reports, event reports Endevaluation (EE) OVERALL INDICATORS (with indicator codes) Source Timing Input Indicators (I1-I11) a. Input indicators human resources 1. Existence of updated steering committee (SC) members list Operations Tools (OT) Baseline assessment (BA) - Quarterly assessment (QA) - End evaluation (EE) 2. Number (and list) of steering committee (SC) members by audience category OT QA - EE 3. % of FTE (full time employment) of technical staff (e.g., 2 staff part-time = 100%) OT BA - Mid-Term Review (MTR) - EE 4. % of FTE of operations support staff OT BA - MTR - EE 5. % of FTE of web development consultant OT BA - MTR - EE 6. % of FTE of technical support consultants OT BA - MTR - EE b. Input indicators institutional resources 7. Existence of updated list of information products sources OT BA - MTR - EE 8. Existence of updated list of research sources OT BA - MTR - EE 9. Existence of updated list of operations tools OT BA - MTR - EE 10. Existence of updated operations tools (stored in folder) OT BA - MTR - EE 11. Existence of budget tracking OT BA - MTR - EE Process indicators (P1-P5) 8 Based on the ECHO proposal. 15

19 1. Existence of selection criteria for external information products and services (quality and OT BA - MTR - EE relevance 2. Existence of updated repository of operations tools OT BA - MTR - EE 3. Level of effort (number of days per month, per Q) for own information products OT EE development (consultants) 4. Level of effort (number of days) for information products sourcing OT MTR - EE 5. Level of effort (number of days) for research sourcing OT EE Output indicators (O1-O34) a. Output indicators information products 1. Number and list of own information products produced, by category, language, new, total Web statistics QA - MTR- EE 2. Number of external information products uploaded, by language, new, total Web statistics QA - MTR - EE 3. Number of country case studies (CCSs) completed OT MTR - EE 4. Number of e-surveys (and m-surveys) of completed OT QA - EE 5. Number of other occasional in-depth studies completed OT MTR - EE b. Output indicators information services 6. Number and list of own information services, by category, language, new, and total OT QA - EE 7. Number and list of external information services, by language OT QA - EE 8. Number of information requests received OT QA - EE 9. Number of information requests answered OT QA - EE 10. Number of monthly updates sent to members OT QA - EE 11. Number of other communications sent to members Web statistics QA - EE 12. Number of quarterly updates sent to SC members OT QA - EE 13. Number of other communications sent to SC members OT QA - EE 14. Number of messages sent by members through the portal Web statistics QA - EE 16. Number of times the CMAM Forum website is visited, by country, language group, total Web statistics QA - EE 17. Number of technical or policy events (conferences, meetings, trainings) attended in OT QA - EE active (resource person, facilitator, presenter) or passive (participant) role 20. Number of advocacy initiatives supported OT QA - EE 21. Number of own information products translated into French (and other languages) Web statistics QA - EE 22. Number of external information products facilitated (advocated for) to be translated OT QA - EE into French (and other languages) 23. Number of taskforces using the portal Web statistics QA - EE 24. Number of adaptations made as per CCS recommendations OT EE c. Output indicators operations 25. Existence of baseline report OT BA - MTR - EE 26. Existence of external end of project evaluation report OT EE 27. Existence of M&E indicator spreadsheets OT BA - MTR - EE 28. Existence of meeting minutes of SC quarterly teleconferences OT QA - EE 29. Existence of mid-term report OT MTR - EE 30. Existence of end-term report OT EE 31. Existence of quarterly monitoring reports OT QA - EE 32. Existence of quarterly website statistics spreadsheets OT QA - EE 33. Existence of slide presentations in repository OT QA - EE 34. Consensus reached on future of CMAM Forum Meeting EE report Audience Indicators (A1-A6) (I.e., Individual or entity that receives CMAM Forum information products and services.) 1. Number of countries with members Web statistics QA - EE 2. Top 10 countries with highest number of users Web statistics QA - EE 3. Top 10 countries with highest number of members (excl headquarters) Web statistics QA - EE 4. Number of users (visits), by country, new, low-high income, language, total Web statistics QA - MTR - CCS - EE 16

20 5. Number of members, by country, new, low-high income, language, total Web statistics QA - MTR - CCS - EE 6. Number (%) of members by audience category Web statistics QA - CCS - MTR - EE Reach Indicators (R1-R12) (I.e., breadth and saturation (extent) of dissemination, distribution or referral. To what extend the information gets into the hands of the intended audience) a. Reach indicators: Primary distribution or Push (I.e., distribution directly from the CMAM Forum to the end user or to a secondary distributor) 1. Number of events where the flier has been distributed Web statistics QA - EE 2. Number of times information sent to members Web statistics QA - EE 3. Number of people engaged in information sharing OT CCS - EE b. Reach indicators: Secondary distribution or Pull (I.e., member-initiated distribution that occurs after the initial mailing announcement) 4. Three most popular downloaded CMAM Forum produced information products Web statistics QA - EE 5. Ten most popular downloaded external information products Web statistics QA - EE c. Reach indicators: Interactions (I.e., means through which people can interact with the CMAM Forum and its members) 6. Number of information product files contributed, overall OT MTR- EE 7. Number of suggestions for information sharing improvements, by category e-/m-survey CCS - EE 8. Number of communication messaging sent between members Web statistics QA - EE d. Reach indicators: Referrals (I.e., various means through which people can find their way to CMAM Forum information through websites, databases, libraries, word of mouth, and other channels) 9. Number of instances own information products indexed or archived in bibliographic Web search EE databases 10. Number of external linkages to CMAM Forum website and products counted Web statistics EE 11. Number of instances own products selected for inclusion in a library Web search EE 12. Number of communications confirmed sharing copies with others e-survey, CCS - MTR - EE anecdotal feedback Usefulness Indicators (U1-U10) (I.e., Usefulness is a proxy for perceived value of an information product or service that is appropriate, applicable and practical and includes aspects such as user satisfaction, quality, innovation and relevance.) a. Usefulness indicators: User satisfaction (I.e., User satisfaction measures how useful users deem a specific product to be overall in providing needed information. It provides a sense of the intended audience s preferences as to the presentation of information as well as their perception of its content.) 1. % of respondents who read or browsed information products, by own or external, e-survey - KII - CCS - Global survey category, overall FGD - WS 2. % of respondents who were satisfied with an own information product or service, by category, overall 3. % of respondents who rated the format or presentation of an own information product or service as usable (design features or presentation), by category, overall 4. % of respondents who rated the content of an information product as useful (perceived relevance and practical applicability of the content), by own or external, category, overall 5. % of respondents who reported knowledge gained from an information product (users feel that they have learned from an information product), by own or external, category, overall 6. % of respondents who reported that an information product changed their views (users views, attitudes, opinions, or beliefs changed as a result of the information in the information product - favourable or unfavourable), by own or external, category, overall 7. % of respondents who were aware of benefits of membership (monthly updates) 8. % of respondents who found the website functions satisfactory b. Usefulness indicators: Information product or service quality (I.e., Information product or service quality, relate to users perceptions of the quality of products and services in terms of authority, credibility, reputability, and trustworthiness.) 9. Number of external citations or references counted to an own product or service Web search EE 10. % of respondents gave a positive appraisal of Authority, Credibility, Reputation, or e-survey - KII and CCS - MTR EE Trust of information products or services FGD - WS Use Indicators (Us1-Us5) (I.e., Use categorized as instrumental, conceptual, and symbolic resulting in: Environment informed; Programmes and practices enhanced; 17

21 Training and education enhanced, Research enhanced; Collaboration facilitated) 1. % of respondents intended to use an information product or service (planned to put to e-survey - KII and CCS - MTR EE use the guidance, concepts, or data from an information product) FGD - WS 2. % of respondents adapted an information product or service (original information has CCS been altered to meet the needs of users in their context) 3. % of respondents used an information product or service to inform policy and advocacy or CCS to enhance programs, training, education, or research (use, outcomes of use) 4. % of respondents used an information product or service to improve their own practice or CCS performance (use and outcomes of the use at the individual level) 5 Number of partner advocacy campaigns that used CMAM Forum information products KII EE Indicators of Collaboration & Capacity Development (C1-C3) 9 (I.e., Contributing to the effectiveness and efficiency of the health intervention and associated health outcomes) 1. Number of instances entities worked together incited by the information products e-survey - KII and FGD - WS 2. Number of instances entities worked together incited by the information services 3. Number and type of capacity development efforts that used information products and services CCS - MTR - EE 9 Collaboration enables partners to share ideas, leverage resources, and jointly develop innovative solutions. Capacity development strengthens organizations and communities and plays a crucial role in the development of sustainable health programs. Both are powerful tools that can mobilize individuals, organizations, communities, and government entities to solve problems faster and generate long-lasting improvements. 18

22 ANNEX 9. Questionnaires A. Questionnaire for e-survey of Members This questionnaire will take approximately 5-7 minutes to complete. Please tick one answer (or more when indicated). [Audience] 1. In which category would you place your current work? Policy and decision-making, or advocacy Health and nutrition management, supervision, or technical advise Service delivery Training or education Research or knowledge management Other (specify) 2. Are you an employee of: Government Academic, training, research or technical institution Donor agency International non-governmental organization National non-governmental organization, faith-based organisation, or community-based organisation IFRC, ICRC, Red Cross/Red Crescent Society Independent UN agency Private sector Student Other (specify) [Reach] 3. How did you first learn about the CMAM Forum? Supervisor External advisor Colleague Friend or peer Internet Other (specify) 4. Have you recommended the CMAM Forum to others? 5. Have you helped anyone else become a member? 6. How often do you check the CMAM Forum website? Never Once a day Once a week Once a month 7. Have you used the CMAM Forum messaging system to send a message to another member? I did not know about the service 8. How many documents did you download in the previous month? ne Fewer than 5 5 or more I did not know I could download documents 9. How many documents did you upload in the previous month? ne Fewer than 5 5 or more I did not know I could upload documents 10. How many documents did you download from the last CMAM Forum Monthly Resources Update you received? ne Fewer than 5 19

23 5 or more I did not know I could download documents 11. Did you share the Monthly Resource Update with others? 12. Did you share documents you downloaded from the CMAM Forum website with others? [Usefulness] 13. From which sources do you obtain information on CMAM (Tick more than one if needed)? CMAM Forum website CMAM Forum Monthly Resource Update WHO website UNICEF website Emergency Nutrition Network Field Exchange or Nutrition Exchange Google Other (specify) 14. Are you satisfied with the CMAM Forum website and the information it provides? Neutral, no opinion 15. Are you satisfied with the quality of the Technical Briefs/FAQs? Neutral, no opinion I have not read the technical briefs 16. Are the topics addressed by the Technical Briefs/FAQs relevant for you? Neutral, no opinion I have not read the technical briefs 17. How do you rate the source of the documents provided on the CMAM Forum website? Reliable (from authors or sources that I trust and find credible or that have a good reputation) Unreliable (from unknown or unprofessional sources) 18. Are key documents from your country missing under the Country Tab? (if yes, please list) I don t know [Use] 19. Which of the Technical Briefs/FAQs have you used in your work? Assessment of Coverage of Management of Acute Malnutrition, Technical Brief No. 1, October 2012 Severe Acute Malnutrition and Infection, Technical Brief No. 2 May 2013 Use of MUAC for Severe Acute Malnutrition, FAQ 1 How Do We Estimate Caseload for SAM and/or MAM for Children Aged 6 59 Months in a Given Time Period? FAQ 2a How Do We Estimate Caseload for Pregnant and Lactating Women 6 Months Post-partum in a Given Time Period? FAQ 2b 20. If you used one or more Technical Briefs or FAQs, for what purpose? To develop policy documents To develop guidelines To develop training materials and use during training To develop technical documents 21. Did you adapt the document(s) to your local context? 22. Did you use the document(s) to change your own practice? 23. Did the CMAM Forum by one of its documents or services facilitate your collaboration with peers or partners? Never Once Many times (give examples of collaborations) [Identified limitations and barriers] 24. Do you face constraints in using the CMAM Forum website? 20

24 : Internet or computer access problems Other (specify) [Suggestions for improvements and proposed action points] 25. What themes would you like to see covered in future Technical Briefs or FAQs: 26. What suggestions, if any, do you have for improving the CMAM Forum website? 27. What other comments, if any, do you have about the CMAM Forum? B. Guidance Questions for Key Informant Interviews or Focus Group Discussions (members, users or neither) [Audience] 1. In which category would you place your current work? Policy and decision-making, or advocacy Health and nutrition management, supervision, or technical advise Service delivery Training or education Research or knowledge management Other (specify) 2. Are you an employee of: Government Academic, training, research or technical institution Donor agency International non-governmental organization National non-governmental organization, faith-based organisation, or community-based organisation IFRC, ICRC, Red Cross/Red Crescent Society Independent UN agency Private sector Student Other (specify) [Reach] 1. Do you have access to a computer with Internet access? 2. Are you aware of the CMAM Forum and its website? 3. Are you a member of the CMAM Forum? 4. Are you a user of the CMAM Forum? If not a user or a member, introduce the CMAM Forum and move to question 20. If a member, ask questions a to h If a user, ask questions a, b, d and f. a. How did you first learn about the CMAM Forum? Supervisor External advisor Colleague Friend or peer Internet Other (specify) b. Have you recommended the CMAM Forum to others? c. Have you helped anyone else to become a member? d. How often do you open the CMAM Forum website? 21

25 Once a day Once a week Once a month Never e. Have you used the CMAM Forum messaging system to send a message to another member? I did not know about the service f. How many documents did you download in the previous month? ne Fewer than 5 5 or more I did not know I could download documents g. How many documents did you upload in the previous month? ne Fewer than 5 5 or more I did not know I could upload documents h. How many documents did you download from the last CMAM Forum Monthly Resources Update you received? ne Fewer than 5 5 or more I did not know I could download documents i. Did you share the Monthly Resource Update with others? j. Did you share documents you download from the CMAM Forum website with others? k. Do you face constraints in using the CMAM Forum website? : Internet or computer access Other (specify) [Usefulness] 5. From which sources do you obtain information on CMAM (Tick more than one if needed)? CMAM Forum website CMAM Forum Monthly Resource Update WHO website UNICEF website Emergency Nutrition Network Field Exchange or Nutrition Exchange Google Other (specify) 6. Are you satisfied with the CMAM Forum website and the information it provides? Neutral 7. Are you satisfied with the quality of the technical briefs? Neutral t applicable because I did not read them 8. Are the topics addressed by the technical briefs relevant for you? Neutral t applicable because I did not read them 9. How do you rate the source of the documents provided on the CMAM Forum website? Reliable (from authors or sources that I trust and find credible or that have a good reputation) Unreliable (from unknown or unprofessional sources) 10. Are key documents from your country missing under the Country Tab? (please list) I don t know [Use] 22

26 11. Which of the Technical Briefs or FAQs have you used in your work (Tick more than one if needed)? Assessment of Coverage of Management of Acute Malnutrition, Technical Brief No. 1, October 2012 Severe Acute Malnutrition and Infection, Technical Brief No. 2 May 2013 Use of MUAC for Severe Acute Malnutrition, FAQ 1 How Do We Estimate Caseload for SAM and/or MAM for Children Aged 6 59 Months in a Given Time Period? FAQ 2a How Do We Estimate Caseload for Pregnant and Lactating Women 6 Months Post-partum in a Given Time Period? FAQ 2b 12. If you used one or more Technical Brief or FAQ, for what purpose (Tick more than one if needed)?? To develop policy documents To develop guidelines To develop training materials and use during training To develop technical documents Other (specify) 13. Did you adapt the document(s) to your local context? 14. Did you use the document(s) to change your own practice? 15. Did the CMAM Forum by one of its documents or services facilitate your collaboration with peers or partners? Never Once Many times (give examples) [Identified limitations and barriers] 16. Do you face constraints in using the CMAM Forum website? : Internet or computer access problems Other (specify) [Suggestions for improvements and proposed action points] 17. What themes would you like to see covered in future Technical Briefs or FAQs: 18. What suggestions, if any, do you have for improving the CMAM Forum website? 19. What other comments, if any, do you have about the CMAM Forum? Question for non-users 20. From which sources do you obtain information on CMAM (Tick more than one if needed)? WHO website UNICEF website Emergency Nutrition Network Field Exchange or Nutrition Exchange Google Other (specify) Questions for ALL 21. What coordination mechanism do CMAM partners use? 22. How do health managers, clinical health workers and nutritionists collaborate in your district/county? 23. How can we reach out to more people who work in nutrition (e.g., health managers, nurses, doctors, training and academic institutions, researchers)? 24. Is there a need to translate key documents into your country s language? (what language?) Questions on the Nutrition Exchange (NEX): 25. Do you know the Field Exchange (FEX)? 26. Do you know the NEX? 23

27 (if no, help link the respondent with NEX, when interested) 27. Do you have field experience related to nutrition that you would like to share in NEX? (if yes, help link the respondent with NEX) C. Questions National Workshop [Audience] 1. Who heard about the CMAM Forum before you were invited for this meeting? 2. Who used or visited the CMAM Forum website? 3. Who received the request to fill the e-survey questionnaire last week? Those are members. 4. Why are some of you users of the site but not members? Following questions are for group work, but can be discussed in plenary when the group is small (< 15) [Reach] 5. What could be opportunities for expanding the audience for the CMAM Forum in Kenya? [Usefulness] 6. Do a SWOT analysis of the CMAM Forum information products and services: identify strengths, weaknesses, opportunities and threats. Discuss access, reach, usefulness, relevance and quality) 7. What is your perceived value of the CMAM Forum website? (Share positive and negative elements). Do you have suggestions for improving it? 8. How do you perceive the value of the own CMAM Forum information products (i.e., technical briefs, FAQs, newsletter): did you read these, are you satisfied with these, are they usable, are they relevant, did they make you change your opinion? [Use] 9. Have you been able to use the CMAM Forum information products: could you absorb their contents or did you make changes to your work? (Give examples) 10. Is there a need for a country-based CMAM Forum as an independent entity? If yes, how would you conceptualise it? D. Questionnaire for Global User Pop-up Survey (To develop) E. Questionnaire for Global Member Survey (To develop) 24