VIGeZ, The Evaluation Matrix: a planning tool for the evaluation of health promotion projects, integrated with the RE-AIM model

Size: px

Start display at page:

Download "VIGeZ, The Evaluation Matrix: a planning tool for the evaluation of health promotion projects, integrated with the RE-AIM model"

Dinah Wells
7 years ago
Views:

1 The Evaluation Matrix: a planning tool for the of health promotion projects, integrated with the RE-AIM model 1

2 PROJECT OBJECTIVES? What project objectives are formulated in your project plan? SMART formulated Main objective / Strategic objective Operational objectives EVALUATION OBJECTIVES What do you and your stakeholders want to know at the end of the? SMART formulated. RE- AIM as a supportive framework for formulating main objectives (see next page). REACH: METHODS How and where (with whom) are you going to collect the required information? Design Source Quantitative / qualitative RESPONDENTS How many respondents are needed? How to select them? Representative ness Sample? AGENT Who executes the? Knowing and being able to TIME LINE When do you execute the? Expected results Design (prepost, ) BUDGET How much is the going to cost? All costs Cost of design Cost of quantitative / qualitative EFFECT(IVENESS): Translate the project objectives into objectives and fit them with RE AIM ADOPTION: IMPLEMENTATION: 2

3 MAINTENANCE: Evaluation objective: VIGeZ, 2014 Control your project objectives for all RE AIM dimensions and, by doing so, explore additional objectives 3

4 An adaptation of questions and levels, measurements, influencing factors, and improvement strategies based on the RE-AIM model Dimension Questions Level Measurement Influencing factors Improvement strategies REACH 1. What % of the target Individual 1. # people actually exposed or Available resources Identify and address potential (improve reach) population would ideally be served / # people ideally exposed (e.g. funding, staff, barriers exposed or served? or served and stakeholder Compare feedback from those (based on the context analysis) 2. Are those exposed representative of your target population? Low SES reflex, gender, age, 3. What are the facilitators or barriers for participation? 2. Compare characteristics between those actually exposed vs. those ideally exposed or vs. the whole population 3. Track for facilitators or barriers of participation support) Perceived benefits vs. costs Mandatory vs. voluntary policy actually exposed vs. those ideally exposed Include modifications and new ideas EFFECT(IVENESS) (effects on behavioural determinants or behaviour) (improve expertise, improve dissemination, ) 1. Will your intervention achieve the outcomes you intend? 2. Are there any unanticipated consequences? E.g.: impact on quality of life, behaviour, costs, negative effects. 3. Are the outcomes consistent across sub-groups of the target population? Low SES reflex, gender, age, 1. Will your intervention achieve the outcomes you intend? 2. Are there any unanticipated consequences? E.g.: impact on expertise, dissemination, costs, negative effects. 3. Are the outcomes consistent across subgroups of the population? Individual 1. Measure consequently: - if your project objectives include behavioural changes measure behavioural outcomes - measuring behavioural determinants 2. Look at unanticipated (+ and -) consequences 3. Examine impact across sub-groups of the target population 1. Measure correctly: fit the definition effect with the objective of the project: - e.g. expertise pre-post - e.g. dispersion - e.g. quality of 2. Look at unanticipated (+ and -) consequences 3. Examine impact across subgroups of organizations or s Intervention approach is evidence-based Strength of - Available resources (time and money) - Relationships with organizational partners Approach is evidencebased Relevant dissemination channels and stakeholders Strength of Develop documentation and tracking systems Allow for ongoing input from your target population Develop documentation and tracking systems Allow for ongoing input from s/organizations 4

5 ADOPTION ((intent to) improve application) IMPLEMENTATION (improve quality of ) Type of organization, urban/rural, size, 1. What % of the target organizations should adopt your intervention? (based on a network analysis) 2. Are the participating organizations representative of the target population? Type, urban/rural, size 3. What are the facilitators and barriers for participation? Within organizations: 4. Is the participating staff representative? Experience, age, gender, 1. What activities are required to implement your intervention? 2. What are the reasons for and barriers of qualitative? 3. What adaptations, if any, are necessary to the initiative? 4. What is the cost (time and money) of your intervention? 1. What products of the intervention are used by the individuals (civilian, student, employee, )? 2. Is the use of products different across subgroups? Low SES reflex, gender, age, different profiles, 1. # settings or s that actually participate / # settings or s that could participate 2. Compare characteristics between settings or s that participate vs. those that don t participate in the intervention 3. Track for facilitators and barriers for participating Define main intervention components or strategies. Monitor subsequently the degree and frequency. 1. Use of different products and strategies / total number of products and strategies. 2. Factors of success and barriers of 3. Suggestions for adaptions 4. Asses time and costs to implement Individual 1. Use of different products / total number of products 2. Comparison between users of products and total target population Fit with the values and capacity of the settings or s Sense of urgency around the issue Evidence of effectiveness Organizational capacity: budget and staff Barriers of Complexity of the intervention Specific type of organisation Specific types and level of staff Costs (time and money) Process of planning Recruitment of individuals and organisations Cooperation process and partnerships Perceived benefits vs. disadvantages Compulsory use vs. optional (policy) Meet with s during the development phase and the formative Ensure compatibility with the values and capacity of settings and s Identify necessary resources and required practices Build organizational support Meet with s during the development phase and formative Analyse when pilot testing the intervention Adapt your intervention to fit local culture and norms Provide staff training, technical assistance and protocols Track and adapt the intervention if necessary Expose barriers and find solutions Compare feedback of users and non-users Foresee adaptations if necessary and new ideas 5

6 MAINTENANCE (improve maintenance of effects) (improve sustainability of ) VIGeZ, Does your intervention produce lasting effects (longer than 6 months)? 2. What is the extent of the drop-out? Low SES reflex 1. What % of the participating organizations continues, maintains and changes the intervention? 2. What are the facilitators and barriers for maintenance? 3. Does your intervention produce lasting effects (if there is measurement of effectiveness on the setting level)? Individual 1. Measure correctly: - if your project objectives include behavioural changes measure behavioural outcomes - measuring behavioural determinants 2. Monitor repeated or continued participation. 1. # settings or s that actually participate / # settings or s that could participate 2. Compare characteristics between settings or s that participate vs. those that don t participate in the initiative 3. Facilitators and barriers for participating Continued social support, policy and environmental supports Perceived benefits vs. disadvantages Perceived benefits vs. disadvantages Available staff training, technical assistance and protocols Level of ongoing funding Repeat monitoring and documentation. Supporting measures (social, financial, policy) Minimize required resources and costs Design an intervention with low complexity or that can be adapted Foresee adaptable initiatives A synthesis of the Flemish Institute of Health promotion and Disease Prevention (VIGeZ) based on: Center for Training and Research Translation - and UNC Center for Health Promotion and Disease Prevention. Van Acker R Dissemination and research in physical activity promotion. Doctors degree in Physical Education. Ghent University. (In Press) Steenhuyzen S An ace in. A practical handbook about in health promotion. Flemish Institute for Health Promotion and Disease Prevention (VIGeZ). Saunders R, Evans M & Joshi P Developing a Process-Evaluation Plan for Assessing Health Promoting Program Implementation: A How-To Guide. Society for Public Health Education, Note: Low SES reflex indicates having attention for target groups of low socio-economic status. This is a priority in the field of health promotion and therefore applicable to the of multiple RE-AIM dimensions. 6

EU-WISE: Enhancing self-care support for people with long term conditions across Europe

EU-WISE: Enhancing self-care support for people with long term conditions across Europe Summary framework and study protocol for the EU-WISE intervention EUGENIE Project partners: SOTON UNWE UNAV UniO