Over 14 million views within a week of posting the video to YouTube. Impressions are useful measure for exposure, but the objective of a safety campaign is behaviour change, not impressions or click-throughs or Facebook shares. So what about that behaviour change?
Leah Waymark, the General Manager of Corporate Relations at Metro Trains, summarizes the evaluation criteria as if we can save one life or avoid serious injury, then that s how we ll measure the success of this campaign I can t find anything elsewhere that describes the campaign s evaluation strategy.
Workshop 2: Selecting and setting Behavioural Goals, and Social Marketing planning Drs Julie Huibregtsen and Professor Jeff French
Its about Behaviour
What is Sociale Marketing? Social Marketing is about creating something that is valued by your audience = citizen based Julie Huibregtsen, Huibregtsen Training & Advies
FREE HEALTHY breakfast club Eating healthy, before entering school Julie Huibregtsen, Huibregtsen Sociale marketing Music Club TV, music & celebrity magazines + a healthy breakfast Passing the doorstep, download some music and eat some sandwiches, while having fun with friends
Social Marketing Social Marketing Techniques Social Marketing Concepts Integrated Intervention Mix Co-creation through Social Markets Competition analysis and Action Systematic planning & evaluation Insight driven Segmentation Behavioral Influence Citizen/ Orientation Focus) Social Offerings (Idea, product, service experience Relationship building The Social Marketing Principle Social Value Creation through Exchange and Social Offerings( Idea, product, service experience
The big frustrating questions for Donors and Governments What is the impact of the funds we invest? What is the ROI? What have we learnt?
Francis Bacon 22 January 1561 9 April 1626 Knowledge is power Lord Kelvin 26 June 1824 17 December 1907 To measure is to know
If you can t measure it you can t manage it. What gets measured gets done
Social Marketing Planning Social Marketing planning models need to be applied alongside the concepts and principles set out above. The four step STELa model is one such planning process. http://www.stelamodel.com/ French 2010 Learn Scope Test Test Enact Learn & Act Act The Rationale Situation Analysis Target Audience Profile Intervention Proposition Initial Marketing Objectives Marketing Intervention Mix Strategies Pre -testing and Piloting Report on the Pilot Programme Full Business Plan Time Frame and Key Mile stones Resources Allocation Stakeholder and Partner Management Evaluation and Monitoring Reporting Dissemination Review and Plan to Build on Learning and Evaluation
The 6 Must Do Tasks French 2012
Step 1 Setting aims and objectives AIM: a broad strategic purpose of a project, AIMS can be long term, medium term or short term. Objective: a specific measurable goal whose achievement will contribute towards the aim
Why is setting behaviour objectives so important?
Why so important? 1.Set clear behavioural goals, everyone (including yourself) knows what you want to achieve and are asking of your target audiance 2.You can messure it 3.You can define what the barriers and benefits are (because it s tangible) 4.It forces you to start with the target audience and determine what the current behaviour is and the desired behaviour must be
Focus on behaviour Set a behavioural goal: 2 % less obesity is not a behavioural goal,taking the stairs is Better awareness of environment isn t, recycling and so really throwing away your glass away is. Julie Huibregtsen, Huibregtsen Training & Advies
Example of desired behaviour: Fun Theory Julie Huibregtsen, Huibregtsen Training & Advies
SMART objectives Specific: Not open to different interpretations Measurable: Can observe & collect objective measures Achievable: Not attempting too much Realistic: Given human & financial resources Time bound: Time scales / stages clearly identified
Objectives can be divided into: Affective objectives, focused of feelings / emotion Cognitive objectives, focused on learning / knowing Psychomotor objectives, focused on doing/ behaviour
OLD KAB MODEL? Knowledge Attitude Behaviour
As a 2 write an Aim
As a 2 write an Behavioural Objective
Objectives can be divided into: Affective objectives, focused of feelings / emotion Cognitive objectives, focused on learning / knowing Psychomotor objectives, focused on doing/ behaviour
C4L Vid Alfie
Behavioural objectives
How to use behavioural objectives to evaluate social marketing programmes
It happens every day Colleagues, I recently ran a multi-channel media communications campaign for promoting colon cancer screening (CCS). I would have preferred an all-inclusive social marketing approach, but time, budget and the small p political considerations happened. One significant component left out of the mix was evaluation. At the time of our initial planning the team had neither funding nor an epidemiologist/evaluator. As the end of the fiscal year draws near, additional funds have been identified for evaluation. I ve been asked to supply evidence or citations to support my arguments I would appreciate any feedback you are willing to provide. Thanks Jo
Evaluation Judgement based on careful assessment and critical appraisal (WHO 1981) Making a judgement about the value of something by looking critically (Ewles and Simnett 1995)
Evaluation can be defined as: A process which sets out to determine systematically and objectively the, effectiveness, efficiency and relevance of activities based on specified objectives. Evaluation is a process for improving current activities, future programmes. Evaluation can also aid decisionmaking about resource allocation.
In 3 s discuss Why do people not evaluate?
Invisible Evaluation In Victoria AUSTRALIA, during 2001, the Government allocated $ 6 MILLION to a new advertising and communications campaign to reduce problem gambling. IN 2002 Calls to the problem gambling helpline increased by 70% and face-to face clients increased by 118%. The Department s 2002-2003 Annual Report stated that an evaluation of the campaign had been funded and was in progress, however, the results of this publication remain unavailable
Spurious Evaluation Cause and effect
In 2 s why do people evaluate? List three reasons:
9 reasons for evaluation 1. To assess the extent to which projects are achieving their aims and objectives 2. To inform the developments of materials, methods and other interventions 3. To ensure ethical practice 4. To optimise use of resources
9 reasons for evaluation 5. To inform programme planning 6. To inform practice and contribute to the evidence base 7. Provide funders with evidence of success 8. Provide feedback for those involved 9. Help build an performance culture
4 step Evaluation 1. Setting Aims and Objectives 2. Identifying Evaluation Indicators 3. Choosing data collection methods 4. Dissemination and Action
What is a metric? A set of numbers that indicate that the objective you have set is being achieved
What kinds of measures or metrics can we use to measure social marketing programmes? Discuss
What Makes a good metric? 1. Rational 2. Relevant 3. Valid 4. Comparable 5. Timely 6. Scalable 7. Easily explained and communicated 8. Not induce perverse incentives or unintended consequences
You have identified a perfect metric when: 1. Nobody complains about it! 2. Users say it helps them choose services & builds confidence 3. Colleagues say it empowers and encourages them 4. Managers use it to improve efficiency 5. Politicians ask you to develop some more!
Evaluation Formative Evaluation: Indicators e.g.: Did we have the right data, did we use the right theory and evidence, did we set the right objectives. Process Evaluation: Indicators: e.g.: How many people took part in or took advantage of a service that was offered, how many people were contacted. Impact Evaluation: Indicators: e.g.: Data on immediate impact of the work. For example: reported increase in knowledge, reported behaviour or an immediate increase in the take up of a service. Outcome Evaluation: Indicators: e.g.: Data on the longer term effects of the project, observed behaviour and its consequences. For example: less obese people or more recycling.
Impact Evaluation
4 Types of Metric
Process metric:
Common Social Marketing Metrics
Cost Per Lead CPL = Cost Per Lead CPL = TPB TLG TPB = Total Programme Budget TLG = Total Leads Generated
Cost Per Change CPC = TPB NODBC 100,00 1000 = 100 CPC = Cost Per Change TPB = Total Programme Budget NODC = Number of Desired Behaviour Changes
ROI (Return On Investment) The ratio of returned for every spent, once the activity has paid for itself e.g. 100,000 Spent 500,000 Gained 400,000 Net Ratio of 1:4
ROI Example estimates of ROI on a proposal to extend smoking cessation clinic opening times Numbers of current attendance annually 1,000 Percentage increase projected by increasing opening 10% times Number of new clients 100 Percentage of new clients expected to quit 25% Number of additional quitters 25 Average cost saving per quitter 3,000 Annual gross cost saving (25 x 3,000) 75,000 Annual costs of providing extended opening 20,000 Net cost savings 55,000 ROI (55,000/20,000) 2.75 : 1 or 275%
Insight driven Evidence driven Consistent Sustained Segmented approach 52
THINK Vid
THINK! CAMPAIGN SUCCESS ROI as social value Saved over 1000 lives and counting Prevented 90,000 injuries and counting Represents 3,494 people (and counting) who are alive and uninjured today who wouldn t have been without THINK! The monetary value to society was 4.2billion. It is estimated that, for every 1 spent on THINK! the campaign has saved society 9.36 54
A sustained programme Educational, enforcement and engineering to improve road safety. Pre Think! total casualties grew by 3%, and total accidents by 2%. Since Think! total casualties and accidents declined by 12% and 11% respectively. For more information see: www.thinkroadsafety.gov.uk
3 more ways to assess efficiency CBA VFM Pay Back
Cost Benefit Analysis The cost of the problem V the cost savings from the intervention
Payback Generating economic value e.g. recruitment of the best teachers improves education and economic output Generating revenue e.g. a campaign to encourage people to inform on tax avoiders Saving public money directly e.g. encouraging more people to complete tax returns online Saving public money indirectly e.g. smoking campaigns reducing the cost on the NHS Spending public money well e.g., reducing administration Costs
Value For Money' (VFM) Assesses if an organisation has obtained the maximum benefit from the resources it controls. Some elements may be subjective, difficult to measure, and intangible. Judgements are required. Takes account of : quality, cost, resources used, fitness for purpose, timeliness, and convenience to judge whether or not, together, they constitute good value.
ROI and VFM 100% Effective 0% Effective 100% Efficient 100% Quality Compliant Range of realistic impact 0% Efficient 0% Quality Compliant
Learning and continuous Range of realistic impact improvement is the major reason for identifying and using performance metrics
When to use SMART Objectives and when to use more general Goals Modified from Ralph D. Stacey: "Complexity and Creativity in Organizations"
Modified from Ralph D. Stacey: "Complexity and Creativity in Organizations"
Modified from Ralph D. Stacey: "Complexity and Creativity in Organizations"
Modified from Ralph D. Stacey: "Complexity and Creativity in Organizations"
MORE GENERAL GOALS AND EVALUATION SMART OBJECTIVES AND PRESCRIBED SYSTEMS AUDIT Source: Stacey RD. Strategic management and organisational dynamics: the challenge of complexity. 3rd ed. Harlow: Prentice Hall,
When to use tight Objectives When to use more general Goals When there is: Strong evidence & consensus, a high degree of specification of objectives and methodology is most effective. When there is : Less consensus on what works the best approach is to set general goals and foster evaluation and research to feed the evidence base.
Why is ownership of evaluation and behaviour goals important for organisations? 1. Self assessment creates ownership and can be a powerful tool for improvement 2. Without ownership target fixing and sabotage is possible
Involving recipients in evaluation 1. Helps to ensure that the project is meeting needs 2. Strengthens community ownership of the project 3. Helps to build community capacity 4. Can contribute to the sustainability of the project
Measurement Culture Performance Culture
Metrics must align with Strategic Goals and be Simple enough that each person sees that their work contributes to achieving the selected metrics.
Summary
SM a good buy? YES An Ounce of Prevention what are the returns? (2 nd Edn 1999) CDC - USA ftp://ftp.cdc.gov/pub/publications/mmwr/other/ozprev.pdf Returns on Investment in Public Health: An epidemiological and economic analysis (2003) Access Economics - Australia http://www.health.gov.au/internet/wcms/publishing.nsf/content/health-pubhlth-publicat-document-roi_eea-cnt.htm Securing good health for the whole population: Final report - February 2004 http://www.dh.gov.uk/en/publicationsandstatistics/publications/publicationspolicyandguidance/dh_4074426 CDC Community Programme http://www.cdc.gov/healthycommunitiesprogram/ National Institute for Health and Clinical Excellencehttp://www.sussedprofessionals.net/files/PH006quickrefguide.pdf
Evaluation of programme objectives is about legacy
Many thanks