So what about that behaviour change?

Transcription

1 Over 14 million views within a week of posting the video to YouTube. Impressions are useful measure for exposure, but the objective of a safety campaign is behaviour change, not impressions or click-throughs or Facebook shares. So what about that behaviour change?

2 Leah Waymark, the General Manager of Corporate Relations at Metro Trains, summarizes the evaluation criteria as if we can save one life or avoid serious injury, then that s how we ll measure the success of this campaign I can t find anything elsewhere that describes the campaign s evaluation strategy.

3 Workshop 2: Selecting and setting Behavioural Goals, and Social Marketing planning Drs Julie Huibregtsen and Professor Jeff French

4 Its about Behaviour

5 What is Sociale Marketing? Social Marketing is about creating something that is valued by your audience = citizen based Julie Huibregtsen, Huibregtsen Training & Advies

6 FREE HEALTHY breakfast club Eating healthy, before entering school Julie Huibregtsen, Huibregtsen Sociale marketing Music Club TV, music & celebrity magazines + a healthy breakfast Passing the doorstep, download some music and eat some sandwiches, while having fun with friends

7 Social Marketing Social Marketing Techniques Social Marketing Concepts Integrated Intervention Mix Co-creation through Social Markets Competition analysis and Action Systematic planning & evaluation Insight driven Segmentation Behavioral Influence Citizen/ Orientation Focus) Social Offerings (Idea, product, service experience Relationship building The Social Marketing Principle Social Value Creation through Exchange and Social Offerings( Idea, product, service experience

8 The big frustrating questions for Donors and Governments What is the impact of the funds we invest? What is the ROI? What have we learnt?

9 Francis Bacon 22 January April 1626 Knowledge is power Lord Kelvin 26 June December 1907 To measure is to know

10 If you can t measure it you can t manage it. What gets measured gets done

11 Social Marketing Planning Social Marketing planning models need to be applied alongside the concepts and principles set out above. The four step STELa model is one such planning process. French 2010 Learn Scope Test Test Enact Learn & Act Act The Rationale Situation Analysis Target Audience Profile Intervention Proposition Initial Marketing Objectives Marketing Intervention Mix Strategies Pre -testing and Piloting Report on the Pilot Programme Full Business Plan Time Frame and Key Mile stones Resources Allocation Stakeholder and Partner Management Evaluation and Monitoring Reporting Dissemination Review and Plan to Build on Learning and Evaluation

12 The 6 Must Do Tasks French 2012

13 Step 1 Setting aims and objectives AIM: a broad strategic purpose of a project, AIMS can be long term, medium term or short term. Objective: a specific measurable goal whose achievement will contribute towards the aim

14

15 Why is setting behaviour objectives so important?

16 Why so important? 1.Set clear behavioural goals, everyone (including yourself) knows what you want to achieve and are asking of your target audiance 2.You can messure it 3.You can define what the barriers and benefits are (because it s tangible) 4.It forces you to start with the target audience and determine what the current behaviour is and the desired behaviour must be

17 Focus on behaviour Set a behavioural goal: 2 % less obesity is not a behavioural goal,taking the stairs is Better awareness of environment isn t, recycling and so really throwing away your glass away is. Julie Huibregtsen, Huibregtsen Training & Advies

18 Example of desired behaviour: Fun Theory Julie Huibregtsen, Huibregtsen Training & Advies

19 SMART objectives Specific: Not open to different interpretations Measurable: Can observe & collect objective measures Achievable: Not attempting too much Realistic: Given human & financial resources Time bound: Time scales / stages clearly identified

20 Objectives can be divided into: Affective objectives, focused of feelings / emotion Cognitive objectives, focused on learning / knowing Psychomotor objectives, focused on doing/ behaviour

21 OLD KAB MODEL? Knowledge Attitude Behaviour

22 As a 2 write an Aim

23 As a 2 write an Behavioural Objective

24 Objectives can be divided into: Affective objectives, focused of feelings / emotion Cognitive objectives, focused on learning / knowing Psychomotor objectives, focused on doing/ behaviour

25 C4L Vid Alfie

26 Behavioural objectives

27 How to use behavioural objectives to evaluate social marketing programmes

28 It happens every day Colleagues, I recently ran a multi-channel media communications campaign for promoting colon cancer screening (CCS). I would have preferred an all-inclusive social marketing approach, but time, budget and the small p political considerations happened. One significant component left out of the mix was evaluation. At the time of our initial planning the team had neither funding nor an epidemiologist/evaluator. As the end of the fiscal year draws near, additional funds have been identified for evaluation. I ve been asked to supply evidence or citations to support my arguments I would appreciate any feedback you are willing to provide. Thanks Jo

29 Evaluation Judgement based on careful assessment and critical appraisal (WHO 1981) Making a judgement about the value of something by looking critically (Ewles and Simnett 1995)

30 Evaluation can be defined as: A process which sets out to determine systematically and objectively the, effectiveness, efficiency and relevance of activities based on specified objectives. Evaluation is a process for improving current activities, future programmes. Evaluation can also aid decisionmaking about resource allocation.

31 In 3 s discuss Why do people not evaluate?

32 Invisible Evaluation In Victoria AUSTRALIA, during 2001, the Government allocated $ 6 MILLION to a new advertising and communications campaign to reduce problem gambling. IN 2002 Calls to the problem gambling helpline increased by 70% and face-to face clients increased by 118%. The Department s Annual Report stated that an evaluation of the campaign had been funded and was in progress, however, the results of this publication remain unavailable

33 Spurious Evaluation Cause and effect

34 In 2 s why do people evaluate? List three reasons:

35 9 reasons for evaluation 1. To assess the extent to which projects are achieving their aims and objectives 2. To inform the developments of materials, methods and other interventions 3. To ensure ethical practice 4. To optimise use of resources

36 9 reasons for evaluation 5. To inform programme planning 6. To inform practice and contribute to the evidence base 7. Provide funders with evidence of success 8. Provide feedback for those involved 9. Help build an performance culture

37 4 step Evaluation 1. Setting Aims and Objectives 2. Identifying Evaluation Indicators 3. Choosing data collection methods 4. Dissemination and Action

38 What is a metric? A set of numbers that indicate that the objective you have set is being achieved

39 What kinds of measures or metrics can we use to measure social marketing programmes? Discuss

40 What Makes a good metric? 1. Rational 2. Relevant 3. Valid 4. Comparable 5. Timely 6. Scalable 7. Easily explained and communicated 8. Not induce perverse incentives or unintended consequences

41 You have identified a perfect metric when: 1. Nobody complains about it! 2. Users say it helps them choose services & builds confidence 3. Colleagues say it empowers and encourages them 4. Managers use it to improve efficiency 5. Politicians ask you to develop some more!

42 Evaluation Formative Evaluation: Indicators e.g.: Did we have the right data, did we use the right theory and evidence, did we set the right objectives. Process Evaluation: Indicators: e.g.: How many people took part in or took advantage of a service that was offered, how many people were contacted. Impact Evaluation: Indicators: e.g.: Data on immediate impact of the work. For example: reported increase in knowledge, reported behaviour or an immediate increase in the take up of a service. Outcome Evaluation: Indicators: e.g.: Data on the longer term effects of the project, observed behaviour and its consequences. For example: less obese people or more recycling.

43 Impact Evaluation

44 4 Types of Metric

45 Process metric:

46 Common Social Marketing Metrics

47 Cost Per Lead CPL = Cost Per Lead CPL = TPB TLG TPB = Total Programme Budget TLG = Total Leads Generated

48 Cost Per Change CPC = TPB NODBC 100, = 100 CPC = Cost Per Change TPB = Total Programme Budget NODC = Number of Desired Behaviour Changes

49 ROI (Return On Investment) The ratio of returned for every spent, once the activity has paid for itself e.g. 100,000 Spent 500,000 Gained 400,000 Net Ratio of 1:4

50 ROI Example estimates of ROI on a proposal to extend smoking cessation clinic opening times Numbers of current attendance annually 1,000 Percentage increase projected by increasing opening 10% times Number of new clients 100 Percentage of new clients expected to quit 25% Number of additional quitters 25 Average cost saving per quitter 3,000 Annual gross cost saving (25 x 3,000) 75,000 Annual costs of providing extended opening 20,000 Net cost savings 55,000 ROI (55,000/20,000) 2.75 : 1 or 275%

51 Insight driven Evidence driven Consistent Sustained Segmented approach 52

52 THINK Vid

53 THINK! CAMPAIGN SUCCESS ROI as social value Saved over 1000 lives and counting Prevented 90,000 injuries and counting Represents 3,494 people (and counting) who are alive and uninjured today who wouldn t have been without THINK! The monetary value to society was 4.2billion. It is estimated that, for every 1 spent on THINK! the campaign has saved society

54 A sustained programme Educational, enforcement and engineering to improve road safety. Pre Think! total casualties grew by 3%, and total accidents by 2%. Since Think! total casualties and accidents declined by 12% and 11% respectively. For more information see:

55 3 more ways to assess efficiency CBA VFM Pay Back

56 Cost Benefit Analysis The cost of the problem V the cost savings from the intervention

57 Payback Generating economic value e.g. recruitment of the best teachers improves education and economic output Generating revenue e.g. a campaign to encourage people to inform on tax avoiders Saving public money directly e.g. encouraging more people to complete tax returns online Saving public money indirectly e.g. smoking campaigns reducing the cost on the NHS Spending public money well e.g., reducing administration Costs

58 Value For Money' (VFM) Assesses if an organisation has obtained the maximum benefit from the resources it controls. Some elements may be subjective, difficult to measure, and intangible. Judgements are required. Takes account of : quality, cost, resources used, fitness for purpose, timeliness, and convenience to judge whether or not, together, they constitute good value.

59 ROI and VFM 100% Effective 0% Effective 100% Efficient 100% Quality Compliant Range of realistic impact 0% Efficient 0% Quality Compliant

60 Learning and continuous Range of realistic impact improvement is the major reason for identifying and using performance metrics

61 When to use SMART Objectives and when to use more general Goals Modified from Ralph D. Stacey: "Complexity and Creativity in Organizations"

62 Modified from Ralph D. Stacey: "Complexity and Creativity in Organizations"

63 Modified from Ralph D. Stacey: "Complexity and Creativity in Organizations"

64 Modified from Ralph D. Stacey: "Complexity and Creativity in Organizations"

65 MORE GENERAL GOALS AND EVALUATION SMART OBJECTIVES AND PRESCRIBED SYSTEMS AUDIT Source: Stacey RD. Strategic management and organisational dynamics: the challenge of complexity. 3rd ed. Harlow: Prentice Hall,

66 When to use tight Objectives When to use more general Goals When there is: Strong evidence & consensus, a high degree of specification of objectives and methodology is most effective. When there is : Less consensus on what works the best approach is to set general goals and foster evaluation and research to feed the evidence base.

67 Why is ownership of evaluation and behaviour goals important for organisations? 1. Self assessment creates ownership and can be a powerful tool for improvement 2. Without ownership target fixing and sabotage is possible

68 Involving recipients in evaluation 1. Helps to ensure that the project is meeting needs 2. Strengthens community ownership of the project 3. Helps to build community capacity 4. Can contribute to the sustainability of the project

69 Measurement Culture Performance Culture

70 Metrics must align with Strategic Goals and be Simple enough that each person sees that their work contributes to achieving the selected metrics.

71 Summary

72 SM a good buy? YES An Ounce of Prevention what are the returns? (2 nd Edn 1999) CDC - USA ftp://ftp.cdc.gov/pub/publications/mmwr/other/ozprev.pdf Returns on Investment in Public Health: An epidemiological and economic analysis (2003) Access Economics - Australia Securing good health for the whole population: Final report - February CDC Community Programme National Institute for Health and Clinical Excellencehttp://

73 Evaluation of programme objectives is about legacy

74

75

76 Many thanks