Program Evaluation Tool

Transcription

1 Prgram Evaluatin Tl

2 Table f Cntents Prgram Evaluatin Tl Sectin 1 Overview Page 3 Steps, Wrksheets and Outcmes Page 4 Step 1 Select Evaluatin Team Page 5 Evaluatin Team Wrksheet Page 6 Step 2 Assess Organizatinal Capacity & Resurces fr Evaluatin Page 7 Evaluatin Resurces Wrksheet Page 9 Step 3 Develp Prgram Lgic Mdel Page 10 Prgram Lgic Mdel Template Page 12 Step 4 Define Evaluatin Questins Page 13 Evaluatin Questins Wrksheet Page 16 Step 5 Build Evaluatin Framewrk Indicatrs Page 17 Evaluatin Design Page 18 Data Cllectin Methds and Tls Page 19 Data Analysis Page 20 Evaluatin Framewrk Template Page 25 Step 6 Dcument the Evaluatin Plan Page 26 Evaluatin Plan Template Page 27 Step 7 Cllect & Analyze Data Page 30 Step 8 Dcument Evaluatin Reprt Page 31 Step 9 Disseminate Findings & Mdify Prgram Page 32 Disseminatin Plan and Actin Plan Templates Page 36 Glssary f Terms Page 38 Sectin 2 Evaluatin 101: Evaluatin Specialists Skills & Experience Page 42 Ptential Rles fr Evaluatin Team Members Page 44 Suggested Wrk Plan fr Evaluatin Team Meetings Page 45 Evaluatin Thermmeter Exercise Page 47 Prgram Lgic Mdels Tutrial Page 50 Prcess and Outcme Evaluatin Questins Page 56 Evaluatin Framewrk Indicatrs, Methds, Tls Page 59 Evaluatin Reprt Guidelines and Template Page 62 Sectin 3 Tips & Resurces: Select Evaluatin Team Page 67 Indicatrs Page 72 Data Analysis Page 74 = see Tips sectin fr mre infrmatin = see Resurces sectin fr mre infrmatin Cpyright 2009 by Twards Evidence Infrmed Practice, Ontari Public Health Assciatin, Trnt, Canada. Permissin t cpy this resurce is granted prvided surce is acknwledged. TEIP Prgram Evaluatin Tl Revised Octber 2009 Page 2

3 Overview The TEIP Prgram Evaluatin Tl prvides user friendly guidelines fr lcal prgram evaluatin. The Tl is designed t help practitiners develp realistic evaluatin plans that include bth prcess and utcme measures. This dcument des nt pretend t cver all aspects f prgram evaluatin a field f study requiring many years t master. Rather, it fcuses n the mre practical and pragmatic aspects f prgram evaluatin. The evaluatin prcess is divided int 9 steps with accmpanying guidelines and wrksheets. In additin, the Evaluatin 101 Sectin reviews key evaluatin cncepts fr thse wh are new t evaluatin r culd benefit frm a brief refresher. Further guidance is available in the Tips & Resurces Sectin. Develpment f the TEIP Prgram Evaluatin Tl Despite the availability f numerus excellent evaluatin resurces, lack f cnfidence, skills, and supprts fr prgram evaluatin is a perennial cncern amng health prmters in Canada. In TEIP hired Dr. Brian Rush, f Virg Evaluatin Planning and Cnsultatin t prvide intensive evaluatin educatin and supprt t the riginal TEIP Pilt cmmunities. Dr. Rush s apprach emphasizes the develpment f a healthy rganizatinal culture fr evaluatin and uses a participatry apprach t invlve prgram stakehlders in evaluatin planning. This Tl is the result f applying Dr. Rush s apprach t building lcal evaluatin capacity. Guiding Principles Infrming Tl Develpment Prgram evaluatin is strengthened by participatry evaluatin prcesses which: Invlve stakehlders thrughut the evaluatin prcess Increase usefulness f evaluatin results Increase rganizatinal evaluatin capacity Evaluatin plans must fit lcal evaluatin capacity and resurces. One must be selective and strategic in deciding what t evaluate. Identificatin f pririty evaluatin questins t demnstrate prgram impact and infrm prgram decisin making is key. Nt everything that can be cunted cunts and nt everything that cunts can be cunted Albert Einstein. Cpyright 2009 by Twards Evidence Infrmed Practice, Ontari Public Health Assciatin, Trnt, Canada. Permissin t cpy this resurce is granted prvided surce is acknwledged. TEIP Prgram Evaluatin Tl Revised Octber 2009 Page 3

4 Steps, Wrksheets and Outcmes The TEIP Prgram Evaluatin Tl uses the analgy f building a huse t illustrate 5 stages in planning and implementing a prgram evaluatin. Steps Wrksheets Outcmes Lay the Grund Wrk 1. Select Evaluatin Team 2. Assess Organizatinal Capacity & Resurces Lay the Fundatin 3. Develp Prgram Lgic Mdel 4. Define Evaluatin Questins Build the Frame 5. Build Evaluatin Framewrk 6. Dcument Evaluatin Plan Cmplete the Interir 7. Cllect & Analyze Data 8. Dcument Evaluatin Reprt Evaluatin Team Wrksheet Evaluatin Resurces Wrksheet Prgram Lgic Mdel Template Evaluatin Questins Wrksheet Evaluatin Framewrk Template Evaluatin Plan Template Evaluatin Reprt Template Evaluatin team established Available resurces identified Realistic timetable develped Visin f healthy evaluatin culture created Prgram activities and bjectives defined Relevant & useful evaluatin questins priritized Link between PLM and evaluatin understd Indicatrs identified Evaluatin design develped Data cllectin methds & tls created Data analysis methds established Realistic & credible evaluatin plan Ethical issues cnsidered Timely data cllectin & analysis Key learnings & recmmendatins identified Evaluatin results reprted Hld Open Huse 9. Disseminate Findings & Mdify Prgram Disseminatin Plan Template Actin Plan Template Audiences identified & infrmatin tailred t audience preferences Stakehlder feedback shapes final evaluatin recmmendatins Cpyright 2009 by Twards Evidence Infrmed Practice, Ontari Public Health Assciatin, Trnt, Canada. Permissin t cpy this resurce is granted prvided surce is acknwledged. TEIP Prgram Evaluatin Tl Revised Octber 2009 Page 4

5 Lay the Grund Wrk Step 1: Select Evaluatin Team Laying the grundwrk fr a successful evaluatin invlves selecting an effective evaluatin team and determining yur rganizatin s capacity and resurces fr evaluatin. A typical evaluatin team cnsists f 3 5 peple wh are cnnected t the prgram and/r have evaluatin expertise. A dynamic, multi disciplinary team supprts capacity building and creates stakehlder buy in frm the utset. Additinal Guidelines Evaluatin 101 Evaluatin 101 Step 1 prvides further guidance n: Skills & Experience f Evaluatin Specialists Ptential Evaluatin Team Members & Rles Suggested Wrk Plan fr Evaluatin Team Meetings Guidelines: Select Evaluatin Team 1. Cmplete Evaluatin Team Wrksheet. Ask prgram staff fr suggestins r cmplete the wrksheet tgether. Seek advice frm clleagues within yur rganizatin r utside netwrks. 2. Develp a ne page summary fr ptential evaluatin team members describing: Prgram t be evaluated Purpse and verall timeline fr the evaluatin Time cmmitment Rles and respnsibilities f the evaluatin team Benefits f being a member f the evaluatin team 3. Develp a draft wrk plan and schedule fr team meetings See Evaluatin 101 Step 1 Suggested Wrk Plan fr Team Meetings Seek input frm evaluatin team members. Cpyright 2009 by Twards Evidence Infrmed Practice, Ontari Public Health Assciatin, Trnt, Canada. Permissin t cpy this resurce is granted prvided surce is acknwledged. TEIP Prgram Evaluatin Tl Revised Octber 2009 Page 5

6 Evaluatin Team Wrksheet Peple Cnnected t the Prgram Inclusin criteria: Name(s) & Cntact Infrmatin Date Cntacted Outcme (Yes / N / Cntact fr feedback nly) Represents intended audience, especially members f traditinally underrepresented grups Impacted by evaluatin results Can supprt implementatin f evaluatin int practice? Invlved in prgram implementatin Cntributes financial / technical resurces t prgram Has experience with similar prgrams Interested in evaluatin results Champin/specialist in the prgram field r intended audience Researcher whse research interest is related t prgram Peple with Evaluatin Expertise Experience develping lgic mdels & evaluatin questins Experience cnducting an evaluatin Expert knwledge f evaluatin cncepts (i.e. evaluatin design, data cllectin methds, data cllectin tls, analysis methds) Experience develping data cllectin tls & evaluatin plans Graduate student needing t cmplete a practicum Cpyright 2009 by Twards Evidence Infrmed Practice, Ontari Public Health Assciatin, Trnt, Canada. Permissin t cpy this resurce is granted prvided surce is acknwledged. TEIP Prgram Evaluatin Tl Revised Octber 2009 Page 6

7 Step 2: Assess Organizatinal Capacity & Resurces fr Evaluatin Realistic evaluatin plans are develped within the cnstraints f lcal evaluatin capacity, expertise and resurces (e.g. time, funds, in kind supprt frm partners and stakehlders 1) Organizatinal Capacity Mving twards a healthy evaluatin culture and develping a shared understanding f evaluatin builds lcal evaluatin capacity. Develping a Culture f Evaluatin A healthy culture f evaluatin: Develps rganizatinal capacity and cnfidence in cnducting evaluatin Prmtes enthusiasm fr evaluatin within the rganizatin Integrates evaluatin int the prgram planning cycle Invlves all interested stakehlders Addresses imprtant issues Ensures evaluatin results are reprted and actins taken The Evaluatin Thermmeter Activity (see Evaluatin 101 Step 2) can be used t: Gauge where peple are in terms f evaluatin practice Identify internal evaluatin champins wh can supprt the evaluatin prcess Build supprt within a newly frmed evaluatin team (i.e. icebreaker activity). Develping a Cmmn Understanding f Evaluatin Lack f cnfidence in evaluatin is a majr barrier t a healthy evaluatin culture. Thse wh are uncmfrtable with evaluatin may resist it r avid invlvement. Be Realistic! A successful evaluatin is ne that yields infrmatin relevant t decisin making while staying within yur rganizatin s cnstraints f skills, time and resurces. Keep it simple Keep it meaningful Ensure yur evaluatin plan is sund by having it reviewed by smene with evaluatin expertise 1 TCHU (2007). Ten Steps t Evaluating a Health Prmtin Prgram. Cpyright 2009 by Twards Evidence Infrmed Practice, Ontari Public Health Assciatin, Trnt, Canada. Permissin t cpy this resurce is granted prvided surce is acknwledged. TEIP Prgram Evaluatin Tl Revised Octber 2009 Page 7

8 Additinal Guidelines Evaluatin 101 Evaluatin 101 Step 2 prvides further guidance n: Evaluatin Thermmeter and Activity Guidelines Understanding Cmmn Types f Prgram Evaluatins Over time, yu can gradually build up yur rganizatin s evaluatin capacity! Guidelines: Determine Evaluatin Resurces 1. Wrk with Evaluatin Team t develp a realistic evaluatin budget. Recrd all resurces (financial r in kind) available t fund the evaluatin using Part One: Evaluatin Budget f the Evaluatin Resurces Wrksheet 2. Develp a realistic timeline. Cmplete Part Tw: Timeline & Human Resurces f the Evaluatin Resurces Wrksheet Set reasnable deadlines fr each evaluatin step in rder t meet the final evaluatin cmpletin deadline 3. Determine whether evaluatin skills are available in huse r must be utsurced. See Part Tw: Timeline & Human Resurces f the Evaluatin Resurces Wrksheet Cnsider n cst r lw cst surces f evaluatin expertise (e.g. cntributins frm cmmunity partners, health prmtin resurce centres and/r academic researchers). Cnsider hw much cnsultatin time yu can affrd. Reflect n which evaluatin step requires extra expertise and cnsider utsurcing this step instead f utsurcing the entire evaluatin prcess. Lk fr internal champins wh have evaluatin skills. See Evaluatin Team Wrksheet. 4. Review evaluatin plans (i.e. timeline and deliverables): If yur evaluatin plans are t ambitins fr the resurces available, it may be necessary t simplify yur plans r search fr additinal resurces. D nt verestimate the availability f staff time. Staff may require temprary relief frm sme f their duties t devte time t the evaluatin. NOTE: = ptential steps t utsurce Cpyright 2009 by Twards Evidence Infrmed Practice, Ontari Public Health Assciatin, Trnt, Canada. Permissin t cpy this resurce is granted prvided surce is acknwledged. TEIP Prgram Evaluatin Tl Revised Octber 2009 Page 8

9 Evaluatin Resurces Wrksheet Prgram Name: Date: Part One: Evaluatin Budget Surce (e.g. rganizatin prviding the resurce) Funds ($) In kind (e.g. printing, evaluatin sftware, data entry, etc.) Cmmunity Partner $2000 Staff will facilitate the fcus grup Part Tw: Timeline & Human Resurces Evaluatin Deliverables (Outputs & Skills) Prgram lgic mdel Evaluatin questins Evaluatin indicatrs Evaluatin framewrk (Data cllectin tls and methds f analysis) Evaluatin plan Oversee evaluatin plan implementatin Cllect data Data entry Analyze and interpret data Write evaluatin reprt Disseminate evaluatin findings Other: Timeline In huse Outsurce Persn(s) Days Persn(s) Days Cpyright 2009 by Twards Evidence Infrmed Practice, Ontari Public Health Assciatin, Trnt, Canada. Permissin t cpy this resurce is granted prvided surce is acknwledged. TEIP Prgram Evaluatin Tl Revised Octber 2009 Page 9

10 Lay the Fundatin Step 3: Develp Prgram Lgic Mdel An imprtant pre requisite t evaluatin planning is the develpment a prgram lgic mdel (PLM) in cllabratin with key stakehlders. The PLM is a diagram that breaks dwn the prgram int its cmpnent parts and illustrates the lgical and sequential links between prgram activities and the anticipated shrt and lnger term utcmes. 2 Nte: A prgram lgic mdel is a snapsht f the prgram at ne mment in time and shuld be peridically revisited as the prgram is revised r updated. Rle f the PLM in Prgram Evaluatin: Prgram evaluatin aims t measure whether a prgram has met its bjectives and t understand any cntributing factrs. The prgram bjectives, as set ut in the PLM, are a useful place t begin planning a prgram evaluatin. If a prgram lacks clearly stated bjectives, r if the bjectives are nt in line with prgram activities, then it is difficult t cnduct a meaningful prgram evaluatin. If a PLM Already Exists: Cnsider the quality f the PLM and whether it needs t be updated. An effective and up t date PLM creates a slid fundatin fr evaluatin planning. Update the PLM when: Imprtant cmpnents f the prgram are missing frm the PLM Prgram gals have significantly changed since the PLM was develped Additinal Guidelines Evaluatin 101 Evaluatin 101 Step 3 prvides further guidance n: Basic Elements f a Prgram Lgic Mdel Characteristics f a Gd Prgram Lgic Mdel (includes an example f a Prgram Lgic Mdel and SMART Outcme Objectives) Limitatins f a Prgram Lgic Mdel 2 OPHA Cre Cmpetencies Preliminary Lng Term Strategy Prject Advisry Cmmittee. (2005) Ontari Public Health Assciatin Cre Cmpetencies Prject. Cpyright 2009 by Twards Evidence Infrmed Practice, Ontari Public Health Assciatin, Trnt, Canada. Permissin t cpy this resurce is granted prvided surce is acknwledged. TEIP Prgram Evaluatin Tl Revised Octber 2009 Page 10

11 Guidelines: G Develp Prgram Lgic Mdel 1. Schedule a half r full day meeting with the evaluatin team and ther prgram stakehlders t develp a first draft f the prgram lgic mdel (PLM). Ensure everyne understands the purpse and the basics f PLM develpment. 2. Write the basic elements f the PLM nt large Pst it ntes. Use separate Pst its fr each prgram Activity and each Intended Audience. Use separate Pst its fr each anticipated Shrt term, Intermediate and Lng term Outcme Objective(s). 3. Identify cnnectins between all f the prgram elements. As a team, mve the elements (Pst its) arund until it seems t wrk. Yu ll knw when it des because yur cllective visin fr the prgram will emerge. There are different ways t categrize the elements and t identify the cnnectins: Tp Dwn Apprach (Frntline practitiners ften prefer this apprach): Spread ut the Intended Audience Pst its in ne line at the tp and place related Activities Pst its directly belw. Place each Shrt term Outcme Objectives Pst it belw its related Activity. Think abut which activity and utcme cmpletes the phrase If we d..., then we will achieve... Fr example, if we distribute prmtinal material, then we will achieve increased awareness amngst the intended audience. Place each Intermediate Outcme Objectives Pst it belw its related Shrt term Outcme and each Lng term Outcme Objectives Pst it belw its related Intermediate Outcme. Cnsider whether related Activities can be gruped int brader categries (e.g. skill building, scial supprts). Write these n Pst its and place them ne line abve its related Activities. These categries becme the prgram Cmpnents. Bttm Up Apprach (Managers ften prefer this apprach): Prceed as abve, but in the ppsite directin, beginning at the bttm with lng term utcme bjectives and wrking yur way up. 4. Dcument the Pst it PLM nt the Prgram Lgic Mdel Template. 5. Seek feedback frm stakehlders t ensure accuracy, clarity, and cmpleteness. 6. Use the PLM as a basis fr develping evaluatin questins (Step 4) Hint: Add additinal bxes and arrws as needed t the template belw t create yur wn PLM. Cpyright 2009 by Twards Evidence Infrmed Practice, Ontari Public Health Assciatin, Trnt, Canada. Permissin t cpy this resurce is granted prvided surce is acknwledged. TEIP Prgram Evaluatin Tl Revised Octber 2009 Page 11

12 Prgram Lgic Mdel Template Prgram Lgic Mdel fr: Date Revised: Cmpnents Intended Audience Activities / Prcess Objectives Shrt term Outcme Objectives Intermediate Outcme Objectives Lng term Outcme Objectives Cpyright 2009 by Twards Evidence Infrmed Practice, Ontari Public Health Assciatin, Trnt, Canada. Permissin t cpy this resurce is granted prvided surce is acknwledged. TEIP Prgram Evaluatin Tl Revised Octber 2009 Page 12

13 Step 4: Define Evaluatin Questins Gd evaluatin questins can fcus the evaluatin prcess and are the basis f a successful evaluatin. Participatry develpment f evaluatin questins results in mre relevant questins and increases the likelihd that evaluatin results will be acted upn. Evaluatin questins link t prgram bjectives in the Lgic Mdel and are f interest and relevance t prgram stakehlders. Cnsider as well lcal internal and external cntextual factrs that can impact the prgram and its utcmes (e.g. plitical climate, existing prgrams with a fcus n the same issues). Imprtant: Participatry develpment f evaluatin questins results in mre relevant questins and increases the likelihd that the evaluatin results will be acted upn Characteristics f Gd Evaluatin Questins Include bth prcess and utcme questins Relate t bth Prcess and Outcmes Objectives frm the Prgram Lgic Mdel (PLM) Address bth quantitative and qualitative issues Are SMART: Specific Asks abut ne issue r tpic Measurable Pssible t btain the data t answer the questin Actinable Infrmatin gathered will be used t make decisins abut the prgram Relevant Infrmatin is needed t increase prgram effectiveness and efficiency (nt just answer a questin that wuld be nice t knw) Timely T Infrmatin is needed nw t guide decisin making Additinal Guidelines Evaluatin 101 Evaluatin 101 Step 4 prvides further guidance n: Prcess vs. Outcme Evaluatin Questins SMART Evaluatin Questins (includes an example f SMART questins) Cpyright 2009 by Twards Evidence Infrmed Practice, Ontari Public Health Assciatin, Trnt, Canada. Permissin t cpy this resurce is granted prvided surce is acknwledged. TEIP Prgram Evaluatin Tl Revised Octber 2009 Page 13

14 1. Use a participatry apprach Guidelines: Define Evaluatin Questins A participatry apprach results in mre relevant evaluatin questins and increases the likelihd that evaluatin results will be acted n. Invlve stakehlders in brainstrming ptential evaluatin questins. Ensure all participants have a say and are familiar with the Prgram Lgic Mdel (PLM) and the relevant external cntext (See #4 belw). In the absence f direct stakehlder input, dcument the infrmatin yu think they wuld need t knw. 2. Cmplete the Evaluatin Questins Wrksheet a. Dcument draft evaluatin questins Classify evaluatin questins as Prcess r Outcme Develp SMART evaluatin questins. (See Evaluatin 101 Step 4 fr examples.) Develp bth qualitative and quantitative evaluatin questins Return t the PLM t cnsider if yu have missed anything imprtant b. Fr each evaluatin questin, dcument the Link t the Prgram Lgic Mdel. Prcess Evaluatin Questins indicate which Intended Audience r Activity the questin is related t. The evaluatin questins drafted in this step help t fcus yur evaluatin prcess (e.g. Was the prgram effective in increasing physical activity?). They are NOT the same as the questins develped fr data cllectin tls t cllect infrmatin (e.g. Hw many minutes are yu physically active a day?). Outcme Evaluatin Questins indicate which Shrt term, Intermediate r Lngterm Outcme Objective the questin is related t. c. Dcument imprtant External Cntextual Factrs related t the evaluatin questin that may impact the results (e.g. plitical climate, existing prgrams). d. Determine ptential uses f evaluatin results. Fr each evaluatin questin, discuss and recrd: Hw will the infrmatin gathered t answer this questin be used? Remember the gal f evaluatin is t imprve utcmes and infrm prgramming decisins. Dn t cllect infrmatin that will nt be used. Cpyright 2009 by Twards Evidence Infrmed Practice, Ontari Public Health Assciatin, Trnt, Canada. Permissin t cpy this resurce is granted prvided surce is acknwledged. TEIP Prgram Evaluatin Tl Revised Octber 2009 Page 14

15 e. Select pririty questins. Due t limited time and resurces, ne cannt usually answer all evaluatin questins. Sme questins can be investigated at a later time. Assign pririty levels f High, Medium, r Lw based n the fllwing: Give higher pririty t questins addressing lcal reprting requirements and/r standards (e.g. Funder cncerns, Public Health Standards) Cnsider stakehlder pririties. What questins may have the greatest impact in their decisin making. Assign SMART evaluatin questins a higher pririty. Include bth prcess and utcme evaluatin questins. 3. Rll up detailed questins int a smaller number f questins by gruping and synthesizing related questins. Imprtant: It is nt pssible t evaluate everything. Priritize evaluatin questins and select thse which are mst relevant, realistic and useful fr guiding decisin making. Detailed evaluatin questins can becme questins asked n a survey r interview guide. Rlled up evaluatin questins express the brader issues the evaluatin aims t understand abut the prgram and its impact. E.g.: Detailed Questins (Many f the detailed evaluatin questins can be later used in develping questins fr the evaluatin tls) Did the flyer g ut? Did the wrkshp get dne? Were the wrkshp facilitatrs sufficiently trained? Did the wrkshp cver all f the materials? Was the flyer handed ut at all f the prmtinal sites? Were the prmtinal items written at an apprpriate reading level fr the intended audience? Was the prmtinal methd effective at reaching the intended audience? What were the characteristics f the audience reached? Hw many individuals did ur prgram reach? Rlled up Questin (The big picture the evaluatin aims t answer) Was the prgram implemented as intended? Did we reach the audience as intended? Cpyright 2009 by Twards Evidence Infrmed Practice, Ontari Public Health Assciatin, Trnt, Canada. Permissin t cpy this resurce is granted prvided surce is acknwledged. TEIP Prgram Evaluatin Tl Revised Octber 2009 Page 15

16 Evaluatin Questins Wrksheet Prcess Evaluatin Questins Key Stakehlders Draft Evaluatin Questins (Frm each stakehlder s perspective What d they want t knw? ) Link t Prgram Lgic Mdel External Cntextual Factrs Ptential Uses f Evaluatin Results: Pririty (High, Medium, r Lw) Rlled up Questin * Cmmunicatins Team Were the flyers effective in recruiting participants? Did Health Prfessinals refer their patients t the prgram? Activities/Prcess Objectives under the Prmtin & Recruitment Other smking cessatin prgrams available Revise cmmunicatin plan High Hw did the participants hear abut the smking cessatin prgram? Outcme Evaluatin Questins Key Stakehlders Draft Evaluatin Questins (Frm each stakehlder s perspective) Link t Prgram Lgic Mdel External Cntextual Factrs Ptential Uses f Evaluatin Results: Pririty (High, Medium, r Lw) Rlled up Questin * Health Prfessinals Is the smking cessatin prgram that patients are referred t successful in getting participants t stp smking? Shrt term and Intermediate Outcme Objectives Smke free by laws Other related prgrams available Imprvements t prgram Advcate fr future funding and referrals High NA * Evaluatin questins fr the Example Prgram Lgic Mdel in Evaluatin 101 Step 3 Cpyright 2009 by Twards Evidence Infrmed Practice, Ontari Public Health Assciatin, Trnt, Canada. Permissin t cpy this resurce is granted prvided surce is acknwledged. TEIP Prgram Evaluatin Tl Revised Octber 2009 Page 16

17 Build the Frame Step 5: Build Evaluatin Framewrk After building the fundatin f yur evaluatin it is time t build the Evaluatin Framewrk and t dcument the Evaluatin Plan. Evaluatin Framewrk The Evaluatin Framewrk cnnects all f the elements needed t cmplete the Evaluatin Plan, including Pririty Evaluatin Questins and External Cntextual Factrs (frm Step 4) Indicatrs, Evaluatin Design, Data Cllectin Methds & Tls and Methds f Data Analysis. This is the mst technically challenging f all the Steps. Dn t despair if yu must review this sectin several times befre it begins t makes sense. 1. Indicatrs What will be measured? Imprtant: Dn t be intimidated by terms like data cllectin methds r evaluatin framewrk. Learning evaluatin is like learning a new language. Once yu understand the vcabulary, it starts t makes sense. Indicatrs are used t measure the success f prgram activities. Fr example, an indicatr f the success f an exercise prgram culd be the change in the participants bdy cmpsitin and/r muscle strength. The change indicates the degree t which prgram bjectives have been met. Indicatrs reflect three things 3: a. What is cnsidered effective b. What is cnsidered a success c. What change is expected 3 Heart Health Resurce Centre (2000). Evaluating Heart Health Prjects Objectives and Indicatrs: Wrkshp Manual. Cpyright 2009 by Twards Evidence Infrmed Practice, Ontari Public Health Assciatin, Trnt, Canada. Permissin t cpy this resurce is granted prvided surce is acknwledged. TEIP Prgram Evaluatin Tl Revised Octber 2009 Page 17

18 T give breath and depth t what yur evaluatin will be measuring, it is imprtant t include a balance f quantitative (numbers) and qualitative indicatrs (text/descriptins). Quantitative indicatrs prvide breath f infrmatin (quantity) and are captured using numbers (i.e. #f cigarettes smked per day). Qualitative indicatrs give yu mre depth f infrmatin (quality) and are captured by descriptive infrmatin (i.e. descriptin f peple s barriers t quitting smking). When relevant, use qualitative indicatrs t cmplement the quantitative indicatrs measured. Fr example, ne f yur quantitative indicatrs culd be the % f participants wh quit smking. T have a deeper understanding f yur prgram s impact, yu can include the descriptin f participants mtivatin t quit r nt quit smking as the qualitative indicatr. This qualitative indicatr will prvide yu with infrmatin that yu can use t strengthen yur prgram and have a greater impact. An Evaluatin Cnsultant can help t identify realistic and apprpriate indicatrs. 2. Evaluatin Design When will the data be cllected? Examples f cmmn evaluatin designs include: Feedback nly Participants prvide feedback n changes made at the cmpletin f a prgram. Infrmatin can be cllected using a pst prgram r wrkshp survey Data is cllected nly nce after the prgram/wrkshp. Pre test/pst test Prvides a mre accurate assessment f changes resulting frm the prgram A brief questinnaire (pre test) is administered as part f the prgram registratin prcess and then repeated upn prgram cmpletin (pst test) Data is cllected at tw separate intervals, befre and after the wrkshp r prgram. Pre test/pst test with 6 mnth fllw up Prvides an assessment f the prgram s impact n the intended audience ver time Data is cllected at three separate intervals, befre, right after and 6 mnths after the wrkshp/prgram. Cpyright 2009 by Twards Evidence Infrmed Practice, Ontari Public Health Assciatin, Trnt, Canada. Permissin t cpy this resurce is granted prvided surce is acknwledged. TEIP Prgram Evaluatin Tl Revised Octber 2009 Page 18

19 NOTE: The evaluatin questin and indicatr influence the chice f evaluatin design. Fr example, if yur evaluatin questin asks hw did participants hear abut the peer nutritin prgram, a feedback nly evaluatin design can be selected. If % bdy fat lst is an indicatr f a healthy weight prgram then a pre test/pst test evaluatin design can be selected. Evaluatin Questin Hw did participants hear abut the peer nutritin prgram? Hw effective is the healthy weight prgram? (Indicatr = % lss bdy fat) Evaluatin Design Feedback nly Pre test / pst test 3. Data Cllectin Methds & Tls Hw will the data be cllected? Examples f cmmn data cllectin methds include: Observatin Activities f intended audience are bserved, recrded and interpreted. Fcus Grup Facilitatr elicits discussin amng participants n specific issues. Survey Participants respnd t a written series f questins. Interview Participants respnd t questins delivered in cnversatin. Dcument Review The cntent f selected dcuments is analyzed and interpreted. The data cllectin tl is the instrument used t gather the infrmatin t measure prgram evaluatin indicatrs. Remember: When develping data cllectin tls, cllect bth quantitative data (numbers) and qualitative data (text/descriptins). The evaluatin design & data cllectin methd determine the evaluatin tls required. Example: Cnnecting Evaluatin Questin t Data Cllectin Methds and Tls Evaluatin Questin Data Cllectin Methd Data Cllectin Tl Hw apprpriate was the prgram fr a specific cultural grup? Wuld past participants recmmend the prgram t thers? Hw d children participate in the activities? Fcus grup Web based survey Observatin Facilitatr s guide tpic list Survey questins Observatin guide & Recrding sheets Cpyright 2009 by Twards Evidence Infrmed Practice, Ontari Public Health Assciatin, Trnt, Canada. Permissin t cpy this resurce is granted prvided surce is acknwledged. TEIP Prgram Evaluatin Tl Revised Octber 2009 Page 19

20 4. Methds f Data Analysis What will be dne with the cllected data? We recmmend cnsulting an evaluatin specialist r epidemilgist if n ne n yur team has previus experience with data analysis r fr mre cmplex analyses. The methds f data analysis differ fr quantitative and qualitative data. Quantitative data analysis Tw categries: Inferential analysis (makes predictins inferences based n the findings) Descriptive analysis (describes the findings) Inferential analysis It is mre cmplex. It lks at the data cllected frm evaluatin participants and makes predictins fr the brader intended audience. It can als be used t answer cause and effect questins. Odds ratis is ne f the mst cmmn types f inferential analysis Odds ratis describe the likelihd that a specific grup will have a specific characteristic in cmparisn t an alternate grup. Fr example, answering the questin Will cmmunities wh receive the Turn n the Tap campaign likely drink less bttled water than thse wh d nt receive the campaign? Descriptive analysis Cmmn types f descriptive analysis are described in the table n the next page. Please see Step 5 Resurces Sectin fr mre guidance n inferential analysis and ther types f descriptive analysis. Cpyright 2009 by Twards Evidence Infrmed Practice, Ontari Public Health Assciatin, Trnt, Canada. Permissin t cpy this resurce is granted prvided surce is acknwledged. TEIP Prgram Evaluatin Tl Revised Octber 2009 Page 20

21 Type f descriptive analysis 4 When t use it 4 Example Frequencies T calculate the number f participants/grups wh: 7 ut f 10 new (numerical cunts) Changed their behaviur, knwledge, mtivatin and/r skills mthers stated they Were satisfied with the prgram/plicy feel cnfident in their Describes hw many Have a particular characteristics (e.g. gender) ability t breastfeed times smething ccurred When # f respnses is small (e.g. < 20) presenting the data as a % culd be misleading. their newbrn (E.g. stating that 60% f participants quit smking and # f participants was 10. Thus 6 peple quit and 6 peple did nt a difference f 2 nly peple). Percentages T calculate the prprtin f participants/grups wh: 72% f participants Describes infrmatin as Changed their behaviur, knwledge, mtivatin and/r skills buy fd frm their part f a whle Were satisfied with the prgram/plicy lcal farmers Have a particular characteristics (e.g. gender) When # f respnses is large (e.g. > 20) presenting data as a % is easier t understand. (E.g. Instead f reprting that 273 f 455 participants quit smking, reprt that 60% f participants quit.) T describe relatinships and cmpares results. (E.g. 83% f wrkshp participants quit smking, while 23% f the participants n the wrkshp waiting list quit smking. T shw the distributin f participants r t classify participants int categries. (E.g. 25% f participants always smke inside their hme, 30% smetimes smke and 45% f participants never smke inside their hme.) 4 Taylr Pwell, E. (1996). Analyzing Quantitative Data. University f Wiscnsin Extensin Prgram, Develpment and Evaluatin Unit. learningstre.uwex.edu/pdf/g pdf Cpyright 2009 by Twards Evidence Infrmed Practice, Ontari Public Health Assciatin, Trnt, Canada. Permissin t cpy this resurce is granted prvided surce is acknwledged. TEIP Prgram Evaluatin Tl Revised Octber 2009 Page 21

22 Type f descriptive When t use it 4 Example 4 analysis 4 Average (Mean) Mst Cmmn Value (Mde) Range Describes the spread/variatin f the data T describe the central characteristic f the participants/grups. I.e.: Number f cigarettes/day Duratin f breastfeeding Percentage f businesses that have a bttle water plicy/municipality Duratin f physical activity pprtunities per week in schls/schl bard T summarize findings frm rating scales. (E.g. wrkshp participants, n average, rated the usefulness f wrkshp cntent as 4.2 n a 1 5 pint scale.) T describe the mst cmmn characteristic f the participants/grups T describe the mst cmmn rating frm a rating scale. (E.g. the mst cmmn rating given by participants n the usefulness f the wrkshp cntent was a 4.0) When wrking with a large # f respnses When the presence f utliers (extreme values) in the data makes calculating the average misleading. (E.g. If 4 participants state their # f cigarettes smked/day was 2, 3, 5 and 34; the average wuld be 11 cigarettes/day. The utlier, 34 cigarettes/day, made the average higher than a majrity f the respnses.) T describe the highest and lwest characteristic f the participants/grups and/r rates n a rating scale. T prvide a better understanding f the results when there is a variety f respnses/characteristics and/r utliers (extreme values). T give mre meaning (depth) t the average/mst cmmn value calculated. Are all participants/grups similar t the mean? Are there any extreme cases/respnses? On average students frm the schls that implemented the physical activity prgram were physically active 45 mins/day The typical family annual incme f thse families that accessed the fd basket prgram was $20,000. After implementing a wrkplace physical activity prgram, emplyee absenteeism decreased by 1.15 t 4.56 days Cpyright 2009 by Twards Evidence Infrmed Practice, Ontari Public Health Assciatin, Trnt, Canada. Permissin t cpy this resurce is granted prvided surce is acknwledged. TEIP Prgram Evaluatin Tl Revised Octber 2009 Page 22

23 Qualitative Analysis Cntent analysis 5 The mst cmmn methd f qualitative data analysis Cntent analysis is like putting tgether a jigsaw puzzle. First yu grup the pieces f the puzzle int categries (e.g. edges f the puzzle, pieces that are part f a similar image). Yu then put all f the categries tgether t reveal the cmplete image. Similarly, in a cntent analysis yu examine the data t identify cmmn themes and patterns (i.e. grup the data int categries). Then, reflecting n all f the categries, yu put the pieces tgether and reveal the stry that the data is telling. Narrative analysis 6 Useful when lking at qualitative data cllected thrugh the interview cllectin methd (e.g. a participant tells a detailed stry f hw the prgram has impacted their life and smene summarizes this stry.) Highlights cntextual factrs that may be different fr each participant Additinal Guidelines Evaluatin 101 Evaluatin 101 Step 5 prvides further guidance n:. Selecting a Data Cllectin Methd Data Cllectin Methds & Tls Sample Data Cllectin Tl frm TEIP Cmmunity An Imprtant Distinctin: What is the difference between research and prgram evaluatin? Research aims t discver universal truths that can be applied beynd the cnfines f the research study. Research methds and tls are designed fr this purpse. Prgram Evaluatin aims t discver lcal truths in rder t influence lcal decisins. 5 My Envirnmental Educatin Evaluatin Resurce Assistant (2008). Step 6: Analyze Data. meera.snre.umich.edu/plan an evaluatin/plnearticlemultipage /step 6 analyzedata 6 Ratcliff, D (n.d). 15 Methds f Data Analysis in Qualitative Research. qualitativeresearch.ratcliffs.net/15methds.pdf Cpyright 2009 by Twards Evidence Infrmed Practice, Ontari Public Health Assciatin, Trnt, Canada. Permissin t cpy this resurce is granted prvided surce is acknwledged. TEIP Prgram Evaluatin Tl Revised Octber 2009 Page 23

24 Guidelines: Build Evaluatin Framewrk 1. List Pririty Evaluatin Questins & External Cntextual Factrs (Step 4 Chart A & B) 2. Select realistic and relevant Indicatrs fr each evaluatin questin. What will be measured? Gd indicatrs are relevant, easy t measure and reflect prgram utcmes. Select bth qualitative and quantitative indicatrs fr each evaluatin questin. Cnsult clleagues, evaluatin specialists, an epidemilgist and/r the literature. Cnsider the apprpriate level f indicatr. D yu intend t measure individual (e.g. abriginals), rganizatinal (e.g. wrkplace), cmmunity (e.g. lw incme neighburhds), r system level (e.g. prvincial) change. 3. Identify Evaluatin Design. When will the data be cllected? Fr each set f evaluatin questins and indicatrs cnsider the mst apprpriate evaluatin design. 4. Identify Data Cllectin Methds & Tls. Hw will the data be cllected? Dcument the Data Cllectin Methd and Data Cllectin Tl t be used. a. Use and/r adapt existing tls, where pssible. Dcument wh yu cntacted and what data cllectin tls they will share. Ask clleagues and search the internet fr existing data cllectin tls. Dcument what yu fund and where yu fund it. Dcument where tls are stred (i.e. hyperlink, where the paper dcuments are stred r/and where the electrnic dcument is saved) If yu create yur wn tl, pilt test it t assess its quality and sensitivity with a fcus grup. Test fr validity (i.e. the tl measures what yu want it t measure) Test fr clarity (i.e. it is easily understd by the intended audience) Test fr reliability (i.e. hw cnsistently the variable can be measured and give the same data each time) 5. Identify Methds f Data Analysis. What will be dne with the cllected data? Nw what? Fr each evaluatin questin and indicatr, cnsider what methd f data analysis can summarize the infrmatin cllected and help t answer the questin. Dcument the methds f data analysis that will be used fr bth qualitative and quantitative data fr each evaluatin questin and fr each indicatr if it differs. Cpyright 2009 by Twards Evidence Infrmed Practice, Ontari Public Health Assciatin, Trnt, Canada. Permissin t cpy this resurce is granted prvided surce is acknwledged. TEIP Prgram Evaluatin Tl Revised Octber 2009 Page 24

25 Evaluatin Framewrk Template Evaluatin Framewrk fr: (Insert Prgram Name) Pririty Evaluatin Questin(s) (Frm Step 4) External Cntextual Factrs (Frm Step 4) Indicatr(s) What will be measured? Evaluatin Design When will the data be cllected? Data Cllectin Methds & Tls Hw will the data be cllected? Methds f Data Analysis What will be dne with the cllected data? * Is the smking cessatin prgram successful in getting participants t stp smking? Smke free by laws Availability f ther smking cessatin prgrams Quantitative: % wh n lnger smke within 30 minutes f waking Reductin in # f cigs/day Qualitative: Descriptin f strategies peple use t quit smking and stay smke free Directly befre and after prgram implementatin and 6 mnths after prgram implementatin (Pre/pst test with 6 mnths fllw up) Pre survey during registratin and pstprgram survey TOOLS: pre/pst questinnaire 6 mnths after pst prgram survey TOOLS: fllw up questinnaire. SURVEY METHOD (e.g. mailut r web based) will be decided based n capacity and participant characteristics. Quantitative descriptive analysis (e.g. mean, percentages and range) using Micrsft Excel Cntent analysis f qualitative data * Evaluatin questins fr the Example Prgram Lgic Mdel in Evaluatin 101 Step 3 Cpyright 2009 by Twards Evidence Infrmed Practice, Ontari Public Health Assciatin, Trnt, Canada. Permissin t cpy this resurce is granted prvided surce is acknwledged. TEIP Prgram Evaluatin Tl Revised Octber 2009 Page 25

26 Step 6: Dcument the Evaluatin Plan The Evaluatin Plan summarizes hw the cmpnents f the evaluatin framewrk will be cmpleted within the given limitatins f time, budget, expertise, and ther resurces. Nte: A detailed Evaluatin Framewrk can substitute fr writing an evaluatin plan IF yu are undertaking a very simple evaluatin. The evaluatin plan acts as a persnalized blueprint t keep yur evaluatin n curse. It will becme an excellent surce f prgram dcumentatin and may be an asset when crrespnding with funding rganizatins. Address each pint in the Evaluatin Plan Template t dcument yur evaluatin plan. Prvide enugh detail t infrm reviewers wh may be unfamiliar with yur prgram and/r cntext. Be as cncise as pssible t encurage brad readership. Guidelines: Dcument the Evaluatin Plan 1. Select smene familiar with the prgram t write the Evaluatin Plan. 2. Interpret the meaning f each heading in the Evaluatin Plan Template Use the questins under each heading f the Evaluatin Plan Template as a guideline n what t include in that particular sectin. 3. Seek assistance frm ther members f yur evaluatin team Yur evaluatin team, as well as representatives frm yur stakehlders may have pertinent infrmatin t cntribute. 4. Write yur respnse under each heading f the Evaluatin Plan Template Identify the key elements t include in yur respnse Include data t supprt yur answers where available Ensure that yur respnses are cmprehensive, yet cncise 5. Cllect and attach supprting dcumentatin fr each respnse Attach cpies f yur prgram lgic mdel, data cllectin tls, and Evaluatin Framewrk. Nte: Writing an Evaluatin Plan may take lnger than expected A first timer cmpleting the Evaluatin Plan Template may require 1 2 days t cmplete the wrk. Experience shrtens the time cnsiderably. The mre cmplete and rganized the prgram dcumentatin, the easier it is t cmplete the evaluatin. Cpyright 2009 by Twards Evidence Infrmed Practice, Ontari Public Health Assciatin, Trnt, Canada. Permissin t cpy this resurce is granted prvided surce is acknwledged. TEIP Prgram Evaluatin Tl Revised Octber 2009 Page 26

27 Evaluatin Plan Template Name f Prgram: Cmmunity/ Organizatin: Plan prepared by: Name: Tel #: Date: Directins: Please prvide apprximately 1/2 page f infrmatin fr all the fllwing questins Prgram Summary Prgram Descriptin Write a cncise descriptin f yur prgram such that a reader can quickly grasp the reasn fr the prgram, the intended audience, the bjectives, the majr activities, hw the prgram is delivered, wh is invlved and any ther imprtant r distinctive features. Ratinale fr Evaluatin Write a cncise descriptin f the perceived need fr the evaluatin and the anticipated use f findings. Describe ratinale fr usefulness f infrmatin frm the Evaluatin Questins Charts. Prvide a cpy f the Develping Relevant and Useful Evaluatin Questins Charts. Overall Evaluatin Objectives What questins are being answered by this evaluatin? What indicatrs will be lked at t answer the pririty evaluatin questins? Include a cpy f the Prgram Evaluatin Framewrk. Remember t write SMART bjectives (please see Evaluatin 101 Step 3 fr examples). Cpyright 2009 by Twards Evidence Infrmed Practice, Ontari Public Health Assciatin, Trnt, Canada. Permissin t cpy this resurce is granted prvided surce is acknwledged. TEIP Prgram Evaluatin Tl Revised Octber 2009 Page 27

28 Methdlgy Data Cllectin Strategy Describe what type f infrmatin will be cllected. Describe the evaluatin design, data cllectin methds and the data cllectin tls. Explain the ratinale behind yur data cllectin strategy. Prvide a cpy f all data cllectin tls. Methd f Data Analysis Describe hw the data yu cllect will be analysed and the ratinale behind that methd f analysis. Dcument wh will cnduct the data analysis. Yur cmpleted Evaluatin Framewrk will be helpful in cmpleting this sectin. Limitatins f the Evaluatin Describe any cautins yu have abut the ptential findings r the results f the evaluatin (i.e. hw t incrprate the findings? What they may mean fr the prgram.). Human Resurces Required Available Resurces Please describe and/r list all human resurces (persnnel) invlved in cmpleting the prgram evaluatin. Include all members f the evaluatin team and utsurced evaluatin specialists. Describe the areas in which each cntributed t the evaluatin and what resurces each prvided. The headings belw can be used t rganize the infrmatin. Yur Evaluatin Team Wrksheets will be helpful in cmpleting this sectin. Suggested Table: Staff Member Psitin / Rle in Evaluatin Types f Resurces Stakehlder / Outsurced (e.g. planning, prgram lgic mdel, Cntributed Specialists evaluatin questins, (e.g. champins in evaluatin) (i.e. Evaluatin cnsultant) implementatin, reprting) Epidemilgist Analysis f Data Champin f data analysis and access t analysis sftware >> >> >> >> >> >> Cpyright 2009 by Twards Evidence Infrmed Practice, Ontari Public Health Assciatin, Trnt, Canada. Permissin t cpy this resurce is granted prvided surce is acknwledged. TEIP Prgram Evaluatin Tl Revised Octber 2009 Page 28

29 Financial Resurces Develp a budget fr the evaluatin. Describe hw much funding is currently available, as well as all funds that may cme available in the future fr this evaluatin prject. List the financial resurces required fr the evaluatin prject, including utsurcing csts, fees fr data cllectin and analysis prgrams that may be used (i.e. survey mnkey: nline survey sftware). The Evaluatin Resurces Wrksheet may be helpful in cmpleting this sectin. Evaluatin Wrk Plan The Evaluatin Wrk Plan acts as a t d list fr the entire evaluatin prcess. It utlines, in a table frmat, the: Evaluatin deliverables Activities required fr meeting each deliverable Individuals respnsible fr cmpleting each activity Resurces needed and available fr each activity Timeline t cmplete each activity When determining the timeline: If data analysis is utsurced, include sufficient time t review and revise. Build in time t disseminate evaluatin results t stakehlders. Refer t Step 9: Disseminate Findings & Mdify Prgram. Suggested Table: Deliverables Activities Persns Resurces Timeline Respnsible Cllectin f data develp fcus Evaluatin lcatin t hld July 2009 grup tpic list Crdinatr fcus grup September 2009 identify and telephne invite fcus grup charges t call participants participants hld fcus grup fd fr fcus grup >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> Cpyright 2009 by Twards Evidence Infrmed Practice, Ontari Public Health Assciatin, Trnt, Canada. Permissin t cpy this resurce is granted prvided surce is acknwledged. TEIP Prgram Evaluatin Tl Revised Octber 2009 Page 29

30 Cmplete the Interir Step 7: Cllect and Analyze Data At this pint yu have built the frame f yur evaluatin by creating an evaluatin framewrk and an evaluatin plan. Nw it is time t put the plan int actin by cllecting and analyzing the data, and dcumenting yur evaluatin results. Cllect the Data Prvide participants with all the infrmatin they need t make a vluntary and infrmed decisin t participate in the evaluatin. The infrmatin shuld be presented in a frmat that is easy t understand. Participants frm whm yu will cllect data have the right t be infrmed f the fllwing: Purpse f the evaluatin Hw lng the data cllectin activity is expected t take. Descriptin f what the evaluatin activity invlves Hw the infrmatin cllected will be stred and reprted. Cnfidentiality A cnfidentiality agreement shuld be signed by all individuals having access t the raw evaluatin data. It states that signatries agree t nt share any infrmatin with anyne utside f the evaluatin team. Once data cllectin is cmpleted, stre the data in a secure lcatin. Ensure that nly thse individuals wh have signed a cnfidentiality agreement have access. Analyze the Data This mst imprtant rule fr any type f analysis is t keep the purpse f the evaluatin and the evaluatin questins in mind. Step by step instructins n hw t perfrm a quantitative data analysis using Micrsft Excel and hw t cmplete a qualitative data analysis are available in Step 7 Resurces. Cpyright 2009 by Twards Evidence Infrmed Practice, Ontari Public Health Assciatin, Trnt, Canada. Permissin t cpy this resurce is granted prvided surce is acknwledged. TEIP Prgram Evaluatin Tl Revised Octber 2009 Page 30

31 Step 8: Dcument Evaluatin Reprt The Evaluatin Reprt dcuments the entire evaluatin and can be used internally r shared t cmmunicate the findings t a brader audience. Creating an Evaluatin Reprt is a majr step twards increasing the availability f practice based learning. The mst imprtant sectins f the Evaluatin Reprt are the Key Learnings and Recmmendatins. Determine key learnings Ask yurself the fllwing questins t identify key learnings frm the evaluatin: What is the answer t the evaluatin questin based n the results? Hw did we meet the prcess bjectives? Were the prgram activities executed as planned? Did the prgram reach match expectatins? Hw did we meet the utcme bjectives? What is the prgress twards achieving the desired changes? If the prcess and utcme bjectives are being met, why? If the prcess and utcme bjectives are nt being met, why nt? Determine recmmendatins The recmmendatins sectin f the Evaluatin Reprt is the mst imprtant sectin. It prvides a list f suggestins fr hw t enhance yur prgram s ability t meet its bjectives based n the evaluatin findings. Lking at the discussin sectin f the evaluatin reprt, reflect n hw the prgram can be mdified in rder t meet its bjectives. Fr example, if the prgram bjectives were nt realistic yur recmmendatins may be t change these bjectives. Additinal Guidelines Evaluatin 101 Evaluatin 101 Step 8 prvides further guidance n: Hw t write the Evaluatin Reprt Evaluatin Reprt Template Imprtant: Mst f the infrmatin fr the Evaluatin Reprt has already been dcumented in the Evaluatin Plan. Simply cut and paste the apprpriate sectins. D nt prepare an elabrate Evaluatin Reprt fr a very small evaluatin. Hwever, if an evaluatin is wrth ding, it is wrth dcumenting t sme degree. Cpyright 2009 by Twards Evidence Infrmed Practice, Ontari Public Health Assciatin, Trnt, Canada. Permissin t cpy this resurce is granted prvided surce is acknwledged. TEIP Prgram Evaluatin Tl Revised Octber 2009 Page 31