Table f Cntents Step 1. Engage Stakehlders... 1 Identifying Evaluatin Stakehlders... 2 Develping an Evaluatin Stakehlder Engagement Plan... 4 Step 2. Describe the Prgram... 6 Prgram Lgic Mdel... 7 Prject Narrative... 9 Step 3. Fcus the Evaluatin...10 Define the Evaluatin Purpse(s)...10 Identify and Priritize Evaluatin Questins...11 Stage f Prgram Develpment... 12 Prcess and Outcme Evaluatin... 13 Priritizing Evaluatin Questins...18 Identifying Apprpriate Evaluatin Methds...23 Evaluatin Methds Matrix...23 Indicatrs... 24 Example... 29 Identifying Data Surces... 32 Existing Data Surces... 33 New Data Surces... 33 Example... 35 Data Cllectin Methds... 37 Quantitative Methds... 38 Qualitative Methds... 38 Example... 38 Preliminary Analysis Plans...41 Example... 42 Disseminatin Plan...46 Refining Yur Evaluatin Budget...49 Develping a Cmprehensive Evaluatin Plan...49 Internal Review Bards...51 Next: Evaluatin Implementatin...51 Tls and Templates...52 Evaluatin Stakehlder Engagement Plan Template...52 Example WISEWOMAN-Specific Prgram Lgic Mdel Template...53 Evaluatin Methds Matrix Template...55 Indicatr Mapping Template...57 Priritizing Evaluatin Questins Template...59
Step 1. Engage Stakehlders RECAP: At this pint, we have intrduced key cncepts related t yur prgram evaluatin: Reasns fr cnducting evaluatin Key features f this evaluatin tlkit Evaluatin standards Centers fr Disease Cntrl and Preventin (CDC) Evaluatin Framewrk Budget resurces Identifying staff t lead yur evaluatin effrts During this step, yu will accmplish the fllwing: Identify key stakehlders. Create a fundatin fr cmmunicatin with stakehlders. Set initial expectatins fr hw each stakehlder will be invlved during the evaluatin. We advise all WISEWOMAN prgrams t engage a grup f stakehlders in evaluatin. Stakehlders are peple r rganizatins wh have an investment (r stake ) in the evaluatin prcess r the findings frm the evaluatin. 5,7,8,9 Members f an evaluatin stakehlder grup include the primary users f the evaluatin results. They generally act as a cnsultative grup thrughut the entire planning prcess, as well as the implementatin f the evaluatin. 5 In evaluatin, engaging stakehlders is cnsistent with utilizatin-fcused and participatry evaluatin appraches. Fr mre infrmatin n these evaluatin mdels, please see the Resurce Guide (Sectin 4 f the WISEWOMAN Evaluatin Tlkit). 1
Wrking with a diverse grup f stakehlders will help yu ensure that varius perspectives are cnsidered as yu Develp and priritize evaluatin questins. Plan evaluatin methds. Select credible data surces. Interpret evaluatin data and results. Use results fr prgram imprvement. Evaluatin stakehlders als can help yu cnduct evaluatin activities and leverage resurces acrss multiple grups fr yur evaluatin activities. This may als help yu use yur evaluatin budget mre efficiently. As yu assemble a grup f stakehlders fr yur evaluatin, cnsider the rles that the stakehlders r their rganizatins might play in an evaluatin. The fllwing are examples f rles that stakehlders may play in cnducting an evaluatin 5 : Why Invlve Stakehlders in Evaluatin? 5,10 Stakehlder engagement in evaluatin helps reduce distrust and fear f evaluatin, increase awareness f the evaluatin, increase cmmitment (r buy in ) t the evaluatin, increase supprt fr evaluatin effrts, increase likelihd that evaluatin findings will be used, enhance the credibility f yur evaluatin findings. Pretest data cllectin instruments. Facilitate data cllectin. Implement evaluatin activities. Disseminate evaluatin findings and ensure that findings are used. Identifying Evaluatin Stakehlders As yu think abut individuals r rganizatins t invlve in yur evaluatin stakehlder grup, yu might cnsider engaging individuals r grups wh are interested in the evaluatin r wh wuld use the evaluatin findings (either directly r indirectly). There is a ntin that in the cntext f public health prgrams, everyne is yur stakehlder. 10 While this may be true, it is imprtant t nte that stakehlders may have diverse and at times cmpeting interests. A single evaluatin cannt address all f the interests raised by stakehlder grups; therefre, we recmmend that yu identify 8 t 10 individuals t represent the stakehlders wh have the greatest stake r vested interest in the evaluatin. 5,7 It is critical that yu priritize yur stakehlders and dcument yur prcess fr priritizing stakehlders in the evaluatin plan. 2
In general, there are fur types f evaluatin stakehlders, as described in Exhibit 1 belw. Exhibit 1. Types f Evaluatin Stakehlders 5,10 Type f Stakehlder Implementers Decisin makers Participants Partners Descriptin Prgram staff and thers directly invlved in the delivery and peratins f the prgram Thse in a psitin t d r decide smething abut the prgram Individuals wh are served by the prgram, their families, r the general public Thse wh supprt r are actively invested in the prgram WISEWOMAN Prgram Example Clinical service prviders Evidenced-based lifestyle prgram staff members Minimum data elements (MDEs) data staff r cntractrs CDC Prject Officers State/tribal chrnic disease prgram directrs Prgram manager WISEWOMAN clients/patients State epidemilgists Diabetes preventin and cntrl prgrams, diabetes preventin prgrams (evidenced-based lifestyle prgrams) American Heart Assciatin State tbacc cntrl prgrams 3
Develping an Evaluatin Stakehlder Engagement Plan Make sure that bth prgram staff and the evaluatin stakehlder grup understand the rle f the stakehlder grup in this evaluatin. 5 An evaluatin stakehlder engagement plan, such as the ne presented in Exhibit 2, can help yu dcument and track stakehlder engagement thrughut the evaluatin. T get started, list each stakehlder s name and affiliatin, alng with hw and when the stakehlder shuld be engaged in the evaluatin. We recmmend setting clear expectatins regarding the invlvement f each stakehlder, given whether it is imprtant fr a given stakehlder t participate frequently r at a few strategic pints during the evaluatin. Setting these Take Time fr Stakehlder Engagement 5 As yu meaningfully engage yur evaluatin stakehlder grup, allw time fr reslving cnflict and cming t a shared understanding abut the evaluatin. This time is wrth the effrt fr a truly participatry and empwering evaluatin. expectatins will depend n yur understanding f yur stakehlders levels f interest, expertise, and availability. Yu can als use the stakehlder engagement plan t dcument each stakehlder s areas f interest in the evaluatin, evaluatin rle, anticipated level f invlvement in the evaluatin, preferred mde f cmmunicatin, and timing fr when the stakehlder wuld best be engaged. If there are specific deadlines fr infrmatin, such as a pending deadline fr midterm r annual reprts, it is imprtant t nte thse as well. Yur stakehlder engagement plan shuld be cnsidered a living dcument that will need t be revisited and revised accrdingly. Explicitly dcumenting and sharing yur stakehlder engagement plan can help yu avid r manage cnflict. This als will allw stakehlders t step up r step back as needed given their availability and need fr invlvement at different pints during yur evaluatin. 4
Exhibit 2. Example Evaluatin Stakehlder Engagement Plan Evaluatin Stakehlder Level f Participatin Evaluatin Interest Rle in the Evaluatin Mde f Cmmunicatin Timing f Cmmunicatin Cmment Suzy Q., Breast and Cervical Cancer Prgram (BCCP) directr Medium Evaluatin questins and results related t utreach, recruitment cmpnents Review and prvide input n evaluatin plan and reprt; BCCP staff will participate in data cllectin effrts in utreach and recruitment Email update, inpersn participatin at the planning and findings review meeting Mnthly It will be especially imprtant t invlve Suzy Q. in meetings when we discuss evaluatin methds and the findings specific t utreach and recruitment, as this will likely have implicatins fr cllabratin with the BCCP Jhn S., Tbacc Cntrl Prgram crdinatr High Methds and utcmes related t referrals fr WISEWOMAN participants Prgram agrees t create an indicatr in their data set fr WISEWOMAN participants and will share prgram data n WISEWOMAN participants fr the evaluatin; prgram staff will be participants fr partnership evaluatin cmpnent In-persn at evaluatin prject meetings Twice mnthly Jane D., RN, MSN, representative frm WISEWOMAN clinical prviders advisry grup High Methds and utcmes related t referring participants Will wrk with cunterparts t help crdinate data delivery fr WISEWOMAN participant behaviral and readiness t change assessments Cnference call fr evaluatin prject meetings Twice mnthly Will be n maternity leave September December; during this time, Barbara M., RN, will serve in her rle Julia R., CDC Prject Officer High Overall quality and cmpliance with CDC requirements Will review and apprve evaluatin plan and final reprt Regular updates during cnference calls, reprts t CDC Mnthly, annual, and interim prgress reprts Will invlve CDC Evaluatin Specialist n a quarterly basis r mre ften as needed 5
Step 2. Describe the Prgram RECAP: At this pint in the evaluatin, yu have accmplished the fllwing: Identified and engaged a grup f evaluatin stakehlders Created a fundatin fr cmmunicatin with stakehlders Set initial expectatins fr hw each stakehlder will be invlved during the evaluatin During this step, yu will accmplish the fllwing: Develp a lgic mdel f the prgram. Create yur prgram narrative. Yur prgram descriptin shuld present a shared understanding f the prgram. Mst prgram descriptins include a prgram lgic mdel and a narrative descriptin f the key cmpnents f the prgram. During the funding pprtunity annuncement (FOA) applicatin prcess fr WISEWOMAN, yu prepared a prject narrative. Fr yur prgram evaluatin, yu will want t update the prject narrative t reflect any recent changes in yur prgram. Yu will als want t make sure that the evaluatin stakehlder grup has a shared understanding and agrees t the prgram descriptin befre launching int evaluatin activities. 5 6
Prgram Lgic Mdel We recmmend that yu begin describing yur prgram by develping a prgram lgic mdel. A prgram lgic mdel visually illustrates the linkages between a prgram s activities and its intended utcmes. 5,6 Lgic mdels als help in guiding evaluatin activities and interpreting the findings. Once yu have a lgic mdel that yur stakehlders understand and agree t, refining yur prject narrative shuld cme relatively easily. Yu may remember reviewing the natinal WISEWOMAN prgram lgic mdel in the FOA. 11 Using that lgic mdel as a starting pint, and in an effrt t assist with yur evaluatin planning, we have develped an example WISEWOMAN-specific prgram lgic mdel template (see Sectin 1 tls and templates). Please nte that the example lgic mdel shuld nt be interpreted as verriding the fur dmains prgram structure. Basic Cmpnents f a Gd Lgic Mdel 5, 7 Displayed n ne page Visually engaging Audience specific Apprpriate level f detail Clear prgram activities Clearly-stated utcmes Presents a lgical flw frm prgram inputs, activities, utputs utcmes, and impact Reflects the cntext in which the prgram perates The lgic mdel template builds n the natinal WISEWOMAN prgram lgic mdel by capturing the prgram-specific activities that ccur rutinely. Yu may wish t cnsult this lgic mdel as a template and develp a lgic mdel f yur wn. Alternatively, yu may tailr the lgic mdel template prvided in this tlkit t reflect yur prgram s specific activities and expected utcmes. The lgic mdel template we have prvided was designed t reflect key activities yu will be invlved with during the cperative agreement. The lgic mdel template als incrprates key inputs, activities, utputs, and utcmes frm the lgic mdel example CDC included in the WISEWOMAN FOA. 11 Yur prgram lgic mdel may be visually represented in a number f ways, such as a flw chart r table. 7 While the frmat used t develp yur prgram-specific lgic mdel may vary, it is imprtant that yu understand the key cmpnents f a lgic mdel s that yur prgram lgic mdel presents an accurate reflectin f the prgram. Exhibit 3 presents a descriptin f key cmpnents that are generally used in a lgic mdel. The mst imprtant aspect f a lgic mdel is that it is useful t yu and yur stakehlders in understanding and describing yur prgram. 7
Cmpnent Exhibit 3. Key Cmpnents f a Prgram Lgic Mdel 5,7 Descriptin 1. Inputs Prgram inputs are resurces that are invested int the prgram (e.g., funding surces, partners, staff, and prgram materials). 2. Activities Activities are the specific events r actins undertaken by prgram staff r partners t prduce desired utcmes (i.e., what yu d). 3. Outputs Outputs are the direct and tangible results r prducts f prgram activities ften things that can be cunted. These are ften represented by dcumentatin f prgress n implementing prgram activities (e.g., prgram materials develped, partnerships frmed, number f prviders trained, wmen screened). 4. Outcmes Outcmes are the desired results f the prgram r what yu expect t achieve. Prgram utcmes may be bserved at an rganizatin, system, r participant level. It is critical t define SMART (specific, measurable, achievable, relevant, and timebund) bjectives fr each f the utcmes yur prgram and evaluatin stakehlders agree upn. i. Shrt-term utcmes ii. Intermediate utcmes iii. Lng-term utcmes Shrt-term utcmes are the immediate effects f yur prgram activities. In general, we define shrt-term utcmes as thse utcmes expected t ccur within a relatively shrt timeframe fllwing the interventin. Shrt-term utcmes shuld lgically lead t intermediate and lng-term utcmes. Fr example, yu might expect t see a change in a participant s knwledge r readiness t make a behavir change befre she tries the State tbacc quit line and subsequently experiences a reduced risk fr cardivascular disease. Intermediate utcmes are effects f the prgram that take mre time fr a change t be bserved. Lgically, yu wuld expect yur intermediate utcmes t take place smetime after yu bserve changes in shrt-term utcmes the specific timeframe will be dependent n the nature f yur interventin (e.g., duratin and number f interventin pints) and the specific intermediate utcmes t be assessed. Typically, yu will find changes in behavirs amng the intermediate utcmes f a prgram. Fr example, nly after yu bserve changes in a participant s readiness fr tbacc cessatin might yu expect t see her try the State tbacc quit line. Lng-term utcmes reflect mre distal effects f a prgram that can take mnths r years t accmplish again depending n the nature f yur interventin and specific lng-term utcmes t be assessed. These changes likely wuld be bserved after yu bserve changes in shrt-term and intermediate utcmes. In ur example with tbacc cessatin, yu might expect t see sustained tbacc cessatin amng the lng-term utcmes. 5. Impact Distinguished frm utcmes, an impact is the ultimate effect--r public health (r ppulatin-level) impact yu hpe t see frm the prgram. Generally, it takes many years befre yu may expect t see ppulatin-level impacts f chrnic disease preventin and cntrl prgrams. 12,13 6. Cntextual factrs Examples f impacts f the WISEWOMAN prgram might include decreased mrbidity and mrtality due t cardivascular disease. Cntextual factrs are characteristics f the plitical, scial, ecnmic, and physical envirnment surrunding yur prgram that may interact with r influence prgram participants. Fr example, cntextual factrs might be similar initiatives being implemented by ther agencies, changes in health care r public health plicies, and scial nrms and values held by prgram participants. 8
Prject Narrative A narrative prgram descriptin is essential fr fcusing the evaluatin design and selecting the apprpriate methds. Often prgrams mve t begin evaluatin activities befre they even have a grasp f r agreement n what the prgram is designed t achieve r what the evaluatin shuld deliver. 5 Even if a prject narrative was included in yur funding applicatin, it is a gd practice t revisit this descriptin with yur evaluatin stakehlder grup t ensure a shared understanding f the prgram and that the prgram is still being implemented as intended. Yur prject narrative shuld include descriptins f the fllwing 3,5 : An issue statement r statement f need t identify the health issue addressed by the prgram SMART bjectives that establish the verall directin and fcus f yur prgram and establish parameters fr the expected utcmes f yur prgram Inputs r prgram resurces available t implement prgram activities Prgram activities that must take place t achieve expected utcmes Stage f develpment f the prgram t reflect prgram maturity (planning, initial implementatin, r prgram maintenance) The envirnmental cntext (i.e., plitical, scial, physical, and ecnmic) within which a prgram is implemented 9
Step 3. Fcus the Evaluatin Develped a tailred lgic mdel fr yur prgram During this step, yu will accmplish the fllwing: RECAP: At this pint in the evaluatin yu have accmplished the fllwing: Drafted a prgram narrative Wrk with yur evaluatin stakehlders t define the purpse f yur evaluatin and develp prcess and utcme evaluatin questins. Establish pririty fcal areas fr yur evaluatin. Determine the mst apprpriate methds and data surces t address yur evaluatin questins. Outline yur preliminary data analysis plan. Cmpile a cmprehensive evaluatin plan. Define the Evaluatin Purpse(s) Identified and engaged a grup f evaluatin stakehlders Created a fundatin fr cmmunicatin with stakehlders Set initial expectatins fr hw each stakehlder will be invlved during the evaluatin Established a shared understanding f the prgram Yur evaluatin purpse may be stated in many ways, but evaluatin purpses generally fall int three categries 5 : Accuntability rendering judgments abut the prgram r its cmpnents. Prgram develpment facilitating imprvements t the prgram Transferability generating knwledge abut the prgram r strategies used in the prgram that can be applied in ther settings r prgrams. 10
Identify and Priritize Evaluatin Questins Evaluatin questins specify what yu expect t learn frm yur evaluatin activities. In the evaluatin f public health prgrams, evaluatin questins generally address the fllwing, as illustrated in Exhibit 4 belw: Planning and implementatin f prgram activities (e.g., reach f the prgram, quantity and quality f prgram activities, barriers and facilitatrs t prgram implementatin) Achieving prgram bjectives Impact f the prgram n participants Impact f the prgram n systems Exhibit 4. Types f Evaluatin Questins and WISEWOMAN-Specific Examples Type f Evaluatin Questin Planning and implementatin f prgram activities WISEWOMAN-Specific Examples What are best practices used by prviders t incrprate WISEWOMAN prtcls int practice wrkflws? What are best practices emplyed by prviders t maximize the reach f the prgram t eligible participants? Achieving prgram bjectives T what extent has the prgram achieved its screening gals? Impact f the prgram n participants T what extent is the risk-reductin cunseling cmpnent being implemented with fidelity amng prviders? T what extent des participatin in the evidence-based lifestyle prgram cntribute t participant adherence t disease management plans (amng participants with hypertensin)? T what extent des participatin in the evidence-based lifestyle prgram cntribute t participant utcmes related t physical activity, nutritin, sdium intake, and tbacc cessatin? Impact f the prgram n systems Hw have WISEWOMAN prgram activities with prviders cntributed t changes in prvider prtcls fr prviding services fr all wmen? It is imprtant that yu engage yur grup f evaluatin stakehlders and slicit their input n evaluatin questins at this stage. Evaluatin questins shuld nt be selected slely based n special interests f stakehlders, but rather n hw the infrmatin generated frm the evaluatin will be used by the prgram and the stakehlders. 5 As yu wrk with yur stakehlders, establish the intentin t use the infrmatin that the evaluatin prduces frm the very beginning f yur evaluatin planning effrt. By engaging yur stakehlders in this prcess, yu will prmte transparency in the evaluatin prcess. This will, in turn, help facilitate stakehlder buy-in, prmte stakehlder acceptance f evaluatin findings, and inspire cntinued supprt fr the prgram. 5 A brainstrming r evaluatin planning sessin with yur evaluatin stakehlders can help facilitate this and can help yu establish as a grup what the evaluatin can and cannt deliver. 5 11
Generally, evaluatin questins shuld align with the prgram bjectives and lgic mdel. Yur prgram gals, as articulated in yur prgram descriptin, shuld serve as a basis fr identifying specific evaluatin questins. 7 While yur evaluatin questins are nt necessarily limited t thse that directly address the prgram bjectives, lgic mdel, gals, and purpse f yur evaluatin effrts, these factrs shuld be cnsidered first and fremst when develping evaluatin questins. Once yu have develped evaluatin questins that align with yur prgram bjectives and cnsidered hw yu will use the findings, yu shuld wrk with yur evaluatin stakehlders t refine yur questins. Cnsider yur evaluatin pririties frm the fllwing perspectives, as further described in the subsequent sectins: Iterative Nature f Evaluatin As yu wrk with yur evaluatin stakehlder grup t fcus the evaluatin design, yu may find that yu need t revisit sme f the activities in Steps 1 and 2. If that happens, just understand that it is a natural ccurrence in develping and implementing an evaluatin. D nt perceive it as a setback. Stage f prgram develpment Prcess and utcme evaluatin Shrt-term versus lng-term evaluatin activities Feasibility and resurces The scpe f yur evaluatin will depend n prgram pririties, stakehlder pririties, available resurces (including financial resurces), staff and cnsultant availability, and the amunt f time available fr implementing the evaluatin. 5 Keep Gd Recrds Once yu have cmpleted an evaluatin and reflected n the findings, yu may find that members f yur evaluatin stakehlder grup r ther prgram stakehlders may questin why sme evaluatin questins that they were interested in are nt reflected in the evaluatin findings. It is imprtant t dcument why certain evaluatin questins were priritized at this stage, s that yu have that infrmatin n hand shuld questins arise at the end f the evaluatin. Stage f Prgram Develpment As yu wrk with yur stakehlders t identify evaluatin questins, it is imprtant that yu cnsider yur prgram frm a develpmental perspective. As mentined in Step 2, prgrams generally mve thrugh the develpmental stages f planning, implementatin, and maintenance. 5 12
Prgrams ften cycle thrugh these stages several times r cre cmpnents f the prgram may be at different stages. Fr example, yur clinical services may be up and running while yur evidence-based lifestyle prgram may just be getting started r underging prgrammatic redesign. It is imprtant that yu discuss these dynamics with yur stakehlders as yu think abut evaluatin questins frm a develpmental perspective. Yur evaluatin questins may vary, depending n yur prgram s maturity as illustrated in Exhibit 5 belw. Exhibit 5. Example WISEWOMAN Evaluatin Questins by Stage f Prgram Develpment Planning Implementatin Maintenance What are the needs f the wmen eligible fr WISEWOMAN? What strategies will help us reach ur pririty ppulatin? Which evidence-based lifestyle prgram will wrk best fr ur WISEWOMAN participants? Are the cmpnents f the prgram being implemented as intended (with fidelity)? T what extent is the riskreductin cunseling cmpnent being implemented with fidelity amng prviders? Des participatin in the evidence-based lifestyle prgram cntribute t participant adherence t disease management plans (amng participants with hypertensin)? T what extent des participatin in the evidencebased lifestyle prgram cntribute t participant utcmes related t physical activity, sdium intake, and tbacc cessatin? T what extent have ur prgram activities with prviders cntributed t changes in prvider prtcls fr prviding services fr all wmen? Prcess and Outcme Evaluatin As presented in Exhibit 6, evaluatins generally fall int fur categries: Frmative evaluatin Prcess evaluatin Outcme evaluatin Impact evaluatin 13
Exhibit 6. Fur Cmmn Types f Evaluatin Type f Evaluatin Descriptin Frmative evaluatin Frmative evaluatin is usually cnducted in the planning stages f a new prgram (r when a prgram is being revised) t help ensure that the prgram is feasible t implement, apprpriate fr the pririty audience(s), and acceptable t prgram stakehlders (including prgram participants). 10 Frmative evaluatin activities include needs assessments, pilt studies, cncept testing, and message r materials testing (e.g., in WISEWOMAN, this culd include testing a risk-reductin cunseling prtcl and materials). Prcess evaluatin Prcess evaluatin is used t determine whether a prgram is being implemented as intended. 5,10,14 Prcess evaluatin fcuses n the left side f the prgram lgic mdel, alng with prgram inputs, activities, and utputs. 5,10,14 Prcess evaluatin is used t establish the plausible links between yur prgram activities and prgram utcmes. By demnstrating with prcess evaluatin that the prgram was implemented as intended, yu can set the stage fr yur expected utcmes as part f utcme evaluatin (e.g., in WISEWOMAN, prcess evaluatin culd include assessing whether evidence-based lifestyle interventins are implemented as designed). Outcme evaluatin Outcme evaluatin fcuses n the shrt-term, intermediate, and smetimes lng-term utcmes f the prgram (i.e., the right side f the prgram lgic mdel). 5 Outcme evaluatin is used t determine the effectiveness f the prgram n yur expected utcmes (e.g., in WISEWOMAN, utcme evaluatin culd invlve assessing whether WISEWOMAN prgram participatin was assciated with change in physical activity behavir). Impact evaluatin Impact evaluatin refers t assessment f the prgram in achieving its ultimate gals tward making a public health (r ppulatin-level) impact (e.g., in WISEWOMAN, this might refer t an assessment f the prgram s cntributin t reduced mrbidity and mrtality due t cardivascular disease r the ecnmic impact f the prgram). 12,13 The purpse(s) f yur evaluatin ften leads directly int the type f evaluatin needed. 7 The specific type f evaluatin can help drive pririties in develping evaluatin questins and evaluatin methds. In this evaluatin tlkit, we will fcus n prcess evaluatin and utcme evaluatin, because these are the mst cmmn types f evaluatin that yu will likely cnduct with yur WISEWOMAN prgram. The subsequent sectins describe prcess and utcme evaluatin in greater detail. We recmmend that yur evaluatin plan include bth prcess and utcme evaluatin, because prcess evaluatin can yield descriptive infrmatin abut the prgram that serves as a fundatin fr accurately interpreting utcme evaluatin findings. 5 14
Prcess Evaluatin Prcess evaluatin is generally cnducted nce prgram implementatin is underway. 7,14 As previusly nted, prcess evaluatin is used t establish whether the prgram is being implemented as intended. It fcuses n evaluating elements f the left side f yur prgram lgic mdel, as illustrated in Exhibit 7 belw. 5,10 Exhibit 7. Cnnectin Between Prcess Evaluatin and the Prgram Lgic Mdel 5 Prcess Evaluatin Inputs Activities Outputs Outcmes Impact Prcess evaluatin can help yu reprt infrmatin n yur prgram activities and use this infrmatin t imprve prgram implementatin. 7,10 In general, it allws yu t address questins related t wh, what, when, hw much, and where. 10 The text bx belw presents sme examples f prcess evaluatin questins fr a WISEWOMAN prgram. Examples f Prcess Evaluatin Questins Hw did prgram staff cllabrate with prviders and cmmunity-based rganizatins t maximize the reach f the WISEWOMAN prgram amng pririty audiences? What are best practices emplyed by prviders t maximize the reach f the prgram t eligible participants? T what extent is the risk-reductin cunseling cmpnent being implemented with fidelity amng prviders? Hw many wmen received WISEWOMAN screenings cmpared t wmen targeted? Hw many wmen started the evidence-based lifestyle prgram? Hw many wmen cmpleted it? What are facilitatrs and barriers f implementing the evidence-based lifestyle prgram? What was ur errr rate fr the MDE submissins this prgram year? 15
Outcme Evaluatin Outcme evaluatin allws yu t address the questin, Did we achieve the utcmes that we expected t achieve? 7 In public health, utcme evaluatin allws yu t dcument healthrelated and behaviral utcmes and identify linkages between expsure t the prgram and quantifiable utcmes. 5,7 The text bx belw presents sme examples f utcme evaluatin questins fr a WISEWOMAN prgram. Examples f Outcme Evaluatin Questins T what extent des participatin in the evidence-based lifestyle prgram cntribute t participant utcmes related t physical activity, sdium intake, and tbacc cessatin? T what extent des participatin in the evidence-based lifestyle prgram cntribute t participant adherence t disease management plans (amng participants with hypertensin)? Hw have ur prgram activities with prviders cntributed t changes in prvider prtcls fr prviding services fr all wmen? Outcme evaluatin is usually cmpleted after a prgram is fairly well established ntice that we use the term cmpleted and nt started. Planning fr utcme evaluatin shuld begin in the planning stages f a prgram. Yu may find that in rder t fully address yur utcme evaluatin questins, yu may need t cllect sme frmative evaluatin data befre yu begin implementing the prgram. As previusly mentined, utcme evaluatin allws yu t determine the effectiveness f the prgram n yur expected utcmes. 5 In ther wrds, utcme evaluatin fcuses n the right side f a prgram lgic mdel, as illustrated in Exhibit 8. T determine the effectiveness f the prgram n specific utcmes, yu als may need t cllect baseline data data cllected befre participants start the prgram. Cllecting baseline data will allw yu t make meaningful cmparisns frm befre participatin t time pints after the prgram has been delivered entirely r fllwing any f several pssible intermediate time pints during the full curse f prgram delivery. 5 16
Exhibit 8. Cnnectin Between Outcme Evaluatin and the Prgram Lgic Mdel 5 Outcme Evaluatin Inputs Activities Outputs Outcmes (Shrt term, immediate, and lng term) Impact Priritizing Shrt-Term Versus Lng-Term Evaluatin Activities Yu and yur evaluatin stakehlder grup may be cncerned with hw much yu may be able t accmplish with yur evaluatin in a given year. Keep in mind that WISEWOMAN is n a 4- year cycle. An advantage f this is that yu d nt have t accmplish all aspects f yur evaluatin right away. Yu might cnsider yur evaluatin in the cntext f shrt-term versus lng-term pririties. Therefre, as yu wrk with yur stakehlders t priritize questins, yu may decide that yu want t address r it s mre apprpriate t address sme parts f yur evaluatin in the shrt term and thers later in the 4-year cycle. Nevertheless, it is imprtant t plan t address bth shrt- and lng-term pririties during the 4-year prgram cycle. Feasibility and Resurces As yu discuss pririty evaluatin questins with yur evaluatin stakehlders, yu will need t address the feasibility f implementing evaluatin activities. Inevitably the questin f csts (including fiscal resurces, materials, and human resurces) will arise. The fllwing text bx highlights questins that yu shuld cnsider when assessing yur evaluatin pririties. 17
Cnsideratins When Assessing the Feasibility f Addressing Evaluatin Questins 2,4,5,8 When d yu need the evaluatin results? Hw frequently will yu need t cllect data and review the results? Wh will cnduct the evaluatin? What skills and levels f staff d yu need in rder t carry ut the evaluatin? D yu have prgram staff members wh can help implement the evaluatin? Can any f yur stakehlders help implement the evaluatin (e.g., data cllectin, analysis)? D yu need the assistance f a cnsultant t address the evaluatin questins? Will yu need t train any prgram staff, partners, r prviders t facilitate data cllectin? D yu already cllect data that can help yu address yur evaluatin questins? D yu have structures in place (e.g., intake prcesses, standard reprting) that can be tweaked t allw yu t cllect data fr use in the evaluatin? Is anther grup already addressing a similar evaluatin questin? Can yu maximize resurces by cllabrating with them? Hw much f a burden will this evaluatin be n yur prgram stakehlders? Will it interfere with standard prcesses? Including yur evaluatin stakehlder grup in priritizing evaluatin questins may yield unfreseen benefits. Fr example, yu may find that certain stakehlders will advcate fr prviding yu with additinal resurces t help address key questins. In sme instances, stakehlders may even cntribute t the evaluatin, allwing yu t address higher-pririty evaluatin questins. 5 Hwever, at times yu simply may nt have the resurces necessary t fund the evaluatin activities fr questins yu wuld like t answer mst. Discussing these cnstraints with yur evaluatin stakehlder grup can help facilitate a shared understanding f what can and cannt be delivered by the evaluatin. 5 Reminder If yu need guidance r technical assistance as yu cnsider the feasibility f addressing yur evaluatin questin, remember that yur CDC Prject Officer and Evaluatin Specialist can help yu cnsider yur ptins. Priritizing Evaluatin Questins Ultimately, the gal in identifying and priritizing evaluatin questins is t wrk with yur evaluatin stakehlders t fcus the evaluatin effrts n feasible and useful evaluatin questins that reflect yur prgram s gals, stage f develpment, and selected purpses fr the evaluatin. 5 The fllwing text bx highlights key questins t address with yur evaluatin stakehlders t help priritize yur evaluatin questins. Yu will ntice that the questins align with the fur standards f the CDC Evaluatin Framewrk. 18
Key Questins t Address When Priritizing Evaluatin Questins 6 Hw will the answers t the evaluatin questins be used? (Utility standard) Will yur prgram be able t make changes in the areas yu are examining? (Utility standard) Which evaluatin questins are feasible t answer? (Feasibility standard) Which questins des yur team have the time, skills, and resurce t investigate? Which evaluatin questins can yu answer using an existing data surce? Can the data needed t answer the evaluatin questin be cllected in accrdance with key guidelines (e.g., maintaining patient cnfidentiality)? (Prpriety standard) Are data available t answer the evaluatin questin with the level f accuracy that is needed t prvide a credible answer t the evaluatin questin? (Accuracy standard) Exhibit 9 illustrates hw yu might rganize the discussin with yur evaluatin stakehlder grup in an evaluatin planning meeting t priritize yur evaluatin. Yur evaluatin stakehlders may help: Organize the discussin by prgrammatic activities f interest. Identify hw the infrmatin will be used (accuntability, prgram imprvement ). Determine at what stage f prgram develpment is certain infrmatin needed. Establish pririties while cnsidering feasibility. As yu select final evaluatin questins, yu will need t balance which questins might be mst useful t yur prgram, while als meeting sme f yur stakehlders infrmatin needs. 5 Yu may find that as yu wrk thrugh priritizing questins with yur evaluatin stakehlders, they will begin t take wnership, which shuld slidify their interest in the evaluatin. Invlving stakehlders in fcusing yur evaluatin als may increase the likelihd that they will secure evaluatin resurces, prvide access t data, help t cllect data, help t interpret the findings, and disseminate findings. 3,8 19
Tpic Risk-reductin cunseling Risk-reductin cunseling Recruitment/enrllment prcesses Purpse/Use (Relevant Prgram Stage) Prgram imprvement T determine whether and hw the risk-reductin cunseling prtcls might be mdified (Implementatin) Achieving prgram bjectives T determine whether prviders need additinal training r technical assistance n this cmpnent f the prgram (Planning, implementatin) Prgram imprvement T guide training and technical assistance t prviders t prmte successful recruitment/enrllment prcedures (Implementatin) Exhibit 9. Priritizing Evaluatin Questins Example* Type f Questin Prcess Planning and implementatin f prgram activities Prcess Achieving prgram bjectives Prcess Planning and implementatin f prgram activities Feasibility Can use MDEs, but will need t cllect sme data frm participants and prviders Still in planning stages and wrking thrugh sme issues t get the lifestyle prgram up and running Can ur prgram epidemilgist help with this? Can address this using the quality assurance site visit reprts Will require us t cllect new data frm prviders Timing Years 2 Year 1 Years 3 4 Evaluatin Questin Hw has risk-reductin cunseling influenced client participatin in the evidence-based lifestyle prgram? T what extent is the risk-reductin cunseling cmpnent being implemented with fidelity amng prviders? What best practices are used by prviders t maximize the reach f the prgram t eligible participants? * The examples presented here are fr illustrative purpses nly. The cntent is meant t represent a range f issues that prgrams may discuss and des nt represent the pinin f the WISEWOMAN prgram r CDC. Pririty Lw High Lw 20
Exhibit 9. Priritizing Evaluatin Questins Example (Cntinued) Tpic Purpse/Use (Relevant Prgram Stage) Type f Questin Feasibility Timing Evaluatin Questin Pririty Evidencebased lifestyle prgram Accuntability T determine whether the ABC lifestyle prgram is effective and whether we will cntinue the prgram in the next FOA (Maintenance) Outcme Impact f the prgram n participants Still in planning stages Will wait until cmpnent is fully peratinal in Year 2 Will need t determine type f data cllected by ABC prgram and whether ABC prgram wuld help us cllect additinal infrmatin if we create an instrument t supplement what they already cllect Years 2 4 T what extent des participatin in the evidence-based lifestyle prgram cntribute t participant utcmes related t physical activity, sdium intake, and tbacc cessatin? High Evidencebased lifestyle prgram Accuntability t determine whether the ABC lifestyle prgram is effective and whether we will cntinue the prgram in the next FOA (Maintenance) Outcme Impact f the prgram n participants Will require data n selfmanagement plans established by prviders Prviders reprt a lt f challenges with abstracting this data frm their electrnic medical recrd (EMR) systems. Fr mst prviders, this is reprted in the prvider ntes fields. Only 2 f 30 prviders have been able t wrk this ut. Years 2 4 T what extent des participatin in the evidence-based lifestyle prgram cntribute t participant adherence t disease management plans (amng participants with hypertensin)? Lw/ medium 21
Exhibit 9. Priritizing Evaluatin Questins Example (Cntinued) Tpic Purpse/Use (Relevant Prgram Stage) Type f Questin Feasibility Timing Evaluatin Questin Pririty Screening Transferability T cntribute t the implementatin science literature regarding implementatin f evidencebased practice and present at an All-Prgram Call (Maintenance) Prcess Impact f the prgram n systems Will necessitate cllectin f new data Need baseline data n current wrkflws While this is may be a cstly endeavr, Suzy Q. feels that this is an imprtant questin in which t invest. She suggested that this is ging t be a pririty fr CDC next year. Years 1 4 What best practices are used by prviders t incrprate WISEWOMAN prtcls int practice wrkflws? High Screening Accuntability T determine whether the prgram has an effect n prvider systems (Maintenance) Outcme Impact f the prgram n systems While this is an imprtant questin, we are nt exactly sure hw t g abut evaluating this need t cnnect with ur Prject Officer and Evaluatin Specialist befre mving frward Year 4 Hw have ur prgram activities cntributed t changes in prvider prtcls fr ffering services fr all wmen? Medium 22
Identifying Apprpriate Evaluatin Methds Once yu have priritized yur evaluatin questins, the next step in the planning prcess is t determine apprpriate evaluatin methds in ther wrds, what yu will d t get the infrmatin yu need t address yur evaluatin questins. 7 Determining apprpriate methds includes defining indicatrs, data surces, and data cllectin techniques and types f data that will be cllected. Smetimes there is a tendency t select methds with which yu r yur evaluatin stakehlders are familiar and cmfrtable and then frce the evaluatin t fit that methd. 5 Hwever, it is imprtant fr yu t select the mst apprpriate methds based n yur evaluatin questins. Here are sme pints t cnsider as yu identify yur evaluatin methds 5 : Reminder It may be helpful fr yu t revisit sme f the earlier steps in the CDC Evaluatin Framewrk as yu identify yur evaluatin methds and cnsider what may r may nt be feasible. Remember, if yu need assistance as yu cnsider yur ptins, feel free t cntact yur CDC Prject Officer r Evaluatin Specialist. Remember the purpse f yur evaluatin, yur prgram s lgic mdel, and the stage f the develpment f the prgram. Track decisins back t yur evaluatin questins. The tls and templates prvided in this evaluatin tlkit will help yu d this. Cnsider the data that yur prgram is already cllecting. Yu may find that yu already have sufficient data, r that with a few minr changes t yur existing data cllectin activities yu can capture sufficient data t address sme f yur evaluatin questins. Think abut what yur stakehlders cnsider as credible evidence. We recmmend that yu btain input and feedback frm yur evaluatin stakehlders t clarify this. Cnsider yur budget. While using existing data may be less expensive than cllecting new data, it may als require sme effrt t abstract r transfrm the existing data. XYZ WISEWOMAN Prgram In this and subsequent sectins f the tlkit, we will refer t an example case study f a fictitius WISEWOMAN prgram, XYZ, t illustrate key pints, decisins, challenges, and ptential slutins. Evaluatin Methds Matrix An evaluatin methds matrix, which may be referred t as an evaluatin grid r evaluatin framewrk, is a systematic way f rganizing and presenting yur evaluatin methds. Exhibit 10 illustrates an evaluatin methds matrix fr sample high-pririty evaluatin questins selected by a fictitius WISEWOMAN prgram, XYZ, that will serve as a case study example thrughut the tlkit. In each f the subsequent sectins, we will walk yu thrugh examples f hw this prgram identified the apprpriate evaluatin methds t address their evaluatin questins. 23
Exhibit 10. Example Evaluatin Methds Matrix Evaluatin Questins Indicatrs Data Surces Methds Timing, Duratin, and Frequency f Data Cllectin Analyses (Preliminary) What best practices are used by prviders t incrprate WISEWOMAN prtcls int practice wrkflws? (i.e., Which measures will answer yur evaluatin questin?) (i.e., Frm whm/ where will yu get the data?) (i.e., What data cllectin techniques and design will yu use?) (i.e., When, ver what perid f time, and hw ften will yu cllect data?) (i.e., What type f analysis will help yu address yur evaluatin questin?) T what extent des participatin in the evidence-based lifestyle prgram cntribute t participant utcmes related t physical activity, sdium intake, and tbacc cessatin? Indicatrs We recmmend that yu begin the prcess f defining yur evaluatin methds by develping indicatrs fr each f yur pririty evaluatin questins. An indicatr is a specific, bservable, and measurable marker f change r accmplishment. 4,7,10 Yu might think f indicatrs as specifying the way in which yu will peratinalize r answer yur evaluatin questins. Yu may need t identify mre than ne indicatr fr a given evaluatin questin, depending n the nature f the questin and what yu hpe t answer with it. Yur indicatrs shuld prvide a clear descriptin f what yu want t measure. An indicatr shuld be smething that is bserved (e.g., a change in behavir), heard r reprted (e.g., shared by prgram participants), r read (e.g., prgram recrds). This is smewhat similar t 24
hw yu might identify SMART i bjectives fr yur prgram. 4 Yur prgram lgic mdel will be especially valuable t yu as yu identify indicatrs fr yur evaluatin. Als, make sure that yur selectin f evaluatin indicatrs is guided by the evaluatin questins as discussed in the fllwing checklist. Checklist fr Defining Indicatrs 10 Engage yur evaluatin stakehlder grup in develping and selecting yur evaluatin indicatrs. Cnsult yur evaluatin questins and use yur lgic mdel as a template fr develping indicatrs. The templates prvided in this tlkit will help facilitate this. Fr prcess evaluatin questins, indicatrs shuld be related t utputs n yur lgic mdel. Fr utcme evaluatin questins, indicatrs shuld be related t the shrt-term utcmes f the lgic mdel. Depending n yur prgram s develpment, yur indicatrs may relate t the intermediate and lng-term utcmes in yur lgic mdel. As apprpriate, cnsider the WISEWOMAN perfrmance measures when identifying indicatrs fr prcess evaluatin questins. Make sure that yur indicatrs are specific, bservable, and measurable. Identify indicatrs that are time-specific t define the time parameters fr yur evaluatin (e.g., as f the end f Prgram Year 1 r between the first lifestyle prgram sessin and 1 mnth after the final lifestyle prgram sessin). Make sure that the selected indicatrs prvide useful infrmatin, are feasible t measure, prvide sufficient infrmatin t allw yu t address yur evaluatin questins. i SMART bjectives are specific, measureable, achievable, relevant, and time-bund, and they lead t evaluatin questins that assess key features f yur prgram. 25
As illustrated in Exhibit 11 belw, we recmmend mapping yur prcess evaluatin questins t the apprpriate utputs n yur lgic mdel. Mapping the questins will help in drafting the prcess indicatrs and in selecting indicatrs. Depending n yur prcess evaluatin questin(s), yu may want t cnsider the WISEWOMAN perfrmance measures as indicatrs. Exhibit 11. Example Indicatr Mapping Exercise, Prcess Evaluatin Prcess Evaluatin Questin What best practices are used by prviders t incrprate WISEWOMAN prtcls int practice wrkflws? Output (frm prgram lgic mdel) Number and percentage f eligible wmen screened Number and percentage f wmen screened wh receive risk reductin cunseling Number and percentage f wmen diagnsed with hypertensin r cardimetablic cnditins wh are referred t and receive fllw-up clinical care Number and percentage f wmen with ne r mre risk factrs referred t evidencebased lifestyle prgrams Number and percentage f wmen with ne r mre risk factrs referred t apprpriate cmmunity-based resurces Indicatr Prcedures and practices used by the tp-perfrming clinical sites as identified by their perfrmance n the fllwing: Number and percentage f eligible wmen screened Number and percentage f wmen screened wh receive risk-reductin cunseling Number and percentage f wmen diagnsed with hypertensin r cardimetablic cnditins wh are referred t and receive fllw-up clinical care Number and percentage f wmen with ne r mre risk factrs referred t evidencebased lifestyle prgrams Number and percentage f wmen with ne r mre risk factrs referred t apprpriate cmmunity-based resurces 26
As illustrated in Exhibit 12 belw, we recmmend mapping yur utcme evaluatin questins t the apprpriate utcmes n yur lgic mdel. Depending n yur stage f prgram develpment, the nature f the intended utcme, and the timing f yur evaluatin activities, shrt-term, intermediate, and in sme cases lng-term utcmes may be applicable. Exhibit 12. Example Indicatr Mapping Exercise, Outcme Evaluatin Outcme Evaluatin Questin T what extent des participatin in the evidence-based lifestyle prgram cntribute t participant utcmes related t physical activity, sdium intake, and tbacc cessatin? Outcmes (frm prgram lgic mdel) Increased participant selfefficacy, readiness t change, and behaviral intent cncerning healthy eating, physical activity, and tbacc cessatin (shrtterm utcme) Increased desirable health behavirs (intermediate utcme) Indicatrs Change in self-efficacy, readiness t change, and intent t engage in 30 minutes f mderate physical activity daily between baseline (i.e., prelifestyle prgram participatin), upn cmpleting the lifestyle prgram, and 6 mnths after cmpleting the lifestyle prgram Change in self-efficacy, readiness t change, and intent t reduce cnsumptin f fast fds between baseline, upn cmpleting the lifestyle prgram, and 6 mnths after cmpleting the lifestyle prgram Change in self-efficacy, readiness t change, and intent t quit using tbacc between baseline, upn cmpleting the lifestyle prgram, and 6 mnths after cmpleting the lifestyle prgram Change in number f minutes per week spent in mderate physical activity between baseline and 6 mnths after cmpleting the lifestyle prgram Change in the number f fast fd meals cnsumed per week between baseline and 6 mnths after cmpleting the lifestyle prgram Change in tbacc use between baseline and 6 mnths after cmpleting the lifestyle prgram It is imprtant that yu cntinue t slicit feedback frm members f yur evaluatin stakehlder grup as yu finalize yur evaluatin indicatrs. This will help yu keep yur stakehlders engaged and maintain stakehlder buy-in fr yur evaluatin findings. 10 This can be accmplished by cnvening yur stakehlders t discuss the indicatrs as a grup r sending 27
dcuments t yur stakehlders fr review and feedback. Yur stakehlders can help yu determine whether the evaluatin indicatrs Prvide useful infrmatin. Are feasible t measure. Prvide sufficient infrmatin t address the evaluatin questins. Belw is a case study f ur example WISEWOMAN prgram s experience with seeking feedback frm stakehlders n evaluatin indicatrs. The experience highlights the imprtance f stakehlder engagement in this step. Case Study: Stakehlder Engagement and Indicatr Selectin Betty S., the prgram manager fr the XYZ WISEWOMAN prgram, has been wrking with an evaluatin cnsultant n the prgram evaluatin plan. After cming up with a list f indicatrs fr each f the pririty evaluatin questins, they decided t send a table utlining their evaluatin questins, related cmpnents f the lgic mdel, and indicatrs t the evaluatin stakehlder grup fr review. In an email reply, Jane D., RN, MSN, a representative frm the clinical prviders advisry grup, pinted ut that the fllwing indicatr may nt yield useful infrmatin fr imprving prgram perfrmance: Descriptin f prcedures and practices emplyed by the tp-perfrming clinical sites based n the fllwing measures: Number and percentage f eligible wmen screened Number and percentage f wmen screened wh received risk-reductin cunseling Number and percentage f wmen diagnsed with hypertensin r cardimetablic cnditins wh were referred t and received fllw-up clinical care Number and percentage f wmen with ne r mre risk factrs referred t evidencebased lifestyle prgrams Number and percentage f wmen with ne r mre risk factrs referred t apprpriate cmmunity-based resurces Betty crdinated with her evaluatin cnsultant t set up a 30-minute call with Jane D. t get sme clarificatin n her cmment. During the cnversatin, Jane expressed cncern that sites nt amng the tp perfrmers may already be using similar practices and prcedures. It might be frustrating t them that they are nt really learning anything new. She als asked fr clarificatin n what tp perfrming really means and what timeframe wuld be used t assess perfrmance. Betty, Jane, and the evaluatin cnsultant wrked tgether t draft a new indicatr fr this questin: Descriptin f unique prcedures and practices emplyed by the tp 10% f clinical sites cmpared t the lwest 10% f clinical sites based n the fllwing measures at the end f Prgram Year 2: Number and percentage f eligible wmen screened Number and percentage f wmen screened wh received risk-reductin cunseling 28
Case Study: Stakehlder Engagement and Indicatr Selectin (Cntinued) Number and percentage f wmen diagnsed with hypertensin r cardimetablic cnditins wh were referred t and received fllw-up clinical care Number and percentage f wmen with ne r mre risk factrs referred t evidencebased lifestyle prgrams Number and percentage f wmen with ne r mre risk factrs referred t apprpriate cmmunity-based resurces The evaluatin cnsultant agreed t revise the table per this feedback, as well as sme relatively minr feedback frm ther stakehlders. Betty then sent the revised table ut t her evaluatin stakehlders fr their recrds. Example In yur evaluatin methds matrix, yu shuld nw have the infrmatin needed t cmplete the first tw clumns, as illustrated in Exhibit 13. 29
Exhibit 13. Example Evaluatin Methds Matrix Indicatrs Evaluatin Questins Prcess Evaluatin Indicatrs Data Surces Methds Timing, Duratin, and Frequency f Data Cllectin Analyses (Preliminary) What are best practices used by prviders t incrprate WISEWOMAN prtcls int practice wrkflws? Descriptin f unique prcedures and practices emplyed by the tp 10% f clinical sites cmpared t the bttm 10% f clinical sites based n the fllwing measures at the end f Prgram Year 2: Number and percentage f eligible wmen screened Number and percentage f wmen screened wh receive risk-reductin cunseling Number and percentage f wmen diagnsed with hypertensin r cardimetablic cnditins wh were referred t and received fllw-up clinical care Number and percentage f wmen with ne r mre risk factrs referred t evidencebased lifestyle prgrams Number and percentage f wmen with ne r mre risk factrs referred t apprpriate cmmunity-based resurces 30
Exhibit 13. Example Evaluatin Methds Matrix Indicatrs (Cntinued) Evaluatin Questins Indicatrs Data Surces Methds Timing, Duratin, and Frequency f Data Cllectin Analyses (Preliminary) Outcme Evaluatin T what extent des participatin in the evidence-based lifestyle prgram cntribute t participant utcmes related t physical activity, sdium intake, and tbacc cessatin? Change in self-efficacy, readiness t change, and intent t engage in 30 minutes f mderate physical activity daily between baseline (i.e., prelifestyle prgram participatin), upn cmpleting the lifestyle prgram, and 6 mnths after cmpleting the lifestyle prgram Change in self-efficacy, readiness t change, and intent t reduce cnsumptin f fast fds between baseline, upn cmpleting the lifestyle prgram, and 6 mnths after cmpleting the lifestyle prgram Change in self-efficacy, readiness t change, and intent t quit using tbacc between baseline, upn cmpleting the lifestyle prgram, and 6 mnths after cmpleting the lifestyle prgram Change in number f minutes per week spent in mderate physical activity between baseline and 6 mnths after cmpleting the lifestyle prgram Change in the number f fast fd meals cnsumed per week between baseline and 6 mnths after cmpleting the lifestyle prgram Change in tbacc use between baseline and 6 mnths after cmpleting the lifestyle prgram 31
Identifying Data Surces By nw yu have cmpleted the first tw clumns f the evaluatin methds matrix. The next step is t identify data surces frm which yu can gather infrmatin related t yur indicatrs and that address yur evaluatin questins (e.g., persns, dcuments, bservatins, administrative databases). 4 Just as yu may have identified mre than ne indicatr fr a given evaluatin questin, yu may identify mre than a single data surce fr a given evaluatin questin. In fact, yu can enhance the credibility f yur findings by using multiple data surces fr a given questin and cmparing the data gathered frm acrss the data surces. 5,6 Yu may have heard f this as triangulating data frm different surces t supprt yur evaluatin findings. 6 Key Questins t Address When Identifying Data Surces 10 What infrmatin d yu need? What data d yu already cllect? Which indicatrs will require yu t cllect new data? When d yu need the data? D yu have access r can yu btain access t the data surce? D members f yur evaluatin stakehlder grup view the data surces as credible? Hw will yu use the data gathered frm the data surce? Alternatively, there is smetimes a tendency t identify many data surces fr yur evaluatin questins, perhaps when multiple evaluatin questins can be addressed using just a few data surces r even a single cmprehensive data surce. In part, determining the number f data surces will be a practical matter. Yu will want t use yur evaluatin funds efficiently by getting mre bang fr yur buck frm a set f carefully selected data surces. As yu identify data surces bth new and existing it is imprtant that yu keep in mind hw yu will use the data gathered in the evaluatin. T that end, it is imprtant that yu strategically select data surces that will answer yur evaluatin questins mst effectively and efficiently. 4,5 All data surces have advantages and disadvantages, given hw data will be cllected, managed, and analyzed. 5,6,10 Therefre, it is imprtant t cnsider yur data surces carefully and keep in mind the key questins yu are trying t address. 5,10 It is beynd the scpe f this tlkit t discuss the different intricacies and cmplexities f using different data surces; hwever, the Resurce Guide in Sectin 4 prvides mre specific infrmatin. We als encurage yu t seek input and feedback frm yur evaluatin stakehlders related t these questins, as they may prvide yu with valuable insight t cnsider abut the data surces. 4 Yu may als cnsult yur CDC Prject Officer and Evaluatin Specialist fr assistance. 32
Existing Data Surces We encurage yu t first lk t data surces that are already available fr yu. Fr example, MDEs are a primary data surce available t WISEWOMAN prgrams. Yu may als have access t prvider claim submissins t determine the types f services prvided, participant intake/infrmatin frms, and participant behavir/readiness t change assessments. Fr sme questins, yu might even cnsider surveillance data (e.g., Behaviral Risk Factr Surveillance Survey data); hwever, if yu use surveillance data, it is imprtant t keep in mind that these data are nt specific t the WISEWOMAN prgram, and are nt cllected fr specific prgram evaluatin effrts. Ptential Existing Data Surces fr WISEWOMAN Prgrams MDEs Prvider claim submissins Participant intake/infrmatin frms Participant behavir/readiness t change assessments Prgram dcuments Surveillance data Existing data surces als include prgram dcuments and administrative recrds, including the fllwing: Training recrds Quality cntrl site visit reprts Staff recrds Plicies and prcedure dcuments Remember, even data that are nt cntained in a frmal database may be useful fr yur evaluatin. Cnsider, fr example, that as part f the intake prcess, participants fill ut a brief behavir/readiness t change assessment that is then used t prvide risk-reductin cunseling. In this example, data cllected frm the brief behavir/readiness t change assessments may nt have been entered int a database, but these data may be a useful surce t help yu answer certain evaluatin questins. New Data Surces Yu will likely need new data surces t fully address yur evaluatin questins, as highlighted in the case study belw. New data cllectin can have a significant impact n yur evaluatin budget. It will be imprtant t balance the need t cllect data frm multiple data surces with the need t cllect new data. Ptential new data surces fr WISEWOMAN prgrams might include the fllwing: Prgram participants Prgram staff Prviders 33
Partners Lifestyle prgram staff New administrative frms Case Study: Stakehlder Engagement and Data Surces The XYZ WISEWOMAN Prgram evaluatin stakehlder grup had riginally planned t use existing quality cntrl site visit reprts t address the fllwing evaluatin questin. The team drafted their indicatr fr this questin and began t identify data surces. Hwever, Dug P., the quality cntrl crdinatr fr the WISEWOMAN and Breast and Cervical Cancer Prgrams (BCCPs), expressed that he des nt think the quality cntrl site visit reprts will prvide sufficient detail t adequately describe the unique prcedures and practices that clinical sites used t incrprate WISEWOMAN prtcls. Evaluatin Questin: What are best practices used by prviders t incrprate WISEWOMAN prtcls int practice wrkflws? Indicatr: Descriptin f unique prcedures and practices emplyed by the tp 10% f clinical sites cmpared t the bttm 10% f clinical sites based n the fllwing measures at the end f Prgram Year 2: Number and percentage f eligible wmen screened Number and percentage f wmen screened wh received risk-reductin cunseling Number and percentage f wmen diagnsed with hypertensin r cardimetablic cnditins wh were referred t and received fllw-up clinical care Number and percentage f wmen with ne r mre risk factrs referred t evidencebased lifestyle prgrams Number and percentage f wmen with ne r mre risk factrs referred t apprpriate cmmunity-based resurces After discussing this issue, the grup agreed that they wuld als need t cllect data frm clinical service prviders t fully address the questin. 34
Example In yur evaluatin methds matrix, yu shuld nw have the infrmatin needed t cmplete the first three clumns as illustrated in Exhibit 14 belw. Exhibit 14. Example Evaluatin Methds Matrix Data Surces Evaluatin Questin Indicatrs Data Surces Methds Timing, Duratin, and Frequency f Data Cllectin Analyses (Preliminary) Prcess Evaluatin What are best practices used by prviders t incrprate WISEWOMAN prtcls int practice wrkflws? Descriptin f unique prcedures and practices emplyed by the tp 10% f clinical sites cmpared t the bttm 10% f clinical sites based n the fllwing measures at the end f Prgram Year 2: Number and percentage f eligible wmen screened Number and percentage f wmen screened wh received risk-reductin cunseling Number and percentage f wmen diagnsed with hypertensin r cardimetablic cnditins wh were referred t and received fllw-up clinical care Number and percentage f wmen with ne r mre risk factrs referred t evidence-based lifestyle prgrams Number and percentage f wmen with ne r mre risk factrs referred t apprpriate cmmunity-based resurces Clinical service prviders Quality cntrl site visit reprts 35
Exhibit 14. Example Evaluatin Methds Matrix Data Surces (Cntinued) Evaluatin Questin Indicatrs Data Surces Methds Timing, Duratin, and Frequency f Data Cllectin Analyses (Preliminary) Outcme Evaluatin T what extent des participatin in the evidence-based lifestyle prgram cntribute t participant utcmes related t physical activity, sdium intake, and tbacc cessatin? Change in self-efficacy, readiness t change, and intent t engage in 30 minutes f mderate physical activity daily between baseline (i.e., prelifestyle prgram participatin), upn cmpleting the lifestyle prgram, and 6 mnths after cmpleting the lifestyle prgram Change in self-efficacy, readiness t change, and intent t reduce cnsumptin f fast fds between baseline, upn cmpleting the lifestyle prgram, and 6 mnths after cmpleting the lifestyle prgram Prgram participants wh have cmpleted at least ne lifestyle prgram sessin Participant behavir/ readiness t change assessments Change in self-efficacy, readiness t change, and intent t quit using tbacc between baseline, upn cmpleting the lifestyle prgram, and 6 mnths after cmpleting the lifestyle prgram Change in number f minutes per week spent in mderate physical activity between baseline and 6 mnths after cmpleting the lifestyle prgram Change in the number f fast fd meals cnsumed per week between baseline and 6 mnths after cmpleting the lifestyle prgram Change in tbacc use between baseline and 6 mnths after cmpleting the lifestyle prgram 36
Data Cllectin Methds Nw that yu have selected data surces fr yur evaluatin questins, it is time t identify the methds that yu will use t cllect data frm yur data surces. The specific methds are driven by each f yur evaluatin questins. 5,10 Nt all methds fit all evaluatin questins. 5 Yu als will want t cnsider the timing, duratin, and frequency f yur data cllectin activities. 10 Addressing sme evaluatin questins will necessitate cllecting data at Reminder Be sure t seek recmmendatins frm yur evaluatin stakehlders, CDC Prject Officer, and CDC Evaluatin Specialist. They may have ideas fr methds that yu have nt cnsidered yet. multiple time pints (e.g., prir t the start f participatin and at a selected fllw-up pint). T address certain evaluatin questins, it may be helpful t cllect data after each f a series f interventin pints (e.g., lifestyle prgram sessins). 1 As discussed in greater detail in the subsequent sectins, there are tw general types f data cllectin methds: quantitative methds and qualitative methds. Yu may als use multiple methds t prvide a cmprehensive answer t a single evaluatin questin, als knwn as mixed methds. 5 We recmmend using multiple quantitative and qualitative methds in yur evaluatin. Questins t Address When Selecting Methds t Cllect Data 4,10 Which methds will allw yu t btain the infrmatin that yu need? That is, which methds will help yu gather infrmatin frm yur selected data surces that directly address yur evaluatin indicatrs, and subsequently yur evaluatin questins? Which methds are mst apprpriate fr use with the data surces that yu have identified? Which methds are mst feasible fr yu t implement, given yur available resurces (mney, persnnel, skill level, etc.)? Which methds are cnsidered credible by yur evaluatin stakehlders? 37
Quantitative Methds Quantitative methds are used t gather numerical data t make calculatins and draw cnclusins. 4 In general, quantitative methds are used t answer the fllwing types f evaluatin questins: Hw much? Hw many? Qualitative Methds Examples f Quantitative Methds Surveys Bimetric tests Quantifiable bservatins Secndary analysis f existing quantitative data Administrative frms Qualitative methds are used t gather data in the frm f ntes, verbal respnses, transcripts, and written respnses. 4 These methds generally allw yu t capture thughts, feelings, and perspectives. 4 In general, qualitative methds are used t answer questins that call fr rich, descriptive infrmatin, such as the fllwing: Why? Hw? Examples f Qualitative Methds Interviews Fcus grups Observatin ntes/diaries Secndary analysis f existing qualitative data Qualitative questins in surveys Testimnials r success stries Example In yur evaluatin methds matrix, yu shuld nw have the infrmatin needed t cmplete the first five clumns, as illustrated in Exhibit 15 belw. 38
Exhibit 15. Example Evaluatin Methds Matrix Methds Evaluatin Questin Indicatrs Data Surces Methds Timing, Duratin, and Frequency f Data Cllectin Analyses (Preliminary) Prcess Evaluatin What best practices are used by prviders t incrprate WISEWOMAN prtcls int practice wrkflws? Descriptin f unique prcedures and practices emplyed by the tp 10% f clinical sites cmpared t the bttm 10% f clinical sites based n the fllwing measures at the end f Prgram Year 2: Number and percentage f eligible wmen screened Number and percentage f wmen screened wh received risk-reductin cunseling Clinical service prviders Quality cntrl site visit reprts Interviews with clinical service prviders Secndary analysis f existing qualitative data Cnduct interviews with clinical service prviders at a single time pint during the first quarter f Prgram Year 3 Number and percentage f wmen diagnsed with hypertensin r cardimetablic cnditins wh were referred t and received fllw-up clinical care Number and percentage f wmen with ne r mre risk factrs referred t evidence-based lifestyle prgrams Number and percentage f wmen with ne r mre risk factrs referred t apprpriate cmmunity-based resurces 39
Exhibit 15. Example Evaluatin Methds Matrix Methds (Cntinued) Evaluatin Questin Outcme Evaluatin Indicatrs Data Surces Methds Timing, Duratin, and Frequency f Data Cllectin Analyses (Preliminary) T what extent des participatin in the evidencebased lifestyle prgram cntribute t participant utcmes related t physical activity, sdium intake, and tbacc cessatin? Change in self-efficacy, readiness t change, and intent t engage in 30 minutes f mderate physical activity daily between baseline (i.e., pre-lifestyle prgram participatin), upn cmpleting the lifestyle prgram, and 6 mnths after cmpleting the lifestyle prgram Change in self-efficacy, readiness t change, and intent t reduce cnsumptin f fast fds between baseline, upn cmpleting the lifestyle prgram, and 6 mnths after cmpleting the lifestyle prgram Change in self-efficacy, readiness t change, and intent t quit using tbacc between baseline, upn cmpleting the lifestyle prgram, and 6 mnths after cmpleting the lifestyle prgram Change in number f minutes per week spent in mderate physical activity between baseline and 6 mnths after cmpleting the lifestyle prgram Change in the number f fast fd meals cnsumed per week between baseline and 6 mnths after cmpleting the lifestyle prgram Change in tbacc use between baseline and 6 mnths after cmpleting the lifestyle prgram Prgram participants wh have cmpleted at least ne lifestyle prgram sessin Participant behavir/ readiness t change assessments Fllw-up telephne surveys with prgram participants Secndary analysis f existing quantitative data Cnduct telephne surveys with prgram participants n an nging basis during Prgram Years 2 4 Cntact participants within 3 weeks f cmpleting the lifestyle prgram Cntact participants 6 mnths after cmpleting the lifestyle prgram 40
Preliminary Analysis Plans As yu define yur evaluatin methds, it is imprtant t think abut yur plans fr analyzing the data. Althugh yur analysis plans might be cnsidered preliminary during Step 3, it is imprtant that yu begin thinking abut the types f analyses that yu will cnduct nw, as this will help yu refine yur methds and set up yur data cllectin effrts later. Yu will finalize yur analysis plans fr the evaluatin and cnduct the analyses during Step 4. The types f analyses that yu prpse at this stage will prbably be general in nature. The type f analyses shuld be driven by the evaluatin questins and the methds that yu intend t use. Cnsider the number f participants frm whm yu will cllect data that is, the size f yur expected sample that will be included in yur evaluatin. It will be imprtant t cnsider whether yur expected sample size will be sufficient t use specific methds fr data analysis. Gauge the types f cnclusins yu may be able t draw frm yur data analyses results if yu have a small sample size. Cnsider whether yu may be able t make brader assumptins t a larger grup f WISEWOMAN participants (r ther larger grup) if yu have a small sample size. Yu may want t cnsult yur CDC Prject Officer r Evaluatin Specialist if yu have specific questins abut the types f data analysis methds that may be apprpriate fr yur expected sample size. It is beynd the scpe f this tlkit t prvide a detailed descriptin f different types f quantitative and qualitative data analysis methds and the intricacies assciated with each type. Hwever, in Exhibit 16 we describe the cmmn quantitative and qualitative analyses that WISEWOMAN prgrams may execute in evaluating yur prgram. In additin, Step 5 describes data analysis prcedures in greater detail. 41
Exhibit 16. Cmmn Quantitative and Qualitative Analyses Evaluatin Methds Quantitative Cmmn Analyses The specific quantitative prcedures that yu will use in evaluatin will vary accrding yur evaluatin questins and indicatrs, as well as the specifics f yur measures. In general, yur quantitative analysis prcedures may invlve the use f descriptive and/r inferential statistics: Descriptive Statistics. Descriptive statistics include frequency cunts, rates, percentages, measures f central tendency (means, medians, and mdes), and measures f dispersin (range, standard deviatin). Inferential Statistics. Inferential statistics will be used t test fr relatinships between variables r cmpare grups with the ntin f drawing cnclusins frm a sample t a ppulatin. These include crrelatinal prcedures (e.g., Spearman, Pearsn, biserial, chisquare, analysis f variance [ANOVA], t tests, and regressin). 15,16 Qualitative Thematic analysis is cmmnly used t analyze qualitative data in prgram evaluatin. This invlves systematically rganizing and reviewing yur data t identify and summarize majr themes (similarities and patterns) in the data. 15,17 Example Yu shuld nw have all f the infrmatin needed t cmplete the evaluatin methds matrix, as illustrated in Exhibit 17 n the fllwing pages. Please nte that Exhibit 17 includes tw additinal evaluatin questins t prvide additinal examples f hw t cmplete the matrix fr different kinds f prcess and utcme evaluatin questins. 42
Exhibit 17. Example Evaluatin Methds Matrix Cmplete Evaluatin Questins Prcess Evaluatin T what extent is the riskreductin cunseling cmpnent being implemented with fidelity amng prviders? What are best practices used by prviders t incrprate WISEWOMAN prtcls int practice wrkflws? Indicatrs Data Surces Methds Number and percentage f wmen screened wh received each f the cre cmpnents f the risk-reductin cunseling sessins bth in written frmat and verbally Screening results Interpretatin f the screening results Recmmendatins in accrdance with natinal clinical care guidelines Descriptin f unique prcedures and practices emplyed by the tp 10% f clinical sites cmpared t the bttm 10% f clinical sites based n the fllwing measures at the end f Prgram Year 2: Number and percentage f eligible wmen screened Number and percentage f wmen screened wh received risk-reductin cunseling Number and percentage f wmen diagnsed with hypertensin r cardimetablic cnditins wh were referred t and received fllw-up clinical care Prgram participants Clinical service prviders Clinical service prviders Quality cntrl site visit reprts Survey f participants Administrative frms Secndary analysis f existing qualitative data Interviews with clinical service prviders Timing, Duratin, and Frequency f Data Cllectin Cnduct survey f participants n an nging (rlling) basis during Prgram Years 2 4 Cntact participants by phne within 2 weeks f riskreductin cunseling Cnduct interviews with clinical service prviders during the first quarter f Prgram Year 3 Cnduct ne interview with each selected prvider Analyses (Preliminary) Descriptive statistics Thematic analysis Thematic analysis cmparing tp 10% sites t bttm 10% 43
Exhibit 17. Example Evaluatin Methds Matrix Cmplete (Cntinued) Evaluatin Questins Indicatrs Data Surces Methds Timing, Duratin, and Frequency f Data Cllectin Analyses (Preliminary) Number and percentage f wmen with ne r mre risk factrs referred t evidence-based lifestyle prgrams Number and percentage f wmen with ne r mre risk factrs referred t apprpriate cmmunity-based resurces Outcme Evaluatin T what extent des participatin in the evidencebased lifestyle prgram cntribute t participant utcmes related t physical activity, sdium intake, and tbacc cessatin? Change in self-efficacy, readiness t change, and intent t engage in 30 minutes f mderate physical activity daily between baseline (i.e., pre-lifestyle prgram participatin), upn cmpleting the lifestyle prgram, and 6 mnths after cmpleting the lifestyle prgram Change in self-efficacy, readiness t change, and intent t reduce cnsumptin f fast fds between baseline, upn cmpleting the lifestyle prgram, and 6 mnths after cmpleting the lifestyle prgram Change in self-efficacy, readiness t change, and intent t quit using tbacc between baseline, upn cmpleting the lifestyle prgram, and 6 mnths after cmpleting the lifestyle prgram Change in number f minutes per week spent in mderate physical activity between baseline and 6 mnths after cmpleting the lifestyle prgram Prgram participants wh have cmpleted at least ne lifestyle prgram sessin Participant behavir/ readiness t change assessments Secndary analysis f existing quantitative data Fllw-up telephne surveys with prgram participants Cnduct telephne surveys with prgram participants n an nging (rlling) basis during Prgram Years 2 4 Cntact participants within 3 weeks f cmpleting the lifestyle prgram Cntact participants 6 mnths after cmpleting the lifestyle prgram Inferential statistics 44
Exhibit 17. Example Evaluatin Methds Matrix Cmplete (Cntinued) Evaluatin Questins Indicatrs Data Surces Methds Timing, Duratin, and Frequency f Data Cllectin Analyses (Preliminary) Change in the number f fast fd meals cnsumed per week between baseline and 6 mnths after cmpleting the lifestyle prgram Change in tbacc use between baseline and 6 mnths after cmpleting the lifestyle prgram Have ur prgram activities cntributed t changes in prvider prtcls fr prviding services fr all wmen? Change in the prprtin f all wmen aged 40 64 (regardless f payment surce) wh receive bld pressure screening between 1 year prir t site initiating WISEWOMAN clinical services and the end f Prgram Year 3, cmpared t nn-wisewoman sites Change in prprtin f wmen aged 40 64 (regardless f payment surce) wh reprt being satisfied with the quality f care prvided between 1 year prir t site initiating WISEWOMAN clinical services and the end f Prgram Year 3, cmpared t nn-wisewoman sites Change in the prprtin f wmen aged 40 64 (regardless f payment surce) wh reprt btaining tailred recmmendatins t cmmunity-based resurces frm their prvider between 1 year prir t site initiating clinical services and the end f Prgram Year 3, cmpared t nn-wisewoman sites Data abstracted fr select clinical sites Prviders Secndary analysis f existing qualitative data Interviews with prviders Cnduct interviews with clinical service prviders during the first quarter f Prgram Year 4 Cnduct 1 interview with each selected prvider Thematic analysis cmparing data preprgram implementatin t end f Prgram Year 3 45
Disseminatin Plan A disseminatin plan describes wh yu will share yur evaluatin findings with, hw yu will share the findings, and when yu will share yur evaluatin findings. Yu may be wndering why we are discussing the disseminatin plan this early in the evaluatin prcess. It is imprtant t think abut yur intended audience fr the evaluatin findings and select disseminatin frmats, channels, and timeframes that will be mst effective in reaching them, as described belw. 4 Yu might cnsider reaching ut t members f yur intended audiences t btain their input n hw they wuld like t receive infrmatin frm the evaluatin. 3,4 Audience: Wh are yu intending t reach? It is imprtant that yu specify all f the audiences that yu intend t reach wh may benefit frm the evaluatin findings. During Step 1, yu created a list f evaluatin stakehlders. This list may be a useful starting pint fr develping yur disseminatin plan. Members f yur evaluatin stakehlder grup may als have suggestins fr audiences that may benefit frm the evaluatin findings. Use: Hw d yu anticipate yur audience using the infrmatin? Nt all findings resulting frm yur evaluatin will be relevant t all f yur prgram stakehlders. 4 Hw yu anticipate the stakehlders using the infrmatin shuld drive what infrmatin yu share and hw yu disseminate the infrmatin. It is mst efficient and effective t nly cllect the data that yu will actually use fr the evaluatin. Frmat/Methds: It is imprtant that yu cnsider the frmats and methds that yur audience prefers and views as credible surces in rder t maximize reach and prmte using the evaluatin findings. Cnsider wh will deliver the evaluatin findings. Remember that peple have different infrmatin-seeking and learning styles, s plan t use multiple disseminatin methds. 4 Yu may als need t use multiple methds t reach a given audience. Methds might include the fllwing: Cmprehensive summary reprt Shrt r executive summary Peer-reviewed jurnal article Press release PwerPint presentatin Presentatin talking pints Strybards r psters 46
Timeframe: When d yu plan t disseminate the findings? As yu think abut the timeframe fr disseminating yur evaluatin findings, cnsider the fllwing questins: When will the evaluatin findings be available? When are members f yur intended audience mstly likely t pay attentin t the evaluatin findings? When are members f yur intended audience mst likely t use the findings? Are there key events t link t such as legislative sessins, r budget planning? Respnsibility: Wh will disseminate the evaluatin findings t the intended audience? The messenger can be just as imprtant as the message with regard t disseminating evaluatin findings. The respnsibility fr disseminating the findings des nt have t rest entirely n the shulders f the prgram manager. Members f yur evaluatin stakehlder grup shuld be invlved in disseminating the findings. Yu might als cnsider the services f cmmunicatin and marketing prfessinals t help yu disseminate the findings. As yu determine wh will assume the respnsibility fr disseminating evaluatin findings, think abut wh the intended audience views as credible. Als, think abut the messengers preferred cmmunicatin styles and their availability. Fr example, if ne f yur stakehlders has severe stage fright, yu might want t delegate the respnsibility fr any ral presentatins t smene wh is mre cmfrtable speaking t large grups. 47
Exhibit 18 belw prvides an example disseminatin plan. Exhibit 18. Example Disseminatin Plan Audience Use Frmat Methd/ Medium Timeframe Respnsibility Julia R., CDC Prject Officer Infrm technical assistance prvided t the prgram Full summary reprt Attachment t prgress reprt April Betty S., prgram manager Partners Identify pprtunities t cllabrate and enhance participant experience Executive summary Email blast April (3 mnths prir t the start f the new prgram year) Prgram manager Prgram staff Identify areas fr prgram imprvement Executive summary with link t dwnlad full summary reprt Presentatin talking pints Email Briefing during a staff meeting April (3 mnths prir t the start f the new prgram year) Prgram manager Prviders Identify areas fr prgram imprvement Implement best practices PwerPint presentatin Webinar fr existing prviders New prvider rientatin July (at start f new prgram year) and rientatin fr new prviders Jane D., clinical prviders advisry grup (Webinar) Prgram manager (new prvider rientatin) Participants Encurage cntinued prgram participatin Prmte prgram t friends and family Strybards/ pster series Psted in waiting rms f clinical sites Starting July, thrughut prgram year Marketing and cmmunicatins liaisn 48
Refining Yur Evaluatin Budget In the Intrductin f this evaluatin tlkit, we recmmended that yu begin thinking abut yur evaluatin budget. Nw that yu have had the pprtunity t wrk with yur evaluatin stakehlders t fcus yur evaluatin and data needs, it is time t cmplete yur evaluatin budget t ensure that yu can implement yur evaluatin plan. 5 Reminder Yu may find that sme f yur planned evaluatin activities are beynd the scpe f yur budget. Shuld this ccur, we recmmend that yu engage yur evaluatin stakehlders again t discuss these cnstraints. It is imprtant that the grup members: See the Sectin 2 Tls and Templates fr an Excel wrksheet that yu can use t develp yur evaluatin budget. Cnsider revising the scpe f the evaluatin questins and indicatrs. Determine whether stakehlders wuld be able and willing t cntribute resurces (e.g., data, persnnel, financial resurces) t supprt evaluatin activities related t evaluatin questins in which they are mst interested. Cnsider reserving sme evaluatin questins t address in anther prgram year. Develping a Cmprehensive Evaluatin Plan Nw that yu have defined yur evaluatin methds and budget, it is time t cmpile all f the infrmatin that yu have wrked n thus far int a cmprehensive evaluatin plan. This can seem like a daunting task, but if yu have fllwed the steps f this tlkit, yu already have the infrmatin that yu need t put tgether a cmprehensive evaluatin plan. In fact, yur evaluatin methds matrix can serve as the primary cmpnent f yur cmprehensive evaluatin plan, as it systematically presents yur evaluatin questins, indicatrs, methds data surces, and preliminary analysis plan. Yu als may cnsider wrking with yur Evaluatin Specialist t btain feedback n early drafts f Key Cmpnents f a Cmprehensive Evaluatin Plan 5 Prgram descriptin Evaluatin gals and questins Evaluatin methds Analysis and interpretatin plan Disseminatin plan yur evaluatin plan. Yu will need t submit yur cmpleted evaluatin plan t yur CDC Prject Officer fr review and apprval. Exhibit 19 belw utlines each f the cmpnents f a cmprehensive evaluatin plan. It als includes references t the tls and templates in the evaluatin tlkit that yu can use t draft yur evaluatin plan. 49
Sectin Title page/cver page Prgram descriptin Evaluatin purpse Methds Analysis plan Disseminatin plan Evaluatin budget Exhibit 19. Evaluatin Plan Outline 5 Cntent/Questins t Answer Prgram name Date Title describing the basic fcus f the evaluatin Intended audience fr the evaluatin plan Prject narrative An issue statement r statement f need Prgram gals and bjectives Inputs r prgram resurces Prgram activities Expected utputs and utcmes Stage f develpment Descriptin f the envirnmental cntext (plitical, scial, physical, and ecnmic) Prgram lgic mdel Acknwledge the members f the evaluatin stakehlder grup wh are cntributing the evaluatin plan and implementatin Evaluatin purpse(s) Evaluatin questins Statement f yur verall apprach t the evaluatin methds Evaluatin methds matrix Overview f preliminary analysis plans (per evaluatin methds matrix) Disseminatin plan that utlines the fllwing: Intended audience Use Frmat Methd/medium Timeframe Respnsibility Ttal budget amunt Itemized budget Relevant Tlkit Tls and Templates N/A Example WISEWOMANspecific prgram lgic mdel template Priritizing Evaluatin Questins template Evaluatin methds matrix template Evaluatin methds matrix template Disseminatin plan template Evaluatin budget wrksheet 50
Internal Review Bards In additin t having yur cmprehensive evaluatin plan reviewed and apprved by yur CDC Prject Officer, it als is imprtant t cnsider whether yu will need t seek any additinal apprvals frm internal stakehlders befre starting yur evaluatin. It will be imprtant t determine whether an IRB review is needed and, if s, what steps are invlved in this prcess. Fr example, yu may need t submit yur data cllectin instruments and prtcl t an institutinal review bard (IRB) t ensure the prtectin f yur participants. Refer t Sectin 2 fr greater detail abut cnsideratins fr wrking with an IRB. Next: Evaluatin Implementatin Nw that yu have a cmprehensive evaluatin plan, it is time t prepare t implement yur evaluatin plan and cllect data. In the next sectin, Evaluatin Implementatin, yu will find guidance t help yu: Establish a data cllectin plan. Cllect data. Manage yur evaluatin data. 51
Tls and Templates Evaluatin Stakehlder Engagement Plan Template Tips fr using the template: List each stakehlder s name and affiliatin alng with hw and when the stakehlder shuld be engaged in the evaluatin. It is als recmmended that yu list the anticipated level f stakehlder participatin (e.g., high, medium, lw) t help yu estimate the frequency and timing f cmmunicatin. Yu can then dcument each stakehlder s areas f interest in the evaluatin, rle, preferred mde f cmmunicatin, and timing f the engagement. The stakehlder engagement plan shuld be reviewed regularly and revised as needed. The template has been partially cmpleted fr a WISEWOMAN Prgram fr illustrative purpses. Refer t Step 1 in the tlkit fr mre infrmatin. Evaluatin Stakehlder Level f Participatin Evaluatin Interest Rle in the Evaluatin Mde f Cmmunicatin Timing f Cmmunicatin Cmments Suzy Q., Breast and Cervical Cancer Prgram (BCCP) directr Medium Evaluatin questins and results related t utreach, recruitment cmpnents Review and prvide input n evaluatin plan and reprt; BCCP staff will participate in data cllectin effrts arund utreach and recruitment Email update, inpersn participatin at the planning and findings review meeting Mnthly It will be especially imprtant t invlve Suzy Q. in meetings when we discuss evaluatin methds and the findings specific t utreach and recruitment as this will likely have implicatins fr cllabratin with the BCCP prgram 52
Example WISEWOMAN-Specific Prgram Lgic Mdel Template Tips fr using the template: This template presents a generic funded prgram-level lgic mdel that has been included fr illustrative purpses. The lgic mdel template builds n the Natinal WISEWOMAN prgram lgic mdel by capturing the prgram-specific activities that ccur rutinely. Yu may wish t cnsult this lgic mdel t develp a lgic mdel f yur wn r yu may tailr it t reflect yur prgram s specific activities and expected utcmes. While the prcess used t develp yur prgram-specific lgic mdel may vary, it is imprtant that it includes the key cmpnents f a lgic mdel and that it is an accurate visual representatin f the prgram. Refer t Step 2 f the tlkit fr mre infrmatin n develping a lgic mdel. Belw is a list f acrnyms t be used as a reference fr the lgic mdel template n the fllwing page. 53
54
Evaluatin Methds Matrix Template Tips fr using the template: An evaluatin methds matrix can be used t systematically rganize and present the evaluatin methds. It is recmmended that yu wrk frm left t right when entering infrmatin in the matrix. Refer t Step 3 in the tlkit fr mre infrmatin abut hw t cmplete the matrix. Evaluatin Questins Indicatrs Data Surces Methds Prcess Evaluatin T what extent is the risk-reductin cunseling cmpnent being implemented with fidelity amng prviders? Number and percentage f wmen screened wh receive each f the cre cmpnents f the risk reductin cunseling sessins bth in written frmat and verbally Screening results Interpretatin f the screening results Recmmendatins in accrdance with natinal clinical care guidelines Prgram participants Clinical service prviders Survey with participants Administrative frms Timing, Duratin, and Frequency f Data Cllectin Cnduct survey f participants n an nging (rlling) basis during Prgram Years 2 4 Cntacting participants by phne within 2 weeks f riskreductin cunseling Analyses (Preliminary) Descriptive statistics Thematic analysis 55
Evaluatin Questins Indicatrs Data Surces Methds Outcme Evaluatin Des participatin in the evidence-based lifestyle prgram cntribute t participant utcmes related t physical activity, sdium intake, and tbacc cessatin? Change in self-efficacy, readiness t change, and intent t engage in 30 minutes f mderate physical activity daily between baseline (prelifestyle prgram participatin), upn cmpleting the lifestyle prgram, and 6 mnths after cmpleting the lifestyle prgram Prgram participants wh have cmpleted at least ne lifestyle prgram sessin Participant behavir/ readiness t change assessments Secndary analysis f existing quantitative data Fllw-up telephne surveys with prgram participants Timing, Duratin, and Frequency f Data Cllectin Cnduct telephne surveys with prgram participants n an nging (rlling) basis during Prgram Years 2 4 Cntact participants within 3 weeks f cmpleting the lifestyle prgram Cntact participants 6 mnths after cmpleting the lifestyle prgram Analyses (Preliminary) Inferential statistics 56
Indicatr Mapping Template Tips fr using template: It is recmmended that yu map yur prcess and utcme evaluatin questins t the apprpriate utputs n yur lgic mdel. Mapping the questins will help in drafting the prcess and utcme indicatrs and in selecting indicatrs. Depending n yur evaluatin questin(s), yu may want t cnsider the WISEWOMAN perfrmance indicatrs. Refer t Step 3 in the tlkit fr mre infrmatin abut hw t cmplete the indicatr mapping template. Prcess Evaluatin Questins What are best practices used by prviders t incrprate WISEWOMAN prtcls int practice wrkflws? Output (frm prgram lgic mdel) Number and percentage f eligible wmen screened Number and percentage f wmen screened wh receive risk reductin cunseling Number and percentage f wmen diagnsed with hypertensin r cardi metablic cnditins wh are referred t and receive fllw-up clinical care Number and percentage f wmen with ne r mre risk factrs referred t evidence-based lifestyle prgrams Number and percentage f wmen with ne r mre risk factrs referred t apprpriate cmmunitybased resurces Indicatr Prcedures and practices used by the tp perfrming clinical sites, shwn by their perfrmance relative t ther clinical sites n the fllwing: Number and percentage f eligible wmen screened Number and percentage f wmen screened wh receive risk reductin cunseling Number and percentage f wmen diagnsed with hypertensin r cardi metablic cnditins wh are referred t and receive fllw-up clinical care Number and percentage f wmen with ne r mre risk factrs referred t evidencebased lifestyle prgrams Number and percentage f wmen with ne r mre risk factrs referred t apprpriate cmmunity-based resurces 57
Outcme Evaluatin Questins Des participatin in the evidence-based lifestyle prgram cntribute t participant utcmes related t physical activity, sdium intake, and tbacc cessatin? Output (frm prgram lgic mdel) Increased participant self-efficacy, readiness t change, and behaviral intent cncerning healthy eating, physical activity, and tbacc cessatin (shrt-term utcme) Increased desirable health behavirs (intermediate utcme) Indicatr Change in self-efficacy, readiness t change, and intent t engage in 30 minutes f mderate physical activity daily between baseline (i.e., pre-lifestyle prgram participatin), upn cmpleting the lifestyle prgram, and 6 mnths after cmpleting the lifestyle prgram Change in self-efficacy, readiness t change, and intent t reduce cnsumptin f fast fds between baseline, upn cmpleting the lifestyle prgram, and 6 mnths after cmpleting the lifestyle prgram Change in self-efficacy, readiness t change, and intent t quit using tbacc between baseline, upn cmpleting the lifestyle prgram, and 6 mnths after cmpleting the lifestyle prgram Change in number f minutes per week spent in mderate physical activity between baseline and 6 mnths after cmpleting the lifestyle prgram Change in the number f fast fd meals cnsumed per week between baseline and 6 mnths after cmpleting the lifestyle prgram Change in tbacc use between baseline and 6 mnths after cmpleting the lifestyle prgram 58
Priritizing Evaluatin Questins Template Tips fr using the template: The template belw illustrates hw yu might rganize the discussin that a WISEWOMAN prgram may have in fcusing yur evaluatin. As yu select final evaluatin questins, yu will need t balance which questins might be mst useful t yur prgram while als meeting sme f yur stakehlders infrmatin needs. The template is partially cmpleted fr illustrative purpses nly. Refer t Step 3 f the tlkit fr mre infrmatin. Tpic Purpse/Use (Relevant Prgram Stage) Type f Questin Feasibility Timing Evaluatin Questin Pririty Evidencebased lifestyle prgram Accuntability T determine whether the ABC lifestyle prgram is effective and whether we will cntinue the prgram in the next FOA (Maintenance) Outcme Impact f the prgram n participants Still in planning stages Will wait until cmpnent is fully peratinal in Year 2 Will need t determine type f data cllected by ABC prgram and whether ABC prgram wuld help us cllect additinal infrmatin if we create an instrument t supplement what they cllect already Years 2 4 Des participatin in the evidencebased lifestyle prgram cntribute t participant utcmes related t physical activity, sdium intake, and tbacc cessatin? High 59