The Poor External Validity of Health Promotion Research & What Can Be Done to Facilitate Translation of Research To Practice

Transcription

1 The Poor External Validity of Health Promotion Research & What Can Be Done to Facilitate Translation of Research To Practice Russell E. Glasgow, Ph.D. Kaiser Permanente Colorado Sheana S. Bull, Ph.D. University of Colorado Health Sciences Center Lisa M. Klesges, Ph.D. University of Tennessee David Dzewaltowski, Ph.D. and Paul A. Estabrooks, Ph.D. Kansas State University

2 Sheana S. Bull, Ph.D. Lisa M. Klesges, Ph.D. Paul A. Estabrooks, Ph.D. Russell E. Glasgow, Ph.D. Audience Internal And External Validity of Behavioral Interventions: A Review of Recent Literature Planning for Translation and Dissemination: RE-AIMing Interventions to Improve Applicability and Enhance Evaluations A Web-Based Resource Center for Program Evaluators and Researchers Concerned with Translating Health Behavior Research to Practice: Conclusions: Cross Cutting Issues and Comments General Discussion

3 MORAL OF THE STORY? IT S S THE DENOMINATOR!

4 ONCE UPON A TIME Target Delivery Sites (GP offices) Effective Intervention 100% of target population could be influenced Entire Target Population

5 Step 1. 50% of offices agree to participate - ADOPTION Target Delivery Sites (GP offices) Effective Intervention 50% of target population could be impacted Entire Target Population

6 Step 2. 50% of staff ADOPTION (Part 2) Target Delivery Sites (GP offices) Effective Intervention 25% of target population could be influenced Entire Target Population

7 Step 3. 50% of eligible participants - REACH Target Delivery Sites (GP offices) Intervention Delivered 12.5% of target population begin participation Entire Target Population

8 Step 4. 50% of protocol dropped - IMPLEMENTATION Target Delivery Sites (GP offices) Intervention not delivered well 6% of target population maintains participation Entire Target Population

9 Step 5. 50% of participants achieve goal after 8 weeks - EFFECTIVENESS Target Delivery Sites (GP offices) Intervention not delivered well 3% of target population influenced positively Entire Target Population

10 Step 6. After 2 years 50% of participants maintain change - MAINTENANCE ntervention not sustained well Target Delivery Sites (GP offices) Entire Target Population 1.5% of target population maintains change in activity

11 RE-AIM FRAMEWORK Adoption Efficacy Effectiveness Reach Building Health Behavior Interventions with a Large Public Health Impact Implementation Maintenance

12 Reach Maintenance Maintenance How do I reach the targeted population How do I intervention? incorporate the How do I know intervention so it my intervention is delivered over is effective? the long-term? How do I ensure the intervention is delivered properly? Implementation How do I develop organizational support to deliver my intervention? Effectiveness Effectiveness Adoption Adoption

13 What is needed to improve translation of research into Health Promotion practice? A review of the literature Sheana Bull, Ph.D., M.P.H. University of Colorado Health Sciences Center

14 Objectives! To discuss and summarize four reviews of recent controlled studies in: Worksites Health Care School Community settings! To present conclusions from reviews regarding current emphasis on internal vs. external validity

15 Objectives! To discuss the opportunities researchers, practitioners, fund agencies, and journal editors have to improve the translation of efficacious and effective interventions into practice

16 Background! Since the 1960 s behavioral medicine and epidemiology have developed ability to identify, explain and address risk factors for disease! Behavioral medicine and epidemiology have also made breakthroughs in development and testing interventions to reduce risk factors

17 Background! While we know a lot about what works to reduce risk factors in certain populations, under certain conditions, we know relatively little about how to disseminate what works to a wide group of practitioners! We also know relatively little about whether what works works for multiple groups with diverse demographic characteristics

18 Background! The standard intervention trial seeks to establish efficacy this is critical to determining what works! Efficacy is insufficient for judging potential effectiveness, reach and sustainability for health promotion Narrowly drawn samples don t reflect larger population; Tightly controlled interventions don t encourage flexibility to apply in multiple settings with multiple groups

19 Background! Over reliance on efficacy studies limits our ability to estimate implications of an intervention for public health practice! The RE-AIM Framework is a useful evaluation tool to assess the extent that components critical for dissemination/translation of interventions to public health practice are present in studies

20 Background! Reach! Efficacy/Effectiveness! Adoption! Implementation! Maintenance RE-AIM

21 Methods! Literature review, using RE-AIM framework to evaluate the potential public health impact of interventions in worksites, health care settings, schools and community settings! Summarized results of controlled intervention studies reported in one of 12 leading public health journals conducted between

22 Methods! Only reviewed health behavior change studies evaluating an intervention with a control condition*! Reviewed studies as part of the Behavior Change Consortium (BCC) and focused on physical activity, smoking and nutrition! 119 studies reviewed addressing one or more of these areas *purposely included controlled studies as well as RCT

23 Results! Dimensions of RE-AIM that are discussed consistently across studies and across behaviors include (n=119): Participation rate (76%, 59-88% range)* Behavioral outcome measure (92%, % range) Attrition rate (79%, % range) *Definition of participation rate varies sometimes includes only those who call in response to an ad, for example

24 PERCENT OF STUDIES REPORTING ON RE-AIM DIMENSIONS Individual Level RE-AIM Dimension/Measure Worksites (n=24) Schools (n=32) Health Care (n=36) Community (n=27) Average REACH Participation rate 88% 59% 69% 88% 76% Representativeness 9% 7% 28% 11% 14% EFFECTIVENESS Behavioral outcome measure 67% 100% 100% 100% 92% QOL or negative outcomes 0% NR 17% 3% 7% MAINTENANCE > 6-month follow-up 4% 26% 86% 30% 36% Attrition at follow-up 54% 74% 87% 100% 79%

25 Results! Few studies of the 119 discuss setting reported criteria related to external validity, or the AIM of RE-AIM: A: Participation rate of settings (16%, 11-25% range) I: Consistency of Intervention delivery (46%, 12-77% range) M: Maintenance (2%, 0-6% range)

26 PERCENT OF STUDIES REPORTING ON RE-AIM DIMENSIONS Setting Level RE-AIM Dimension/Measure Worksites (n=24) Schools (n=32) Health Care (n=36) Community (n=27) Average ADOPTION Participation rate site 25% 15% 11% 11% 16% level Representativeness of settings 0% 0% 0% 7% 2% IMPLEMENTATION Treatment Delivery * 12% 37% 77% 59% 46% Time or cost 0% NR 31% 63% 31% MAINTENANCE Setting continuation 4% 0% 6% 0% 2% * Often from efficacy studies where treatment delivered by research staff

27 Discussion! Recommendations for researchers Involve target audience in planning to enhance reach, adoption, implementation and maintenance Focus on representation at individual, setting and clinician levels Enhance recruitment procedures Replicate interventions to assess reach Study consistency of implementation in replication and impact on behavioral outcome

28 Discussion! Recommendations for journal editors/reviewers Place equal weight on external as well as internal validity Develop a standard for reporting reach,effectiveness, adoption, implementation and maintenance Develop a standard for reporting more detail on representation at individual and setting levels at enrollment and follow-up and compare participants to non participants Develop a standard for reporting on intervention delivery processes

29 Discussion! Recommendations for Funding Organizations solicit proposals that: Investigate interventions in multiple settings Require standard reporting of exclusions, participation rates (individual and setting levels) and representation of population Require a maintenance/sustainability phase and encourage processes for institutionalization of efficacious interventions Add potential for translation as review criterion

30 Planning for Translation and Dissemination: RE-AIMing Interventions to Improve Applicability and Enhance Evaluations Lisa M. Klesges, Ph.D. University of Tennessee Health Sciences Center Paul A. Estabrooks Kansas State University Community Health Institute 24th Annual Scientific Sessions of the Society of Behavioral Medicine, 2003

31 Objectives! Describe design elements related to translation and dissemination that can be incorporated in planning phase of studies! Early in the planning phase: Encourage researchers and program planners to consider key questions regarding public health impact Anticipate barriers and methods to improve the generalizability, adoption, impact, and sustainability of behavior change interventions

32 Background! Behavior change interventions that are applicable and generalizable to diverse settings and populations are needed to achieve population health goals! Participatory planning methods can facilitate the potential translation of interventions to practice by engaging practitioners, service providers, and community members in the various stages of research planning and execution

33 Key Planning Questions Reach Maintenance Maintenance What is the potential cost and sustainability of the intervention in practice settings? Implementation Who is the targeted population? Who is intended to benefit from this intervention? Is the intervention feasible to implement? Can it be consistently delivered in practice? Will the intervention be effective in practice? Can many settings easily adopt this intervention? Effectiveness Effectiveness Adoption Adoption

34 Reaching the targeted population?! Consider how to include hard to reach groups! Formative evaluation of recruitment strategies and intervention focus! Assess and reduce barriers to participation! Plan to assess the representativeness of your sample to evaluate the generalizability of your intervention

35 Have you reached your intended audience? Target Population (y) Those Exposed to Recruitment [a] Those Responding to Recruitment (May be differential by Race/Ethnicity, Sex, Age, Income, Education) [b] Those Eligible [c] Those Who Agree to Participate [d]

36 Intervention effectiveness in practice?! Design robust interventions that minimize nonadherence and attrition and maximize effects! Incorporate social and environmental supports to enhance effectiveness! Define broad outcomes e.g., adverse consequences and quality-of-life to estimate risk-to-benefit effectiveness! Assess costs of intervention (e.g., materials, equipment, personnel, time, and space) to aid policy decisions and costs analysis

37 Wide adoption of intervention?! Plan to target settings having wide Reach into hard to find" populations! Conduct formative evaluation to design interventions that are feasible and attractive to targeted adopters! Design customization into the intervention! Assess the representativeness of the targeted settings to judge context and generalizability

38 Have you reached your intended settings? Targeted Settings for Adoption (z) Those Eligible to Participate [e] Contacted to Participate [f] Those Who Agree to Participate [g]

39 Feasible & Reliable Tx Implementation?! Test intervention ideas and materials to judge acceptability and value to the target population! Feasibility studies of the capabilities of delivery agents and preferences of adopting agencies! Design intervention protocols and training materials for delivery agents with diverse backgrounds & training! Plan to assess process of intervention delivery

40 Sustainability of intervention and effects?! Consider what will occur after study period ends for both individual maintenance and program sustainability! Build interventions with boosters, follow-up, community linkages to maintain behavior change! Conduct pre-intervention needs assessment w/ organizational and community settings to modify programs to enhance sustainability

41 Planning Tools & Resources! Website ( to provide detailed information on framework, calculations, and on-line data resources! Guided assessments to prioritize RE-AIM elements to use in planning! Future Menu options and customized approaches within planning framework

42 RE-AIM Planning Summary! Enhance capabilities of researchers and program adopters to plan, implement, and evaluate behavior change interventions with future translation and dissemination in mind! Produce participatory linkages between researchers, program adopters, delivery agents, and communities in planning behavior change interventions with high public health impact

43 RE-AIM Planning Goal Adoption Efficacy Effectiveness Reach Planning Health Behavior Interventions with a Large Public Health Impact in Mind Implementation Maintenance

44 A Web-Based Resource Center David A. Dzewaltowski, Ph.D. Paul A. Estabrooks. Ph.D. Kansas State University Community Health Institute

45

46 Visits - Number of visits to your Web site. Each visit by each visitor is counted, even if the visitor came to your Web site many times. Visitors Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb

47 Unique Visitors - Number of individuals who visited your site during the report period. If someone visits more than once, they are counted only the first time they visit. 800 Unique Visitors Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb

48 Average Visitor Minutes Month Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Visitor Minutes 0 00:27:53 00:21:17 00:07:51 00:02:36 00:22:12 00:09:43 00:03:27 00:08:20 00:06:27 00:04:54 00:27:51

49 Top Regions in North America 5, % 2. Western Europe % 3. Asia % 4. Australia (AU) % 5. Middle East % 6. South America % 7. Eastern Europe % 8. Northern Europe % 9. Region Unspecified % 10. Pacific Islands % 11. Region Not Known % 12. North Africa %

50 Top States or Provinces in California 1, % 2. Kansas 1, % 3. Massachusetts % 4. New York % 5. Ontario % 6. Texas % 7. Colorado % 8. Maryland % 9. New Jersey % 10. Tennessee % 11. Georgia % 12. Virginia % 13. Oregon % 14. Pennsylvania % 15. Washington %

51 Future Developments! Online Communities! Interactive Calculators & Comparisons Reach Effectiveness/Efficacy Adoption Implementation Maintenance! Data sharing

52 Conclusions and Comments Russell E. Glasgow, Ph.D. Kaiser Permanente Colorado

53 Something may seem okay in theory but once put into practice, it just doesn t work.

54 ! RE-AIM has been useful in summarizing the literature and identifying research needs Glasgow RE (1999) Outcomes of and for diabetes education research. Diab Educ 25(6) Suppl: Noell J, et al (1999) Interactive technology applications for behavioral counseling: Issues and opportunities for health care. Am J Prev Med 17: Glasgow RE & Eakin EG (2000) Medical office-based interventions. In: FJ Snoek & TC Skinner (Eds) Psychology in diabetes care. New York. John Wiley & Sons, pp Eakin EG, et al (2000) Review of primary are-based physical activity intervention studies: Effefctiveness and implications for practice and future research. J Fam Pract 49(2): France EK, et al (2001) Smoking cessation interventions among hospitalized patients: What have we learned? Prev Med 32(4): Glasgow RE, et al (2002) Making a difference with interactive technology: Considerations in using and evaluating computerized aids for diabetes self-management education. Diabetes Spectrum 14(2): Eakin EG, et al (2002) Reaching those most in need: A review of diabetes self-management interventions in disadvantaged populations. Diabetes Metab Res Rev 18(1): Glasgow RE, et al (2002) Behavior change intervention research in health care settings: A review of recent reports, with emphasis on external validity. Am J Prev Med 23: Estabrooks PA, et al (2003) Reporting of validity from school health promotion studies published in 12 leading journals, J School Health 73: Bull SS, et al (In press) Worksite health promotion research: To what extent can we generalize the results and what is needed to translate research to practice? Health Education and Behavior.

55 !RE-AIM has been used to help design and evaluate studies Glasgow RE, et al (2000) A brief smoking cessation intervention for women in low-income Planned Parenthood Clinics. Am J Pub Health 90(5): Glasgow RE, et al (2000) Brief, computer-assisted diabetes self-management counseling: Effects on behavioral, physiological and quality of life outcomes. Medical Care 38: Lando HA, et al (2002) Promoting smoking abstinence in pregnant and postpartum patients: A comparison of 2 approaches. Am J Managed Care 7: Glasgow RE, et al (2002) Implementation, generalization, and long-term results of the Choosing Well diabetes self-management intervention. Patient Educ Counseling 48: Toobert DJ, et al (2002) If you build it, will they come? Reach and adoption associated with a comprehensive lifestyle management program for women with type 2 diabetes. Patient Educ Counseling 48: Toobert DJ, et al (2002) Enhancing support for health behavior change among women at risk for heart disease: The Mediterranean Lifestyle Trial. Health Educ Res 17: Estabrooks PA, et al (2003) Evaluating the impact of behavioral interventions that target physical activity: Issues of generalizability and public health. Psychol Sport Exerc 4: Estabrooks, PA, et al (in press). The relationship between delivery agents physical activity level and the likelihood of implementing a physical activity program. Am J of Health Promo. Glasgow RE, et al (In press) The D-Net diabetes self-management program: Long-term implementation, outcomes, and generalization results. Prev Med

56 ! Will RE-AIM be useful for program planning and for quality improvement efforts?! Can it be communicated so is understandable to non-researchers?! Can it provide effective solutions to concerns identified?

57 ! The RE-AIM website appears used, but: " Will it be useful for planners and decision makers? " Can it help create a Network of Innovators? " Are there other or better avenues?

58 ! Does the RE-AIM model need modification or refinement?