EDM Forum: Analytical Methods for Learning Health Systems

Transcription

1 EDM Forum: Analytical Methods for Learning Health Systems Erin Holve (moderator), Ph.D., M.P.H., M.P.P., AcademyHealth; Michael A. Stoto, Ph.D., Georgetown University School of Nursing and Health Studies; Lucy Savitz, Ph.D., M.B.A., Intermountain Healthcare; Neil S. Fleming, Ph.D., Baylor Scott and White Health and Hankamer School of Business, Baylor University; Elisa Priest, Dr.PH., Baylor Scott and White Health Center for Clinical Effectiveness; Brian S. Mittman, Ph.D., VA Center for Implementation Practice and Research Support and Kaiser Permanente Southern California Department of Research and Evaluation September 9, 2014

2 Welcome Erin Holve, Ph.D., M.P.H., M.P.P. Senior Director of Research & Education, AcademyHealth Principal Investigator of the EDM Forum egems Editor-in-Chief Follow the conversation on 2

3 AcademyHealth: Improving Health & Health Care AcademyHealth is a leading national organization serving the fields of health services and policy research and the professionals who produce and use this important work. Together with our members, we offer programs and services that support the development and use of rigorous, relevant and timely evidence to: 1. Increase the quality, accessibility and value of health care, 2. Reduce disparities, and 3. Improve health. A trusted broker of information, AcademyHealth brings stakeholders together to address the current and future needs of an evolving health system, inform health policy, and translate evidence into action. 3

4 Evidence, Data, and Methods to Build Learning Health Systems (LHS) of the Future Advancing learning and collaboration in big data and big science in healthcare Working with cutting-edge research and quality improvement networks that reach more than 1 in 10 American across priority populations and conditions Sign up for updates: [email protected]

5 Check Out egems! egems (Generating Evidence and Methods to improve patient outcomes) Free, peer-reviewed, open access, e- publication Focus: generalizable lessons learned within analytic methods, clinical informatics, governance, and the learning health system 53 publications with over 24,500 downloads! Want updates? with subject Add to EDM Update We are actively accepting submissions and recruiting qualified peerreviewers. Learn more at repository.academyhealth.org/egems

6 The audio and slide presentation will be delivered directly to your computer Speakers or headphones are required to hear the audio portion of the webinar. If you do not hear any audio now, check your computer s speaker settings and volume. If you need an alternate method of accessing audio, please submit a question through the Q&A pod. 6

7 Technical Assistance Live technical assistance: Call Adobe Connect at (800) Refer to the Technical Assistance box in the bottom left corner for tips to resolve common technical difficulties. Please turn off your pop-up blocker in order to take a survey 7

8 Questions may be submitted at any time during the presentation To submit a question: 1. Click in the Q&A box on the left side of your screen 2. Type your question into the dialog box and click the Send button 8

9 Learning Objectives This webinar will discuss approaches to using observational data to improve performance in learning healthcare system. At the conclusion of the session, participants will be able to: Understand approaches to study design that can be applied to electronic health data in learning healthcare systems Understand how to frame research questions to effectively leverage electronic health data Learn about methods from a variety of fields that can be used to analyze electronic health data in learning healthcare systems. 9

10 Today s Faculty Lucy Savitz, Ph.D., M.B.A., Intermountain Healthcare Michael A. Stoto, Ph.D., Georgetown University Neil S. Fleming, Ph.D., Baylor Scott and White Health and Hankamer School of Business, Baylor University Elisa Priest, Dr.PH., Baylor Scott and White Health Center for Clinical Effectiveness Brian S. Mittman, Ph.D., VA Center for Implementation Practice and Research Support and Kaiser Permanente Southern California Department of Research and Evaluation 10

11 Overview of Learning Health Systems & Types of Studies September 9, 2014 EDM Forum Webinar Analytic Methods for Learning Health Systems Lucy A. Savitz, Ph.D., MBA Director of Research and Education Intermountain Institute for Health Care Delivery Research Research Professor, Clinical Epidemiology Director, Patient Centered Research Methods Core University of Utah, School of Medicine

12 EBM Defined Evidence-based medicine (EBM) is a set of principles and methods intended to ensure that to the greatest extent possible, clinical practice guidelines and medical decisions are consistent with evidence of efficacy, effectiveness, and benefit.

13 Evidence-Informed Decision Making What supports do health system organizations have in place to facilitate evidence-informed decision making? (Ellen et al., 2013) Decisions regarding the structure & process of care are often made without the input of timely and reliable evidence 57 interviews conducted in 25 organizations in Ontario & Quebec

14 What Was Learned Organizational efforts that can increase the use of evidence in decision making: Facilitate roles that actively promote research use within the organization; Establish ties to researchers & opinion leaders outside the organization; Support a technical infrastructure that provides access to research evidence (e.g., databases) Provision and participation in training programs to enhance staff s capacity building

15 Intermountain Healthcare Not-for-profit hospitals, physician group, and health plan $6 billion organization Founded in Hospitals 180+ Clinics Serves about 60% of Utah s population of about 2.9 million

16 Driving Health System Improvement: Role of the Learning Commons A Vital Resource to Support the Clinical Enterprise in Achieving Mission Critical Performance

17 Role of Research at Intermountain Priorities: 1. High operational utility 2. Of interest to clinical investigators 3. Of interest to close partners 4. Of interest to external investigators/entities If we knew what we were doing, we wouldn t call it research. A. Einstein

18 Need for a Healthcare System that Learns We require a sustainable system That gets the right care to the right person at the right time and then Captures the results for making improvements and Knowing what works.

19 21 st Century Health Care Information-rich, patientfocused enterprises Evidence is continually refined as a by-product of care delivery 21 st Century Health Care Information and evidence transform interactions from reactive to proactive (benefits and harms) Actionable information available to clinicians AND patients just in time

20 Case for Continuous Improvement Incorporating Innovation Disciplined QI Evaluation Critical to finding new designs and solutions to close the gaps and meet the goals of optimizing: Patient experience Health of the population Controlling cost/reducing waste.

21 Scientific Approach to QI IOM: Selker, H et al., 10/11. Clear, measurable process & outcomes goals Basis in evidence Iterative testing Appropriate analytic methods Documented results

22 QI: Role in Driving Evidence Base Quality Improvement Implementation Science Clinical Effectiveness Program Evaluation Outcomes Research Quasiexperimental Intervention Studies (Trials) Driving the science of change/innovation

23 Designs & Methods Knowing What Works, requires that we design a study and use those methods that allow us to answer key questions. Design Approaches: Intervention Quasi-experimental Observational Time Series Methods: Quantitative Qualitative Mixed

24 Example Study Advances in the Prevention and Control of Health Care-Associated Infections, JB Battles, JL Cleeman, KK Kahn, DA Weinberg (editors) Savitz, LA, SL Moore, W Biffl, C Price, H Gilmartin: A Participatory Research Approach to Reducing Surgical Site Infections (SSIs): Development of an Automated SSI Surveillance Tool, available at AHRQ publication No , June, 2014.

25 Delivery System Science A rigorous means to promote a learning commons to answer questions & test hypotheses around: Organizational Factors Care Delivery Attributes Customer Impact design, infrastructure, policies, payment processes, culture, standards, controls expectations, knowledge, empowerment, outcomes, satisfaction

26 Improvement through Understanding Pawson, R., N Tilley. Realistic Evaluation, Los Angeles, CA: Sage Publications, Experimentalists (aka trial-based designs) have pursued too single-mindedly the question of whether a program works at the expense of knowing why it works.

27 Realistic Research Design Approach Base strategy of producing a clear theory of program mechanisms, contexts, and outcomes, and then using them to design the appropriate empirical measures and comparisons

28 Generative Causation An Action Is causal only if Mechanism Context Outcome(s) it s outcome is triggered by a mechanism acting in context

29 Long History of Collaboration in Learning Commons

30 IOM Roundtable on Value & Science Driven Health Care Fundamental concepts of a learning healthcare organization: Need CEOs to get all stakeholders involved as partners Manage limited bandwidth in terms of research they can support and institutional energy Create a culture that values rapidly deployed research that informs outcomes and efficiency Develop shared research assets to conduct studies and create a community of researchers and stakeholders who reuse resources

31 Establishing Research Priorities Stevens & Ovretreit, Conduct environmental scan of literature and initiatives 2. Field a stakeholder survey 3. Expert panel for Delphi Consensus Exercise Priority identification Adopted as research agenda to guide strategy

32 Selected References Dopson, S and L Fitzgerald. Evidence-based Health Care in Context, Oxford University Press, Ellen, M et al. What Supports Do Health System Organizations Have in Place to Facilitate Evidence-informed Decision-making? A Qualitative Study, Implementation Science, 8(84), Institute of Medicine (IOM). Health System Leaders Working Toward High-Value Care through Integration of Care and Research, The National Academies, Workshop in Brief, June, Stevens, KR and J Ovretveit. Improvement Research Priorities: USA Survey and Expert Consensus, Nurs Res Prac, 2013.

33 Analytical methods for a learning healthcare system Michael A. Stoto for the EDM Forum Methods Collaborative

34 Introduction Learning healthcare systems use routinely collected electronic health data (EHD) to support continuous learning (QI) and advance knowledge (Okun, 2013) disease monitoring and tracking target services for improved health outcomes and cost savings inform patient and clinician decision-making avoid harm and costs of repeat testing and unsuccessful treatments accelerate and improve the use of research in routine medical care to answer medical questions more effectively and efficiently Indeed, in an era of big data, some feel that with enough data, RCTs are no longer necessary ( How big data will save your life, Mearian, Computerworld, 2013)

35 Introduction Despite big data enthusiasm, this is not easy Whether used to manage care, conduct QI or comparative effectiveness research (CER) Neither the breadth, timeliness and volume of information, nor machine learning algorithms or sophisticated visualizations can overcome problems of inaccurate/incomplete data confidently infer cause and effect relationships from observational data Without randomization, research results can be biased by selection effects and confounding with factors not recorded in existing electronic health records (EHR) Some feel that without RCTs we know nothing (Gina Kolata, New York Times, Feb. 2, 2014)

36 Introduction Depending on the research question, careful study design and appropriate analytical methods can improve the utility of EHD in a learning healthcare system This presentation will discuss how observational data can be used to improve performance in learning healthcare systems how to frame the research question the basic elements of study design and analytical methods that can help to ensure rigorous results Most are aware of some, but not most, of these methods and approaches

37 Presentation outline Matching the method to the question What is the question? Observational studies as a complement to RCTs Assessing cause and effect Study design and analysis Analytical methods for individual-level data Delivery system science Role of qualitative methods

38 What s the question? 1. What is the current (and likely future) situation? Prevalence of a disease or condition What s the prevalence of diagnosed and undiagnosed diabetes among adults in practice X? in county Y Occurrence, timing, and patterns of adverse events What are the primary causes of healthcare acquired infections in hospital Z? How does the rate of central line-associated blood stream infections (CLASBI) compare to peer institutions? Cost and utilization of healthcare Who are the high users of healthcare ( frequent flyers ) in the system? Causal inference not relevant 2. Does intervention improve outcomes of interest? 3. Translation and spread (delivery system science)

39 What s the question? 1. What is the current (and likely future) situation? 2. Does intervention improve outcomes of interest? what was the impact of the 2006 Massachusetts healthcare reform on health care utilization, morbidity, and mortality? Compared to what? What is the risk of short-term mortality associated with initiation of conventional vs. atypical antipsychotic medication (APM)? Effectiveness vs. efficacy How does the risk of mortality with initiation of APM vary between primary care providers and other providers in actual practice? Safety What is the risk of intussusception after vaccination with secondgeneration rotavirus vaccines? Consider external as well as internal validity 3. Translation and spread (delivery system science)

40 What s the question? 1. What is the current (and likely future) situation? 2. Does intervention improve outcomes of interest? 3. Translation and spread (delivery system science) Example: Patient-centered medical home (PCMH) approach to managing patients with diabetes, using Team-based care Health IT and registry functionality Care coordination and management Quality-adjusted financial incentives How and why does the intervention work? What works for whom and in what contexts? How can a model be amended to work in new settings? Causal inference not the main issue

41 Randomized clinical trials (RCTs) RCTs are the gold standard in health research for assessing cause and effect Especially important when the focus is on question #2 (Does the intervention work?) For some questions, other study designs more appropriate for ethical and practical reasons Especially evaluation, translation and spread of of delivery system changes (question #3)

42 Randomized clinical trials (RCTs) RCTs are limited even for Does it work By restrictions on subjects to obtain homogeneity RCTs often don t represent real-world patient populations By limited sample size, for cost reasons Affecting their ability to detect adverse effects In duration, limiting ability to see long-term consequences When the focus is on question #2 (Does the intervention work?), RCTs are strong for internal validity weak for external validity

43 Observational studies Observational studies using existing EHD offer the opportunity to investigate interventions and outcomes sometimes at lower costs Strengths Large, diverse populations under observation Can help estimate heterogeneous treatment effects Relatively complete information in EMR on Treatments administered Health outcomes experienced Reporting bias minimized because data are collected for operational rather than research purposes Data already in electronic form quicker results and possibly lower costs

44 Observational studies Weaknesses due in part to lack of randomization possibility (likelihood?) of selection bias & confounding incomplete information on confounders needed to adjust for treatment and control group differences data quality issues When the focus is on question #2 (Does the intervention work?), observational studies are strong for external validity weak for internal validity How do we balance the rigor and internal validity of RCTs relevance and external validity of observational studies Complement to, not substitute for, RCTs

45 What s the question? 1. What is the current (and likely future) situation? 2. Does intervention improve outcomes of interest? Compared to what? Effectiveness vs. efficacy Safety 3. Translation and spread (delivery system science) How and why does the intervention work? What works for whom and in what contexts? How can a model be amended to work in new settings?

46 Assessing cause and effect Evidence-based medicine requires evidence of a cause and effect relationship Otherwise not sure that interventions will have the desired effect in other circumstances RCT s address the three criteria for a contributory cause (Riegelman) Cause precedes effect in time outcomes observed after randomization Individual-level association between cause and effect randomization ensures that there are no systematic differences between treatment and control groups, even in unobserved variables Altering the cause alters the effect deliberate allocation to treatment and control group deliberate allocation is also a feature of natural or quasi experiments

47 Estimating the counterfactual RCTs essential in studying the effect of medications and health services provided to individuals because patient outcomes vary and are unpredictable Only some respond to even most effective medications, and some people get better even without medication Selection effects of income, education, insurance, etc. RCTs control group tells us the counterfactual What would have been the outcome in those who received the medication if they hadn t receive it Randomization also balances selection factors Not needed if the counterfactual is clear jumping from a plane without a parachute or for some systems level changes E.g. Michigan hospital checklist study median rate of catheter-related decreased from 2.7 infections infections per 1,000 catheter days at baseline to zero, sustained for 15 months of follow-up (Pronovost)

48 Presentation outline Matching the method to the question What is the question? Observational studies as a complement to RCTs Assessing cause and effect Study design and analysis Analytical methods for individual-level data Delivery system science Role of qualitative methods

49 Study design and analysis Potential for bias in non-experimental studies comparison groups do not have the same underlying risk for the outcome potential for confounding by unmeasured covariates Minimize potential for bias by selecting data sources, patient populations, and comparators study designs epidemiological study designs quasi- and natural experiments multifactorial and other complex designs Interrupted time series model for analysis Use of logic model

50 Using a logic model to clarify a causal chain Program goals Antecedent & mediating variables Intervening events Target population Program elements Program outputs Intermediate outcomes Long-term outcomes Available resources Causal assumptions Rival activities

51 Quasi- and natural experiments Quasi-experiment Intervention that potentially effects health outcomes Implementation within the control of researchers, but allocation to treatment and comparison group not random Natural experiment Intervention (e.g. change in law or policy) that potentially effects health outcomes Implementation beyond control of researchers, so can t randomly assign treatment to some & not others Observational studies focus on ensuring observed association is not confounding or observation bias Quasi- and natural experiments focus on situations where the change in assignment breaks usual links that could lead to selection bias, reverse causation, etc.

52 Quasi- and natural experiments Advantages Capitalize on deliberate policy and other changes unrelated to factors influencing outcomes Cause (treatment) precedes effect Changing cause changes outcome Use tracking data gathered for other purposes, so no observation bias With a strong design, can estimate counterfactual Disadvantages Cannot attribute the effect to the policy change (as opposed to other things that might have changed) Does not indicate what about the policy change made a difference

53 Mean # of Rxs per patient ITS logic and parameter estimation 8 6 Anticipatory Change due to cap demand point Absolute: excluded (-2.69, from -2.04) estimate Relative: -46% of policy (-50%, -42%) effects β 0 4 β 2 2 Assumptions: 1. Linearity 2. Normality 3. Autocorrelation Structure 0 Dec-79 Jun-80 Jan-81 Aug-81 Feb-82 Sep-82 Mar-83 Oct-83 Apr-84 Study Month β 4 β 3 β 5 Y t β 0 β β 4 1 * time * policy2 t t β β 2 5 * policy1 * time t β 3 * time after policy2 t after policy1 e t 61 t

54 Interrupted Time Series (ITS) summary Advantages Intuitive visual display Direct estimate of effects Controls common threats to validity confounding, selection, statistical regression, instrumentation, history or maturation Limitations Requires reasonably stable series Boundary problems Need relatively long segments Sensitive to points near end of segment No patient-level adjustment (can adjust rates) 62

55 Simplified logic model: Massachusetts healthcare reform Insurance coverage Primary care utilization Healthcare amenable mortality Affordability Hospital & ER utilization Self-reported health status

56 Unadjusted mortality rates for adults aged 20 to 64 years in Massachusetts versus control group ( ). The shaded band designates the beginning of the Massachusetts state health care reform that was implemented starting in July 2006.

57 Summary: study design and analysis Use a logic model to position data in context Quasi- and natural experiments Find setting where exogenous changes are unlikely Appropriate controls (Multiple) pre and post measurements Interrupted time series (ITS) analysis Individual-level statistical analysis Controlling for relevant confounders, etc. Instrumental variables, propensity scores, etc. Use pre-existing data to control recall bias Medical records rather than patient recall about use of a drug thought to cause adverse effects Simulation models to analyze complex patterns Simulate time-shift in incidence following introduction of cancer screening programs

58 Presentation outline Matching the method to the question What is the question? Observational studies as a complement to RCTs Assessing cause and effect Study design and analysis Analytical methods for individual-level data Delivery system science Role of qualitative methods

59 Analysis of individual-level data Potential for bias in non-experimental studies groups do not have the same risk for the outcome potential for confounding by unmeasured covariates Adjust for bias and confounding through Regression approaches Instrumental variables Propensity score methods Structural equation models Analyzing observational data like RCTs

60 Causal inference framework For a given individual, the effect of a treatment is the comparison of the outcomes that would be observed if the person receives the treatment the person receives the comparison condition instead Fundamental problem: no single individual can receive both the treatment and the comparison condition In In our case study of the Medicare Part D program, the treatment group members (dual eligibles) are likely sicker, older, and may additionally have a support network helping them negotiate the Medicare and Medicaid bureaucracy Randomization solves this by creating a control group that, on average, is no different from the treatment group no selection effects or confounding

61 Causal inference assumptions Regression approaches assume that the relationship between treatment, outcome, and other variables is properly specified all variables available for analysis and measured without error error term is independent and identically distributed. Instrumental variables find instruments that affect treatment but unrelated to outcomes estimate how much of the variation in the treatment variable that is induced by the instrument - and only that induced variation - affects the outcome measure Propensity score methods model factors influencing treatment decisions use to find similar patients with whom to compare outcomes

62 Summary: Analysis of individual-level data Potential for bias in non-experimental studies comparison groups do not have the same underlying risk for the outcome potential for confounding by unmeasured covariates Approaches minimize potential for bias by selecting data sources, patient populations, and comparators choice of study designs statistically adjust for differences in observed characteristics All of these approaches involve assumptions about data, causal relationships, and biases that are generally not directly observable but in some cases may be tested Solutions conduct sensitivity analyses for what cannot be observed use multiple approaches with different assumptions

63 Presentation outline Matching the method to the question What is the question? Observational studies as a complement to RCTs Assessing cause and effect Study design and analysis Analytical methods for individual-level data Delivery system science Role of qualitative methods

64 What s the question? 1. What is the current (and likely future) situation? 2. Does intervention improve outcomes of interest? Compared to what? Effectiveness vs. efficacy Safety 3. Translation and spread (delivery system science) How and why does the intervention work? What works for whom and in what contexts? How can a model be amended to work in new settings?

65 Evaluation of health care improvement initiatives Question is not Does it work? rather How and in what contexts does the new model work or can be amended to work? Fixed protocol RCTs are ill-suited for healthcare QI initiatives, which are complex and context sensitive iterative in nature, and vary depending on initial innovative stage more developed testing stage

66 Evaluation of health care improvement initiatives Requires theory-driven formative evaluation that considers the full path of the intervention from activities to engage participants and change how they act to the expected changes in clinical processes and outcomes a stage-specific evaluation approach rapid-cycle feedback of findings to practitioners qualitative as well as quantitative components realist evaluation perspective

67 Stage-specific evaluation approach Innovation phase Aim: generate or discover a new model of care and estimate achievable improvement Methods: quantitative estimate of variations in content theory, qualitative interviews to understand underlying concepts Testing phase Aim: test whether a model works or can be amended to work in specific contexts Methods: quantitative estimate of variations in content theory in specific contexts including ITS and control charts, steppedwedge designs; qualitative interviews to understand how teams learn in local context Improvement phase Aim: assess whether execution theory worked and should be amended to increase uptake Methods (primarily quantitative): process measures overall impact, plus unintended consequences

68 Summary: delivery system science methods Strength: delivery system science methods study impact of deliberate interventions in real-world settings deliberate intervention critical for determining causation Question is not Does it work? rather How and in what contexts does the new model work or can be amended to work? Fixed protocol RCTs ill-suited for healthcare QI initiatives, which are complex and context sensitive iterative in nature, and vary depending on improvement phase Requires a stage-specific evaluation approach rapid-cycle feedback of findings to practitioners qualitative as well as quantitative components realist evaluation perspective

69 Conclusions Big data analytics vs. RCTs False distinction - Observational studies complement RCTs Need to match the method to the question What is the current (and likely future) situation? Does intervention improve outcomes of interest? Translation and spread Depending on question, use combinations of Study design and analysis Analytical methods for individual-level data Qualitative methods

70 Analytic Methods for Learning Health Systems Neil Fleming, PhD Elisa L Priest, DrPH

71 Data alone are not sufficient for learning

72 Care Navigation Evaluation Inpatient referral to Clinic Staff Unconnected (Pts. do not make follow-up visit) Outcomes Tracking Clinic Staff enrolls eligible pts. Baseline data collected: Demographics Comorbidities Home status Other variables BCC Irving Medical Home Connected (Pts. establish follow-up in clinic) Usual Care + Care Navigation Intervention 1:3 Randomization Usual Care Comparative Analyses Outcomes Tracking

73 Implementing a bundle for Intensive Care Unit Delirium Prospective, observational All eligible ICU patients in 2 hospitals Compared to other ICUs Intervention Training Decision support tools Modified work flow EHR template for tracking* IT and Human component

74 Analytic Methods Propensity Score Comparative effectiveness research using electronic health records: impacts of oral antidiabetic drugs on the development of chronic kidney disease Masica AL et al Pharmacoepidemiology & Drug Safety 2013 Selection bias Subject specific propensity score Survival models to estimate treatment effects

75 Challenges Interdisciplinary team needed Traditional timeline for research QI vs research Dissemination of findings

76 Challenges Implementation * Efficacy = Effectiveness R E Glasgow et al Evaluating the public health impact of health promotion interventions: the RE-AIM framework. AJPH Learning from Big Health Care Data Sebastian Schneeweiss, M.D., Sc.D, NEJM June 2014

77 Kaiser Permanente RESEARCH Achieving the Vision of Learning Healthcare Delivery Systems: Professional, Institutional and Policy Barriers September 9, 2014 Brian S. Mittman, PhD Center for Implementation Practice and Research Support, Dept of Veterans Affairs Dept of Research and Evaluation, Kaiser Permanente Southern California School of Medicine, University of California at Los Angeles DEPARTMENT OF RESEARCH AND EVALUATION

78 What is a Learning Healthcare Delivery System? 1. Desired, required features? organizational structure, design policies, procedures infrastructure, resources programs, activities 2. Frameworks, guidance general features and design research and evaluation-specific features Department of Research and Evaluation

79 How does a system become a LHDS? 1. Internal stakeholder understanding, endorsement, commitment senior leadership departmental leadership technical staff front-line/operations staff 2. Domains of interest staff capacity, skills, training organizational resources Department of Research and Evaluation

80 How does society facilitate LHDS development and success? 1. External stakeholder understanding, endorsement, commitment payors, regulators, users/customers advocacy groups research funding agencies prof l assns, training programs 2. Research designs, methods identification, guidance Innovation: refinement, development Department of Research and Evaluation

81 Submitting Questions To submit a question: 1. Click in the Q&A box on the left side of your screen 2. Type your question into the dialog box and click the Send button 89

82 Thank You Please take a moment to fill out the brief evaluation which will appear in your browser.