Applying the Analytic Hierarchy Process to Health Decision Making: Deriving Priority Weights

Transcription

1 Applying the to Health Decision Making: Deriving Priority Weights Tomás Aragón, MD, DrPH Principal Investigator, Cal PREPARE,. CIDER UC Berkeley School of Public Health Health Officer, City & County of San Francisco San Francisco Department of Public Health Blogger at Ferenc Dalnoki-Veress, PhD Scientist-in-Residence & Adjunct Professor James Martin Center for Nonproliferation Studies Monterey Institute of International Studies 1 Tomás Aragón, MD, DrPH, Cal PREPARE Appling AHP to Public Health Decision Making

2 Outline Introduction and Background 1 Introduction and Background Tomás Aragón, MD, DrPH, Cal PREPARE Appling AHP to Public Health Decision Making

3 Quote Introduction and Background Not everything that counts can be counted and not everything that can be counted, counts Albert Einstein 3 Tomás Aragón, MD, DrPH, Cal PREPARE Appling AHP to Public Health Decision Making

4 Acknowledgements Introduction and Background Lisa Goldberg, MPP, MPH, DrPH (c), Graduate Student, Johns Hopkins Bloomberg School of Public Health Ferenc Dalnoki-Veress, PhD, Lead Investigator, Monterey Institute of International Studies, James Martin Center for Non-proliferation Studies 4 Tomás Aragón, MD, DrPH, Cal PREPARE Appling AHP to Public Health Decision Making

5 What process do we/you use to... Make decisions? Set priorities? Allocate resources? Prioritize budgets? Resolve conflict? Achieve consensus? 5 Tomás Aragón, MD, DrPH, Cal PREPARE Appling AHP to Public Health Decision Making

6 How do we commonly set priorities? Informal methods Organizational traditions Leadership preferences Politics and advocacy Categorical funding Personal interests Formal methods Conduct a needs assessment Define core services Conduct economic evaluations (CBA, CBA) Conduct multi-criteria decision-making (MCDM) process 6 Tomás Aragón, MD, DrPH, Cal PREPARE Appling AHP to Public Health Decision Making

7 Challenges to setting health priorities Our understanding of complex health and social systems is limited Qualitative attributes (values, ethics, and equity) are difficult to combine with quantitative attributes (e.g., burden of disease) Inclusion of diverse stakeholders with competing interests and values Goals, criteria, or alternatives are not always well specified Timely decisions must be made with limited evidence Ideal analyses are rarely complete, timely, or feasible Decision-making process may not be transparent Our understanding of prioritization methods may be limited Criteria may have different measurement scales or no scale at all 7 Tomás Aragón, MD, DrPH, Cal PREPARE Appling AHP to Public Health Decision Making

8 The analytic hierarchy process will help us... Make decisions Set priorities Allocate resources Achieve consensus Resolve conflicts Improve communications Justify and defend decisions 8 Tomás Aragón, MD, DrPH, Cal PREPARE Appling AHP to Public Health Decision Making

9 Analytic hierarchy process in 4 steps 1 Define the goal 2 Select, organize, and weight criteria 3 Apply criteria to alternatives and rank 4 Conduct sensitivity analysis 9 Tomás Aragón, MD, DrPH, Cal PREPARE Appling AHP to Public Health Decision Making

10 Analytic hierarchy process (e.g., choosing a leader) 10 Tomás Aragón, MD, DrPH, Cal PREPARE Appling AHP to Public Health Decision Making

11 Analytic hierarchy process Introduction and Background Figure: Generic analytic hierarchy process model 11 Tomás Aragón, MD, DrPH, Cal PREPARE Appling AHP to Public Health Decision Making

12 Derive criteria weights using pairwise comparisons Pairs to compare Criterion A vs. Criterion B Criterion A vs. Criterion C Criterion B vs. Criterion C Measurments 1 Nominal categories 2 Ordinal categories 3 Interval scale 4 Ratio scale* * Includes numerical and ordinal categories with ratio properties 12 Tomás Aragón, MD, DrPH, Cal PREPARE Appling AHP to Public Health Decision Making

13 The fundamental scale for pairwise comparisons Intensity Table: The fundamental scale Definition 1 Equal importance 3 Moderate importance 5 Strong importance 7 Very strong (or demonstrated) importance 9 Extreme importance 2, 4, 6, 8 For compromise between above values 13 Tomás Aragón, MD, DrPH, Cal PREPARE Appling AHP to Public Health Decision Making

14 Apply the fundamental scale for pairwise comparisons Which criterion is more important? How much more important? Figure: Pairwise comparisons of Criteria A, B, and C using the fundamental scale 14 Tomás Aragón, MD, DrPH, Cal PREPARE Appling AHP to Public Health Decision Making

15 Create comparison matrix Introduction and Background Which criterion is more important? How much more important? Criterion A Criterion B Criterion C Criterion A 1 Criterion B 1 Criterion C 1 15 Tomás Aragón, MD, DrPH, Cal PREPARE Appling AHP to Public Health Decision Making

16 Complete comparison matrix with integer ratios Which criterion is more important? How much more important? Criterion A Criterion B Criterion C Criterion A 1 5 Criterion B 1 Criterion C Tomás Aragón, MD, DrPH, Cal PREPARE Appling AHP to Public Health Decision Making

17 Complete comparison matrix with reciprocal ratios Which criterion is more important? How much more important? Criterion A Criterion B Criterion C Criterion A 1 5 1/3 Criterion B 1/5 1 1/7 Criterion C Tomás Aragón, MD, DrPH, Cal PREPARE Appling AHP to Public Health Decision Making

18 Derive criteria priority weights To derive the criteria priority weights we solve for the normalized right eigenvector of the comparison matrix (free software available) Table: Criteria priority weights Criterion (j) Priority Weight (p j ) A B C Total (In)consistency ratio The consistency ratio is the amount of inconsistency in the judgements. A CR < 0.10 is considered acceptable. 18 Tomás Aragón, MD, DrPH, Cal PREPARE Appling AHP to Public Health Decision Making

19 Example 1: Establishing health priorities (see tutorial) Now consider a local health department (LHD) that is committed to becoming a high performance, learning organization through robust strategic effectiveness, performance management, and quality and equity improvement. They have adopted AHP to improve their decision making, priority setting, and resource allocation. The planning unit has developed a priority setting tool to assist in prioritizing health programs so that they align with the agency s strategic directions. 19 Tomás Aragón, MD, DrPH, Cal PREPARE Appling AHP to Public Health Decision Making

20 Example 1: Establishing health priorities AHP for Prioritizing Health Programs (Adapted from Mitton, 2011, PMID ) 20 Tomás Aragón, MD, DrPH, Cal PREPARE Appling AHP to Public Health Decision Making

21 Example 1: Establishing health priorities The executive team completed a session where they defined and approved the criteria. Each executive had been provided with a criteria scoring tool. They had been instructed to use their experience, expert judgement, and understanding of existing evidence to score the criteria based on pairwise comparisons. If they judged two criteria to be equal in importance that pair would be get an intensity score of 1. If not, they would use the fundamental scale to express how much more important one criteria is compared to another. 21 Tomás Aragón, MD, DrPH, Cal PREPARE Appling AHP to Public Health Decision Making

22 Example 1: Establishing health priorities Dr. Juan Nieve is a public health officer and believes that Health Impact is the most important criterion. He has no strong feelings on the other criteria and considers them equal. His scoring tool results are displayed in Figure. 22 Tomás Aragón, MD, DrPH, Cal PREPARE Appling AHP to Public Health Decision Making

23 Example 1: Establishing health priorities Dr. Juan Nieve Dr. Juan Nieve is a public health officer and believes that Health Impact is the most important criterion. He has no strong feelings on the other criteria and considers them equal. His scoring tool results are displayed in previous Figure. Dr. Donald Trumpini In contrast, Mr. Donald Trumpini is a finance officer and he stays up-to-date on the anticipated fiscal impacts of health care reform. He recommends that programs that generate revenue should be weighted moderately higher to ensure a financially sustainable health system. (His score sheet is not shown.) 23 Tomás Aragón, MD, DrPH, Cal PREPARE Appling AHP to Public Health Decision Making

24 Example 1: Establishing health priorities Dr. Juan Nieve s comparison matrix HA SA OI FI Health Impact (HI) Strategic Alignment (SA) 1/ Organizational Impact (OI) 1/ Financial Impact (FI) 1/ Dr. Donald Trumpini comparison matrix HA SA OI FI Health Impact (HI) /3 Strategic Alignment (SA) /3 Organizational Impact (OI) /5 Financial Impact (FI) Tomás Aragón, MD, DrPH, Cal PREPARE Appling AHP to Public Health Decision Making

25 Example 1: Establishing health priorities Aggregating Individual Priority (AIP) weights Table: Criteria priority weights, including normalized geometric mean Nieve Trumpini Aggregated Criterion (j) Weights Weights Weights p 1j p 2j p j Health Impact Strategic Alignment Organizational Impact Financial Impact Total (In)consistency ratio Tomás Aragón, MD, DrPH, Cal PREPARE Appling AHP to Public Health Decision Making

26 Example 1: Establishing health priorities An alternative approach is aggregating individual judgment (AIJ) weights. In the AIJ method we aggregate the comparison matrices first then derive the criteria priority weights. Aggregating Individual Judgment (AIJ) weights HA SA OI FI Health Impact (HI) Strategic Alignment (SA) Organizational Impact (OI) Financial Impact (FI) Tomás Aragón, MD, DrPH, Cal PREPARE Appling AHP to Public Health Decision Making

27 Example 1: Establishing health priorities Table: Comparison of aggregated priority weights using the AIP and AIJ methods AIP AIJ Arithmetic Geometric Geometric Criterion Mean Mean Mean Health Impact Strategic Alignment Organizational Impact Financial Impact Total Tomás Aragón, MD, DrPH, Cal PREPARE Appling AHP to Public Health Decision Making

28 Example 2: Selecting a high school 28 Tomás Aragón, MD, DrPH, Cal PREPARE Appling AHP to Public Health Decision Making

29 Example 2: Selecting a high school: Luis vs. Mami vs. Papi Radial chart of criteria weights. Can you match color to family member? 29 Tomás Aragón, MD, DrPH, Cal PREPARE Appling AHP to Public Health Decision Making

30 Example 3: UCLA Hazard Risk Assessment (HRA) Instrument In general, expected impact of an adverse event: E [Impact] = (Risk of Event) (Impact of Event) = P (Haz)P (Exp Haz)P (Ev Exp) Impact Risk management Avoid risk; Mitigate risk (eliminate/minimize risks); Transfer risk (buy insurance, outsource); Consequence management Detect early and control (monitor and control) Mitigate impacts (minimize impacts: protection, resiliency); Plan contingencies (activate alternate plans); 30 Tomás Aragón, MD, DrPH, Cal PREPARE Appling AHP to Public Health Decision Making

31 Example 3: UCLA Hazard Risk Assessment (HRA) Instrument 31 Tomás Aragón, MD, DrPH, Cal PREPARE Appling AHP to Public Health Decision Making

32 Why are multi-criteria decision making methods not used? Lack of familiarity with decision sciences Lack of understanding of computers or mathematics Concerns about losing control over decision-making 32 Tomás Aragón, MD, DrPH, Cal PREPARE Appling AHP to Public Health Decision Making

33 Concluding thoughts Introduction and Background Decision making, priority setting, or resource allocation requires ranking and selection of alternatives. To achieve this we must develop and weight criteria that are applied to these alternatives. Using the analytic hierarchy process (AHP), criteria can be weighted by conducting pairwise comparisons of each criterion with respect to importance, likelihood, preference, or other factor of interest. AHP allows us to combine our interpretations of evidence (from data, testimony, etc.) with qualitative attributes such as preference or other intangibles. This fact alone makes AHP incredibly powerful and practical. At worse, AHP improves our decision making. Finally, AHP can be used as a research tool to measure decision making, learning, interpretation and much more. 33 Tomás Aragón, MD, DrPH, Cal PREPARE Appling AHP to Public Health Decision Making

34 Bibliography Introduction and Background 1 Decision Sciences page at Med Epi blog ( Register to receive updates. 2 Decision by Objectives: How to Convince Others that You are Right, by Ernest H. Forman Permalink: Also available for free PDF download from edu/profforman/decisionbyobjectives/default.html 3 Decision Making for Leaders: The for Decisions in a Complex World, New Edition 2001 (Analytic Hierarchy Process Series, Vol. 2), by Thomas L. Saaty Permalink: 34 Tomás Aragón, MD, DrPH, Cal PREPARE Appling AHP to Public Health Decision Making