Planning Your Analysis. Copyright 2012 The University of Utah - All Rights Reserved

Transcription

1 Planning Your Analysis

2 Learning Objectives Establish a pattern of analytical thinking Compare analysis to reporting Discuss data types and data issues Establish the steps of data collection Discuss data types and scales Explain basic data cleanse Discuss random sampling and segmentation

3 Why Analysis? Never let the facts get in the way of a good opinion.!!!! We must become more data-driven so we can act on fact Support evidence-based medicine Narrow our choices...and so on You re entitled to your own opinion, but not your own facts. Creativity isn't the conclusions you have drawn but how you pry them back open. It ain't what you don't know that gets you into trouble. It's what you know for sure that just ain't so. There s a difference between an opinion and an informed opinion.

4 What are some of the substitutes for analytical thinking? Gut (instinct, experience) Story telling I remember this one patient Common sense Reaction vs. Reflection (the age of the smart phone) Groupthink HiPPO (highest paid person s opinion) On your project teams, you may need to combat these. To do so, establish a simple pattern of analytical thinking.

5 The pattern of analytical thinking Data Analysis Insights Deliberation Decision so what? what if? General Methodology Framework 1.! Project Definition 2.! Baseline Analysis 3.! Investigation 4.! Improvement Design 5.! Improvement Implementation 6.! Monitoring

6 When conducting an analysis, start with the end in mind! Data Analysis Insights Deliberation Decision Decision Deliberation what if? Insights so what? Analysis Data What decision has to be made? What are the options? What are the current arguments, if any? What insights are needed to help us act on fact? What analyses are needed to gain those insights? What data sets are most appropriate to feed the analyses? We work in a data-rich environment; not an environment rich with information and insights. It pays to think ahead.

7 Planning Your Analysis Decision Deliberation Insights Analysis Data Data Analysis Insights Deliberation Decision! Of drills and holes.! Start broadly.! A narrow request: I want the average length of stay (LOS) for appendectomies.! A broad request: I want to understand why some appendectomies LOS are short and some are long, and then see if we can do anything about the long ones.

8 The analysis vs. value S-curve: Analysis isn t free. Data Analysis Insights Deliberation Decision Decision Deliberation Insights Analysis Data Value (insights) analysis paralysis Gut, etc. Analytical Energy (time, cost, frustration)

9 How to hit the sweet spot Decision Deliberation Insights Analysis Data Data Analysis Insights Deliberation Decision Working backward through the pattern will avoid the extremes. Decision Deliberation Insights Analysis Data! When we have,, and analyses in front of us, we ll have what we need to make a decision. 6#$4+).'"2'-7&25 94&0)+&(: #"#$%2'2)) 8#,#$%2'2! If you, with your analyst, can fill in those blanks before the analysis, you re in the right spot on the S-curve.!"#$%&'(#$)*"+,-%).&'/+0)(12&0)3,42&,#&'1"5

10 Reporting is not Analysis Reporting period Reporting Data Reports metrics How re we doing? good need to improve Analysis Decision Deliberation Insights Analysis Data what if? so what? Data Analysis Insights Deliberation Decision so what? what if?

11 Reporting is not Analysis Analysis Analysis cannot be automated. Analysis is often less visually appealing. Analysis is more comprehensive Requires some interpretation. Might look at the same data many ways. May come with a narrative explaining it and what it means. Reporting Reporting is a regularly scheduled activity with a template. Metrics (and dashboards and scorecards ) are reports. Reporting is often flashy. Reporting is more straightforward. Can be understood in a nanosecond. Standard reports or metrics are sometimes born of comprehensive analysis.

12 Where are you going to get your data?

13 Collecting Data: What to do when the data you need hasn t been collected? Start Design a data analysis plan Is the data you need already collected? No Is there a suitable substitute data set? No Design a data capture plan Decision Deliberation Insights Analysis Data Yes Yes Alter your analysis plan accordingly Train data collectors Data Analysis Insights Deliberation Decision Alter your analysis plan accordingly Design data capture tools Execute data capture plan Pilot data capture plan Go get it Analyze Stop

14 Collecting Data: Checksheets and Data Collection Forms Design a data capture plan Alter your analysis plan accordingly Design data capture tools Pilot data capture plan Train data collectors Execute data capture plan Figure out exactly the data you want to collect Refer to your process map and pinpoint where and when a data collector needs to be to capture the occurrences Design a form for easy capture Hard to design so it s easy to use Now make it easier to use Does the form follow the workflow and decision flow? Is your use of font, white space, borders, colors, etc. easy on the eye? Does it makes sense to convert to electronic? Pilot the system The first version is rarely perfect Train the data collectors Execute You probably only have one chance at this. If you have to go back for more due to a mistake, your welcome on the unit will be worn thin. Lots of great examples out there to borrow from.

15 Data Types Better Continuous Discrete Characteristics Indefinite, Variable Countable, Categorical, Attribute Examples Duration of intubation Age Weight Length Likert scales Episodes of intubation Gender Age group Zip code Readmission rates

16 Types of Scales Better Scale Characteristic Example 1. Ratio à A known zero point à Revenue 2. Interval à No known zero point à Temperature 3. Ordinal à Discrete with order à Ed. level 4. Nominal à No order à Marital status You may never be required to distinguish between these

17 Data Issues: Outliers '" '%!($)*+,*-./%*0.)!%")& You were set up.!"#$%& %' %& ## #$! Same data! Exactly the same dispersion! Different order! Different scales! A few biases built in!"#"("!"#"'%!"#"&"!" ( # (( (# &( &# Comic Book Price per Page! Use a set definition for outliers! Occurrences > 3!! Don t depend on your (biased) eyes or your hunches! Show results with and without outliers (and if there s no difference, leave them in.)!"#"$%!"#""" Occurrence Average

18 Data Issues: Sampling I need to draw your blood population! You cannot draw a sample that is larger than the population.! Hierarchy: Universe Population Sample!! Overlapping populations is okay Sample must be measured in the same units as population Why analyze samples instead of populations?! Cost! Sometimes it s impossible RN

19 Data Issues: Sampling Are the following data sets samples or populations? 1. SSI rates of 100% of general surgery patients from 2009 and Room temperature of all the rooms on your unit 3. Satisfaction survey results from last week, n=16 Answers: 1. It depends 2. It depends 3. It depends Analysis of samples allows us to say something about the population; allows us to make inferences about the population. If you re only interested in the data you have, and don t care to make any inferences based on your data, then you have population data. But this is rarely the case. (More on inferential statistics later.)

20 Random Sampling: Every member of the population is equally likely to be selected Literary Digest polls correctly predicted every presidential winner since 1920 (Squire 1988) In 1936 presidential election, Literary Digest predicted a landslide for Alf Landon based on 2.4 million responses (by US Post) Landon carried only Vermont and Maine: 8 electoral votes, a record low that still holds Literary Digest went out of business shortly after What happened? Got three lists to generate their survey targets: Its own subscription list Registered automobile owners People with registered telephone numbers 2.4 million isn t much better than 1500 if it s a truly random sample Sometimes stratification is required to produce homogenous sub-groups: Laparoscopic surgeries vs. all surgeries Still needs to be random

21 Data Issues: Segmentation of intubated patients Let s look at all patients from the last 5 years with at least one episode of intubation Well, things change fast, so let s go with six months We re really only interested in surgery patients And actually only patients in SICU We want to differentiate between patients who were intubated for greater than and less than 48 hrs Segment into single intubation episodes and multiple...because we re targeting the patients who were positive for VAP Not homogenous enough. n = 20,000 n = 2,000 n = 800 n = 450 <48: 300 >48: 150 S: 250 M: 50 S: 90 M: 60 P: 2 N: 248 P: 3 N: 47 P: 2 N: 89 P: 4 N: 54 Segmented down to 11 cases: Too small for a strong argument. P: 2 P: 3 P: 2 P: 4

22 Data Issues: Scrubbing for integrity and accuracy Duplicates Our charting systems, when queried, can produce many Excel: Data à Remove duplicates Similar categories ED = ER = EmerDept = Emergency Dept particularly when dealing with text fields Missing data Small samples: Can you find the missing data? Probably don t need to delete these. Excel skips them. Bad numbers Run a simple analysis to look for weird highs and lows Excel: =max(a1:a2000) =min(a1:a2000) Sort and plot. See anything weird? Did your blanks become zeros? Anomalies in the data

23 Data Collection Issues More fields is usually better Always save an unedited version of your raw data Cause of death vs. dead: Y/N Get continuous when you can, as opposed to categorical DOB vs. Years old. Continuous data can be categorized if you need, but it s hard to go the other way. Be very careful manipulating data It s easy to corrupt your data with SORT and FILTER You ll likely need to do both Record the bread crumbs of your analysis in a separate tab or a separate file. This requires personal discipline Six months from now, you ll want to retrace your steps

24 Final Word on Analysis Decision Deliberation what if? Insights so what? Analysis Data Data Analysis Insights so what? Deliberation what if? Decision!! There are many tools available to you.! Not a single one is magic! Don t get lost in the trees! Start with your analytical thinking pattern! Make a plan, refer to it often, change it when it makes sense There s no substitute for direct observation of the process! Watch for work-arounds that have become normal! Watch for mistake catching instead of the preferred mistake preventing! Watch for frustration! Watch for the seven wastes