Transparency of Hospital Productivity Benchmarking

Transcription

1 Transparency of Productivity Benchmarking (Research-in-Progress) S. Laine, Department of Computer Science and Engineering, Aalto University Laine, Sami, Niemi, Erkka (2013), Transparency of Productivity Benchmarking in Two Finnish Districts, In the Proceedings of the 29th annual Patient Classification Systems International (PCSI) Conference, Helsinki, Finland. (Best Paper Award, Full Paper Download)

2 Personal background combines technical, human and healthcare perspectives University of Turku, Finland Information systems Empirical field studies in hospital focusing on the use of IT. Turku University, Finland Healthcare datawarehousing Project management, system and service design. Aalto University, Finland Usability Research Healthcare data quality research across contexts.

3 The Finnish Productivity Benchmarking has a long history but it is not used in decision making Data Results The benchmarking results are produced by National Institute for Health and Welfare (THL) on annual basis. The background, implementation and future plans of the BMS have been described earlier by Linna and Häkkinen. They noted that policymakers and managers do not regularly use efficiency analyses and the main reason appears to be concern about data quality. Linna, M. and Häkkinen, U. (2007) Benchmarking Finnish s. In Evaluating Policy and Performance: Contributions from Policy and Productivity Research, pp

4 Benchmarking claimed significant productivity differences in neurology specialty Where do the figures come from? Pirkanmaa District District of Southwest Finland What do the figures actually mean? What is excluded? What is included?

5 The QUALIDAT project ( ) uses three complementary research approaches to study the same Information Production Process (IPP) Management approach Studies the information management and governance best practices. Usability approach Studies the users and their hands-on work situations in care pathways. Data analytics approach Tracks the entire information flow from data entry to data utilization.

6 Information Production Process (IPP) consists of three phases based on Total Quality Management (TQM) DATA SUPPLY DATA MANUFACTURING DATA CONSUMPTION Enters data for primary purpose Builds data sets for secondary use Analyses and reports data Interprets data and makes decisions Electronic Patient Record Medical Imaging System collect data collect data Data Warehouse produce internal reports produce data external sets Internal Service Reports Data Warehouse produce external reports Finnish Productivity Benchmarking Wang, R. Y., Lee, Y. W., Pipino, L. L. and Strong, D. M. (1998) Manage Your Information as a Product. Sloan Management Review, 39, 4, pp

7 Productivity figures are complex combination of care pathways and information production processes! DATA SUPPLY DATA MANUFACTURING DATA CONSUMPTION Enters data for primary purpose Electronic Care Patient Record pathways Medical Imaging System Financial Issues collect data collect data Builds data sets for secondary use Data Warehouse produce internal reports produce data external sets Data Warehouse Analyses and reports data Information production process produce external reports Interprets data and makes decisions Productivity Formula Internal Service Reports Finnish Productivity Benchmarking

8 Data Research Explanatory Research Constructs algorithms and derives results out of data. Aims Manipulates algorithm to explain variations in results. This opens up a new perspective, since we look the same phenomena from completely opposite direction and turn around previous assumptions. Assumes randomness and Gaussian distribution Errors Seeks systematic reasons for variations

9 Systematic biases in measurement During the explanatory research project, we will identify and explain mechanisms for potential systematic biases By systematic bias we mean unintended or undesirable inherent characteristics that have effects to the benchmarking results. Systematic biases have many undesirable consequences for validity of benchmarking results, quality of healthcare, and incentives of hospital management.

10 There exists systematic biases in productivity formula that are caused by mechanisms Casemix mechanisms Selecting patients Dumping patients Service Fragmentation Patient Casemix Human Incentives Data Scope Documentation Level Errors Fixes Fragmentation Mechanisms Splitting services Labelling services

11 Fragmentation bias rewards splitting and heterogeneity District District

12 Fragmentation bias rewards splitting and heterogeneity Less production but more health for same money! Episode A DRG X Episode A DRG X District District DRG A Episode 1 DRG B More production but less health for same money! Splitting X to 3 parts AND DRG C Renaming them to A,B&C.

13 Fixing bias causes hidden semantic heterogeneity and leads to surprising impacts District District

14 Fixing bias causes hidden semantic heterogeneity and leads to surprising impacts District District Software suggests additional diagnosis codes AND Software searches for and fills in alternative data EHR SUGGESTS DATA Episode A Episode A Episode A WAREHOUSE SUBSTITUTES SELECTED SELECTED SELECTED Diagnosis A Diagnosis B Diagnosis B+C C Episode A Episode A Episode B+C! A MISSING! A DATA WAREHOUSE SUBSTITUTES

15 All bias mechanisms can be built-in to healthcare services or information production processes! DATA SUPPLY DATA MANUFACTURING DATA CONSUMPTION Service Model Enters data for primary purpose Electronic Patient Record Medical Operative Imaging System Practice collect data collect data Software Feature Builds data sets for secondary use Fragmentation Data Warehouse Casemix Incentives Scope produce internal reports Documentation produce data external sets Errors Fixing Data Warehouse Analyses and reports data produce external reports Human Action Interprets data and makes decisions Internal Service Reports Finnish Productivity Benchmarking

16 Bias mechanisms cause systematic variations rather than random errors DATA SUPPLY DATA MANUFACTURING DATA CONSUMPTION Enters data Data Entry for primary Error Bias purpose Builds data Interpretation sets for Mismatch Bias secondary use Analyses and reports data Interprets data and makes decisions Electronic Application Patient Record Suggestion Bias Medical Imaging System collect data collect data Complex Data Architecture Warehouse Biases produce internal reports Scripting Error produce data Bias external sets Internal Service Reports Data Warehouse produce external reports Bias variations can be cumulative or they diminish each others -> artificial gains or losses! Finnish Productivity Benchmarking

17 Benchmarking suffers from obscurity DATA SUPPLY DATA MANUFACTURING DATA CONSUMPTION Enters data Data Entry for primary Errors purpose Builds data Scripting sets for Error secondary use Analyses and reports data Interprets data and makes decisions Obscurity Electronic Patient Record Application Feature Bias Medical Imaging System collect data collect data Obscurity Architecture Data Warehouse Bias produce internal reports produce Scripting Error data external sets Internal Service Reports Obscurity that leads to problems in recognizing, preventing and fixing biases. Data Warehouse Obscurity produce external reports Finnish Productivity Benchmarking

18 Obscurity is a problem, because black boxes are unpredictable! X A*80% A Y A*135%

19 Obscurity is a problem, because black boxes are unpredictable! X A*80% 1,12 A Y A*135% s actual productivity and perceived productivity lose their connection to each other because of bias mechanisms in black boxes.

20 More information about the entire benchmarking system should be made visible for all stakeholders to avoid organizational silos and hidden bias ransparent Care Pathway Transparency should cover all significant influencing factors in result themes: productivity formula, healthcare service production, information production process. Details are important! A small detail can have huge cumulative impacts. Transparency should open all IPP phases in and between all participating organizations: data supply, manufacturing and consumption Semantic error can occur in any phase! Errors can increase, diminishing or change direction depending on the internal calculations! Black boxes are unpredictable. Transparent Information Production Process

21 More information about the entire benchmarking system should be made visible for all stakeholders to avoid organizational silos and hidden bias ransparent Care Pathway Only in this way, one can evaluate the validity of decisions in patient care, hospital administration, policy making, and medical research. Transparent Information Production Process

22 The Next Steps in QUALIDAT Project

23 QUALIDAT tracks care pathways and data flows to identify critical factors that affect productivity results! DATA SUPPLY DATA MANUFACTURING DATA CONSUMPTION Referral User Interface Screen 1 Attribute X Attribute X Visit Attribute X Attribute X Admission User Interface Screen 2 Attribute X Discharge User Interface Screen 3 Attribute X Attribute X Finnish Productivity Benchmarking

24 Questions?

25 Additional References to Transparency of Healthcare Information Management Laine, S. (2014) Open Data Critical Capability in Healthcare Information Production Processes, in the seminar of Openness and the Future of Healthcare IS by Service Factory, Aalto University School of Business at 18th of March 2014.