HEALTH SYSTEM PERFORMANCE INTERACTIVE INDICATORS WEBSITE PUBLIC ENGAGEMENT SUMMARY REPORT

Transcription

1 HEALTH SYSTEM PERFORMANCE INTERACTIVE INDICATORS WEBSITE PUBLIC ENGAGEMENT SUMMARY REPORT

2 PAGES TABLE OF CONTENTS INTRODUCTION 1 KEY FINDINGS: ONLINE AND IN-PERSON ENGAGEMENT 2 FINDINGS: ONLINE ENGAGEMENT 3 FINDINGS: IN-PERSON ENGAGEMENT 11 ENGAGEMENT METHODOLOGY 13 NEXT STEPS 14 APPENDICES APPENDIX INTRODUCTION 16 i

3 INTRODUCTION An Interactive Health System Performance Website for Canadians The Canadian Institute for Health Information (CIHI) is a leading source of unbiased, credible and comparable information about health and health care in Canada. A challenge for CIHI is how to make reports on health indicators more accessible and useful to Canadians. To address this issue, CIHI will be launching a series of interactive websites that provide information in visual and engaging ways on the performance of the health system. The first of these interactive websites, for use by the general public, will launch in the fall of The site will feature a small number (15) of comparable indicators that will help Canadians understand the relative performance of the health system in their province or region compared to other parts of the country, as well as in Canada as a whole. The website will be based on the areas of performance that matter most to Canadians; it is intended to serve as a national reference to support better health system decisions and enhance transparency of the health system for all Canadians. Engaging Canadians on the Interactive Website CIHI understands that as users and funders of the health system, Canadians want to be involved in discussions about how well the health system is performing. However, crosscountry consultations recently conducted by CIHI suggest there is confusion about health system performance, with a large volume of information reported by different organizations in an uncoordinated and sometimes contradictory fashion. In order for the planned website to be most useful for Canadians, it must focus on the areas of system performance that are of greatest interest and meaning to them, while featuring available measures that are reliable and credible. Public participation means to involve those who are affected by a decision in the decision-making process. It promotes sustainable decisions by providing participants with the information they need to be involved in a meaningful way, and it communicates to participants how their input affects the decision. International Association for Public Participation (IAP2) With this in mind, CIHI began the website design project by engaging Canadians to identify what types of indicators or measures of health system performance - they wished to see and access. The Institute partnered with Hill+Knowlton Strategies to conduct a robust pan-canadian consultation designed to identify the areas of performance measurement that matter most to Canadians. This was an exercise in public participation a new approach to decision-making that seeks to involve people in decisions in which they are affected or interested. In February 2013, CIHI, together with H+K Strategies, held a series of 5 small group dialogues with people across the country. At the same time, we engaged a randomized, representative sample of over 3000 Canadians using a deliberative online Choicebook. This complementary online and in-person approach provided direction on the areas of performance measurement of most interest to Canadians. This report provides a summary of this public engagement, and the key messages heard from Canadians. More detailed research data is also available in the report s appendix. 1

4 KEY FINDINGS: ONLINE AND IN-PERSON ENGAGEMENT Highest Interest in Outputs and Outcomes In our research process, participants were introduced to the four main quadrants of the CIHI Health System Performance Framework: Inputs, Outputs, Outcomes and Social Determinants of Health. They were asked to indicate their level interest in each of the quadrants on a scale from one to seven (where one represented not interested at all and seven represented very interested ). All quadrants received high ratings from participants, with health system outcomes and health system outputs receiving the highest scores at the end of the process (6.2. and 6.1 out of 7 respectively). Access to Care is Highest-Rated Performance Dimension Participants were also asked to prioritize health system performance dimensions within each of the four quadrants. The highest rated dimensions among Choicebook participants were: 1. Access (Outputs) 2. Equity (Outcomes) 3. Responsiveness (Outcomes) 4. Quality (Outputs) 5. Health promotion and disease prevention (Outputs) 6. Value for money (Outcomes) When probed further, Canadians identified wait times as the most important component of access, and prioritized access to family physicians and specialists as measurement areas of high interest. Canadians Share Common Priorities Overall, our findings were conclusive and consistent across all groups of Canadians. Only small differences in priorities were identified when breaking down results by age, province or territory, urban/rural place of residence, and whether the person was a frequent user of the health system or a caregiver. Findings Consistent Between In-Person and Online Research Channels Overall, there was a high degree of congruence between the online and in-person findings. The measurement areas of most interest to online participants were almost identical to those identified during the in-person sessions. The major difference was that in-person participants showed a greater level of interest in health system innovation as an area of performance measurement. 2

5 FINDINGS: ONLINE ENGAGEMENT We used a deliberative online Choicebook to gather the perspectives of a large, representative sample of Canadians. The Choicebook was a 30-minute interactive experience in which participants learned about options for measuring health system performance. Engagement Approach For both the online and in-person engagement, CIHI s new Health System Performance Framework was used as a starting point to frame the issues for discussion. Based on scientific literature, the framework maps out the main dimensions of health system performance, demonstrating how they interconnect and relate to each other, and how they contribute to overall performance goals, such as improved health status and better value for money. Participants were introduced to the four main quadrants of the framework, Inputs, Outputs, Outcomes and Social Determinants of Health, as well as some key performance dimensions within these quadrants. They were probed about which areas of performance they valued the most and wanted to see included in the public website. Figure 1: Health System Performance Quadrants and Corresponding Dimensions Included in Public Engagement Process Health System Inputs and Characteristics refer to the structural characteristics of the health system including governance and leadership, innovation, the resources available for use, the distribution and allocation of those resources as well as the capacity of the health system to adjust and adapt to meet population health needs: Efficient allocation of resources Innovation Planning the right services to meet population needs Health System Outputs are immediate results of services delivered to patients and to the population. Common outputs include the delivery of accessible, timely and effective health and health care services: Access Appropriateness of care Efficiency of health services Patient experience Health promotion & disease prevention Quality and safety of health services Health System Outcomes correspond to the intrinsic goals of the health system. These outcomes are the improvement of both the level and distribution of the health of the population, responsiveness to needs and demands of Canadians, and value for money to ensure health system sustainability: Health status Responsiveness Equity Value for money Social Determinants of Health represents the factors, outside the health system, that influence the health of a population. In this framework these include genetic endowment, social position, life conditions and physical environment. We broke down these social determinants into two performance dimensions: Neighbourhood planning People s circumstances 3

6 To make these concepts more accessible to Canadians, we developed fictional scenarios, or micro-stories, to help illustrate the performance dimensions with concrete examples and put a human face on them. Figure 2 is an example of a scenario from the engagement. Figure 2: Example Scenarios to Illustrate Access and Efficiency 4

7 High interest in all performance quadrants, but highest for Outcomes and Outputs. Figure 3: Health System Performance Areas, Pre Post Interest (Likert Scale 1-7, Average Score) Pre Post Pre Post Pre Post Pre Post At the beginning and the end of the online Choicebook process, we asked participants to indicate their level of interest in each of the quadrants on a scale from one to seven (where one represented not interested at all and seven represented very interested. ) As illustrated in Figure 3, there was a high level of interest in all of the quadrants. However, interest was highest for two Outcomes and Outputs. Preference for these quadrants was equally high or higher at the end of the process, after participants had learned more about these areas of performance measurement. To explore participants measurement interests further, we focused on 16 performance dimensions, organized under the four quadrants. Due to the large number of dimensions in the output quadrant, relative to others, we divided outputs into two groups. Figure 4 outlines the division of dimensions. 5

8 Figure 4: Health System Performance Dimensions, Organized by Quadrant Inputs Outputs (Group 1) Outputs (Group 2) Outcomes Factors Influencing Health 1. Efficient allocation of resources 2. Innovation 3. Planning the right services to meet population needs 4. Access 5. Appropriateness 6. Efficiency 7. Patient experience 8. Health promotion & disease prevention 9. Quality 10. Safety 11. Value for money 12. Equity 13. Health status 14. Responsiveness 15. Neighbourhood planning 16. People s circumstances Prioritizing performance dimensions To measure relative priorities within each of the quadrants, we used a question structure that asked participants to allocate 100 fictional dollars ( CIHI Bucks ) between the performance dimensions. They could choose to allocate all 100 CIHI Bucks to a single measurement area, or distribute among any other combination provided that all 100 dollars were allocated. For example, participants were given 100 CIHI Bucks to distribute among the four dimensions in the Outcomes basket: value for money, equity, health status, and responsiveness. Figure 5 presents a summary picture of Canadians preferences. 6

9 Figure 5: Online "CIHI Bucks" Allocation, Weighted Results Outputs 1 Outputs 2 Outcomes Inputs Factors Influencing Health Access Appropriateness Efficiency Pa ent Experience Health Promo on & Preven on Quality Safety Value for Money Equity Health Status Responsiveness Alloca on of Resources Innova on Planning the Right Services People's Circumstances Neighbourhood Planning Weighted $ Alloca on Results were weighted to $50 to facilitate comparison between dimensions. The sample was also weighted to be representative of the Canadian population from a demographic perspective. This exercise demonstrated that Access was by far the most important priority for Canadians in terms of performance measurement, followed by Equity, Responsiveness, Health Promotion and Disease Prevention, Quality, and Value for Money. Figure 6 presents an interesting story, as well as a potential starting point for information to be presented on CIHI s health system performance website. It shows the relative investment in performance dimensions, against interest in the category or quadrant that it belongs to. The top right corner of this chart illustrates the dimensions with the highest investments in the quadrants of highest interest. These performance areas cumulatively received over $50 mean dollars in the allocation exercise, as well as a Likert score above 6 (out of 7) for their respective quadrants. 7

10 Figure 6: Online "CIHI Bucks" Allocation by Average Post Interest by Quadrant Access Mean Alloca on of $ (Weighted) 55 People's Circumstances Allocation of Resources Appropriateness Quality Health Promotion & Disease Prevention Patient Experience Efficiency Responsiveness Equity Value for Money Planning the Right Services 5.75 Neighbourhood Planning Innovation Safety Health Status 35 Average Post Interest in the Four Categories The cumulative Choicebook results identified the following individual performance dimensions of greatest interest to Canadians: 1. Access (Outputs) 2. Equity (Outcomes) 3. Responsiveness (Outcomes) 4. Quality (Outputs) 5. Health promotion and disease prevention (Outputs) 6. Value for money (Outcomes) Choicebook Findings by Variable Additional analysis was also conducted to break down results by age, province or territory, urban/rural place of residence, and whether the person was a frequent user of the health system or a caregiver. Overall, the Choicebook results were remarkably consistent across all groups of Canadians, with only small differences found based on demographic profile. Nevertheless, there were the following minor variations: 8

11 Age + While results were generally consistent across age groups, the biggest differences were seen between youngest (under 35) and oldest (65 and older) respondents. + When looking at output measures, younger Canadians (adults under 35) were considerably more interested in health promotion and disease prevention than Canadian seniors. + Younger Canadians were also more interested in health status as an outcome measure, while older Canadians rated value for money relatively higher. Province/Territory + When asked about social determinants of health, British Columbians were slightly less interested in neighborhood planning as an area of performance measurement than Canadians as a whole, and slightly more interested in people s circumstances. + Atlantic Canadians had a slightly higher preference than the national average for health promotion and disease prevention measurement, and were mildly less interested in patient experience. Urban/Rural Residence + Rural residents were considerably more interested in people s circumstances and less in neighborhood planning than people living in urban parts of Canada. + Access also received a slightly higher score from Canadians living in rural areas; health status on the other hand, was rated a bit lower than it was for urban Canadians. Caregivers + Participants who said they were caregivers for sick, elderly or disabled friends or family members also gave higher scores to the performance dimension of access to care, at the expense of efficiency of health services. + Caregivers were also more interested in measurement of patient experience and planning the right services to meet people s needs. Frequent Users + Frequent users of the health system - or those who had more than 3 medical visits a year for the same condition - gave slightly higher relative ratings to access, patient experience, safety, and equity performance dimensions than low users of the system, and lower scores to health promotion and disease prevention. Understanding Access Findings As the performance area of highest interest to Canadians, it was important to probe further to understand what access means to Canadians. In a series of activities that focused specifically on this performance dimension, participants were asked to allocate 100 CIHI Bucks to measure relative interest in access-related themes. First, participants were asked which potential barriers to access were of most interest to them for measurement purposes: affordability, geography, navigation, or wait times. As shown in Figure 7, wait times were universally 9

12 viewed as the barrier of most interest. This was the case regardless of a participant s age, experience with the health system, and whether one is a caregiver. Figure 7: Access "CIHI Bucks" Allocation In a follow-up question, participants were asked to identify their level of interest in access to specific parts of the health system: family doctors, specialists, cancer treatments, emergency wait times, long-term care, and elective surgeries. The results show a clear hierarchy of interest, as illustrated in Figure 8, that likely mirrors the experiences Canadians have with the health system with the most common ranking highest, such as access to family doctors and specialists. Overall, all Canadians share this order of priorities. Older Canadians, however, allocated slightly more CIHI Bucks for access to elective surgeries and long-term care. Rural Canadians allocated slightly fewer CIHI Bucks to emergency wait times. 10

13 Figure 8: "CIHI Bucks" Allocation for Access to Specific Parts of the Health System FINDINGS: IN-PERSON ENGAGEMENT At each of our five in-person dialogues, we asked participants many of the same questions posed in the online consultation. This allowed us to compare the results of the two engagement channels and to measure whether, and to what extent, participant perspectives differ between a shorter, individual online engagement, and a longer, in-person dialogue that involved direct interaction amongst participants. Overall, there was a high degree of congruence between the online and in-person findings. The measurement areas that received the highest CIHI Buck allocations in the in-person dialogues were almost identical to those that received the highest allocations online: access, equity, responsiveness, quality, health promotion and disease prevention, value for money, allocation of resources. The major difference was that in-person participants showed a greater level of interest in both the Inputs quadrant more generally, and the innovation dimension within it more specifically. Understanding Why? In addition to including some of the same questions as the online Choicebook, the in-person dialogues asked participants additional questions to further understand their perspectives and relative interest in performance measurement areas. In-person dialogue participants had an opportunity to explain their CIHI Bucks allocation decisions. Their detailed and thoughtful explanations provide insight into the variation in the levels of interest across the measures both those that received higher and lower levels of interest. 11

14 + Access (Higher Interest): Participants indicated that access is the Access is most important because most important characteristic of the health system; that, unless without access to the service patients Canadians could access the right services when they needed need, they could die or become them, that other measures of the system, such as quality of permanently injured services, were irrelevant. They expressed an interest in information about access to inform their health system journeys, and to foster greater accountability for improvement in accessing health services. + Equity (Higher Interest): Participants felt strongly that fairness and equity are fundamental principles of the health system. They expressed an interest in information about how well the system is doing to improve equal access to good health and health care services. The system has to adapt according to the growth and aging of the population + Responsiveness to Canadians Needs (Higher Interest): Those who participated in our in-person dialogues believe that the health system should adapt to provide the services that Canadians need as the population and its health requirements change. They are interested in seeing information that would show how effectively the system is responding to changing needs in their communities. The scenario used to illustrate responsiveness, which featured mental health services to meet the needs of the homeless, also resonated strongly with the public. + Value for Money (Higher Interest): While participants were interested in how well the system is working to improve the health of Canadians and deliver timely services of high quality, they also expressed an interest in understanding how their health dollars are spent, and whether they are getting good value for investments made to achieve those health improvements. Value for money measures the problem in Canadian health care. Ratio of resources to treatment. Decisions are often not value for money and quality is sometimes lacking. + Quality (Higher Interest): Participants believed that quality was a central performance criterion for the health system. They feel that quality services have the biggest impact on improving health outcomes. This is the main reason for health care - to get better! + Health Promotion and Disease Prevention (Higher Interest): Participants suggested that health promotion and disease prevention is essential for ensuring the health system s financial sustainability. They believe that preventing illnesses is a more efficient use of scarce resources than treating these illnesses later. + Health Status (Lower Interest): Participants acknowledged the importance for decision-makers and the public to have information about the overall health of the Canadian population. However, they suggested that there would be limited value in including this information on the planned CIHI website because data on health status is readily available from other sources, such as on news media and government websites (e.g. Statistics Canada). 12

15 + Appropriateness (Lower Interest): This measure received relatively low CIHI Bucks allocations online and inperson. During the in-person dialogues, participants commented that the appropriateness of health services could at times be subjective and difficult to measure. They preferred to focus on measurement areas seen as more objective, such as wait times for priority health services. + Patient Safety (Lower Interest) Participants were asked about their interest in patient safety relative to three other output dimensions, including quality of patient care. They prioritized quality over patient safety, because they reasoned that if the system is focused on delivering the best quality of care to patients, it should by definition also be delivering safe and appropriate care. + Patient Experience (Lower Interest): While participants agreed that it is important for patients to have a good care experience, they also consider patient experience to be more subjective than the other measurement areas they were asked to chose from, such as quality, and health promotion and disease prevention. Good or bad experience can be subjective but a good outcome is what matters. ENGAGEMENT METHODOLOGY We used a complementary approach of in-person dialogues with small groups of randomly recruited participants in diverse locations across Canada, and an online Choicebook. This complementary two-channel approach allowed our research team to hear a broad cross-section of views online, as well as to explore the subject matter in more depth and time through our in-person dialogues. We partnered with EKOS Research Associates ProbIt online panel to engage a randomized, representative sample of Canadians in the online Choicebook. We heard from over 3000 Canadians (with the exception of Quebec), from coast-to-coast, with an over-representation from provinces with smaller populations to allow us to identify any potential differences based on geographic location. Indicator Framework for Engagement Our engagement methodology was founded on the principle of informed participation a belief that the public seeks information and context to participate and provide their informed feedback and advice. The key online engagement tool was a Choicebook, which used fictional scenarios and accessible language to present participants with a set of measures from which to identify their priorities. In-person similar information was presented in a Conversation Guide. Participants were presented performance areas to consider, as defined in CIHI s new Performance Measurement Framework for the Canadian Health System (Figure 1 above). The framework uses four inter-related quadrants as organizing groups of performance measurement: health system outcomes, social determinants of health, health system outputs, and health system inputs and characteristics. 13

16 Online Choicebook We used a deliberative online Choicebook to gather the perspectives of a large number of Canadians. The Choicebook was a 30-minute interactive experience in which participants learned about options for measuring health system performance through videos and fictional scenarios before identifying which were most interesting to them through open and closed-ended questions. Results from the Choicebook were gathered over a three-week period in February In-Person Dialogues As a complement to the online engagement channel, we held a series of five in-person dialogues in different regions of Canada between February 4 and February 15, Again working with EKOS and its ProbIt panel, we recruited representative groups of residents in each of Ottawa (Ontario), Bathurst (New Brunswick), St. John s (Newfoundland and Labrador), Prince Albert (Saskatchewan), and Vancouver (British Columbia). At each of these 3.5-hour events, we engaged socially diverse groups of local residents about the health system performance measures of greatest interest to them. To foster deliberation and informed participation, each participant was given a Conversation Guide that presented them with the same learning information as the online Choicebook, and asked to identify their priorities using the same question formats to allow direct comparison across engagement channels. Our research methodology uses electronic voting keypads to generate quantitative data on participants views. The difference between the online and in-person engagement channels is that at the in-person events, participants were provided with an opportunity to discuss and explore the performance measurement framework in small groups. This allowed us to measure whether, and to what degree, in-person participants views changed after talking about measuring health system performance with other Canadians. Further details on our engagement methodology are available in the Appendix. NEXT STEPS On behalf of CIHI, we wish to thank all those who participated in our engagements. We are now using the feedback and advice to shape the overall design and features of the interactive indicators website. These are the next steps in the project: Indicator Selection In order to provide Canadians with the most reliable available measures of health system performance, CIHI is working with a group of experts from across the country to select health indicators for inclusion in the public web site. The experts will provide their advice on the best available indicators in the areas identified as most important to the public. Criteria for selection also include validity, reliability, and whether the indicator is pan-canadian in scope. 14

17 Interactive Website Design H+K Strategies, together with CIHI, are currently working to design the interactive website in a way that is visually engaging and easy-to-use, while providing Canadians with the health system performance information of most interest to them. Testing by Canadians We plan to continue to engage Canadians in the indicators website. Once the organization has a preliminary website available, we will be inviting Canadians to test the site and provide us with feedback about their experience. Website Launch CIHI will be officially launching the interactive indicators website for use by Canadians in the fall of Production of this report is made possible by financial contributions from Health Canada and provincial and territorial governments. The views expressed herein do not necessarily represent the views of Health Canada or any provincial or territorial government. 15

18 APPENDIX INTRODUCTION This appendix contains more information about our online and in-person methodologies, including sampling and weighting, as well as more results from the online Choicebook and in-person dialogues. It can be found as a companion document on the CIHI website. 16