The Nursing Home Five Star Rating: How Does It Compare to Resident and Family Views of Care?

Transcription

1 The Gerontologist Advance Access published May 21, 2014 The Gerontologist, 2014, Vol. 00, No. 00, 1 10 doi: /geront/gnu043 Research Article The Nursing Home Five Star Rating: How Does It Compare to Resident and Family Views of Care? Anthony Williams, MGS, MBA, 1, * Jane K. Straker, PhD, 2 and Robert Applebaum, PhD 2 1 Otterbein Senior Lifestyle Choices, Otterbein Retirement Community, Lebanon, Ohio. 2 Scripps Gerontology Center, Miami University, Oxford, Ohio. *Address correspondence to Robert Applebaum, PhD, Scripps Gerontology Center, Miami University, 396 Upham Hall, Oxford, OH applebra@miamioh.edu Received January ; Accepted April Decision Editor: Rachel Pruchno, PhD Abstract Purpose of the Study: In 2008, the Centers for Medicare and Medicaid Services (CMS) implemented a five-star rating system of nursing homes in the United States. These star ratings have been widely publicized both by CMS and the national and state media. Although the components of the star rating system take into account various dimensions of quality, satisfaction of nursing home residents and their families is not taken into consideration. Design and Methods: The current study compares the CMS star rating system to nursing home satisfaction data reported by residents and their families in Ohio. Results: Findings indicate that the star rating system does not adequately reflect consumer satisfaction. Implications: We recommend that the star system be refined to include a consumer component. Key words: Long-term care, Nursing homes, Quality of care, Quality of life, CMS data sets (OSCAR, MDS) In 1998, the Centers for Medicare and Medicaid Services (CMS) launched an online quality report card that was designed to provide consumers with information on every Medicare- and Medicaid-certified nursing home in the United States ( html; Castle, 2009; Castle & Ferguson, 2010). This report card, called Nursing Home Compare (NHC), originally reported only on basic nursing home characteristics but has since grown to include an array of quality information (Grabowski & Town, 2011). In 2002, CMS launched a nationwide media campaign consisting of television and newspaper advertisements that promoted NHC as a major source of information on nursing home quality (Grabowski & Town, 2011; Office of Inspector General, 2004). Today, the site publicly rates nursing homes on a five-star scale based on data related to staffing, performance on quality indicators taken from Minimum Data Set (MDS) information, and information gathered through health inspections (Centers for Medicare and Medicaid Services [CMS], 2010). Although these measures include important elements of nursing home quality, two crucial pieces of information missing are data on family and resident satisfaction. Little research has been done to study how the CMS star ratings compare to the satisfaction scores of residents and family The Author Published by Oxford University Press on behalf of The Gerontological Society of America. All rights reserved. For permissions, please journals.permissions@oup.com. Page 1 of 10

2 The Gerontologist, 2014, Vol. 00, No. 00 Page 2 of 10 members. Simply put, do facilities with high star ratings also report high levels of resident and family satisfaction? Quality Indicators and Report Cards Quality indicators for nursing homes have often been subject to the criticism that they are too heavily focused on clinical aspects and lack focus on quality of life measures (Castle & Ferguson, 2010). The current quality measures reported on the NHC website all focus on resident s physical and clinical services with none examining resident quality of life (CMS, 2010). Although clinical data are undoubtedly important to examine in nursing homes, for many residents the facility is where they live for a long period of time, making the evaluation of quality of life measures important as well. Quality report cards are often designed with the intention of helping consumers compare quality across nursing homes, especially among dimensions that may not be easily observable when visiting a facility (Castle & Lowe, 2005; Mukamel & Spector, 2003; Mukamel et al., 2007; Stevenson, 2006). They can serve as valuable tools to consumers when they offer accurate and useful data to compare indicators of quality across nursing homes (Castle & Lowe, 2005; Mukamel & Spector, 2003; Stevenson, 2006). The NHC star rating system composed of three main components state health surveys, quality data from the national minimum data set, and staffing information has received criticism from providers, consumers, and researchers. The health inspection domain has been criticized for using survey data that have poor reliability and high levels of variability. A 2003 report published by the Office of the Inspector General concluded that there was variability in how states went about determining citation practices as a result of inconsistent survey focuses, high staff turnover, unclear guidelines, and the lack of a common review process for draft survey reports (Office of Inspector General, 2003). More recently, a study comparing CMS federal surveys to state inspections found that 7 in 10 state surveys missed at least one deficiency citation; the average number of deficiencies missed also varied significantly across states (Government Accountability Office [GAO], 2008). When examining variations in inspection data, a recent study found that different survey districts within states also had variability beyond what could be explained simply by regional variations in quality (Miller & Mor, 2008). Within the quality domain, the primary concern is that the data are based on the MDS, which is collected and submitted by nursing home staff. Nursing home administrators argue that variations in data coding, unaccounted differences in case mix, and unusual events affect performance on these quality measures (Mukamel et al., 2007; Rahman & Applebaum, 2009). Research has backed these beliefs, identifying variability between facilities in the way that MDS resident assessments are completed as well as the presence of potential issues with the risk adjustment performed on the data (Mor, 2005). The staffing domain is also subject to the self-report criticism faced by the quality domain. There has also been concern that the 2-week period from which staffing data are obtained may not be representative of staffing that occurs year round (PointRight Inc., 2008). A review of nursing home report cards conducted in 2009 found six states publicly reporting family and/or resident satisfaction data; of these, only two states, Rhode Island and Ohio, reported satisfaction data from both groups (Castle, Diesel, & Ferguson, 2011). Despite the increased use of resident and family satisfaction measures, our review found only three studies that have addressed this linkage. One of the studies reviewed investigated the relationship between NHC ratings and family satisfaction data in Maryland (Çalıkoğlu et al., 2011). Results suggested a relationship between the family satisfaction data and NHC overall rating as well as the health inspection and staffing domains, whereas no relationship was found between family satisfaction and the NHC quality domain. The second examined the relationship between both family and resident satisfaction scores from a small sample of nursing homes as they related to health inspection deficiency citations received by the same facilities. Results showed a moderate positive correlation between both family and resident satisfaction data and the number of deficiency citations received by nursing homes (Tellis-Nayak, Shiverick, & Hernandez, 2010). Finally, one recent study with 32 nursing homes in the greater Detroit area compared quality of life indicators with the CMS Star rating (Kim et al., 2014). The study reported that quality of life elements of nursing home living, such as privacy, autonomy, and relationships with support workers, was not associated with the CMS star rating of a facility. Despite the recognized importance of consumer input, there are some longstanding challenges associated with such efforts (Applebaum, Straker, & Geron, 2000; Uman et al., 2000). A lack of consensus on how to define and measure satisfaction has been a consistent barrier to success. Concerns about respondent bias both as a result of social desirability pressures and consumer fear that a negative response could affect services received represent ongoing difficulties (Applebaum, Uman, & Straker, 2006). A major concern in the satisfaction literature and a challenge for this work is the tendency for satisfaction to be positively skewed. Data from the 2010 Ohio Nursing Home Family Satisfaction Surveys used in this study are on average positive, but the survey does demonstrate a range of responses. For example, the overall facility

3 Page 3 of 10 The Gerontologist, 2014, Vol. 00, No. 00 satisfaction scores ranged from 67.9 to 99.6 with a mean of Four items had an average score below 80%, 19 items scored 90% or higher, and 27 items scored between 76% and 89%. (Straker, Chow, Mwangi, & Reddecliff, 2011). By comparison the most recent Ohio Department of Health survey found facility deficiency compliance rates ranging from 53.7% to 100%, with an average of 86.2% (LTCOhio.org). Methods Several data sources were utilized in the current study. These included family and resident satisfaction scores that were collected for inclusion in Ohio s Long-Term Care Consumer Guide, as well as archived data that were previously published on NHC in An overview of each of these data sets is provided subsequently. The Ohio Nursing Home Resident and Family Satisfaction Surveys were developed in 2001 for public reporting of satisfaction information on Ohio s long-term care consumer guide ( They have undergone extensive testing and refinement since first administered over a decade ago (Ejaz, & Castle, 2007; Ejaz, Straker, Fox, & Swami, 2003; Straker, Ejaz, McCarthy, & Jones, 2007; Wheatley et al., 2007). The surveys are administered to residents and families in alternating years. The 2010 Ohio Nursing Home Family Satisfaction survey consisted of 17 background questions and 54 close-ended satisfaction questions in11 domains. The domains included were activities, administration, admissions, choices, direct care and nursing staff, environment, laundry, meals and dining, social services, therapy, and general questions. Nursing homes mailed written survey packets to families of their current residents. Surveys were returned from family members of residents in 931 (97%) of the 961 nursing homes provided with survey packets. The total number of family surveys returned was 29,873; this is estimated to be about a 47% response rate (Straker et al., 2011). The 2009 Ohio Nursing Home Resident Satisfaction Survey consisted of 1 screening question and 50 close-ended satisfaction questions. These questions encompassed 10 domains including activities, administration, choice, direct care and nurse assistants, environment, laundry, meals and dining, social services, therapy, and overall/general satisfaction. Surveys were administered to consenting residents in a private location through face-to-face interviews conducted by interviewers. Residents were randomly selected from a census provided 2 weeks prior to the interview team s visit. Residents who were in isolation or whose legal guardian declined participation on the behalf of the resident were not included in the sampling process. In all, 955 Ohio facilities participated in the resident survey. Within these facilities, a total of 28,412 interviews were requested with residents and 23,462 residents were willing to participate, yielding a resident participation rate of 83%. Interviews were completed with 22,822 residents. Started, but incomplete, interviews resulted from resident inability to respond (n = 347), resident refusal to continue (n = 137), fatigue (n = 72), and other reasons (n = 84; Vital Research, 2010). Overall satisfaction scores from families and residents are calculated as an average of all resident and family responses to all survey items. In addition to use on the consumer guide, overall satisfaction scores are also part of a quality payment formula used for Ohio Medicaid nursing facility reimbursement. Our second major source of data is NHC, which was launched in 1998 by CMS to provide consumers with information on each Medicare- and Medicaid-certified nursing home in the United States (Castle, 2009; Castle & Ferguson, 2010). Initially, the report only provided information on deficiency citations, facility characteristics, and resident characteristics (Castle, 2009; Stevenson, 2006). In 2000, staffing information was also added to NHC (Castle, 2009). Following a pilot study in six states, including Ohio, quality measures were added nationally to the report card in 2002 (Castle, 2009). These measures have changed over time and currently nine quality measures are reported; seven long stay resident items examining activities of daily living, pressure sores, catheter use, physically restraints, urinary tract infection, moderate to severe pain and falls with major injury, and two short stay resident measures, pressure ulcers, and moderate or severe pain. (Castle, 2009; Castle & Ferguson, 2010; CMS, 2010). Through consultation with an expert panel consisting of individuals from academia, patient advocacy groups, and nursing home provider groups, CMS developed and implemented a significant change to the NHC system in December of 2008 (CMS, 2008). This change was the implementation of a five-star rating system on NHC. These star ratings serve as the primary NHC data utilized for this study. This system includes an overall star rating developed from ratings in three domains: (a) Health inspections, (b) quality measures (QMs), and (c) staffing (CMS, 2010). Each rating is based on a five-point scale with 1 representing much below average and 5 representing much above average (CMS, 2010). NHC overall and domain star ratings from 2009 and 2010 (CMS, 2009) were merged with satisfaction data from 2009 and 2010 for residents and families, respectively. After cleaning, the final data set consisted of 918 nursing homes, with 880 facilities (95.9%) reporting from all four primary datasets. To align the NHC data with the family satisfaction results, we used the NHC information that was

4 The Gerontologist, 2014, Vol. 00, No. 00 Page 4 of 10 archived in December Although MDS 3.0 was implemented in 2010, the QM component of the five-star system was held constant until July 2012 and did not affect our analyses (Abt Associates, 2013). The revised MDS 3.0 does include a greater emphasis on direct resident communication and has a more resident centered focus, suggesting that a replication analysis based on the revised version would be warranted. Results Overall Star Rating in Relation to Consumer Satisfaction As noted, the overall star rating is calculated by combining a facility s score on the health inspection, quality measures, and staffing domains. Averages from the resident and family satisfaction scores were then calculated for the facilities included in each of the five-star categories. Mean scores for both resident and family respondents tended to increase as the number of stars received increased (See Table 1). Resident satisfaction scores ranged from 84.2 (one star) to 87.2 (five stars). Mean family satisfaction scores ranged from 84.5 in one star facility to 89.8 in five star nursing homes. Despite this finding, a comparison of satisfaction averages associated with each star rating found that mean satisfaction scores did not always vary from the star above or below (e.g., one star did not have a significantly different mean satisfaction score than two stars, etc.). For example, Table 1 includes p values that test for significant differences between mean scores across the five-star groupings. For the resident satisfaction data, mean satisfaction scores at one star were significantly less than the mean satisfaction scores at three, four, and five stars. No other significant differences emerged. For family satisfaction, the mean score at one star was significantly less than the mean satisfaction scores at three, four, and five stars, but not significantly different from the mean satisfaction score at two stars. Similarly, the mean satisfaction score at five stars was significantly greater than the mean satisfaction scores at three, two, and one star, but not significantly different from the mean satisfaction score at four stars. To further examine the relationship between satisfaction and star ratings, facilities were divided into quintiles based on their resident and family satisfaction scores. The five categories ranged from very low to very high satisfaction and used the same sample distribution proportions as the CMS five star breakdown. As is done in the star rating, the top 10% were in the highest quintile, the bottom 15% were in the lowest quintile, and the middle 70% were divided proportionally into quintiles two through four. Comparisons between consumer satisfaction categories and the NHC overall ratings that one might expect to correspond to the greatest extent (e.g., very low satisfaction and one star ) are indicated by boxes within each table. As shown in Table 2, of the nursing homes that received one star on the NHC overall rating, 59.9% were appropriately categorized as having either very low or low resident satisfaction. In contrast, 19.3% that received one star on the NHC overall rating were categorized as having high or very high resident satisfaction. Of the nursing homes that received five stars on Table 1. Satisfaction by Overall Star Rating for Ohio Nursing Homes Overall star rating Resident satisfaction mean p value between resident satisfaction means 1 star **.000**.000** 2 stars stars stars stars Overall star rating Family satisfaction mean p value between family satisfaction means 1 star **.000**.000** 2 stars **.000**.000** 3 stars * 4 stars stars Notes: Resident data from 2009 Nursing Home Compare and 2009 Ohio Nursing Home Resident Satisfaction Surveys. N = 907. Family data from 2010 Nursing Home Compare and 2010 Ohio Nursing Home Family Satisfaction Surveys. N = 892. *p <.05. **p <.01.

5 Page 5 of 10 The Gerontologist, 2014, Vol. 00, No. 00 Table 2. A Comparison of Satisfaction Categories by Overall Star Rating for Ohio Nursing Homes Overall star rating Resident satisfaction category Very low satisfaction 36.2% 24.1% 21.8% 20.6% 19.7% Low satisfaction 23.7% 27.1% 22.3% 17.1% 19.7% Moderate satisfaction 20.7% 22.2% 22.8% 24.1% 14.5% High satisfaction 15.9% 22.7% 23.4% 27.6% 19.7% Very high satisfaction 3.4% 3.9% 9.6% 10.6% 26.3% Total percentage 100.0% 100.0% 100.0% 100.0% 100.0% Number with star rating Family satisfaction category Very low satisfaction 41.1% 27.6% 11.9% 11.0% 5.1% Low satisfaction 22.2% 25.5% 27.3% 15.9% 14.3% Moderate satisfaction 16.7% 22.9% 21.1% 25.6% 21.4% High satisfaction 13.9% 17.7% 30.9% 31.7% 36.7% Very high satisfaction 6.1% 6.3% 8.8% 15.9% 22.4% Total percentage 100.0% Number with star rating Notes: Resident data from 2009 Nursing Home Compare and 2009 Ohio Nursing Home Resident Satisfaction Surveys. N = 907. Family data from 2010 Nursing Home Compare and 2010 Ohio Nursing Home Family Satisfaction Surveys. N = 892. the NHC overall rating, 46% were categorized as having either very high satisfaction or high satisfaction. In contrast, 53.9% receiving five stars in the NHC overall rating were categorized as having moderate, low, or very low resident satisfaction. Table 2 also shows that nursing homes that received one star on the NHC overall rating were also categorized as having very low family satisfaction in four of 10 facilities (41.1%). Of those that received one star, 63.3% were categorized as having either very low or low family satisfaction. In contrast, 20.0% that received one star on the NHC overall rating were categorized as having high or very high family satisfaction. Of those that received five stars, 59.1% were categorized as having either very high satisfaction or high satisfaction. In contrast, 40.8% that received five stars on the NHC overall rating were categorized as having moderate, low, or very low family satisfaction. Nearly 6 of 10 five star facilities were appropriately categorized. Star Rating by Subareas As noted the CMS five-star rating is composed of three sub areas: A health inspection, based on the state survey, a quality rating based on indicators collected in the MDS, and direct care staffing levels. In the following sections, we compare resident satisfaction results to each of the individual subareas to see whether a specific area had a different outcome compared with resident and family satisfaction. The results show a similar pattern across each of the three areas. We provide the detailed data tables as an example in the health area, but the tables have not been included for the quality and staffing areas. Health Inspection Domain in Relation to Consumer Satisfaction The health inspection domain includes information based on 3 years of standard and complaint surveys at each facility. Ratings reflect the criteria that facilities must meet to achieve Medicare certification (CMS, 2010). Table 3 provides mean comparisons for family and resident satisfaction data across levels of star ratings within the NHC health inspection domain. These tables also show mean satisfaction scores at each star rating level, as well as provide an overview of where significant differences exist between the mean satisfaction scores at different levels. Mean satisfaction scores for both family and resident data tended to increase as the number of stars received in the NHC health inspection domain increased. As shown in Table 4, the pattern of results was similar to the overall findings. Within the resident data, of the nursing homes that received one star in the NHC health inspection domain, 58.3% were categorized as having either very low satisfaction or low resident satisfaction. In contrast, 22.1% that received one star in the NHC health inspection domain

6 The Gerontologist, 2014, Vol. 00, No. 00 Page 6 of 10 Table 3. Satisfaction by Health Inspection Star Rating for Ohio Nursing Homes Health inspection star rating Resident satisfaction mean p value between resident satisfaction means 1 star **.000**.000*** 2 stars *.018* 3 stars ** stars ** stars ** Health inspection star rating Family satisfaction mean p value between family satisfaction means 1 star **.000**.000**.000** 2 stars *.000** 3 stars stars stars Notes: Resident data from 2009 Nursing Home Compare and 2009 Ohio Nursing Home Resident Satisfaction Surveys. N = 907. Family data from 2010 Nursing Home Compare and 2010 Ohio Nursing Home Family Satisfaction Surveys. N = 892. *p <.05. **p <.01. ***p <.001. Table 4. A Comparison of Satisfaction Categories by Health Inspection Rating for Ohio Nursing Homes Health inspection star rating Resident satisfaction category Very low satisfaction 35.1% 20.7% 16.2% 13.4% 15.2% Low satisfaction 23.2% 27.6% 25.7% 20.4% 20.3% Moderate satisfaction 19.5% 26.3% 27.1% 23.1% 12.7% High satisfaction 17.8% 21.7% 23.3% 29.6% 29.1% Very high satisfaction 4.3% 3.7% 7.6% 13.4% 22.8% Total percentage 100.0% 100.0% 100.0% 100.0% 100.0% Number with star rating Family satisfaction category Very low satisfaction 42.7% 20.7% 10.9% 12.3% 6.0% Low satisfaction 25.7% 27.9% 22.4% 22.4% 12.0% Moderate satisfaction 14.6% 22.1% 26.4% 24.1% 26.5% High satisfaction 9.9% 25.5% 29.9% 29.4% 37.3% Very high satisfaction 7.0% 3.8% 10.4% 11.8% 18.1% Total percentage Number with star rating Notes: Resident data from 2009 Nursing Home Compare and 2009 Ohio Nursing Home Resident Satisfaction Surveys. N = 907. Family data from 2010 Nursing Home Compare and 2010 Ohio Nursing Home Family Satisfaction Surveys. N = 892. were categorized as having high or very high satisfaction. Of the nursing homes that received five stars in the NHC health inspection domain, 22.8% were also categorized as having very high satisfaction. In contrast, 48.2% that received five stars in the NHC health inspection domain were categorized as having moderate, low, or very low resident satisfaction. Among nursing homes that received one star, 68.4% were categorized as having either very low or low family satisfaction. In contrast, 16.9% that received one star in the NHC health inspection domain were categorized as having high or very high family satisfaction. Of the nursing homes that received five stars in the NHC health inspection domain, 55.4% were categorized as having either very high

7 Page 7 of 10 The Gerontologist, 2014, Vol. 00, No. 00 satisfaction or high family satisfaction. In contrast, 44.5% of facilities that received five stars in the NHC health inspection domain were categorized as having moderate, low, or very low family satisfaction. Quality Domain in Relation to Consumer Satisfaction Information included in the quality domain is from the Minimum Data Set, reflects physical and clinical care of residents, and includes such things as the prevalence of pressure sores (CMS, 2012). There was no significant relationship between family or resident satisfaction data and the star rating received in the NHC quality domain. The ranges in both satisfaction surveys were very small, with mean satisfaction scores for family satisfaction ranging from 86.4 (one star) to 87.6 (five stars) and mean satisfaction scores for resident satisfaction ranging from 84.9 (five stars) to 85.8 (two stars). Within the resident data, of the nursing homes that received one star in the NHC quality domain, 23.4% were also categorized as having very low satisfaction. Of those that received one star, 46.8% were categorized as having either very low satisfaction or low satisfaction. In contrast, 23.8% that received one star in the NHC quality domain were categorized as having high or very high satisfaction. Of the nursing homes that received five stars in the NHC quality domain, 10.2% were also categorized as having very high satisfaction. Of those that received five stars, 16.3% were categorized as having either very high satisfaction or high satisfaction. In contrast, 83.7% that received five stars in the NHC quality domain were categorized as having moderate, low, or very low satisfaction. Within the family data, of the nursing homes that received one star in the NHC quality domain, 19.3% were also categorized as having very low satisfaction. Of those that received one star, 46.7% were categorized as having either very low satisfaction or low satisfaction. In contrast, 24.4% that received one star in the NHC quality domain were categorized as having high or very high satisfaction. Of the nursing homes that received five stars in the NHC quality domain, 17.3% were also categorized as having very high satisfaction. Of those that received five stars, 36% were categorized as having either very high satisfaction or high satisfaction. In contrast, 64.0% that received five stars in the NHC quality domain were categorized as having moderate, low, or very low satisfaction. Staffing Domain in Relation to Consumer Satisfaction The staffing rating is based on the average number of hours of care provided by nursing and nursing home aides per resident over a 2-week time period. Mean satisfaction scores for resident satisfaction showed a gradual increase, with the exception of a decrease between one and two stars. Resident satisfaction scores ranged from 84.8 (two stars) to (five stars). Mean satisfaction scores for family satisfaction tended to increase as the number of stars received in the NHC staffing domain increased. Family satisfaction scores ranged from (one star) to 88.9 (five stars). In resident satisfaction comparisons, 51.8% of the nursing homes that received one star in the NHC staffing domain were categorized as having either very low or low satisfaction. In contrast, 27.4% that received one star in the NHC staffing domain were categorized as having high or very high satisfaction. Of the nursing homes that received five stars in the NHC staffing domain, 22.2% were also categorized as having very high satisfaction. Of those that received five stars, 44.4% were categorized as having either very high satisfaction or high satisfaction. In contrast, 55.6% that received five stars in the NHC staffing domain were categorized as having moderate, low, or very low satisfaction. Within the family data, of the nursing homes that received one star in the NHC staffing domain, 56.5% were categorized as having either very low satisfaction or low satisfaction. In contrast, 20.6% that received one star in the NHC staffing domain were categorized as having high or very high satisfaction. Of the nursing homes that received five stars in the NHC staffing domain, 51.6% were categorized as having either very high satisfaction or high satisfaction. In contrast, 48.4% that received five stars in the NHC staffing domain were categorized as having moderate, low, or very low satisfaction. Exploring the Relationship Between Common Domains A final piece of the analysis involved a comparison of the specific star subarea with resident and family items that corresponded to that specific domain. For example, some of the resident and family items asked specifically about direct care staff at the facility and we compared the results of those questions to the star staffing rating. As shown in Table 5, about half of the facilities receiving one star on the staffing domain were in the two lowest satisfaction categories. Four in 10 were in the two highest satisfaction quintiles. In looking at the facilities receiving a five-star rating for staffing, 55% had residents in the two highest satisfaction categories. These patterns were similar for the resident survey and across each of the other subareas. Implications for the Overall Star Rating The overall star rating on NHC receives considerable attention from the media and consumers. In some years CMS

8 The Gerontologist, 2014, Vol. 00, No. 00 Page 8 of 10 Table 5. Resident Direct Care Staffing Category by Staffing Star Rating for Ohio Nursing Homes Resident satisfaction category Staffing star rating Very low satisfaction 25.7% 25.8% 26.5% 21.8% 24.5% Low satisfaction 25.7% 23.6% 16.1% 17.9% 19.0% Moderate satisfaction 16.4% 20.6% 30.0% 22.2% 22.1% High satisfaction 29.3% 26.6% 26.1% 35.0% 30.8% Very high satisfaction 2.9% 3.4% 1.3% 3.0% 3.6% Total percentage 100.0% 100.0% 100.0% 100.0% 100.0% Number with star rating Notes: Data from 2009 Nursing Home Compare and 2009 Ohio Nursing Home Resident Satisfaction Surveys. Totals may not sum up to exactly 100% due to rounding. has posted results in major newspapers across the nation. Given how the star rating is used, the relationship to resident and family satisfaction ratings is a critical question to both CMS and to consumers. Although it was promising to find that nursing homes receiving higher ratings on NHC tended to have higher consumer satisfaction scores, the limited relationship between consumer satisfaction scores and the NHC star ratings is cause for concern. For example, although the mean satisfaction score for those nursing homes receiving one star in the NHC overall domain was significantly lower than the mean satisfaction score for those nursing homes receiving five stars, the categorical comparisons between the NHC overall ratings and the consumer satisfaction categories demonstrated the inconsistencies that exist. Many nursing homes that received five stars on the NHC overall rating had moderate to very low consumer satisfaction (41% compared with family satisfaction and 54% compared with resident satisfaction), and many nursing homes that received one star on the NHC overall rating had high to very high consumer satisfaction (20.0% compared with family satisfaction and 19% compared with resident satisfaction). With such high rates of inconsistency between the NHC overall ratings and consumer satisfaction scores, it is difficult to say that consumer s views of nursing homes are adequately reflected in the NHC overall rating. The three NHC domains that make up the NHC overall rating vary in how well they reflect consumer satisfaction ratings. The health inspection and staffing domains were at least generally similar to consumer ratings; that is, satisfaction scores were often higher, but not significantly, for nursing homes receiving more stars within each of these two domains. Within the health inspection and staffing domains, family mean satisfaction scores at one star were significantly lower than mean satisfaction scores at five stars. Within the health inspection domain, the mean resident satisfaction score at one star was also significantly lower than mean satisfaction score at five stars. This suggests that, to some extent, the ratings in these domains are able to detect consumer satisfaction at the extremes. Those scenarios where significant differences between one star mean satisfaction scores and five star mean satisfaction scores were not present require further examination. It is possible that the resident mean satisfaction scores in the staffing domain are not significantly different due to the varying perceptions of staffing between families and residents. Although families may observe the quantity of staff in a facility and thus make a similar rating as that on NHC, residents likely focus more on the quality of care they are receiving or the relationships they have with staff, regardless of the quantity of staff present. The quality domain, which poorly reflected satisfaction among both family and residents, may vary due to the different nature of the dimensions of quality being assessed (e.g., clinical care vs. quality of life). All three of these domains demonstrated a great deal of inconsistency in the categorical comparisons. Within the health inspection and staffing domains, the levels of disagreement ranged from 16.9% to 55.6%. The level of disagreement within the quality domain was even higher, ranging from 23.8% to 89.8%. At these levels, inferring how satisfied consumers might be with nursing homes based on NHC star ratings is not viable. Family and Resident Views In comparing the overall family and resident satisfaction scores, the correlation while statistically significant was.35. The relatively low correlation between family and resident satisfaction scores is likely due to the differing perspectives of each consumer group. Residents may provide subjective views of their experiences, (e.g., perceived quality of staff), whereas family members may be looking to more objective measures (e.g., quantity of staff) to establish their views. It is possible to see this phenomenon occurring by examining the levels of disagreement within the categorical

9 Page 9 of 10 The Gerontologist, 2014, Vol. 00, No. 00 comparisons. As family members in most cases had higher levels of agreement with the NHC ratings, it is possible to assume that family member opinions focus more on those dimensions of quality captured by the NHC system than do residents. This finding suggests that although furthering efforts to obtain input from both families and residents is important, future efforts might be more valuable if they give priority to obtaining input from residents rather than families because families are more aligned with the information already captured in Nursing Home Compare. Rationale for Differences and Potential Solutions for CMS There are a number of possible reasons why the ratings provided on NHC do not accurately reflect consumer satisfaction. First and foremost, the NHC star rating system is not designed to directly assess consumer satisfaction. Rather, it is focused on assessing other dimensions of quality. Although one might argue that performance on health inspections, the quality of clinical care, and staffing might be predictive of consumer satisfaction, this study shows that those measures are by no means a substitute for consumer input. Additionally, performance on the measures utilized in the calculations of the NHC ratings may not adequately reflect year-round nursing home practice. In addition to those topics discussed in the literature review, a number of factors can account for the discrepancies between NHC and the consumer satisfaction ratings. For instance, nursing homes generally know the time frame when state surveyors will arrive. They can prepare for this, add extra staffing, and correct issues that may be faulty during other times throughout the year. As a result, family and residents may be assessing nursing homes based on their long-term experiences with facilities, whereas NHC is only assessing nursing homes based on data collected at limited periods in time. As such, nursing home consumer satisfaction may have the ability to offer a more holistic view of consumers experiences within nursing homes compared with the information offered on NHC. At a minimum, the complete exclusion of consumer satisfaction data from NHC is cause for concern. In order to serve as a more comprehensive and reliable resource for consumers, CMS could work to better integrate the thoughts and opinions of consumers into its NHC star rating system. If large-scale surveying, such as that conducted in Ohio, is not a viable option for nationwide implementation, residents should at least be provided the opportunity to provide some input into how their nursing homes are rated. Such information could be obtained by introducing a satisfaction component to the state survey process wherein, at the very least, surveyors could ask a core set of satisfaction questions to a sample of the residents in each nursing home. Although incorporating resident satisfaction into the star ratings presents some major challenges, these data indicate that the failure to include consumers is a serious limitation in the current star rating system. References Abt Associates & Colorado Foundation for Medical Care. (2013). Nursing home compare five-star quality rating system: Year three report. Baltimore, MD: Centers for Medicare and Medicaid Services. Retrieved March 10, 2013, from Certification/CertificationandComplianc/Downloads/FSQRS- Report.pdf Applebaum, R. A., Straker, J. K., & Geron, S. (2000). Assessing satisfaction in health and long-term care. New York, NY: Springer Publishing Company Applebaum, R. A., Uman, G. C., & Straker, J. K. (2006). Capturing the voices of consumers in long-term care: If you ask them they will tell. In S. R. Kunkel & V. Wellin (Eds.), Consumer voice and choice in long-term care (pp ). New York, NY: Springer Publishing Company. Çalıkoğlu, Ş., Christmyer, C. S., & Kozlowski, B. U. (2011). My eyes, your eyes the relationship between CMS five-star rating of nursing homes and family rating of experience of care in Maryland. Journal of Health Care Quality, 34, doi: /j x Castle, N. G. (2009). The nursing home compare report card: Consumers use and understanding. Journal of Aging and Social Policy, 21, doi: / Castle, N. G., Diesel, J., & Ferguson, J. C. (2011). The evolution of nursing home report cards. Journal of Applied Gerontology, 30, doi: / Castle, N. G., & Ferguson, J. C. (2010). What is nursing home quality and how is it measured? The Gerontologist, 50, doi: /geront/gnq052 Castle, N. G., & Lowe, T. J. (2005). Report cards and nursing homes. The Gerontologist, 45, doi: /geront/ Centers for Medicare and Medicaid Services (CMS). (December 2008). Questions and answers: Improving the Nursing Home Compare web site: The five-star nursing home quality rating system: Volume 1. Retrieved from /CertificationandComplianc/Downloads/QsandAsFinal.pdf Centers for Medicare and Medicaid Services (CMS). (December 23, 2009). Nursing Home Compare Archived Dataset [Archived Dataset]. Retrieved from /qualityinitiatives- geninfo/amcd/itemdetail.asp?filtertype=none&filterbydid=- 99&sortByDID=1&sortOrder= ascending&itemid=cms &intNumPerPage=10 Centers for Medicare and Medicaid Services (CMS). (July 2010). Design for nursing home compare five-star quality rating system Technical users guide. Retrieved October 18, 2011, from gov/certificationandcomplianc/downloads/usersguide.pdf Centers for Medicare and Medicaid Services (CMS). (2012). Nursing home compare. Retrieved January 10, 2012, from Medicare.gov/nhcompare/

10 The Gerontologist, 2014, Vol. 00, No. 00 Page 10 of 10 Ejaz, F. K., & Castle, N. G. (2007). Resident satisfaction with longterm care services. Journal of Aging and Social Policy, 19, 1 8. doi: /j031v19n02_01 Ejaz, F. K., Straker, J. K., Fox, K., & Swami, S. (2003). Developing a satisfaction survey for families of Ohio s nursing home residents. The Gerontologist, 43, doi: /geront/ Government Accountability Office. (May 2008). Federal monitoring surveys demonstrate continued understatement of serious care problems and CMS oversight weaknesses (Publication No. GAO ). Retrieved from d08517.pdf Grabowski, D. C., & Town, R. J. (2011). Does information matter? Competition, quality, and the impact of nursing home report cards. Health Services Research, 46, doi: /j x H.B. 403, 123rd Gen. Assem., Reg. Sess. (OH. 2000). Kim, S. J., Park, E., Kim, S., Nakagawa, S., Lung, L., Choi, J. B., Yoo, J. W. (2014). The association between quality of care and quality of life in long-stay residents with preserved cognition. Journal of the American Medical Directors, 15, doi: /j.jamda Miller, E. A., & Mor, V. (2008). Balancing regulatory controls and incentives: Toward smarter and more transparent oversight in long-term care. Journal of Health Politics, Policy and Law, 33, doi: / Mor, V. (2005). Improving the quality of long-term care with better information. Milbank Quarterly, 83, doi: /j x Mukamel, D. B., & Spector, W. D. (2003). Quality report cards and nursing home quality. The Gerontologist, 43(Special Issue II), doi: /geront/43.suppl_2.58 Mukamel, D. B., Spector, W. D., Zinn, J. S., Huang, L., Weimer, D. L., & Dozier, A. (2007). Nursing homes response to the nursing home compare report card. Journal of Gerontology: Social Sciences, 62B, S218 S225. Office of Inspector General. (2003). Nursing home deficiency trends and survey and certification process consistency (OIG publication No. OEI ). Washington, DC: Office of Inspector General. Retrieved from oei pdf Office of Inspector General. (2004). Inspections results on nursing home compare: Completeness and accuracy (OIG publication No. OEI ). Washington, DC: Office of the Inspector General. Retrieved from PointRight Inc. (2008). Review of CMS five-star nursing home rating system [PowerPoint Slides]. Retrieved from facility_operations/survey_certification /Documents/Review%20 of%20cms%205%20star%20rating%20system.pdf Rahman, A. N., & Applebaum, R. A. (2009). The nursing home minimum data set assessment instrument: Manifest functions and unintended consequences Past, present, and future. The Gerontologist, 49, doi: /geront/gnp066 S.B. 264, 129 th Gen. Assem., Reg. Sess. (OH. 2011) Stevenson, D. G. (2006). Is a public reporting approach appropriate for nursing home care? Journal of Health Politics, Policy, and Law, 31, doi: / Straker, J. K., Chow, K., Mwangi, S., & Reddecliff, L. (2011). Implementation of the 2010 Ohio nursing home family satisfaction survey. Oxford, OH: Miami University. Straker, J. K., Ejaz, F. K., McCarthy, C., & Jones, J. A. (2007). Developing and testing a satisfaction survey for nursing home residents. Journal of Aging and Social Policy, 19, doi: /j031v19n02_05 Tellis-Nayak, V. V., Shiverick, B. N., & Hernandez, M. (2010). Where allies part ways and strangers converge: Nursing home performance in the eyes of residents, families, and state surveyors. Seniors Housing & Care Journal, 18, Uman, G. C., Hocevar, D., Urman, H. N., Young, R., Hirsch, M., & Kohler, S. (2000). Satisfaction surveys with the cognitively impaired. In J. Cohen-Mansfield, F. K. Ejaz, & P. Werner (Eds.), Satisfaction surveys in long-term care (pp ). New York, NY: Springer Publishing Company. Vital Research. (2010). Implementation of the 2009 long-term care resident satisfaction survey. Los Angeles, CA: Vital Research, LLC. Wheatley, M. V., Choi, K., Hirsch, M., Walley, J. E., Lee, C. S., Urman, H. N., & Uman, G. C. (2007). Implementation and results of the statewide Ohio nursing home resident satisfaction survey. Journal of Aging & Social Policy, 19, doi: / J031v19n02_06