Quality Ratings and Premiums in the Medicare Advantage Market

Transcription

1 Quality Ratings and Premiums in the Medicare Advantage Market Ian M. McCarthy Department of Economics Emory University Michael Darden Department of Economics Tulane University January 2015 Abstract We examine the response of Medicare Advantage contracts to published quality ratings. We identify the effect of star ratings on premiums using a regression discontinuity design that exploits plausibly random variation around rating thresholds. We find that 3, 3.5, and 4-star contracts in 2009 significantly increased their 2010 monthly premiums by $20 or more relative to contracts just below the respective threshold values. High quality contracts also disproportionately dropped $0 premium plans or expanded their offering of positive premium plans. Welfare results suggest that the estimated premium increases reduced consumer welfare by over $250 million among the affected beneficiaries. JEL Classification: D21; D43; I11; C51 Keywords: Medicare Advantage, Premiums, Quality Ratings, Regression Discontinuity Emory University, Rich Memorial Building, Room 306, Atlanta, GA 30322, ian.mccarthy@emory.edu 206 Tilton Memorial Hall, Tulane University, New Orleans, LA mdarden1@tulane.edu 1

2 1 Introduction The role of Medicare Advantage (MA) plans in the provision of health insurance to Medicare beneficiaries has grown substantially. Between 2003 and 2014, the share of Medicare eligible individuals in an MA health plan increased from 13.7% to 30%. 1 To better inform enrollees of MA quality, in 2007, the Center for Medicare and Medicaid Services (CMS) introduced a five-star rating system that provided a rating of one to five stars to each MA contract a private organization that administers potentially many differentiated plans in each of five quality domains. 2 For the 2009 enrollment period, CMS began aggregating the domain level quality scores into an overall star rating for each MA contract in which each plan offered by a contract would display the contract s quality star rating. Since in 2012, contracts have been incentivized to earn high quality star ratings through star-dependent reimbursement and bonus schemes. Early studies on the effects of the star rating program focus on the informational benefits to Medicare beneficiaries. To this end, the program has been found to have a relatively small positive effect on beneficiary choice, with heterogeneous effects across star ratings (Reid et al., 2013; Darden & McCarthy, forthcoming). However, one area thus far overlooked concerns the supply-side response to MA star ratings, where a natural consequence of the star rating program could be for contracts to adjust premiums and other plan characteristics in response to published quality ratings. 3 Indeed, while the quality star program is often presented as a potential information shock to enrollees, the program could also serve as an information shock to health insurance contracts, better informing them of competitor quality and better informing contracts of their own signal of quality to the market. For example, learning that its plans have the highest quality star rating in a market in 2009, a contract may choose to price out its quality advantage in 2010 by raising plan premiums. Conversely, a relatively low-rated contract may lower its 2010 premium in response to its 2009 quality star rating. More generally, the extent to which policy may cause health insurance companies to adjust premiums is a central question in health and public economics. 4 The current paper provides a comprehensive analysis of 2010 premium adjustments to the 2009 publication of MA contract quality stars. We investigate the specific mechanisms by which contracts can adjust their premiums in response to their quality ratings, and we calculate the corresponding welfare effects. We adopt a regression discontinuity (RD) design that exploits plausibly random variation around 2009 star thresholds, allowing us to separately identify the effect of reported quality on price 1 Kaiser Family Foundation MA Update, available at 2 For example, one domain on which contracts were rated was Helping You Stay Healthy. 3 Preliminary evidence of a supply-side response to the publication of MA quality stars was found in Darden & McCarthy (forthcoming), albeit with a restricted sample of contract/plan/county/year observations. 4 For example, see Pauly et al. (2014) on the effects of the Affordable Care Act on individual insurance premiums. 2

3 from the overall relationship between quality and price. Our data on contract/plan market shares, reported contract quality, plan premiums, and other plan characteristics come from several publicly available sources. Our results suggest strong premium adjustments following the 2009 star rating program, with average to above average star-rated contracts significantly increasing premiums from 2009 to When we conduct our analysis at the contract level, we find that 3, 3.5, and 4-star contracts increase their average premiums across existing plans by $33.60, $29.30, $31.85, respectively, relative to contracts with 2009 ratings just below the respective threshold values. At the plan level, we estimate mean increases of $19.40, $41.99, and $31.52 for 3, 3.5, and 4-star contract/plans, respectively. These effects are sizable compared to overall average premium increases of between $9 and $15. The results are also broadly consistent across a range of sensitivity analyses, including consideration of alternative bandwidths, falsification tests with counter-factual threshold values, and the exclusion of market-level covariates. While an MA contract may directly adjust its plans premiums in response to quality stars, the contract may also adjust the mix of plans it offers within a market (county). For example, in response to the published star ratings, a contract could alter the number of zero-premium plans; adjust the number of plans that include Medicare Part D coverage; change the drug deductible in plans that offer part D coverage; or add/drop plans entirely. Indeed, our data show that nearly all of the regional variation in plan premiums is due to selection of plan offerings by contracts, as opposed to contracts charging different premiums in different areas of the country. We find that contracts just above the 3 and 3.5-star thresholds in 2009 are more likely to drop $0 premium plans in 2010, with 3.5-star contracts also more likely to introduce positive premium plans into new markets. We find no such disproportionate change in $0 or positive premium plans among contracts with a 4-star rating in Meanwhile, low quality contracts (those just above the 2.5-star threshold in 2009) maintain their 2009 plan offerings at largely the same premium levels in 2010, while contracts just below the 2.5-star threshold in 2009 are much more likely to exit the market altogether in Overall, our results suggest that the star rating program in 2009 may have caused low quality contracts to drop plans while generating large premium increases among contracts receiving 3-star ratings and above. Adopting the consumer welfare calculations used in Town & Liu (2003) and Maruyama (2011), our estimated increases in premiums imply a reduction in consumer surplus of over $250 million among those beneficiaries enrolled in the relevant plans. To the extent that higher quality plans are replacing low quality plans at reasonable premium levels, plan entry and exit behaviors induced by the star-rating program may partially offset this welfare loss; however, given the number of new plans estimated to have entered the market due to the star ratings, such offsets are likely relatively small (Maruyama, 2011). 3

4 In what follows, we discuss the institutional details of Medicare Advantage and the recent star rating program in Section 2. The data and methods are discussed in Sections 3 and 4, respectively. We present our results in Section 5, with a series of robustness checks discussed in Section 6. Section 7 examines the potential mechanisms underlying our estimated premium adjustments, and Section 8 summarizes the welfare effects associated with our estimated premium increases. The final section concludes. 2 Institutional Background Since Medicare s inception, beneficiaries have had the option to receive benefits through private health insurance plans. The Balanced Budget Act of 1997 (BBA) classified all private Medicare health insurance plans as Medicare Part C plans, and it allowed for additional types of business models including Preferred Provider Organizations (PPOs), Provider-Sponsored Organizations (PSOs), Private fee-forservice (PFFS) plans, and Medical Savings Accounts (MSAs). Later, in addition to the beneficiary entitlement to prescription drug coverage, the Medicare Modernization Act of 2003 renamed Medicare Part C plans as Medicare Advantage (MA) plans. In each year since 2003, Medicare beneficiaries choose to enroll in traditional fee-for-service (FFS) Medicare or an MA plan during an open enrollment period from November 1st through December 31st. By enrolling in an MA plan, enrollees must pay Medicare Part B premiums in addition to any additional premium charged by the plan. In exchange, MA plans provide at least (often more than) the services covered by traditional FFS Medicare. In 2009, 38% of MA plans charged no additional premium, while 77% of plans also offered prescription drug coverage. Given the generosity of plan coverage at possibly no additional cost relative to traditional Medicare FFS, the MA has grown dramatically in recent years with share of Medicare eligible individuals in an MA plan increasing from 13.7% in 2003 to 30% in Broadly, an MA contract is an agreement between a private insurance company and CMS whereby the company agrees to insure Medicare beneficiaries in exchange for reimbursement. A contract is approved by CMS to operate in specific counties, and an approved contract typically offers a menu of MA plans that are differentiated by premium, prescription drug coverage, and, if covered, the prescription drug deductible. Most MA contracts are required to offer at least one plan that includes prescription drug coverage. For the 2015 enrollment year, 78% of all Medicare beneficiaries live in a county with access to at least one plan that offers prescription drug coverage (MA-PD) and charges no additional premium (above the Part B premium). 6 In 2009, the mean number of MA plans available to 5 Kaiser Family Foundation MA Update, available at

5 beneficiaries was roughly 11 plans per county. 7 However, there exists considerable regional variation in the availability of MA plans, and enrollments in MA plans are concentrated in a few national contracts. Indeed, according to the Kaiser Family Foundation (KFF), 60% of all plans offered in 2015 are affiliated with just seven health insurance companies. 8 Staring in the 2007 enrollment year, CMS began collecting and distributing a one to five-star quality rating in each of five quality domains (e.g., Helping You Stay Healthy ). Each domain was itself an aggregation of many individual quality metrics such as the percentage of enrollees with access to an annual flu vaccine. These individual quality metrics are calculated based on data from a variety of sources, including HEDIS, the Consumer Assessment of Healthcare Providers and Systems (CAHPS), the Health Outcomes Survey (HOS), the Independent Review Entity (IRE), the Complaints Tracking Module (CTM), and CMS administrative data. Starting in enrollment year 2009, CMS began aggregating the domain level quality stars to an overall contract rating of between one and five stars (in half-star increments). 9 And since 2011, CMS constructs the contract-specific quality ratings as a function of Part D coverage, when relevant. Our focus is on the 2009 and 2010 enrollment years - the first two years of the overall contract star rating program and the years in which all contracts, including those offering prescription drug coverage, were rated based on the same underlying quality metrics. The literature on the MA quality rating initiatives has generally focused on the enrollment effects. Recently, Reid et al. (2013) find large effects of increases in star-ratings on enrollment that are homogeneous across the reported quality distribution, but results from that paper fail to disentangle the effects of quality from quality reporting on enrollment. Attempting to disentangle these effects, Darden & McCarthy (forthcoming) find heterogeneous effects of the quality star rating program on MA plan enrollment in 2009 and no significant effect in At the plan level, they find that a marginally higher rated contract at the lower end of the quality distribution (e.g., a 3 as compared to 2.5 starred contract) realized a positive and significant enrollment effect equal to 4.75 percentage points relative to traditional FFS Medicare in 2009 enrollments. This effect diminishes for higher rated contracts, and vanishes for the 2010 enrollment year. The lack of an enrollment response to 2010 quality stars suggests that the 2009 star ratings may have acted as a one-time informational event, or that there was a supply-side response in 2010 based on the 2009 ratings. Generally, the potential for supply-side responses to Medicare Advantage policy has received little attention from researchers. One recent exception is Stockley et al. (2014), who examine how MA plan premiums and benefits respond to variation in the benchmark payment rate - the subsidy received 7 Author s calculation. See Section 3 for a presentation of our data. 8 See 9 For a complete discussion of the star rating program, see Darden & McCarthy (forthcoming). 5

6 by the MA contract for each enrollee. Those authors find that contracts do not adjust premiums directly as a result of changes in benchmark payment rates, but rather contracts adjust the generosity of plan benefits in response. Conversely, Darden & McCarthy (forthcoming) find that contract/plans in 2010 raise premiums in response to higher 2009 contract-level quality star ratings. However, the sample used to estimate the supply-side response of contracts in 2010 was restricted to just those contract/plans with a.) 10 or more enrollees in both 2009 and 2010 and b.) nonmissing quality ratings in Furthermore, that paper only focuses on direct premium increases, ignoring the possibility of indirect premium adjustments such as changing the number of zero-premium plans or adjust the plan-mix within a county. The current paper provides a comprehensive examination of the supply-side response to quality star ratings, examining the full population of approved MA contracts to evaluate several potential response mechanisms as well as potential welfare consequences. 3 Data We collect data on market shares, contract/plan characteristics, and market area characteristics from several publicly available sources for calendar years 2009 and As a base, we use the Medicare Service Area files to form a census of MA contracts that were approved to operate in each county in the United States in 2009 and To these contract/county/year observations, we merge contract/plan/county/year data on enrollment and other contract characteristics. 11 To our market share data, we merge further information on MA contract quality ratings, contract/plan premiums, countylevel MA market share, CMS benchmark rates, fee-for-service costs, hospital discharges, and census data. The CMS quality information includes an overall summary star measure; star ratings for different domains of quality (e.g., helping you stay healthy); as well as star ratings and continuous summary scores for each individual metric (e.g., percentage of women receiving breast cancer screening and an associated star rating). Data are not available for the overall continuous summary score (i.e., the score rounded to generate an overall star rating), but we are able to replicate this variable by aggregating the specific quality measures following CMS instructions. We explain this process thoroughly in Appendix B. Hospital discharge data are from the annual Hospital Cost Reporting Information System (HCRIS), and CMS benchmark rates and average FFS costs by county are publicly available from CMS. Finally, county-level demographic and socioeconomic information are from the American Community Survey (ACS). 10 See Appendix C for a detailed discussion of our dataset and specific links. 11 CMS suppresses enrollment counts for contract/plans with 10 or fewer enrollees, but we keep these observations and impute enrollment. The Service Area files are needed because the enrollment files do not account for migration. For example, it is possible for the enrollment file to contain a positive enrollment record for a contract/plan in a county even if that contract is not approved to operate in the county. See Appendix C for futher details. 6

7 Our enrollment data is available monthly; however, there is little variation in enrollments across months due to the nature of the open enrollment process at the end of each calendar year. Furthermore, all other variables of interest are specific to a calendar year. Therefore, we take the average enrollment of each plan across months in a given year. The resulting unit of observation is the contract/plan/county/year. Our analysis focuses only on health maintenance organizations (HMO), local and regional preferred provider organizations (PPO), and private fee-for-service (PFFS) contracts. We exclude all special needs plans and employer/union-specific plans (also known as 800-series plans), and we drop all observations that pertain to United States Territories and Outlying Areas. Our final sample includes 247,978 contract/plan/county/years. Table 1 provides summary statistics for our final dataset at the plan, county, and contract level. The data consist of 51,442 and 34,642 plan/county observations in 2009 and 2010, respectively, with an increase in average MA enrollment per plan from 292 in 2009 to 361 in The county-level summary statistics also reveal an increasing penetration of MA in the overall Medicare market, from 15.6% in 2009 to 16.5% in 2010, alongside a decrease in the number of plans offered per county, an increase of just over $15 in average premiums, an increase in the percentage of plans offering prescription drug coverage, and an increase in the proportion of HMO and PPO plans relative to PFFS plans. Finally, the bottom panel of Table 1 illustrates a slight rightward shift in the distribution of star ratings from 2009 to 2010, with 1.5-star contracts either improving in rating in 2010 or exiting the market, and with a relative increase in the percentage of 4.5 and 5-star contracts in Table 1 4 Methodology Since star ratings are assigned to contracts (rather than specific plans operating within a contract), our initial analysis follows Town & Liu (2003), Cawley et al. (2005), Dafny & Dranove (2008), Frakt et al. (2012) and others in aggregating plan characteristics to the contract level by taking the mean values across plans within a contract (in the same county). We then examine the relationship between a contract s quality star rating in 2009 and the contract s premiums in Denoting the vector of mean characteristics in market m (county) for contract c by ȳ cm = {ȳ cm,1,..., ȳ cm,k }, we specify the 12 As indicated in Table 1, enrollment data are not available for all plans as CMS does not provide enrollment counts for plans with 10 or fewer enrollments. As such, the mean enrollment figures presented are higher than the true mean as they exclude a large number of plans with missing enrollment data. 7

8 mean characteristic k for contract c as follows: ȳ cmk = f (q c, X cm, W m ) + ε cmk, (1) where q c denotes the contract s star rating in 2009, X cm denotes other contract characteristics, W m denotes 2010 market-level data on the age, race, and education profile of a given county, and ε cmk is an error term independently distributed across characteristics and markets. 13 Given our focus on premiums, our plan characteristics of interest consist of the average premium and the proportion of the contract s plans (in the same county) charging a $0 premium. 14 The CMS quality rating system relies on a continuous summary score between 1 and 5 which is rounded to the nearest half. A contract with a 2.24 summary score is therefore rounded down to a 2-star rating, while a contract with a 2.26 summary score is rounded up to a 2.5-star rating. Intuitively, these two contracts are essentially identical in quality but received different quality ratings. We propose to exploit the nature of this rating system using a regression discontinuity (RD) design. 15 More formally, denote by R c the underlying summary score, by ˆR the threshold summary score at which a new star rating is achieved (e.g., ˆR = 2.25 when considering the 2.5 star rating), and by Rc = R c ˆR the amount of improvement necessary to achieve an incremental improvement in rating. We then limit our analysis to contracts with summary scores within a pre-specified bandwidth, h, around each respective threshold value, ˆR. For example, to analyze the impact of improving from 2.0 to 2.5 stars, the sample is restricted to contracts with summary scores of 2.25 ±h. To implement our approach, we specify plan/contract quality as follows: ( q c = γ 1 + γ 2 I R c > ˆR ) + γ 3 R ( c + γ 4 I R c > ˆR ) R c, (2) where γ 2 is the main parameter of interest. Incorporating this RD framework into equation (1), and adopting a linear functional form for f(.), yields the final regression equation ȳ ckm ( = γ 1 + γ 2 I R c > ˆR ) + γ 3 R ( c + γ 4 I R c > ˆR ) R c +β c X cm + β m W m + ε ckm, (3) where W m and X cm are as discussed previously. Our baseline analysis estimates equation 3 using ordinary least squares with a bandwidth of h = We consider alternative bandwidths in Section 13 We cluster standard errors by contract; however, the results are qualitatively unchanged when clustering standard errors at the county level. 14 The overall plan type (e.g., HMO versus PPO) is typically contract-specific and therefore does not vary across plans within the same contract. 15 See Imbens & Lemieux (2008) for a detailed discussion of the RD design and its application in economics. 8

9 6 as well as a more traditional RD design with a triangular kernel Imbens & Lemieux (2008). Changes in mean premiums at the contract level can arise in several ways, most directly via changes to premiums among specific plans. To investigate this possibility, we also estimate a regression of 2010 plan premiums as a function of the plans 2009 premiums, 2009 star ratings and other contract-level variables, and 2009 county characteristics. This analysis is akin to estimating equation 3 but where our analysis is at the plan level rather than aggregating to the contract level. For this analysis, we examine only plans that were available in the same county in both 2009 and Results 5.1 Average Premiums at the Contract Level Table 2 presents the results of a standard OLS regression of mean contract characteristics in 2010 on the 2009 mean value, the contract s 2009 star rating, as well as additional county and contractlevel covariates. To the extent that contract quality is already reflected in the contract s mean plan characteristics, we would expect the effects of increasing star ratings to be relatively small in magnitude. This is the case in Table 2, where we see small decreases in average premiums among 2.5 and 4-star contracts with small increases in premiums among 3 and 3.5-star contracts (relative to contracts with one-half star lower ratings). Note that, in order to better reflect the premium charged to a given enrollee in a specific contract, our analysis of average premiums at the contract level excludes plans with 10 or fewer enrollments. 16 Our analysis at the plan-level makes no such exclusion. Table 2 The OLS results say little about the specific effects of an increase in reported quality on premiums. To address this question directly, Table 3 presents the initial RD results at the contract level for a bandwidth of h = The results suggest a large premium increase for contracts receiving a 3, 3.5, or 4 star rating in 2009, with these contracts increasing average premiums by between $29 and $34 per month from their 2009 levels relative to contracts with one-half star lower ratings. By contrast, contracts receiving a 2.5-star rating showed no statistically significant increase in premiums. By virtue of the RD design and the nature of the CMS star rating program, we argue that these estimates can be interpreted as the causal effect of a one-half star increase in quality ratings separate from the quality 16 Not surprisingly, low star-rated plans with 10 or fewer enrollments also charge much higher premiums relative to the same quality plans with higher enrollments. For example, in 2010, the average premium among 2.5-star plans with 10 or fewer enrollments was $63, compared to just $32 among 2.5-star plans with 11 or more enrollments. The results are nonetheless consistent when we include all plans and an indicator variable for missing enrollment data. 9

10 of the contract itself. For example, 3.5-star contracts of comparable true quality to 3-star contracts were able to increase their premiums on average $29 per month. Looking purely at sample averages, all other contracts receiving a 3.5-star rating in 2009 increased their premiums by an average of $12, while 3-star contracts falling just below the 3.25 threshold increased their premiums by just over $3. We provide extensive robustness and sensitivity analyses for these results in Section 6. Table Premiums at the Plan Level Table 4 summarizes the RD results for 2010 plan premiums as a function of 2009 premiums, countylevel covariates, as well as the contract s quality rating as specified in equation 2. This analysis therefore estimates premium changes at the plan level (for the same plans offered in both 2009 and 2010), rather than analyzing average premiums at the contract level as in Table 3. For the same plan/county/contract, the results again show a large and statistically significant increase in premiums for 3, 3.5, and 4-star contracts, with premiums increasing by between $19 and $42 per month for the same plans. Table 4 6 Robustness and Sensitivity Analysis The appropriateness of our proposed RD design depends critically on whether contracts can sufficiently adjust their summary scores. Intuitively, it is unlikely that contracts can manipulate their scores because the star ratings are calculated based on data two or three years prior to the current enrollment period. Contracts would therefore not have the opportunity to manipulate other observable plan characteristics in response to their same-year star ratings. To test this formally, McCrary (2008) proposes a test of discontinuity in the distribution of summary scores around the threshold values. The resulting t-statistics range from 0.15 to 0.96, suggesting no evidence of a discontinuity in the running variable at any of the threshold values. In the remainder of this section, we investigate the sensitivity of our results along several other dimensions, including: 1) bandwidth selection; 2) inclusion of covariates; and 3) falsification test with counter-factual threshold values. 10

11 6.1 Choice of Bandwidth The choice of bandwidth is a common area of concern in the RD literature (Imbens & Lemieux, 2008; Lee & Lemieux, 2010). To assess the sensitivity of our results to the choice of bandwidth, we replicated the local linear regression analysis from Tables 3 and 4 for alternative bandwidths ranging from 0.1 to 0.25 in increments of The results for mean plan premiums at the contract level (Table 3) are illustrated in Figure 1, where each graph presents the estimated star-rating coefficient, ˆγ 2, along with the upper and lower 95% confidence bounds. Similar results for plan-level premium adjustments are presented in Figure 2. In general, our results are consistent across a range of alternative bandwidths. Figure Inclusion of Covariates The RD literature generally advises against including covariates in a standard RD design (Imbens & Lemieux, 2008; Lee & Lemieux, 2010). The intuition for this advice is as follows: if treatment assignment is random within the relevant bandwidth, then the covariates should also be randomly assigned to the treated and control groups. However, in our setting, purely randomized quality scores at the contract level would not necessarily imply randomization in county-level variables. As such, we argue that county-level covariates belong in our analysis in order to control for geographic variation influencing contract location and plan offerings. Nonetheless, we assess the sensitivity of our analysis to the exclusion of these covariates by estimating a more traditional RD model with right-hand side variables presented in equation 2. We estimate the effect of a one-half star increase in quality ratings with a triangular kernel and our preferred bandwidth of h = The results, summarized in Table 8, are generally consistent with our initial findings in Tables 3 and 4, where we again see large increases in average premiums among 3, 3.5, and 4-star contracts relative to contracts just below the respective star-rating thresholds. One exception is the estimated effect on individual plan premiums for 4-star versus 3.5-star contracts presented in the bottom right of Table 8. In this case, unlike the estimates in Table 4, we find no significant increase in premiums among 4-star contracts along with a reduction in the magnitude of the estimated effect. This is perhaps not surprising given the location of higher rated contracts throughout the country, where 4-star contracts are more concentrated in specific geographic areas relative to lower star-rated contracts. 11

12 Table Falsification Tests Finally, it is possible that the observed jumps at threshold values of 2.25, 2.75, etc. are driven more by specific contracts that happen to fall above or below the threshold versus the star rating system itself. As a test, we therefore considered a series of counter-factual threshold values above and below the true threshold values. Intuitively, we should not see any jumps in premiums around these thresholds. Figure 3 presents the results of this analysis for mean premiums at the contract/county level, where we estimated the effects just as we did for Figure 1 and Table 3. The results support 2.75 and 3.25 as the true threshold values, with the largest premium increases occurring just above those thresholds. The results for 2.25 and 3.75 thresholds are less conclusive, with apparent jumps in premiums for what should be irrelevant thresholds such as 1.9, 3.65, and Figure 3 7 Mechanisms for Premium Adjustment Comparing our contract-level (Table 3) and plan-level (Table 4) analysis, we see larger premium increases at the plan level for 3.5-star contracts and smaller increases at the plan level for 3-star contracts. These results suggest that increases in average premiums at the contract level do not arise solely from increases in premiums of the same plans from 2009 to Rather, the results suggest that contracts also alter their plan mix from one year to the next (e.g., dropping plans within a contract, introducing new plans under the same contract, or expanding plans to new counties). Table 5 summarizes the exit behaviors from 2009 to 2010 by star rating, where we see low quality plans were significantly more likely to exit their respective markets than plans associated with higher star ratings. In particular, we see almost all 1.5-star plans leave the market from 2009 to 2010, with very little exit among 4 and 4.5-star plans. 17 Regarding plan entry, Table 5 shows that of the contracts receiving a 1.5-star rating in 2009 that still operate in 2010, 37% of the underlying plans entered into a new county in Similarly, 55% of 2-star plans (in 2009) entered into a new county in 2010, while higher rated contracts were relatively less likely to enter into new markets. Collectively, the exit and 17 The 1.5-star contracts that stayed in the market from 2009 to 2010 also had a marginally higher star rating in As such, there are no 1.5-star contracts remaining in 2010 (see Table 1). 12

13 entry figures reflect larger turnover in plan offerings among lower rated contracts relative to higher rated contracts. This is perhaps expected as higher rated contracts may be more deliberate in their market entry/exit decisions and less likely to quickly cycle through new plans from one year to the next. Table Analysis of Plan Exit To examine plan exit more directly, we follow Bresnahan & Reiss (1991), Cawley et al. (2005), Abraham et al. (2007), and others in assuming that an insurance company will only offer a plan in a given county if the plan positively contributes to the contract s profit. Assuming profit is additively separable across geographic markets (counties), our observed plan choice indicator becomes: 1 if π c(j)m = g ( ) W m, X c(j)m + εc(j)m 0 y c(j)m = (4) 0 if π c(j)m < 0 where W m again denotes county-level demographics, X c(j)m denotes contract and plan characteristics (including the contract s 2009 quality, q c, plan premium, Part D participation, etc.), and ε c(j)m is an error term independently distributed across markets and plans. We adopt a reduced form, linear profit specification with covariates including the benchmark CMS payment rates, 2009 contract quality (q c ), the plan s enrollments in 2009, the number of physicians in the county, the average Medicare FFS cost per beneficiary in the county, and plan characteristics such as premiums, whether the plan offers prescription drug coverage, and indicators for HMO or PPO plan type. Within this specification, we also consider the RD design from equation 2. We estimate equation 4 with a linear probability model where y c(j)m = 1 indicates that the contract continued to offer the plan in 2010 and y c(j)m = 0 indicates the plan was dropped. By definition, this analysis is based on existing plans as of The results of our RD analysis of plan exit are summarized in Table 6. The top panel presents results for all plans, while the remaining panels present results for plans with $0 premiums and plans with positive premiums, respectively. Overall, we see that 2.5-star contracts are significantly less likely to exit markets than 2-star contracts of similar overall quality. Relative to 2.5-star contracts, 3-star contracts show no significant differences in exit behaviors, but they are significantly more likely to drop their $0 premium plans and less likely to drop positive premium plans. Somewhat surprisingly, 13

14 contracts receiving a 3.5-star rating are more likely to drop plans overall; however, from the middle panel of Table 6, we see that this result is entirely driven by 3.5-star contracts dropping their $0 premium plans. Finally, 4-star contracts are significantly less likely to exit overall, particularly for their positive premium plans. 18 Table Analysis of Plan Entry An important and relatively unique aspect of the MA market concerns the distinction between plan and contract-level decisions. Specifically, contracts must obtain CMS approval in order to be offered in a given county; however, conditional on receiving CMS approval, the decision of which plan(s) to offer in a county is relatively less regulated. As a result, we argue that the fixed costs of entry are primarily incurred at the contract level while the plan-level entry/exit decisions are based on the variable profits per enrollee (i.e., regardless of market share). With regard to plan entry, this unique CMS approval process alleviates many of the traditional econometric issues surrounding multiple equilibria or endogeneity of other players actions in models of market entry with incomplete information (Berry & Reiss, 2007; Bajari et al., 2010; Su, 2012). Conditional on plan characteristics, our entry analysis therefore need only consider variable cost shifters and should be largely independent of the number or type of competing plans in the county. 19 The full set of plans available to a contract in a given market m is identified by taking all plans offered under that contract across the entire state in the same year. All such plans are therefore considered eligible to be operated in any given county, and the contract must choose which of those plans to offer in each county, where y c(j)m = 1 indicates that the plan was added to the county (under that contract) in 2010, and y c(j)m = 0 indicates that the plan was not offered. As with our analysis of plan exit, we estimate the entry-equivalent to equation 4 using a standard linear probability model, with entry considered as a function of 2010 county and plan characteristics as well as 2009 contract quality as in equation 2. Table 7 summarizes the results of our RD analysis for plan entry. Note that these results only apply to markets in which the contracts previously operated (i.e., we do not consider the contract-level 18 The robustness of our plan exit results to bandwidth selection is summarized in Appendix D. The overall results (top panel of Table 6) at the 2.75 threshold appear relatively sensitive to bandwidth selection, with the statistical significance, magnitude, and sign of the point estimates changing within bandwidths from 0.1 to 0.2. In terms of hypothesis testing, we interpret this as evidence in favor of the null that the star rating has no effect on plan exit at the 2.75 threshold. As such, the qualitative findings from our point estimates in Table 6 are unchanged. 19 Results are robust when we weaken this assumption and allow predicted 2010 market shares to influence entry behaviors. The results are excluded for brevity but available upon request. 14

15 entry decisions and instead focus specifically on the plan-level entry of pre-existing contracts). The RD results indicate that a one-half star improvement for 3 or 3.5-star contracts makes them significantly more likely to expand their plans into new markets. The bottom panels of Table 7 further reveal that the increase in probability of plan entry occurs for the positive premium plans, with 3.5-star contracts significantly less likely to enter new markets with their $0 premium plans. 20 Table 7 8 Welfare Effects To examine the welfare effects of our estimated premium increases in Section 5, we follow Town & Liu (2003) and Maruyama (2011) in estimating a standard Berry-type model of plan choice based on market-level data (Berry, 1994). Specifically, let the utility of individual i from selecting Medicare option c(j) in market area m be given as U ic(j)m = δ c(j)m + ξ c(j)m + ζ ig + (1 σ)ɛ ic(j)m, (5) where δ c(j)m and ξ c(j)m represent the mean level of utility derived from observed and unobserved contract-plan-market area characteristics, respectively. We include in δ c(j)m observed characteristics at the contract and plan level, including premiums, plan type (HMO, PPO, or PFFS), and the underlying summary score of the contract. Similar to Town & Liu (2003), we partition the set of Medicare options into two groups: 1) MA plans that offer prescription drug coverage (MA-PD plans); and 2) MA plans that do not offer prescription drug coverage (MA-Only). Traditional Medicare FFS is taken as our outside option. In addition to the i.i.d. extreme value error ɛ ic(j)m, individual preferences are allowed to vary through group dummies ζ ig. This nested logit structure relaxes the independence of irrelevant alternatives assumption and allows for differential substitution patterns between nests. The nesting parameter, σ, captures the within-group correlation of utility levels. Following Berry (1994) and others, the parameters in equation 5 can be estimated using marketlevel data on the relative share of MA plans. Specifically, our estimation equation is as follows: ln(s c(j)m ) ln(s 0m ) = x c(j)m β αf c(j) + σln(s c(j)m g ) + ξ c(j)m, (6) 20 The robustness of our plan entry results to bandwidth selection is summarized in Appendix D. 15

16 where x c(j)m denotes observed plan/contract characteristics, and ξ c(j)m denotes the mean utility derived from unobserved plan characteristics. We estimate the parameters of equation 6 using two-stage least squares (2SLS) due to the endogeneity of within-group shares, S c(j)m g, and plan premiums, F c(j). We take as instruments the number of contracts operating in a county, the number of hospitals in a county, the Herfindahl-Hirschman Index (HHI) for hospitals in a county (based on discharges), and the number of physicians in the county. The results of this regression are presented in Appendix D. With estimates of the mean observed utility, ˆδ c(j)m, and the within-group correlation, ˆσ, estimated monthly consumer surplus for a representative MA beneficiary is then derived as follows (Manski & McFadden, 1981; Town & Liu, 2003; Maruyama, 2011): W i = (1 ˆσ) ln 1ˆα j J m exp ( ˆδc(j)m + ˆξ ) c(j)m. (7) 1 ˆσ Our results yield an estimated $120 reduction in yearly consumer surplus per beneficiary for every $10 increase in premiums (all else equal). In 2010, there were approximately 1,080,000 beneficiaries enrolled in a 3, 3.5, or 4-star MA plan with a summary score just above the relevant threshold value. Assuming a $20 increase in premiums from 2009 to 2010 (the smallest estimated effect in Tables 3 and 4), this yields a total reduction in consumer surplus of approximately $259 million. 9 Discussion The potential supply-side response of MA contracts to the CMS quality rating system is critical both from a policy perspective as well as a consumer welfare perspective. If contracts can take advantage of improved quality scores by increasing premiums (holding the contract s true quality constant), then this suggests a lack of competitiveness in the MA market with contracts raising prices without any true improvement in quality. Building on the initial results of Darden & McCarthy (forthcoming), the current paper finds strong evidence of such premium increases among average to above average star-rated contracts. Based on the results in Section 5 and the range of sensitivity analyses in Section 6, we conclude that the increases in premiums for 3-star versus 2.5-star contracts (the 2.75 threshold) as well as 3.5- star versus 3-star contracts (the 3.25 threshold) are not due to chance but are instead reflective of a true increase in premiums following an increase in reported quality. Meanwhile, we find no consistent changes in premiums for 2.5 relative to 2-star contracts. We find some initial evidence for increases in premiums among 4-star contracts relative to 3.5-star contracts; however, this finding is sensitive to bandwidth specification, and the effect does not persist in our falsification tests. Plan-level results for 16

17 4-star rated contracts are also sensitive to the inclusion of market-level covariates, There are likely several reasons for a contract to increase 2010 premiums in response to its prioryear quality ratings. One natural reason is pure rent extraction - contracts may seek to capitalize on their high reported quality by charging a higher price to its existing customers. However, contracts may also increase premiums in order to better curb adverse selection. In this case, contracts of higher reported quality but comparable true quality may want to price-out certain customers from the market, particularly if sicker beneficiaries are more likely to make decisions based in-part on the quality ratings. With market level data, we cannot empirically identify either of these effects individually. Nonetheless, our results generally suggest that the perceived benefits of the star rating program in terms of beneficiary decision-making are at least partially offset by the supply-side response of higher premiums. 17

18 References Abraham, Jean, Gaynor, Martin, & Vogt, William B Entry and Competition in Local Hospital Markets. The Journal of Industrial Economics, 55(2), Bajari, Patrick, Hong, Han, Krainer, John, & Nekipelov, Denis Estimating static models of strategic interactions. Journal of Business & Economic Statistics, 28(4). Berry, Steven, & Reiss, Peter Empirical models of entry and market structure. In: Armstrong, M., & Porter, R. (eds), Handbook of industrial organization, vol. 3. Amsterdam: Elsevier. Berry, Steven T Estimating discrete-choice models of product differentiation. The RAND Journal of Economics, Bresnahan, Timothy F, & Reiss, Peter C Journal of Political Economy, Entry and competition in concentrated markets. Cawley, John, Chernew, Michael, & McLaughlin, Catherine HMO participation in Medicare+ Choice. Journal of Economics & Management Strategy, 14(3), Dafny, L., & Dranove, D Do report cards tell consumers anything they don t already know? The case of Medicare HMOs. The Rand journal of economics, 39(3), Darden, M., & McCarthy, I. forthcoming. The Star Treatment: Estimating the Impact of Star Ratings on Medicare Advantage Enrollments. Journal of Human Resources. Frakt, Austin B, Pizer, Steven D, & Feldman, Roger The Effects of Market Structure and Payment Rate on the Entry of Private Health Plans into the Medicare Market. Inquiry, 49(1), Imbens, G.W., & Lemieux, T Regression discontinuity designs: A guide to practice. Journal of Econometrics, 142(2), Lee, David S, & Lemieux, Thomas Regression Discontinuity Designs in Economics. Journal of Economic Literature, 48, Manski, Charles F, & McFadden, Daniel Structural analysis of discrete data with econometric applications. Mit Press Cambridge, MA. Maruyama, Shiko Socially optimal subsidies for entry: The case of medicare payments to hmos*. International Economic Review, 52(1),

19 McCrary, Justin Manipulation of the running variable in the regression discontinuity design: A density test. Journal of Econometrics, 142(2), Pauly, Mark, Harrington, Scott, & Leive, Adam Sticker Shock in Individual Insurance under Health Reform. Tech. rept. National Bureau of Economic Research. Reid, Rachel O, Deb, Partha, Howell, Benjamin L, & Shrank, William H Association Between Medicare Advantage Plan Star Ratings and EnrollmentStar Ratings for Medicare Advantage Plan. JAMA, 309(3), Stockley, Karen, McGuire, Thomas, Afendulis, Christopher, & Chernew, Michael E Premium Transparency in the Medicare Advantage Market: Implications for Premiums, Benefits, and Efficiency. Tech. rept. National Bureau of Economic Research. Su, Che-Lin Estimating discrete-choice games of incomplete information: Simple static examples. Quantitative Marketing and Economics, Town, Robert, & Liu, Su The welfare impact of Medicare HMOs. RAND Journal of Economics,

20 A Appendix A: Star Rating Metrics The star rating system consists of five domains, with the names of each domain, the underlying metrics in each domain, and the data sources for each metric changing over the years. The metrics and relevant domains for 2009 are listed in Table 9. Table 9 20

21 B Appendix B: Star Rating Calculations Although the domains and individual metrics changed from year to year, the way in which overall star ratings were calculated was consistent across years. The calculations follow in five steps, as described in more detail in the CMS technical notes of the 2009, 2010, and 2011 star rating calculations: 1. Raw summary scores for each individual metric are calculated as per the definition of the metric in question. As discussed in the text, these scores are derived from a variety of different datasets including HEDIS, CAHPS, HOS, and others. The resulting summary scores are observed in our dataset. 2. The summary scores in each metric are translated into a star rating. For most measures, the star rating is assigned based on percentile rank; however, CMS makes additional adjustments in cases where the distribution of scores are skewed high or low. Scores derived from CAHPS have a more complicated star calculation, based on the percentile ranking combined with whether or not the score is significantly different from the national average. The resulting stars for each individual metric are observed in our dataset. 3. The star values from each metric are averaged among each respective domain to form domain level stars, provided a minimum number of metric-level scores are available for each domains. For example, in 2009 and 2010, a domain-level star was only calculated if the contract had a star value for at least 6 of the 12 individual measures. The domain-level star ratings are observed in our dataset. 4. Overall Part C summary scores are then calculated by averaging the domain-level star ratings and adding an integration factor (i-factor). The i-factor is intended to reward consistency in a plan s quality across domains, and is calculated as follows: (a) Derive the mean and variance of all individual metric summary scores for each contract. (b) Form the distribution of the mean and variance across contracts. (c) Assign an i-factor of 0.4 for low variance (below 30th percentile) and high mean (above 85th percentile), 0.3 for medium variance (30th to 70th percentile) and high mean, 0.2 for low variance and relatively high mean (65th to 85th percentile), and 0.1 for medium variance and relatively high mean. All other contracts are assigned an i-factor of Overall Part C star ratings are then calculated by rounding the overall summary score to the nearest half-star value. 21