Instructions for Analyzing Data from CAHPS Surveys:

Instructons for Analyzng Data from CAHPS Surveys: Usng the CAHPS Analyss Program Verson 4.1 Purpose of ths Document...1 The CAHPS Analyss Program...1 Computng Requrements...1 Pre-Analyss Decsons...2 What Does the CAHPS Analyss Program Analyze?...6 Preparng Data for Analyss...8 SAS Data Set Requrements... 12 Usng the CAHPS Analyss Program... 20 Interpretng the Results... 36 Small Data Set Example... 45 Explanaton of Statstcal Calculatons... 49 Hypothess Tests and Assgnment of Fnal Ratngs... 61 Examnng Sample Sze Issues for CAHPS Surveys... 63 Appendx Summary of Features Included n Each Verson of the CAHPS Analyss Program... 66 Lst of Exhbts and Tables Table 1. Descrpton of test data set varables... 22 Table 2. Arguments for CAHPS 4.1 Macro... 28 Table 3. Effect sze detected wth 80 percent power (alpha = 0.05) by number of plans and sample sze per plan... 64 Table 4. Effect sze detected wth 80 percent power (alpha = 0.05) by number of plans and sample sze for one plan (n = 300 for all other plans)... 64 For addtonal gudance, please e-mal cahps1@ahrq.gov or call the Help Lne at (800) 492-9261.

Purpose of ths Document CAHPS Surveys and Instructons Ths document explans how the CAHPS Analyss Program works and how sponsors and vendors can use the program to nterpret the results of ther CAHPS survey. Whle the program was ntally desgned for the CAHPS Health Plan Survey, you can use t to analyze data from any of the CAHPS surveys. For most CAHPS surveys, the nstructons nclude a document wth analyss gudance specfc to that survey. The CAHPS Analyss Program The goal of the CAHPS Analyss Program often referred to as the CAHPS macro s to provde the user wth a flexble way to analyze CAHPS survey data n order to make vald comparsons of performance. Wrtten n SAS, the CAHPS Analyss Program s desgned to assst CAHPS survey users n mplementng two knds of statstcal adjustments. Comparng tems and compostes. CAHPS surveys collect consumers and patents reports and ratngs of a number of dmensons of health care. Comparng performance based on all the ndvdual CAHPS survey tems s a very complex task. Moreover, ndvdual survey tems are often less relable than multple tem combnatons. To smplfy the nterpretaton of the data and enhance the relablty of the results, questons that measure smlar topcs are grouped together. These groups of questons, called compostes, facltate comparsons of performance across your unt of analyss (e.g., health plan, medcal practce, clncan). (Note: The nstructons avalable for each survey nclude a document that lsts the tems n that survey s reportng measures,.e., composte measures and ratngs.) Adjustng for case mx. The CAHPS Team recommends that you adjust the survey data for respondent age, educaton, and general health status. Ths makes t more lkely that reported dfferences are due to real dfferences n performance, rather than dfferences n the characterstcs of enrollees or patents. Computng Requrements The CAHPS Analyss Program was developed usng SAS software. SAS s a data management, analyss, and presentaton product produced by the SAS Insttute, whch s headquartered n Cary, North Carolna. The operaton of SAS requres a Base system, but a number of ndvdual modules can be added to perform more complex analyses and data manpulaton. The CAHPS Analyss Program requres Base SAS and the SAS/STAT module. Base SAS, whch s requred to use any SAS product, provdes the data manpulaton, prnt commands, smple plottng capabltes, and procedures for descrptve statstcs. Base SAS ncludes the CORE module. The SAS/STAT module adds several statstcal procedures for use by SAS. The CAHPS Analyss Program uses the SAS regresson procedure, PROC REG, to do part of ts case-mx calculatons. If case-mx calculatons wll not be used, then the macro requres Base SAS only. Page 1

The CAHPS Analyss Program was wrtten n verson 6.12 of PC SAS and verson 6.12 of SAS/STAT. The program should work on all SAS platforms that have verson 6.0 or later. It has been extensvely tested on UNIX and Wndows SAS products and has performed well. Pre-Analyss Decsons The CAHPS Analyss Program offers the user a number of optons for analyzng the survey data. Before preparng to run the program, analysts should make sure that the project team has agreed upon answers to the followng questons. Ther mplcatons for the CAHPS Analyss Program are revewed below. Havng these questons answered early wll save tme when dong the analyses. What s the reportng unt (entty)? Any analyss of CAHPS data s ntended to assess, compare and report on some type of reportng unt. Examples of possble such unts nclude health plans, hosptals, provder groups, clncs, stes of care, and ndvdual physcans. To avod confuson, we use the neutral term entty n these nstructons to refer to the unt whose data wll be aggregated nto a summary measure. Users of the Analyss Program wll have to specfy whch varable dentfes the entty to whch each response wll be attached. Note that there mght be alternatve ways of analyzng the same data wth dfferent enttes, and f the data collecton desgn s sutable, more than one of them mght be vald. For example, a dataset mght be analyzed to compare provder groups and then, wth a dfferent entty varable, the same data mght be used to assess ndvdual doctors. The CAHPS Analyss Program was ntally wrtten for the CAHPS Health Plan Survey. Therefore, varable names, examples, and other references throughout the program often refer to health plans. Ths has no bearng on the sutablty of the program for analyzng data on other types of enttes. For specfc nstructons on adaptng the program for other surveys, please refer to Preparng and Analyzng Data (https://www.cahps.ahrq.gov/surveys-gudance/cg/~/meda/fles/ SurveyDocuments/CG/12%20Month/Prep_Analyze/1035_Preparng_analyzng_data_ from_cg.pdf) n the nstructon documents avalable for the survey you are usng. Ths document s not avalable for all CAHPS surveys. Wll you analyze specfc populaton groups separately? If the project team has collected data for dfferent groups of people, the team needs to decde whether to analyze the data separately or together. Subgroups that you may be consderng for separate analyses can be defned by payer (e.g., Medcare, Medcad, prvately nsured), geographc regon (e.g., state, county, regon), or other factors. If the groups are to be analyzed together, no changes to the CAHPS Analyss Program are necessary. If a team decdes to analyze the groups separately and the data fle contans more than one group, t s mportant to set up selecton crtera n the CAHPS Analyss Program or splt the data set. Page 2

Wll adult and chld surveys be analyzed together or separately? The Analyss Program allows users to specfy how chld and adult surveys wll be analyzed. The project team needs to decde whether to analyze surveys about adults and chldren separately or together. If adult and chld survey data are to be analyzed together, the team must also decde whether to consder nteracton effects. Interacton effects are mportant to consder n an analyss when the mpact of age or health status on one of the reportng tems depends on whether an adult or chld survey s beng analyzed. See the secton called Explanaton of Statstcal Calculatons for a more detaled dscusson of nteracton effects. We recommend that you consder nteracton effects when combnng adult and chld data. If the team collected only adult or chld surveys, users stll need to specfy an opton lsted below n the CAHPS Analyss Program. There are four optons dependng on whch surveys are n the data set and how the user wants to analyze them: Opton value 0 Surveys n data set Adult and chld surveys, only adult surveys, or only chld surveys 1 Adult and chld surveys 2 3 Adult and chld surveys or only chld surveys Adult and chld surveys or only adult surveys Analyss method Combne adult and chld survey data; do not consder nteracton effects Combne adult and chld survey data; consder nteracton effects Chld surveys only Adult surveys only Wll hgh and low users of health care servces be analyzed together or separately? The CAHPS Analyss Program allows users to analyze the data for survey respondents wth a hgh or low number of outpatent vsts separately or combned. The crteron for establshng low and hgh users of outpatent servces s based on Queston 7 of the CAHPS Health Plan Survey Adult Commercal Questonnare or Queston 4 of the CAHPS Clncan & Group Surveys Adult 12-Month and Vst Questonnares. Hgh users are defned as respondents who ndcated that they have had three or more vsts for ther own care to a doctor s offce or clnc (not ncludng emergency room vsts). Low users ndcated that they have had fewer than three vsts to a doctor s offce or clnc. The CAHPS Consortum recommends reportng data for global ratngs (e.g., respondent s ratng of ther personal doctor or nurse, specalsts, health care n the last 12 months, and/or health plan) accordng to outpatent utlzaton. It s up to the project team to decde whether to analyze the survey data for low and hgh users Page 3

separately or together. You can then choose from the correspondng Opton Values lsted below for the CAHPS Analyss Program. Opton value Analyss decson 1 Combne low and hgh users. 2 Low users only (< 3 vsts) 3 Hgh users only ( 3 vsts) What level of sgnfcance (p-value) wll you use n the analyss? The CAHPS Consortum recommends a p-value of 0.05 to test for statstcally sgnfcant dfferences between the enttes beng compared. The p-value the team chooses must be specfed n the CAHPS Analyss Program. What, f any, level of substantve (practcal) sgnfcance wll you use to compare performance? Substantve sgnfcance refers to an absolute dfference between the enttes beng compared (e.g., health plans, provder groups, ndvdual physcans) that must be acheved before a change s consdered meanngful. If two health plans, for example, had sgnfcantly dfferent average scores based on the p-value crtera, the dfference between the plans average scores may stll not be large enough to be meanngful. The CAHPS Analyss Program has two optons that allow the user to specfy a dfference that s substantve. You can use these optons smultaneously or specfy only one. Frst method. The team decdes on a percentage of the dstance to the nearest bound that s meanngful. The example presented below explans ths concept. Assume the analyss of a global ratng queston (one that uses a 0-10 ratng scale) has the followng mean scores for a global ratng queston across all enttes: Global Ratng Queston overall mean (0-10 scale) = 6 To determne a level of dfference between enttes that s substantvely large: 1. Determne the dstance from the mean to the nearest bound: a) Compute the dstance from the lower bound Mean (6) - Lower bound (0) = 6 b) Compute the dstance from the mean to the upper bound Upper bound (10) - mean (6) = 4 Page 4

c) Determne the smaller dfference Mnmum (6,4) = 4 2. The dstance from the nearest bound s 4. Now the project team must decde what percentage of ths dstance s a meanngful dfference between enttes. Ths fracton s entered n the CAHPS Analyss Program. Second method. A much smpler method avalable n the Analyss Program s to specfy an absolute dfference that must exst between the entty mean and the mean for all enttes n the analyss for a dfference to be consdered sgnfcant. For ths method, the user needs only to specfy the absolute dfference consdered to be meanngful. Do you need to adjust the results for case mx? Case mx refers to the respondents health status and socodemographc characterstcs, such as age or educatonal level, that may affect the ratngs that the respondent provdes. Wthout an adjustment, dfferences between enttes could be due to case-mx dfferences rather than true dfferences n qualty. Each project team must determne f t s approprate to adjust ts data to account for case-mx dfferences. What to adjust for If the project team decdes to adjust the data for case-mx, t must choose the approprate adjusters. The CAHPS Consortum recommends usng general health status, age, and educaton. Indvduals n better health and older ndvduals tend to rate ther care, plans, and provders hgher. There s also evdence from a number of studes that educaton affects ratngs, wth more educated ndvduals gvng lower ratngs. Mssng data for case-mx adjusters If case-mx adjusted results wll be used, the project team must decde whether or not to mpute mssng data for the adjusters at each adjuster s entty-level mean. Rsk of out-of-range values for case-mxed means In the specal cases where there are very few records for an analyss varable or all respondents answered n only one or two response categores, there s the possblty that the case-mx adjusted values wll be out of range. For example, f all respondents to a Health Plan Survey answered Yes, where 0= No and 1= Yes to a yes/no queston, and the adjustment for that entty s up, the adjusted mean for that entty would be greater than 1 and the adjusted frequences would be less than zero for the No category and greater than 1 for the Yes category. Page 5

The macro does not force a change n these values, snce t would change the mean of the means on the adjusted scores but not on the unadjusted scores. It s recommended that, n reports of CAHPS results, you set these out-of-range values to the mnmum or maxmum value for that category. Then a manual adjustment could be made to the adjacent category f necessary. For example, n the case of three response categores, where the mnmum frequency should be zero and the maxmum value s 100, the case-mxed frequency results are as follows: category 1 = -2.0, category 2 = 25.0 and category 3 = 77.0 The results could be adjusted so that category 1 = 0.0, category 2 = 23.0 and category 3 = 77.0 Do results need to be analyzed usng weghtng and stratfcaton? As dscussed above, the survey samplng plan can be desgned to select dsproportonately potental respondents from certan geographc or demographc groups n the populaton. Alternatvely, stuatons can arse after samplng s complete that create the need to combne data for certan samplng unts. For example, ths can occur when two enttes merge ther operatons and the survey sponsor chooses to report ther results as a combned score. Whether enttes are mergng ther operatons or a dsproportonate stratfed samplng desgn was used, the CAHPS Analyss Program can perform the approprate analyses, provded the user specfes the correct strata to be combned and the number of members n each stratum out of the entre populaton. What Does the CAHPS Analyss Program Analyze? The CAHPS Analyss Program s desgned to analyze mult-tem compostes and sngle tems from the CAHPS survey data. The output from the program compares the performance of an entty to the overall performance of all enttes. The macro accepts fve varable types. For four of these, the macro cleans ther response values wthn the expected mnmum and maxmum range. For the ffth type, the mnmum and maxmum response values must be entered as an argument. Page 6

The followng are the varable types: Varable type Mn Max response values 1 Dchotomous 0 1 2 Global ratng 0 10 3 How often or other 4-pont 1 4 response scale 4 3-pont response scale 1 3 5 Other mn_resp max_resp Global ratngs are based on survey tems that ask the respondents to rate dfferent aspects of health care on a scale from 0 to 10. For example, the global ratngs tems n the CAHPS Health Plan Survey 4.0 Adult Commercal Questonnare are: Health care n the last 12 months (Queston 8) Personal doctor (Queston 15) Specalst (Queston 19) Health plan (Queston 27) Overall Ratngs Response Format: 0-10 Q8 Q15 Q19 Q27 Usng any number from 0 to 10, where 0 s the worst health care possble and 10 s the best health care possble, what number would you use to rate all your health care n the last 12 months? Usng any number from 0 to 10, where 0 s the worst personal doctor possble and 10 s the best personal doctor possble, what number would you use to rate your personal doctor? We want to know your ratng of the specalst you saw most often n the last 12 months. Usng any number from 0 to 10, where 0 s the worst specalst possble and 10 s the best specalst possble, what number would you use to rate the specalst? Usng any number from 0 to 10, where 0 s the worst health plan possble and 10 s the best health plan possble, what number would you use to rate your health plan? Compostes are groupngs of two or more questons that measure the same dmensons of health care or health nsurance plan servces. Compostes usually are developed for survey tems that have the same response optons. Page 7

For example, the compostes n the Health Plan Survey represent the experences of respondents (adult enrollees or the parents/guardans of enrolled chldren) n the followng areas: Gettng needed care (2 questons for adults; 2 questons for chldren). Gettng care quckly (2 questons for adults; 2 questons for chldren). How well doctors communcate (4 questons for adults; 5 questons for chldren). Health plan nformaton and customer servce (2 questons for adults; 2 questons for chldren). The followng table llustrates how tems n the Health Plan Survey 4.0 are grouped nto the frst of these compostes. Gettng Needed Care Q17 Q21 In the last 12 months, how often was t easy to get appontments wth specalsts? In the last 12 months, how often was t easy to get the care, tests, or treatment you thought you needed through your health plan? Response Format Never Sometmes Usually Always A document lstng each survey s reportng measures compostes and ratngs s avalable n the nstructons provded for that survey. Preparng Data for Analyss Pror to applyng the CAHPS Analyss Program, you must perform several tasks to transform raw data from the completed questonnares nto data that the SAS analyss programs can use. (Gudance on determnng when a questonnare s complete s avalable for most CAHPS surveys n the appendx of the nstructons on feldng that survey.) Task 1: Identfy and exclude nelgble cases. Task 2: Code and enter the data. Task 3: Clean the data. Task 4: Conduct an audt. Page 8

Many nterm fles wll be created along the way. Before begnnng ths process, you must take steps to preserve the orgnal data fle created when the raw survey responses were entered. Any changes and correctons made durng the cleanng and data preparaton phase should be made on duplcate fles. There are three reasons for ths acton: 1. The orgnal data fle s an mportant component of the complete record of the project. 2. Havng an orgnal fle wll allow you to correct data errors that were made durng the cleanng process. 3. The exstence of an orgnal fle s crtcal f the vendor or sponsor wants to go back later and conduct other analyses or tests, such as extent of error tests or tests of skp patterns. Data Fle Specfcatons The data fle contans the raw data from responses to the survey. Short tem handles for the tems n each questonnare can be found n tables provded n the document called Overvew of the Questonnares at http://cahpscms.westat.com/surveys-gudance/ CG/~/meda/Fles/SurveyDocuments/CG/12%20Month/Get_Surveys/1350_cg_ overvew_of_questonnares.pdf. The responses to each queston must use the code numbers, or precodes, contaned n the questonnares. Users should construct a separate data fle for each verson of the survey. Do not nclude data from dfferent survey nstruments n the same data fle. For example, do not nclude responses to the Clncan & Group Adult 12-Month Survey and Chld 12-Month Survey n the same data fle. If you are nterested n submttng your data to the CAHPS Database, please refer to the data fle specfcatons for submsson at https://www.cahps.ahrq.gov/cahps- Database/Submttng-Data/CG-Data.aspx. The number and scope of the data preparaton tasks and the way they are carred out depend on the data collecton protocol and the way n whch the data were recorded. For example: If the vendor/sponsor collected data wth a self-admnstered maled questonnare, dd respondents record answers on optcal scan forms 1 or record them drectly on the CAHPS-formatted questonnares? If the vendor/sponsor collected data through telephone ntervews, dd the ntervewer use computer-asssted telephone ntervewng (CATI) or paper-and-pencl forms? 1 Optcal scan forms are answer sheets n whch respondents fll n the crcle that corresponds to ther answer choce. These forms are fed through an optcal scannng machne, and the data are automatcally captured by a computer. Standardzed tests for students, such as the SAT, generally use optcal scan forms. Page 9

Task 1: Identfy and Exclude Inelgble Cases Several stuatons render a case nelgble for analyss. One common scenaro that vendors must be prepared to handle occurs when the respondent reports he or she has not vsted the sampled entty (e.g., a physcan or medcal group). Ths mght be ndcated by a no response to Queston 1 (e.g., Our records show that you got care from the provder named below n the last 12 months. Is that rght? ). Other questonnares may be consdered ncomplete and excluded from analyss but are not excluded from the denomnator used to calculate the response rate. For example: If someone else asssted the respondent or answered the questons (as a proxy) or If at least half of the key tems on the questonnare were not flled n. (For most CAHPS surveys, a lst of key tems s avalable n the appendx of the nstructons on feldng the survey.) Task 2: Code and Enter the Data There are a varety of possble methods that can be used to enter data from CAHPS surveys. The exact level of codng requred wll depend on the method used to capture the data (e.g., questonnares that requre data entry versus questonnares that are scanned by a computer). Your codng specalst should revew each questonnare to see whether the responses are legble and whether any responses need to be coded. Each tem should have a correspondng code, even tems that were not answered. The table below shows examples of recommended codng. If you are nterested n submttng your data to the CAHPS Database, follow the data fle and codng specfcatons avalable at https://www.cahps.ahrq.gov/cahps-database/submttng- Data/CG-Data.aspx. After codng s completed, enter the data nto a computer fle. Response Code Actual response Use the correspondng survey precode ndcated besde the response opton. Precodes typcally begn wth 1 and number consecutvely for each response opton. Item was approprately skpped Code as 7 (or 77 f precodes exceed 7) Item showed more than one response Code as 8 (or 88 f precodes exceed 7) opton completed when only one s approprate (.e. multple marks) Item was left blank and should not have Code as 9 (or 99 f precodes exceed 7) been skpped Page 10

If you use optcal scan forms for your maled questonnares, the scannng equpment automatcally enters the data nto a computer-readable fle. If you do not use optcal scan forms, the mal questonnares are desgned for drect data entry wthout the need for codng most respondent answers. However, f t s unclear whch answer the respondent selected (e.g., the respondent s pencl mark does not neatly ft wthn a sngle answer category, or two responses are marked), then your codng specalst wll have to make a decson about whch response the respondent ntended. If t s not readly apparent what the respondent ntended, the codng specalst should ndcate that the answer be entered as mssng or multple marks, as approprate. If you use a CATI system for a telephone survey, data are entered drectly nto a data fle that has already been programmed to refuse unlkely and nvald responses. If you use paper questonnares to record answers gven n a telephone ntervew nstead, the process for codng and data entry s the same as for the standard paper verson of the maled questonnares. To ensure qualty, answers from paper-and-pencl questonnares should be keyentered by two separate data entry specalsts. The results from the two should be compared to dentfy and correct data entry errors. At the end of the codng and data entry process, you wll have an electronc data set of responses to all the questonnare tems. Addtonal codng and recodng may be necessary pror to usng the CAHPS Analyss Program. Refer to the SAS Data Set Requrements below for recodng varables for use wth the CAHPS Analyss Program. Task 3: Clean the Data In many cases, the data set you have created wll have mperfectons. You wll have to take several steps to fx these mperfectons before any results are reported. Check for out-of-range values. Out-of-range responses occur when respondents provde napproprate responses for a partcular queston. For example, f the vald response choces for a queston are 0 or 1, a value of 2 would be out of range. Smlarly, f a respondent crcled two categores when he or she was supposed to provde only one answer, the response s out of the acceptable range of the queston. To detect out-of-range values, you need to revew queston frequences. Ths can be done ether by vsually scannng a report showng the tem dstrbutons or frequences or by runnng the questonnare data through a computer program., Both are often used to mprove the qualty of the data. If a value s found that s mpossble (or unlkely) gven the response optons, then the questonnare should be revewed and revsons made to the data. These revsons often nvolve settng the out-of-range values to mssng, whch drops them from the data analyss for that partcular queston. Carefully document the results from ths revew process, ncludng any changes to the data set. Mantan an audt tral (electroncally and on paper) so t s possble to go back to the orgnal data fle. Check for skp pattern problems. Response nconsstences generally arse when a respondent msunderstands a queston or does not successfully follow nstructons to Page 11

skp questons. An example of a response nconsstency would be f a respondent answered that he or she had no doctor vsts n the past 6 months, but then answered followup questons about vsts n the past 6 months. If there are nconsstences between the response to the screener queston and the followng queston s response, assume the screener response s correct. Check agan for nelgble cases. Identfy any questonnares that are not elgble for analyss and remove them from the data set used for the CAHPS SAS program. Questonnares n whch fewer than half of key tems are answered should not be counted as completed surveys. (Gudance on determnng when a questonnare s complete, ncludng a lst of key tems, s avalable for most CAHPS surveys n the appendx of the nstructons on feldng that survey.) Check for duplcates. The number of records n the data fle should match the number of completes and partals n the sample fle. Duplcates can occur f the vendor conducts a followup phone ntervew, f the mal questonnare arrves at the same tme or soon after, and the case slps through the recept control system, or f there are errors n data entry. Your polcy should be to keep the frst questonnare that comes n. Task 4: Conduct an Audt Whether surveys are collected n standard paper format, as optcally scanned forms, or as paper telephone questonnares, a small random sample of the entered data should be audted by comparng hard-copy forms wth the results of data entry. Ths enables you to catch any systematc errors. For example, f the optcal scannng program was ncorrect, Queston 5 may be entered n the fle where Queston 6 was supposed to be. These types of systematc errors wll show up consstently across all questonnares. SAS Data Set Requrements Before runnng the CAHPS Analyss Program, make sure that the structure and propertes of the data fle meet the specfcatons lsted below. If the data set does not meet these requrements, the SAS program wll not work properly. Many of the varable codng and cleanng requrements are demonstrated n the next secton on usng the CONTROL.SAS Analyss Program. Data Fle Specfcatons Each row or case n the SAS data set represents the data for a unque questonnare. If data from dfferent CAHPS questonnares are n the same data set and are to be analyzed together, each questonnare s lsted on a separate row. If data from adult and chld questonnares are n the data set, the adult and chld questonnares are also lsted on separate rows. Page 12

If data from dfferent CAHPS questonnares are n the same data set, responses for equvalent questons are lsted under the same varable names. Sample Sze Requrements Number of enttes (.e., such as health plan or provders). The data set must have surveys from at least two enttes. If there s only one entty n the data beng analyzed, statstcal comparsons cannot be performed and some parts of the program wll not work properly. If the CAHPS macro s run wth data for one entty, a couple of warnng messages and notes wll appear n the log fle that would not be produced f two or more enttes were represented. All the reports wll stll be produced, though some of the results wll be of lmted value. Responses. At least two responses per entty are requred by the Analyss Program. We recommend analyss of at least 100 responses for each entty. The program flags enttes wth fewer than 100 responses for an ndvdual measure, but the analyss s performed on all enttes wth at least two records. Includng enttes wth very lttle data tends to reduce the precson of comparsons between ndvdual enttes or provders and the overall means. The user can consder removng enttes wth fewer than 100 responses from the data fle before analyss. Note: When analyzng unts of analyss such as medcal groups or ndvdual physcans, follow the mnmum response gudance n the nstructons for feldng the survey. Snce the program was ntally desgned for the CAHPS Health Plan Survey, you wll receve the program flags n your results when there are fewer than 100 responses even f the target number of completed responses s less for your survey. Varable Codng and Cleanng Requrements Numerc varables. All analytc varables used by the CAHPS Analyss Program must be numerc. Analytc varables nclude any questonnare tem used to compute CAHPS reportng tems, case-mx adjustment varables, the dchotomous varable used to dentfy chld and adult surveys, and the varable used to dentfy hgh and low users of outpatent servces. These varables are dscussed n more detal below. To ensure that an error does not occur n the SAS program, all varables created from survey questons should be coded numercally. If the user recodes character varables to numerc, there should be a mnmum length of 4. A length of 8 s recommended for the recode. WARNING: The varables PLAN, CHILD, VISITS, and SPLIT are varable names needed by the CAHPS macro. If the data set has other varables wth these names and they do not conform to the specfcatons below, the macro may produce errors n the log fle and the results may be erroneous. Varable PLAN. The varable PLAN must be ncluded n the data set. Note that ths varable represents a numerc code or text descrptor for each entty n the data set. Ths s the only varable that does not have to be coded numercally. The SAS program accepts alphanumerc, character, and numerc formats for ths varable. The maxmum varable length for PLAN s 40 characters. Page 13

Even f you are not analyzng health plan data, you must use the varable name PLAN to refer to your unt of analyss. Ths s because the SAS macros use that varable name. However, the varable can be any unt of analyss. For example, PLAN can represent the names of the medcal groups (Group A, Group B, etc.) or ndvdual physcans (Dr. A, Dr. B, etc.). Varable CHILD. The numerc varable CHILD needs to be n the data set f subsettng the data between adult and chld records. Ths varable s used to dstngush between adult and chld surveys n the SAS program. CHILD should be coded 0 for adult surveys and 1 for chld surveys. If ths varable s mssng from the data set, the CAHPS macro sets CHILD = 0 when ADULTKID has the values 0, 1, or 3, and sets CHILD = 1 when ADULTKID = 2. Varable VISITS. The varable VISITS needs to be n the data set f usng the VISITS parameter. Ths varable s used to dentfy hgh and low users of health care servces (e.g., refer to Queston 7 from the CAHPS Health Plan Survey 4.0 -- Adult Commercal Questonnare). The table that follows shows the response values based on tem 7 n the CAHPS Health Plan Survey 4.0 -- Adult Commercal Questonnare. We recommend that you use these values whle codng your questonnares. If the VISITS varable s mssng from the data set, the CAHPS macro wll work as long as the VISITS parameter s not equal to 1. 7. In the last 12 months, not countng tmes you went to an emergency room, how many tmes dd you go to a doctor s offce or clnc to get care for yourself? Response value Label/descrpton 0 None 1 1 tme 2 2 tmes 3 3 tmes 4 4 tmes 5 5 to 9 tmes 6 10 or more tmes All other values Not analyzed by the SAS program Varable SPLIT. The numerc varable SPLIT needs to be n the data set f you are dong separate case-mx adjustments on two dfferent populatons as ndcated by the macro parameter SPLITFLG = 1. For most cases, the default value 0 for SPLIT does not need to be modfed. An example of splttng the case-mx adjustments separately on two populatons s when comparng Medcad Fee-for-Servce populatons wth Medcad Managed Care populatons. Yes/No Varables. Varables wth yes/no response categores to be used n the analyss are coded as shown n the table below. Any varable wth dchotomous response optons should be coded n ths manner. For easer nterpretaton of the Page 14

results, the postve response should have the hghest value. Raw data for ths type of varables wll need to be recoded as the precodes typcally set the values of the responses to 1 and 2 rather than 0 and 1. Response value Label/descrpton 0 No 1 Yes All other values Not analyzed by the SAS program Three Response Optons. Any varable wth three response optons should be coded as shown n the table below. For easer nterpretaton of the results, the postve response should have the hghest value. Reverse codng may be necessary to ensure that the most postve response has the hghest value for example, where Yes, defntely s the most postve response. Response value Label/descrpton 1 Yes, defntely 2 Yes, somewhat 3 No All other values Not analyzed by the SAS program Four-Pont Frequency scale. Varables wth never to always response optons are coded as shown n the table below. Any varable wth four response optons should be coded n ths manner. For easer nterpretaton, the postve response should have the hghest value. Reverse codng may be necessary to ensure that the most postve response has the hghest value for example, where never s the most postve response. Response value Label/descrpton 1 Never 2 Sometmes 3 Usually 4 Always All other values Not analyzed by the SAS program Response value Label/descrpton 1 Defntely no 2 Somewhat no 3 Somewhat yes 4 Defntely yes All other values Not analyzed by the SAS program Page 15

Global Ratngs. Global ratng tems wth 0-10 response optons are coded as shown n the table below: Response value Label/descrpton 0 Worst 1 2 3 4 5 6 7 8 9 10 Best All other values Not analyzed by the SAS program Codng for Adjuster Varables. If the project team decdes to case-mx adjust the CAHPS survey results, numerc varables must also be properly coded for each adjuster varable. The CAHPS Consortum recommends adjustng the data for age, educaton, and general health status; however, the program allows for a flexble number of adjuster varables. The user can choose the proper specfcaton for each varable used to adjust the data. Specfcatons used for age, educaton, and general health status n the CAHPS Analyss Program are descrbed below. Users may also specfy the varables as dchotomous wth reference categores (dummy varables). It s mportant to remember that the SAS program prevously cleaned out-of-range values for these varables. However, the added flexblty of a user-specfed number of varables and specfcaton of the varables makes t necessary to code out-of-range values to mssng before runnng data through the macro. The codng specfcaton for the numerc varable EDUCATION s ncluded n the data set. Educaton refers to the respondent s hghest level of school completed. Ths varable and ts response codes should be coded based on the responses to the educaton tem (such as Queston 35 n the CAHPS Health Plan Survey -- Adult Commercal Questonnare). Page 16

35. What s the hghest grade or level of school that you have completed? Response value Label/descrpton 1 8 th grade or less 2 Some hgh school, but dd not graduate 3 Hgh school graduate or GED 4 Some college or 2-year degree 5 4-year college graduate 6 More than 4-year college degree All other values Code to mssng The codng specfcaton for the numerc varable GENERAL HEALTH RATING (GHR) s ncluded n the data set. The GHR s a ratng of the survey respondent s overall health status. Ths varable and ts response codes should be based on the responses to the health status tem (for example, Queston 28 n the CAHPS Health Plan Survey -- Adult Commercal Questonnare). 28. In general, how would you rate your overall health now? Response value Label/descrpton 1 Excellent 2 Very good 3 Good 4 Far 5 Poor All other values Code to mssng Page 17

The numerc varable AGE s ncluded n the data set, representng age groupngs based on data from the CAHPS survey. Ths varable and ts response value codes should be based on questons about age, such as Queston 33 of the CAHPS Health Plan Adult Commercal Questonnare and Queston 32 of the CAHPS Health Plan Chld Commercal Questonnare. The response values for these questons need to match the values for AGE as follows: 33. What s your age? / 32. What s your chld s age? Response value Label/descrpton (years) For chld surveys: 0 < 1 1 1-3 2 4-7 3 8-12 4 13-17 For adult surveys: 1 18 to 24 2 25 to 34 3 35 to 44 4 45 to 54 5 55 to 64 6 65 to 74 7 75 or older All other values Code to mssng Stratfed Data. If you want to combne data for reportng from dfferent samplng strata, you must create a text fle that dentfes the strata and ndcates whch ones are beng combned and the dentfer of the entty obtaned by combnng them. Some examples llustrate stuatons n whch ths feature mght be used: Two health plans are merged that were formerly separate and were treated as such n the survey. A hosptal decdes to sample 200 medcal and 200 surgcal patents, although ths s not proportonal to the numbers of dscharges n the two servces. A survey desgned to assess provders samples 80 patents from each regardless of the number of sessons each has, but the sponsor decdes to also use the data to assess provder groups. If no fle s specfed, the macro creates one usng the PLAN varable n the data set to set the Orgnal Plan and New Plan equal to the PLAN varable, the Populaton Sze equal to 1, and Subsettng Code equal to 1. If stratfcaton s part Page 18

of your survey desgn, an ASCII data set needs to be created wth columns separated by one or more spaces for these four varables: Orgnal Plan a unque dentfer of the unts or strata before they are combned. Ths varable can be coded as alphanumerc, but t cannot exceed 16 characters. Ths varable s the frst column of the data table. New Plan dentfer for the enttes that wll be created by combnaton of strata. Ths varable can be coded as alphanumerc, but t cannot exceed 16 characters. Ths varable s the second column of the data table. If no stratfcaton s beng done, ths column may look dentcal to the column for orgnal plan. Populaton Sze a numerc varable that ndcates the sze of the populaton for the unt or stratum. Ths varable s used to create the weghts for combnng the strata. The populatons for the combned strata should equal the total populaton of the new plan. Ths varable s the thrd column of the data table. If no stratfcaton s beng done, ths column may be set to 1s. Subsettng Code dentfer for the subset (.e., regon, state, county ) that the entty belongs n. Ths varable can be coded as alphanumerc. If no subsettng s to be done, ths column may be set to 1s. The ASCII fle for the plan detals should not contan any mssng data and each column of data should be separated by spaces. If tabs are used, the macro may not read n the data correctly. Also, be sure to not have any extra records at the bottom of the ASCII fle. If you want to make a quck sample plan detal fle from the CAHPS data set, use the program make_plandtal_dat.sas as a startng pont and change varable names and paths as needed. If the number of plans s small, t s probably easer to create the fle by hand. An example of the plan detal data set s provded for the test program (test.sas). The data fle s called plandtal.dat and looks lke the text below: HMO_A_URBAN HMO_A 5000 Northeast HMO_B_URBAN HMO_B 8000 Northeast HMO_C_URBAN HMO_C 15000 Atlantc HMO_B_RURAL HMO_B 2000 Northeast HMO_C_RURAL HMO_C 3000 Atlantc Page 19

The TEST data set provdes an example for three health plans (2 nd column): HMO_A, HMO_B, and HMO_C. The urban/rural strata for HMO_B and HMO_C are weghted together and the combned plans compared to HMO_A, whch had members only n urban areas. The frst column provdes a unque dentfer for each plan/regon combnaton (orgnal plan). The second column, the new plan varable, ndcates whch unts wll be combned. The thrd column, or the unt populaton sze, s used to compute the weghts for the plans. Unts wth greater populaton szes receve more weght than smaller unts n the combned plan. The fourth column s the regon (subset) of the country n whch each plan does busness. Adult and Chld Interactons (Macro Parameter ADULTKID) When the macro parameter ADULTKID equals 1, the macro creates adult and chld nteractons for the adjuster varables. The macro creates addtonal adjuster varables, wth the a set namng conventon, AC1, AC2,..., ACn, where n s the total number of adjusters orgnally submtted n the macro call parameter ADJUSTER. When there s an adult and chld nteracton, the macro creates the ACx varables by loopng through the lst of adjusters. For example: If &ADJUSTER = GHR AGE EDUCATION, then the followng addtonal nteracton adjuster varables are created: AC1 = GHR * CHILD AC2 = AGE * CHILD AC3 = EDUCATION * CHILD Usng the CAHPS Analyss Program The steps descrbed below assume a basc knowledge of how to use the SAS system. Step 1: Loadng the programs and test data Verson 4.1 of the CAHPS Analyss Program contans three core components: a SAS control program, a SAS macro, and a Plan Detal data fle. Page 20

All program and data sets needed for the CAHPS Analyss Program, Verson 4.1, are avalable for downloadng n the nstructons provded for every CAHPS survey. Below s a descrpton of the purpose of each fle. Each fle should be coped to a project folder related to the CAHPS data set that s to be analyzed. CONTROL.SAS CONTROL.SAS s a SAS program that contans examples of the macro call parameter arguments that need to be specfed to produce the recommended reportng measures for CAHPS surveys. For most surveys, specfc examples of the macro call are provded n separate nstructons on analyzng the results of that survey. The program also demonstrates the varable cleanng and codng steps needed to perform the analyses for entty-level comparsons. Modfcatons most lkely wll need to be made to ths program to reflect how varables are named, how varables are coded or formatted, whether or not entty stratfcaton s used, and whether or not the data set ncludes chld surveys, among the many possble combnatons. CAHPS41.SAS CAHPS40.SAS s the core SAS macro program that performs the analyses the user specfes n the control program. The macro fle should not be modfed. PLANDTAL.DAT Ths s a sample data set that s used by the CAHPS macro when runnng the TEST data set. It contans the unque plan names, combned strata names, strata weght, and subsettng code. If ths fle s not ncluded n your control programs, the macro wll create ths data set based on the PLAN varable n the nput data set. SMALLTEST.SAS Ths program creates a small data set wth ten records that can be used to better understand what the CAHPS macro s dong. More detals on how ths data progresses through the macro can be found n the Small Data Set Example secton n ths document. FORMAT.SAS FORMAT.SAS s the SAS program that creates formats helpful to vew the data wth Englsh words nstead of the data values assgned n the TEST data set. The formats have the essence of the tables descrbed n the secton SAS Data Set Requrements. The program creates the formats n the lbrary named LIBRARY as defned by the lbname LIBRARY statement n CONTROL.SAS and TEST.SAS programs. The fle, currently set up to work wth the test data programs, can be modfed for use wth other data. Modfcatons, such as changng the values of the formats, addng new formats or deletng formats, are the most common. Page 21

TEST.SAS TEST.SAS s a test control program for the CAHPS Analyss Program macros. It was desgned for use wth the test data sets descrbed below. Ths program was constructed to provde users wth a short program and data set that demonstrated the analyss optons and output for Verson 4.1 of the CAHPS Analyss Program. The hypothetcal example was desgned to ncorporate a dsproportonate samplng desgn of ndvduals n rural and urban areas for three health plans. One health plan (HMO_A) has members only n urban areas, whle the other two plans (HMO_B, HMO_C) have members n both urban and rural areas. To make comparsons across the three plans, the data for the plans wth members sampled from rural and urban regons need to be combned usng weghts. Varous optons are used n the test program to demonstrate the analyss features avalable to the user. Analyses are performed for all three types of reportng tems (sngle questons, global ratngs, and compostes). A varety of analyss features are also used, ncludng a varyng number of case-mx adjustment varables, turnng off the opton that creates output data sets, and the weghtng opton. TEST.SAS7BDAT, TEST.SSD01, TEST_wndows.SAS7BDAT, TEST.SD2 TEST*.* fles are SAS data sets that contan the same varables. TEST.SSD01 and TEST.SAS7BDAT are for use wth UNIX SAS programs, and TEST.SD2 and TEST_wndows.SAS7BDAT are desgned for use wth the Wndows verson of SAS. The table below descrbes the varables n the data sets and provdes value labels for each varable. (Note that the test data fles and sets were created for the CAHPS Health Plan Survey 3.0 and have not yet been updated.) Table 1. Descrpton of test data set varables Varable Descrpton Response optons ID Respondent dentfcaton number Unque numerc value PlanID Plan dentfcaton number 1 = HMO_A_URBAN ; 2 = HMO_B_URBAN ; 7 = HMO_C_URBAN ; 4 = HMO_B_RURAL ; 5 = HMO_C_RURAL ; 6 = HMO_BE_1 ;. = Mssng Q31 Global ratng of care 0 (worst) - 10 (best).=mssng 98=Inapplcable 99=No Answer Gven Page 22

Varable Descrpton Response optons Q38 Global ratng of plan 0 (worst) - 10 (best).=mssng 98=Inapplcable 99=No Answer Gven Q06 Q10 Q22 Q23 Q05 Q15 Problem to get doctor or nurse that you were happy wth Problem to get a referral to see a specalst that you needed Problem n gettng the care you or doctor beleved was necessary Problem wth delays n health care whle watng for approval from plan Was t easy to fnd a personal doctor or nurse How often got the help or advce you needed 1=A Bg Problem 2=A Small Problem 3=Not a Problem.=Mssng 98=Inapplcable 99=No Answer Gven 1=A Bg Problem 2=A Small Problem 3=Not a Problem.=Mssng 98=Inapplcable 99=No Answer Gven 1=A Bg Problem 2=A Small Problem 3=Not a Problem.=Mssng 98=Inapplcable 99=No Answer Gven 1=A Bg Problem 2=A Small Problem 3=Not a Problem.=Mssng 98=Inapplcable 99=No Answer Gven 1=Yes 2=No.=Mssng 98=Inapplcable 99=No Answer Gven 1=Never 2=Sometmes 3=Usually 4=Always.=Mssng 98=Inapplcable 99=No Answer Gven Page 23

Varable Descrpton Response optons Q17 How often got appontment as soon as you wanted 1=Never 2=Sometmes 3=Usually 4=Always.=Mssng 98=Inapplcable 99=No Answer Gven Q19 How often got needed care as soon as you wanted 1=Never 2=Sometmes 3=Usually 4=Always.=Mssng 98=Inapplcable 99=No Answer Gven Q24 How often wat n doctor s offce 1=Never 2=Sometmes 3=Usually 4=Always.=Mssng 98=Inapplcable 99=No Answer Gven Q39 General health ratng 1=Excellent 2=Very Good 3=Good 4=Far 5=Poor.=Mssng 98=Inapplcable 99=No Answer Gven Q40 Age of respondent 1=18 to 24 2=25 to 34 3=35 to 44 4=45 to 54 5=55 to 64 6=65 to 74 7=75 or older.=mssng 98=Inapplcable 99=No Answer Gven Q41 Gender 1=Male 2=Female.=Mssng 98=Inapplcable 99=No Answer Gven Page 24

Varable Descrpton Response optons Q21 Offce and clnc vsts n the past 6 months 1=None 2=1 tme 3=2 tmes 4=3 to 4 tmes 5=5 to 9 tmes 6=10 or more tmes.=mssng 98=Inapplcable 99=No Answer Gven Step 2: Modfyng CONTROL.SAS CONTROL.SAS s a SAS program that nvokes and executes the macro fle, CAHPS41.SAS, to perform basc analyses for the CAHPS surveys. Ths program can be modfed to perform the analyses that the team has decded to conduct. Statements from CONTROL.SAS are demonstrated below. The user can alter ths program to perform analyses on other data sets. The modfcatons demonstrated below apply to the test SAS program, TEST.SAS, as well. TEST.SAS demonstrates many of the key concepts for a lmted number of varables descrbed earler. (Note that the test data fles and sets were created for the CAHPS Health Plan Survey 3.0 and have not yet been updated.) Identfyng the Data Set, Macros, Program, and Output Fle Locatons The program statements below specfy the lbrary reference and fle names for the macros and data sets. These statements should be modfed based on the locaton of the fles. Note: The flename statements creatng logfle and outfle are not necessary unless the user wants to save the log nformaton to a fle named CONTROL.LOG and the prnted results to a fle named CONTROL.TXT. The lbname statement creatng out statement s requred to dentfy the locaton where the data sets of the summary results the program creates wll be placed. %let ProgramName = control ; %let root = /data/cahpsmmc/analyss_program/verson_4.1 ; lbname n &root./sasdata/ ; lbname out &root./sasdata/control/ ; lbname lbrary &root./sascatalog/ ; flename logfle &root./output/logs/&programname..log ; flename outfle &root./output/&programname..txt ; flename cahps &root./programs/cahps41.sas ; flename plan_dat &root./data_other/plandtal.dat ; Page 25

Output Table Ttles The followng code clears the SAS ttles and footnotes from the output data tables. Ths step ensures that any ttles and footnotes prevously created n a SAS sesson are cleared before you execute the control program or macro. ttle ; footnote ; Prnt the Output to a Fle The followng statements can be removed f the user does not want to save the results to an explctly stated external fle. proc prntto prnt = outfle new log = logfle new ; run ; To return the log and output to the default sources, nclude the followng lnes of code at the end of the control fle. proc prntto ; run ; Data Set Specfcatons The followng statements prepare the test data set accordng to the specfcatons outlned under Computng Requrements. You may need to make modfcatons to the followng statements dependng on the varable names and varable response optons n the data set. It s very mportant that all varables n that secton are n a temporary or permanent SAS data set that wll be used for the analyss. General health status (Q39) and age (Q40) varables are prepared as case-mx adjusters for llustratve purposes. 1. Set permanent or temporary SAS data set. data adult ( drop = ) ; set n.test ; Page 26

2. Recodes numerc plan varables to character to smplfy nterpretaton of the result tables. length plan $ 16 ; f pland = 1 then plan = HMO_A_URBAN ; else f pland = 2 then plan = HMO_B_URBAN ; else f pland = 7 then plan = HMO_C_URBAN ; else f pland = 4 then plan = HMO_B_RURAL ; else f pland = 5 then plan = HMO_C_RURAL ; else f pland = 6 then plan = HMO_BE_1 ; 3. Creates vsts varable. vsts = q21 ; 4. Creates chld varable by codng t to 0 for all surveys. chld = 0 ; 5. Recodes dchotomous varables from 1-2 to 1-0. array yn q05 q13; do = 1 to dm ( yn ) ; f yn [] = 0 then yn [] =. ; else f yn [] = 2 then yn [] = 0 ; end ; 6. REVERSE codes tem n whch never s a postve response and always s a negatve response. array rev q24 ; do = 1 to dm ( rev ) ; f rev [] n (1, 2, 3, 4) then rev [] = 5 - rev [] ; else rev [] =. ; end ; 7. Verson 1.5 and hgher of the CAHPS program does not automatcally clean case-mx adjustment varables as prevous versons dd because t allows for a varyng number and specfcaton of the adjusters. If adjusters are used n the analyss they must be cleaned frst. age = q40 ; ghr = q39 ; f ghr not n (1, 2, 3, 4, 5) then ghr =. ; f age not n (1, 2, 3, 4, 5, 6, 7) then age =. ; CAHPS41.SAS Specfyng Arguments and Optons The followng statement ncludes the macro code CAHPS41.SAS. %nclude cahps ; The macro call statements for CAHPS41.SAS n CONTROL.SAS requre that at least sx arguments, (VAR, VARTYPE, NAME, ADULTKID, DATASET, and OUTNAME), be specfed for t to work properly. These arguments, along wth the 20 optonal arguments, are lsted n the table below wth the vald value ranges. The sx arguments must be specfed for each analytc run of the global ratngs and Page 27

compostes. If usng adjusters, then the ADJUSTER argument s requred. The macro call can be repeated any number of tmes n the SAS program for the dfferent compostes and ratngs the user wants to compare. Users can also analyze the same composte or global ratng more than once by usng dfferent macro arguments. For each composte, the user needs to specfy the tems lsted below n CONTROL.SAS. Arguments wth an astersk (*) are optonal and are needed only n specfc cases. Table 2. Arguments for CAHPS 4.1 Macro Opton or argument Descrpton Values Var Name of varable(s) n composte, or global ratng Name of varable(s) from SAS data set to nclude n composte or global ratng. For compostes, separate the varable names by a sngle space. Vartype Type of varable 1 = Dchotomous Scale (yes/no 0-1) 2 = Global Ratng Scale (0-10) 3 = How Often Scale or other four-pont response scale ( never to always scale 1-4) 4 = Any type of three-pont response scale (1-3) 5 = Other Scale (Must assgn a value to mn_resp and max_resp arguments) Page 28

Opton or argument Descrpton Values * Recode Recodes the global ratng* and the How Often scales down to three categores before performng the casemx adjustment and the statstcal tests the default value s 0 0 = For the statstcal tests, use default response optons for the varables n the Var argument. For the Percent of each response table and report, splt the Ratng scale nto three categores wth the followng break ponts, 0-6 7-8 9-10 or 1-2 3 4 for the How Often scale. Recode opton s not needed n the CAHPS macro call f t = 0. 1 = For the statstcal tests, recode the Global Ratng Scale (0-10, vartype = 2) and the How Often scale (1-4, vartype = 3) as: Ratng How Often 0 6 = 1 1 2 = 1 7 8 = 2 3 = 2 9 10 = 3 4 = 3 If vartype s not equal to 2 or 3, then no recodng occurs for the statstcal tests. For the Percent of each response table and report, splt the Ratng scale nto three categores wth the followng break ponts, 0-6 7-8 9-10 or 1-2 3 4 for the How Often scale. 2 = For the statstcal tests, use default response optons for the varables n the Var argument. For the Percent of each response table and report, splt the Ratng scale nto three categores wth the followng break ponts, 0-7 8-9 10 or 1-2 3 4 for the How Often scale. Page 29

Opton or argument Descrpton Values * Recode (contnued) 3 = For the statstcal tests, recode the Global Ratng Scale (0-10, vartype = 2) and the How Often scale (1-4, vartype = 3) as: Ratng How Often 0 7 = 1 1 2 = 1 8 9 = 2 3 = 2 10 = 3 4 = 3 If vartype s not equal to 2 or 3, then no recodng occurs for the statstcal tests. For the Percent of each response table and report splt the Ratng scale nto three categores wth the followng break ponts, 0-7 8-9 10 or 1-2 3 4 for the How Often scale. * Mn_resp Used wth vartype = 5 only the mnmum response value * Max_resp Used wth vartype = 5 only the maxmum response value Can be any numerc value. It wll be used as the low value for the vald response optons. Can be any numerc value. It wll be used as the hgh value for the vald response optons. Name Descrpton of composte or global ratng Note: Ths argument s lmted to 40 characters and can be numerc, text, or a combnaton of both. * Adjuster Name(s) of adjuster varables * Adj_bars Flag ndcatng f the frequences for the response values are to be case-mx adjusted for the trple stacked bar the default value s 0 Name(s) of case-mx adjuster varables separated by a space f usng more than 1. 0 = Do not case-mx adjust the trple stacked bars. 1 = Case-mx the trple stacked bars and store the adjusted frequences along wth the unadjusted frequences. Page 30

Opton or argument Descrpton Values * Bar_stat Flag ndcatng f permanent data sets for the case-mxed frequences should be 0 = Do not case save the statstcal results n data sets for the case-mx adjusted trple stacked frequency bars. saved the default value s 0 1 = Save the case-mx adjusted statstcal results n permanent data sets for the trple stacked frequency bars. * Impute Imputaton of mssng data for adjuster varables the default value s 0 * Even_wgt Determnes how to weght composte tems the default value s 1 * K Assgn a target mnmum response sze for equal weghtng for compostes (even_wgt = 1) - the default value s 1. * Kp_resd Flag used to make the resdual values from the SAS work data set RES_4_ID n the STD_DATA module. The resduals are the response values after case-mx adjustments have been made the default value s 0 0 = Do not mpute mean values by plan for all adjuster varables. 1 = Impute mean values by plan for all adjuster varables. 0 = Use tem weghtng for compostes. 1 = Use equal weghtng for compostes (1 / # of Items). 2 = Apply the respondent level weght, n WGTRESP, to the tem weghtng for compostes. Number 0. 0 = Do NOT save the resdual response values. 1 = Save the resdual response values n a permanent data set. Page 31

Opton or argument Descrpton Values Adultkd Specfes how to analyze chld and adult surveys 0 = Combne adult and chld surveys n analyss; do not consder nteracton effects n case-mx adjustment. Ths opton can also be used f the data set contans only a sngle type of survey. 1 = Combne adult and chld survey data n analyss; consder nteracton effects between chld and each case-mx adjuster varable. 2 = Analyze chld data only. 3 = Analyze adult data only. * Vsts Specfes whether to analyze hgh and low users together or separately (based on VISITS varable) the default value s 1 * Pvalue Level of sgnfcance for comparsons the default value s 0.05 * Change Level of practcal sgnfcance based on a percentage dfference from the mnmum absolute theoretcal dfference from the overall mean (can be used only wth p-value crtera) the default value s 0 * Meandff Level of practcal sgnfcance based on absolute dfference between plan mean and mean of all plans (can be used only wth p-value crtera) the default value s 0 1 = All vsts. 2 = Low users only (< 3 vsts per 6 months). 3 = Hgh users only ( 3 vsts per 6 months). 0.05 recommended, but vald values are between 0 and 1. Value between 0 and 1 (.e., 25% s entered as 0.25). Number 0. Page 32

Opton or argument Descrpton Values * Wgtdata Specfes whether samples 1 = Do not nvoke weghtng macro are stratfed wthn health plan the default value s 1 2 = Combne strata, weghtng * Wgtresp Name of the varable storng the weght values for ndvdual respondents. the default value s blank * Wgtmean Name of the varable storng the weght values for the plan means the default value s blank * Wgtplan Specfes whether to use plan weghts for the plan level statstcal test or not. The default value s zero. * Id_resp If there s unque varable n the data set that dentfes each ndvdual respondent, then ths varable name may be entered here the default value s blank Blank or the name of a varable n the data set Blank or the name of a varable n the data set 0 = Do not use the plan weghts when computng the overall mean for the comparson of plan means. Equal weghtng wll be used as n prevous versons of the macro. 1 = Use the sum of the weghts to the plan level of the varable specfed n the parameter wgtmean. Ths weght s used for weghtng the overall and grand means used n the statstcal comparsons of the plan means. Blank or the name of a varable n the data set. Ths varable s ncluded n the resdual data set when kp_resd = 1. The varable wll be a character and have a maxmum of 50 characters. Page 33

Opton or argument Descrpton Values * Subset Perform the case-mx adjustments and 1 = No subsettng done. Global case-mx model and centerng. statstcal test based on each subset of plans; the subset code s a column 2 = Global case-mx model wth centered means for each subset before performng statstcal tests. n the plan detal fle the default value s 1 3 = Subset case-mx model wth centered means for each subset. * Spltflg The default value of 0 lets the macro run the data set as usual wth every plan centered to the same mean and the case-mx beng run once. If the flag s set to 1 then the data set must contan the varable SPLIT and the values of ths varable n the data set must be 0 and 1 for each plan subset. * Smoothng Assgn weght for pooled varance estmate n smoothng varances - the default value s 0. Dataset SAS data set name to be used n the analyss (varables recoded and renamed accordng to Computng Requrements ) * Outregre A flag that ndcates whether or not to nclude the regresson output text created by SAS, n the fnal text report fle the default value s 0 0 = Run macro wth one case-mx model 1 = Run macro wth two case-mx models Example: Managed Care plans splt=0 and for the other plans, Fee for Servce, splt=1. If there are any mssng values for ths varable, then these records are dropped from the analyss; the default value s 0. Value greater than 0. For a detaled explanaton of how ths value s selected, see the secton called Explanaton of Statstcal Calculatons. Data set name depends on how you called n the fle. 0 = No regresson output appears n the text report fle. 1 = SAS prntout from the PROC REG s ncluded n the text report fle. Outname Part of SAS data set name for output tables created for summary results Maxmum length s fve characters f usng SAS 6, can be longer for SAS 8 or later. If the user does not want SAS data sets created enter. Remember, the results tables wll stll be created for the.out fle. Page 34

Examples of usng these arguments wth the TEST data set are lsted below. * Executes CAHPS macro wth global ratng scale varable, no adjusters and combnng strata. %cahps(var = q38, vartype = 2, name = Ratng Health Plan, adjuster =, adultkd = 3, vsts = 1, wgtdata = 2, dataset = test, outname = rplan ) ; * Executes CAHPS macro wth How Often composte varables and the recode opton = yes, tem weghtng opton = yes and 2 adjusters; %cahps(var = q15 q17 q19 q24, vartype = 3, recode = 1, name = Gettng Care Quckly, adjuster = age ghr, mpute = 1, adultkd = 3, wgtdata = 2, dataset = test, outname = quck ) ; * Executes CAHPS macro wth global ratng scale varable, age and ghr adjusters, combnng strata, and smoothng varances. Note: smoothng = 25 s as an example. For a detaled explanaton of how ths value s determned, see the secton called Explanaton of Statstcal Calculatons. %cahps(var = q38, vartype = 2, name = Ratng Health Plan, adjuster = age ghr, adultkd = 3, vsts = 1, wgtdata = 2, smoothng = 25, dataset = test, outname = rplan ) ; Page 35

Interpretng the Results The CAHPS Analyss Program prnts the results of the analyses performed for each composte and global ratng. The program produces entty-level estmates of mssng data for the analyss tems and case-mx adjusters, calculates the percentage of responses n each category, compares performance of enttes on the reportng tem, and flags enttes wth fewer than 100 responses. If adjusters are used, then the coeffcents and the regresson analyss are produced for each adjuster tem. Examples of results tables from the test data set for global ratng scales are revewed below. Please note that the results tables are also output to SAS data sets. These data sets mplement the followng namng conventons where &OUTNAME s the text assgned by the user to the varable outname n the CAHPS macro call. Percentage Items Mssng: Percentage of Each Response: (for Global Ratng* aggregates to 0-6, 7-8, and 9-10, or 0-7, 8-9, 10 for How Often Scale aggregates to 1-2, 3, and 4) Regresson Coeffcents: R-Squared Values Resdual Values (only f KP_RESID = 1) Overall Statstcs for All Enttes: Star Ratngs for All Enttes: Plans dropped by macro wth only 0 or 1 record P_&OUTNAME N_&OUTNAME C_&OUTNAME R2&OUTNAME Y_&OUTNAME OA&OUTNAME SA&OUTNAME DP&OUTNAME * See the FAQs on the CAHPS Web ste (https://www.cahps.ahrq.gov) to learn more about the cutponts for ths scale. If the stratfed weghtng opton = 2, then the followng data sets wll be created for each unstratfed entty. Percentage Items Mssng: Percentage of Each Response: Overall Statstcs for All Enttes: Star Ratngs for All Enttes PW&OUTNAME NW&OUTNAME OW&OUTNAME SW&OUTNAME Page 36

If the keep permanent data sets for case-mx adjusted frequences opton = 1 and the stratfed weghtng opton = 1, (no stratfed weghtng), then the followng data sets wll be created. Overall statstcs for all enttes for frst bar/frequency: Star ratng detals for all enttes for frst bar/frequency: Overall statstcs for all enttes for second bar/frequency: Star ratng detals for all enttes for second bar/frequency: Overall statstcs for all enttes for thrd bar/frequency (not for dchotomous varables): Star ratng detals for all enttes for thrd bar/frequency (not for dchotomous varables): F1&OUTNAME B1&OUTNAME F2&OUTNAME B2&OUTNAME F3&OUTNAME B3&OUTNAME If the keep permanent data sets for case-mx adjusted frequences opton = 1 and the stratfed weghtng opton = 2 (do stratfed weghtng), then the followng addtonal data sets wll be created. Overall statstcs for unstratfed data for frst bar/frequency: Star ratng detals for unstratfed data for frst bar/frequency: Overall statstcs for unstratfed data for second bar/frequency: Star ratng detals for unstratfed data for second bar/frequency: Overall statstcs for unstratfed data for thrd bar/frequency (not for dchotomous varables): Star ratng detals for unstratfed data for thrd bar/frequency (not for dchotomous varables): FA&OUTNAME BA&OUTNAME FB&OUTNAME BB&OUTNAME FC&OUTNAME BC&OUTNAME For a detaled descrpton of the computaton and statstcal analyses used to develop these results, see the secton Explanaton of Statstcal Calculatons. At ths pont, all data elements have been collected to perform the testng on the hypothess. Page 37

Warnngs and Parameter Info The followng page of the SAS text output shows each of the parameter settngs for the Analyss Program. You can use ths to dentfy tems you may want to consder when nterpretng the results produced by the program, such as enttes wth fewer than 100 responses to an tem after consderng mssng adjusters and analyss tems. No SAS data set s produced that contans all ths nformaton. Ratng Scale (0-10): Rate Plan Analyss = ADULTS ONLY - Vsts = COMBINE LOW AND HIGH USERS *---------------------------------------------* CAHPS SAS Analyss Program Verson 4.1 Report run on 20 May 2011 at 14:04:45 *---------------------------------------------* ********** WARNING NOTE ********** PLANS WITH FEWER THAN 100 CASES -------------------------------------------- Plan ID 2 - HMO_B_RURAL - 95 Cases Plan ID 4 - HMO_C_RURAL - 68 Cases -------------------------------------------- The Varable Item = q38 The Varable Type = 2 The 2 Adjuster Varables = q40 q39 Global Case Mx Model Global Centerng of Means The RECODE parameter = 0 The MIN_RESP parameter = 0 The MAX_RESP parameter = 10 The NAME parameter = Ratng Health Plan The ADJ_BARS parameter = 1 The BAR_STAT parameter = 0 The IMPUTE parameter = 0 The EVEN_WGT parameter = 1 The KP_RESID parameter = 0 The ADULTKID parameter = 3 The VISITS parameter = 1 The PVALUE parameter = 0.05 The CHANGE parameter = 0 The MEANDIFF parameter = 0 The WGTDATA parameter = 2 The WGTRESP parameter = The WGTMEAN parameter = The WGTPLAN parameter = 0 The ID_RESP parameter = The SUBSET parameter = 1 The SPLITFLG parameter = 0 The data set used = test The OUTREGRE parameter = 0 The output data set suffx = rplan Page 38

Percent of Items Mssng by Health Plan Ratng Scale (0-10): Ratng Health Plan Analyss = ADULTS ONLY - Vsts = COMBINE LOW AND HIGH USERS PERCENT ITEMS MISSING BY HEALTH PLAN Health Plan Total # of Respondents Global Ratng of Plan Age Range General Health Ratng HMO_A_URBAN 345 30.43% 1.16% 0.58% HMO_B_RURAL 134 29.10% 1.49% 0.75% HMO_B_URBAN 530 24.15% 1.70% 0.94% HMO_C_RURAL 90 23.33% 0.00% 2.22% HMO_C_URBAN 874 29.18% 0.69% 0.92% Report run on 20 May 2011 at 14:04:45 CAHPS SAS Analyss Program Verson 4.1 Data Set out.p_rplan Defnton of column headngs Column Descrpton Health plan Health plan names based on the varable PLAN recoded from PLANID. Total number of respondents Global ratng of plan (Label for the varable tem Q38) General health ratng (Label for the adjuster Q50 GHR) Age of adult (Label for the adjuster Q51 - AGE) Total number of respondents n the data set by health plan. For llustraton purposes, an entty wth less than 100 respondents has been left n the sample output. Percent of cases or records wth each tem response mssng. At least two tems must have a vald response to be ncluded n the computaton. The only answers counted as nonmssng by the SAS program for the global ratng scales are the responses wth nteger values from 0 to 10. All other values are converted to the SAS mssng value.. Computaton of percent mssng s based on the total number of patents n each plan. Answers counted as nonmssng by the SAS program are the responses excellent (coded 1), very good (coded 2), good (coded 3), far (coded 4), and poor (coded 5). Computaton of percent mssng s based on the total number of patents n each plan. Answers for adult surveys counted as nonmssng by the SAS program are the responses 18-24 (coded 1), 25-34 (coded 2), 35-44 (coded 3), 45-54 (coded 4), 55-64 (coded 5), 65-74 (coded 6), and 75+ (coded 7). Answers for chld surveys counted as nonmssng by the SAS program are the responses <1 (coded 0), 1-3 (coded 1), 4-7 (coded 2), 8-12 (coded 3), and 13-17 (coded 4). Computaton of percent mssng s based on the total number of patents n each plan. Page 39

Percent of Composte Responses by Category 2 Ratng Scale (0-10): Rate Plan Analyss = ADULTS ONLY - Vsts = COMBINE LOW AND HIGH USERS PERCENT RESPONSE TYPE - NO IMPUTATIONS Number of Total # of Respondents % Ratng % Ratng % Ratng Health Plan Respondents Analyzed 0-6 7-8 9-10 HMO_A_URBAN 345 238 17.23% 37.39% 45.38% HMO_B_RURAL 134 95 21.05% 38.95% 40.00% HMO_B_URBAN 530 393 15.78% 34.35% 49.87% HMO_C_RURAL 90 68 16.18% 33.82% 50.00% HMO_C_URBAN 874 613 15.33% 38.99% 45.68% Health Plan Adjusted Bar 1 Adjusted Bar 2 Adjusted Bar 3 HMO_A_URBAN 17.81% 37.92% 44.27% HMO_B_RURAL 20.84% 39.38% 39.78% HMO_B_URBAN 15.92% 34.22% 49.87% HMO_C_RURAL 15.51% 32.94% 51.55% HMO_C_URBAN 15.49% 39.04% 45.47% Report run on 20 May 2011 at 14:04:45 CAHPS SAS Analyss Program Verson 4.1 Data Set out.n_rplan Defnton of column headngs Column Descrpton Health plan Health plan names based on the varable PLAN recoded from PLANID. Total number of respondents Number of respondents analyzed Percent Ratng 0 6 Percent Ratng 7 8 Percent Ratng 9 10 Adjusted Bar 1 Adjusted Bar 2 Adjusted Bar 3 Total number of respondents n the data set by health plan. Number of respondents wth nonmssng values (as defned n Percent of Mssng Items on page 44). Percent of Ratng tem wth response values from 0 to 6, by health plan. Percent of Ratng tem wth response values equal to 7 or 8, by health plan. Percent of Ratng tem wth response values equal to 9 or 10, by health plan. Case-mx adjusted frequences for bar 1, Percent of Ratng tem wth response values from 0 to 6, by health plan. Case-mx adjusted frequences for bar 2, Percent of Ratng tem wth response values equal to 7 or 8, by health plan. Case-mx adjusted frequences for bar 3, Percent of Ratng tem wth response values equal to 9 or 10, by health plan. 2 For a detaled explanaton of how these calculatons were performed, see the secton called Explanaton of Statstcal Calculatons. Page 40

Case-mx Adjuster Regresson Coeffcents Ratng Scale (0-10): Rate Plan Analyss = ADULTS ONLY - Vsts = COMBINE LOW AND HIGH USERS REGRESSION COEFFICIENTS FOR ADJUSTER VARIABLES Varable Subset Name splt Name Q38 Q40 Q39 0 0 GLOBAL GLOBAL 0.3113-0.4191 Report run on 20 May 2011 at 13:32:32 CAHPS SAS Analyss Program Verson 4.1 Data Set out.c_rplan Defnton of column headngs Column Varable name Q39 = General health ratng 1 Excellent 2 Very good 3 Good 4 Far 5 Poor Descrpton Q40 = Age 1-18-24 2-25-34 3-35-44 4-45-54 5-55-64 6-65-74 7-75+ Splt Subset name Q38 If the SPLITFLG parameter equals 0, then there s no splt for the case-mx regresson and splt s 0. If SPLITFLG equals 1, then the two regressons are run, one for each splt, where splt equals 0 and 1. If subsettng s used for case-mx adjustment, the subset name or code s found n ths column. Otherwse t defaults to GLOBAL; that s, t used all records n the case-mx adjustment regresson. Regresson coeffcents for the Global Ratng of Plan varable. Page 41

R-Squared Values for Dependent Varables Ratng Scale (0-10): Rate Plan Analyss = ADULTS ONLY - Vsts = COMBINE LOW AND HIGH USERS R-SQUARED VALUES for DEPENDENT VARIABLES splt Subset Name Dependent varable R-squared Adjusted r-squared 0 GLOBAL Q38 0.0768 0.0755 Report run on 20 May 2011 at 13:32:32 CAHPS SAS Analyss Program Verson 4.1 Data Set out.r2rplan Defnton of column headngs Column Descrpton Splt If the SPLITFLG parameter equals 0, then there s no splt for the case-mx regresson and splt s 0. If SPLITFLG equals 1, then the two regressons are run, one for each splt, where splt equals 0 and 1. Subset Name Dependent Varable R-squared Adjusted R-squared If subsettng s used for case-mx adjustment, the subset name or code s found n ths column. Otherwse t defaults to GLOBAL; that s, t used all records n the case-mx adjustment regresson. Global Ratng of Plan varable. The R-squared value from the regresson for the dependent varable. The Adjusted R-squared value from the regresson for the dependent varable. Page 42

Overall Statstcal Test Results Ratng Scale (0-10): Rate Plan Analyss = ADULTS ONLY - Vsts = COMBINE LOW AND HIGH USERS P-Value For Contrast = 0.05 - Change > 0 - Meandff > 0 Overall Statstcs from t-test Ho: Plan Means All Equal Subset Overall Name Mean DFR DFE F-Statstc P-Value GLOBAL 7.9671 4 1,400 1.4508 0.2150 Report run on 20 May 2011 at 13:32:32 CAHPS SAS Analyss Program Verson 4.1 Data Set out.oarplan Defnton of column headngs Column Descrpton Subset Name If subsettng s used for case-mx adjustment, the subset name or code s found n ths column. Otherwse t defaults to GLOBAL; that s, t used all records n the case-mx adjustment regresson. Overall Mean DFR DFE F-Statstc P-Value The mean of all the plan means. The numerator degrees of freedom. The denomnator degrees of freedom. The results of the F-test on the null hypothess. One mnus the probablty of the F dstrbuton. NOTE: Ho presents a global test of the null hypothess that all plans have the same adjusted mean ratng. Page 43

Statstcal Test Performance by Health Plan Ratng Scale (0-10): Ratng Health Plan Analyss = ADULTS ONLY - Vsts = COMBINE LOW AND HIGH USERS P-Value For Contrast = 0.05 - Change > 0 - Meandff > 0 ALL PLANS Plan Dff. # of Unweghted Weghted Adjusted From Total # of Respondents Unadjusted Unadjusted Plan Overall Plan Name Respondents Analyzed Plan Mean Plan Mean Mean Mean HMO_A_URBAN 345 238 7.9622 7.9622 7.9189-0.0482 HMO_B_RURAL 134 95 7.7053 7.7053 7.7120-0.2551 HMO_B_URBAN 530 393 8.1501 8.1501 8.1434 0.1763 HMO_C_RURAL 90 68 7.9559 7.9559 8.0093 0.0422 HMO_C_URBAN 874 613 8.0620 8.0620 8.0519 0.0848 Std Error +/- 95% Conf. Varance Varance of Lmt of of the of the Plan Name Dfference Dff. Mean Mean - old Ratng Plan Weght HMO_A_URBAN 0.1356 0.2657 0.0125 0.0125 ** 1.00 HMO_B_RURAL 0.1982 0.3884 0.0348 0.0360 ** 1.00 HMO_B_URBAN 0.1155 0.2263 0.0071 0.0071 ** 1.00 HMO_C_RURAL 0.2265 0.4440 0.0476 0.0497 ** 1.00 HMO_C_URBAN 0.1042 0.2041 0.0045 0.0045 ** 1.00 Report run on 20 May 2011 at 13:32:32 CAHPS SAS Analyss Program Verson 4.1 Data Set out.sarplan Defnton of column headngs Column Plan Name Total Number of Respondents Number of Respondents Analyzed Unadjusted Plan Mean Adjusted Plan Mean Plan Dff. from Overall Mean Std Error of Dfference Varance of the Mean Ratng Descrpton Health plan names based on the varable PLAN recoded from PLANID. Total number of patents n the data set by health plan. Number of respondents wth nonmssng values (as defned above) for the composte. Average health plan composte not adjusted for age or health status. Average health plan composte adjusted for age and health status. Overall Mean mnus Adjusted Plan Mean. Standard error of Plan Dfference From Mean. Varance of the plan means. Star ratng of plan performance for the global plan ratng based on a comparson of plan s Adjusted Plan Mean to Overall Mean Identfes statstcally meanngful dfferences. * = Health Plan was sgnfcantly below average ** = Health plan was not sgnfcantly above or below average *** = Health plan was sgnfcantly above average Page 44

Small Data Set Example CAHPS Surveys and Instructons Ths secton uses a small data set wth ten records, two enttes, two questons, two adjusters and one weght varable to walk through an example of what happens to the data set as t moves through the CAHPS macro. The perods (.) n the table below represent mssng values. The observaton numbers are not a part of the data set; they are used only for reference purposes later. The sample tables after ths one use the shorter column headngs (Obs, Plan, Q1, Q2, A1, A2). Please note that a varable PLAN refers to your unt of analyss. Varables n tables below are used nsde of the macro. SAS data set SMALLTST Obs Observaton Plan Q1 Queston 1 Q2 Queston 2 A1 Adj 1 A2 Adj 2 Weght 1 A 2 4 1 1 40 2 A 3. 2 2 50 3 A 4 2 3. 6 4 A 4 3.. 8 5 A 3 3 2 3 10 6 B 3 3 2 3 3 7 B.. 4 5 5 8 B 2 2 5 4 3 9 B 3 2 6 3 5 10 B 7 3 3 3 3 The plan detal fle s created by the macro and looks lke the data set below. Ths data set s used by the macro to dentfy the plans t needs to analyze and create sequental plan numbers for use wthn the macro. Plan A s assgned the value of 1 and Plan B s assgned the value of 2. The macro needs the numerc values to perform loopng functons at varous ponts. In the ALLCASES secton of the macro, t merges ths nformaton wth the SMALLTST data set. In ths example, we do not perform any strata weghtng or subsettng of the data, so the values for these felds are set to 1. For the remander of ths example, the plan names A and B are used and the nternal macro dentfcatons are gnored. Plan detals data Observaton Orgnal plan New plan Populaton sze (strata weght) Subsettng code 1 A A 1 1 2 B B 1 1 Page 45

Ths example follows two paths for the analyss of the composte measure Q1 and Q2. One uses no adjuster varables, Run 1, and the other uses the two adjusters, A1 and A2, wthout mputaton of mssng values of the adjuster s mean wthn plan, Run 2. The macro cleans (makng sure the values are wthn the vald range for the gven varable type) the tems beng analyzed, Q1 and Q2. In the macro call they were ndcated as beng a type 3 varable, whch means the response values must be a 1, 2, 3, or 4. Any other response value s set to mssng. In our small data set (observaton 10), Q1 has a value of 7 so t s set to mssng; all other values are fne. The adjuster values are not cleaned n the macro, so all values are accepted. The frst step n the macro that begns to prepare the data for the reports s the USABLE secton of the macro. Ths checks for mssng values n each observaton and determnes whether to keep the record based on the macro arguments. The results may dffer dependng on whether adjusters are used and whether mssng adjusters get an mputed mean value. The observatons that are dropped for Run 1 and Run 2 after the USABLE secton are noted as follows: Obs Plan Q1 Q2 A1 A2 Run 1 - No Adj Run 2 - Wth Adj 1 A 2 4 1 1 2 A 3. 2 2 3 A 4 2 3. A2 Mssng 4 A 4 3.. A1 & A2 Mssng 5 A 3 3 2 3 6 B 3 3 2 3 7 B.. 4 5 Q1 & Q2 Mssng Q1 & Q2 Mssng 8 B 2 2 5 4 9 B 3 2 6 3 10 B. 3 3 3 The next few sectons of the macro use the records retaned from the USABLE secton, nne records for Run 1 and seven records for Run 2. These sectons smply report and summarze that data for low number of respondents, percent mssng for each varable, and the percent breakdown of the response categores. The next step s to standardze each analyss varable n the data to a mean of zero and perform the case-mx adjustment f there are adjusters, as n Run 2. Page 46

Next we must determne the case-mx adjusters for each plan and the resduals for each tem to obtan the adjusted means and calculate the varance for each plan. In Run 1, we are not dong any case-mx adjustments, so the adjustment to the means s zero and the resduals are the standardzed values n the above table. In Run 2, there are adjusters, so the macro performs the regresson necessary to get the adjustments for the means and the resduals. Adjustments (on mean = 0) Plan Run 1 Run 2 A 0.00-0.25 B 0.00 0.25 Resduals Run 1 Run 2 Obs Plan Q1 Q2 Q1 Q2 1 A -1.20 1.00-0.67 0.50 2 A -0.20. 0.33. 3 A 0.80-1.00 NA NA 4 A 0.80 0.00 NA NA 5 A -0.20 0.00 0.33-0.50 6 B 0.33 0.50 0.33 0.50 7 B NA NA NA NA 8 B -0.67-0.50-0.67-0.50 9 B 0.33-0.50 0.33-0.50 10 B. 0.50. 0.50 Page 47

The next step s to multply each resdual by the tem s equal weght, 1/number tems n the composte (0.50) and dvde by the total number of vald responses wthn plan and tem n the composte. Then we can sum the results of the weghted resduals for each composte record. Obs Plan Run 1 Run 2 1 A 0.005-0.09 2 A -0.020 0.06 3 A -0.045 NA 4 A 0.080 NA 5 A -0.020 0.03 6 B 0.118 0.04 7 B NA NA 8 B -0.174-0.12 9 B -0.007 0.06 10 B 0.063-0.02 Next we sum the squared composte weghted resduals by plan. Plan Run 1 Run 2 A 0.009 0.01 B 0.048 0.02 The fnal step to prepare for the statstcal test s to calculate the varance wthn the plan usng the above results by plan, multpled by the number of usable records, dvded by the number of usable records mnus one. Plan Run 1 Run 2 A 0.012 0.017 B 0.064 0.027 At ths pont, all data elements have been collected to perform the testng on the hypothess. Page 48

Explanaton of Statstcal Calculatons CAHPS Surveys and Instructons The purpose of ths secton s to descrbe the CAHPS macros n suffcent detal so that a statstcally sophstcated reader can understand what analyses have been appled and some of the essental detals of the mplementaton. Note: In ths secton, a plan represents an entty (e.g., health plans, provder groups, ndvdual physcans). General Descrpton The CAHPS macros are desgned to carry out a seres of standard analyses on cleaned CAHPS data sets. Inputs to the macro. The nput to the macro s a SAS data set wth one observaton per survey respondent. The data set may contan only chld responses, only adult responses, or a mxture of the two. (If there s a mxture, an opton must be selected to ndcate whch group(s) are ncluded n the analyss.) See Step 2, Modfyng CONTROL.SAS for more detals on the preparaton of the fles. Outputs and statstcal tests. The macro produces the followng prnted outputs for each summary scale (0-10 sngle tem scale) and composte evaluaton tem: 1. A summary of warnngs and parameter nformaton. 2. A summary of mssng data (tem nonresponse) rates by tem and plan. 3. A summary table of responses by tem and plan for each category (wth never and sometmes combned for How Often Scale and 0-6, 7-8 and 9-10 combned for the Global Ratng Scale). 4. If adjusters are used, a prntout of the regresson for each adjuster, a table of the adjuster coeffcents and the dependent varable R-squared value. 5. Overall results of a summary hypothess test ndcatng the strength of the evdence that the plan means are not all dentcal on the gven tem or composte. 6. A table summarzng the statstcal analyss, whch ncludes unadjusted and adjusted plan means (percent yes for yes/no tems), dfferences from the overall mean and standard error of those dfferences, and star-ratngs (one to three stars) ndcatng statstcally sgnfcant dfferences. Data Subsettng and Checkng Subsettng. If only adults or only chldren are beng analyzed but the data fle combnes both groups, the approprate records are selected. Hgh users can be selected by specfyng the approprate macro argument. Response opton checkng. Item responses are checked to make sure that they conform to the response optons for that varable tem. All other responses are converted to mssng values. After ths recodng operaton, the percent of mssng tems by plan s calculated and prnted for each tem n the analyss. Page 49

Number of responses. The number of cases s calculated for each entty. If the number of cases n the analyss for an entty falls below a cutoff value (100 cases), a warnng message s prnted. Ths s only a warnng and does not affect any further analyses. Weghtng Algorthm for Compostes Once the SAS program groups the questonnare tems, t then computes means for each entty for each composte and global ratng. Because the compostes nclude more than one tem, a more elaborate computaton s requred to develop the mean. CAHPS uses tem weghts to compute the means of the compostes for each entty. Two methods are avalable for computaton of the tem weghts. Frst, the CAHPS macro now ncludes an opton to use equal tem weghtng n the composte, even_wgt = 1, where the tem weght equals one dvded by the total number of tems. So f equal weghtng was chosen and there were four tems n the composte, the tem weght s 1/4 = 0.25 for each tem. By default, f even_wgt s not specfed n the macro argument, then the composte uses even weghtng. An advantage of ths approach s that the relatve weghts of the tems n the composte are consstent among survey admnstratons. Furthermore, survey sponsors may regard each tem as equally mportant even f some are answered more frequently than others. A dsadvantage of ths opton s a possble loss of statstcal precson f an tem wth few responses s combned, equally weghted, wth an tem wth many responses. The CAHPS 4.1 macro offers some optons that solve ths problem by downweghtng of low-response tems. The frst modfcaton s motvated by the fact that responses to dfferent tems n the same composte often have dfferent mean values for a varety of reasons, ncludng how frequently problems arse n dfferent knds of nteractons and servces and how the questons are worded. If the tems are weghted the same way for every unt to calculate the composte, the effect of these unequal means across unts s mnmal. However, f tems are not weghted equally, ths could gve rse to varatons unrelated to varatons n qualty. Thus, we frst modfy calculaton of weghted compostes to mnmze the mpact of such dfferences n tem means on expected scores. To explan the need for ths modfcaton, suppose y s the mean score for tem at a gven unt, and µ s the mean score for tem across all unts. Wth weghts w that sum to 1, the composte score wy for a specfc plan, and f that plan s at the average on all measures, ts score s s w µ. If the overall means µ dffer, ths last expresson wll depend on w ; n other words even two plans that are average on every measure wll receve dfferent composte scores f the compostes are calculated wth dfferent weghts. To remove ths dependence, we center the scores at ther means before combnng them. Suppose now that w represents the weght for tem at a partcular unt, and w 0 represents some standard weghts common to the entre report. Now defne a composte score as w( y µ ) + w0 µ. Any unt that s average (y =µ ) on every tem Page 50

wll receve the same composte score w 0µ regardless of the weghts w, so bas due purely to weghtng s removed even f dfferent unts are scored wth dfferent weghts. Note that the second term of ths composte score expresson s the same for every unt; t s ncluded only to brng the average back to an nterpretable level as an average score of overall means. Gven ths modfcaton, we can now consder modfyng tem weghts for dfferent unts. The man requrement s that the weght must be zero (w =0) when there are no responses for tem ; we also want the weghts to be equal (or at least to approach equalty) when there s adequate sample for every tem. One smple weghtng mechansm meetng these requrements follows: Set w 0 =1/I, =1,, I, where I s the number of tems n the composte. Choose a cutoff number of observatons K; weghts wll not be modfed for tems wth at least K observatons. Defne unt-specfc weghts w = mn( n, K) mn( n ', K), where n s the ' = 1,..., I number of responses from the unt for tem I, and mn( n, K ) s the lesser of n and K. Calculate composte scores as descrbed above. Ths procedure has the followng desrable propertes: For each unt, all tems wth at least K responses are gven equal weght. Consequently there s no modfcaton to equal tem weghtng for unts wth large samples. Items wth no responses n a gven unt are gven no weght, so the composte score can stll be calculated. Items wth low numbers of responses (<K) are gven reduced weght so ther effect on varance s mtgated. The crteron for determnng whether an tem wll be downweghted s very smple to descrbe. The procedure can easly be modfed for unequal baselne weghts w 0. Page 51

Examples The followng table llustrates the calculaton of tem weghts for varous scenaros n a composte wth three tems, assumng that the target mnmum sample sze K=20. Sample szes n mn(n,k) Calculaton of weghts w Weghts w smplfed Interpretaton 60, 70, 80 20, 20, 20 20/60, 20/60, 20/60 1/3, 1/3, 1/3 0, 22, 24 0, 20, 20 0/40, 20/40, 20/40 0, 1/2, 1/2 10, 22, 34 10, 20, 20 10/50, 20/50, 20/50 1/5, 2/5, 2/5 2, 3, 5 2, 3, 5 2/10, 3/10, 5/10 2/10, 3/10, 5/10 Every tem has adequate sample so equal weghtng s OK Item wth no responses gets no weght One tem has low response and s downweghted If all samples are small, weght each tem proportonal to number of responses to mprove effcency of estmaton The followng llustrates the calculaton of the centered weghted average n a unt n whch one tem of the composte has few responses (thrd lne of table above), agan assumng K=20. Descrpton Symbol Item 1 Item 2 Item 3 Baselne equal weghtng w 0 1/3 1/3 1/3 Overall (all unts) mean µ 3.45 2.75 2.65 Mean n a specfc unt y 3.55 2.80 2.75 Sample szes n that unt n 10 22 34 Weghts n that unt w 1/5 2/5 2/5 Centered unt means y µ 0.10 0.05 0.10 The baselne weghtng s assumed to be equal for the three tems. Thus the overall mean composte score s (3.45+2.75+2.65)/3 = 2.95. Because at the specfc unt of nterest there are only 10 responses for Item 1, t s gven half the weght of each of the other tems. The weghted mean for the unt s then (1/5) 3.55 + (2/5) 2.80 + (2/5) 2.75 = 2.93. Note that ths s below the overall mean composte score, despte the fact that the unt s above the mean on each tem, because the tem that generally has a hgh score s downweghted. To calculate the score by the proposed method, we frst calculate the centered means (last lne of table), whch are all postve. Ther weghted mean s (1/5) 0.10 + (2/5) 0.05 + (2/5) 0.10 = 0.08. We then add ths mean devaton from mean and add t to the overall Page 52

mean, 0.08 + 2.95=3.03, whch s the reported score. Ths correctly reflects the superorty of ths unt across all the tems. A second approach weghts tems unequally, n proporton to the number of respondents. To use tem weghtng by number of responses, the argument even_wgt = 0 must be entered nto the macro call. The followng bref explanaton of the ratonale for selectng ths method of computng composte means s followed by a descrpton of how the SAS program carres out the computaton. Because there are dfferences among respondent experences and, consequently, dfferences n the rates at whch respondents use varous servces (expressed n skp patterns for varous tems), there are often dfferent numbers of responses for the tems that make up a composte. To reflect the dfferng numbers of responses by tem, each tem may be assgned a dfferent weght n the composte score (an tem weght). Items that receve a greater number of respondent answers count more toward the composte score. Thus, an tem such as how often dd you need to see your personal doctor or nurse generally has greater weght than how often dd you need to see a specalst because more respondents are lkely to need to see ther personal doctors than a specalst. One not very satsfactory way to create tem weghts for the compostes s to develop a dfferent set of weghts for each entty beng evaluated, proportonal to the number of tem responses at that entty. Ths means, however, that entty composte scores depend not only on the means on each tem at the entty but also on the number of vald responses on each tem avalable for that entty. Therefore, an entty could actually have a lower composte score just because t had a hgh number of responses to an tem wth generally low scores. Because such effects could dstort comparsons among enttes, we do not recommend ths approach and t s not mplemented n the macro. Another opton accounts for the dfferent numbers of vald responses for each tem wthn a composte but does so across all enttes by standardzng the tem weght. We consder ths a vald approach to tem weghtng for the compostes and have ncorporated t nto the Analyss Program. Ths approach prevents enttes from farng worse or better just because they have fewer avalable vald responses because of skp patterns. For compostes, the results for the several tems must be weghted together. For each composte, a set of data-determned tem weghts s calculated and used for all enttes. The number of vald responses obtaned for each tem determnes these weghts. Page 53

The followng s a formal descrpton of the calculaton of unequal tem weghts. Let n p = number of vald responses obtaned from entty p on tem, n = p np = total number of vald responses obtaned on tem, w = n ( n ) = weght for tem, and F pr = fracton of responses for tem for entty p that fall nto response category r. The sums are over tems that are part of ths composte. Then the weghted fracton for response category r n entty p s w F pr Case-Mx Adjustment Another mportant polcy decson regardng the analyss of CAHPS data s whether and how to adjust the data for dfferent case-mx patterns. The CAHPS Development Team has studed ths ssue, and ts recommendatons for the adjustment procedure are ncorporated nto the SAS code, although the user may choose what varables to adjust for. The followng explans the mportance of the adjustment and how t s mplemented. When comparng enttes on the bass of the ratngs by ndvduals covered by those plans, t s mportant to adjust the data for patent characterstcs known to be related to systematc bases n the way people respond to survey questons. Ths s called case-mx adjustment. It s automatcally performed by the CAHPS Analyss Program f adjusters are specfed. For example, f you know that people of a partcular age group are reluctant to report problems and persons of that group are dsproportonately represented n certan enttes, t may be desrable to account for that when comparng data among enttes. However, t s mportant to recognze that dfferences n patterns of responses may reflect real dfferences n qualty of care as well as systematc bases. There s no way to separate these two types of dfferences based purely on statstcal analyss of satsfacton data. The most popular methods for adjustng the data to account for dfferences n patent characterstcs related to systematc bases are regresson, stratfcaton, and propensty score analyss, wth regresson beng by far the most commonly used method. Page 54

Health status and age are two patent characterstcs frequently found to be assocated wth patent reports about the qualty of ther medcal care. People n worse health tend to report lower satsfacton and more problems wth care than do people n better health. Older patents tend to report more satsfacton and fewer problems than do younger patents, although ths assocaton s usually not as strong as the one between health status and ratngs. Results from numerous CAHPS surveys n enttes of all types confrm these general fndngs. For example, consumer ratngs about health care were consstently hgher for those n better health. Health status may be related to ratngs of care because scker persons are more lkely to gve negatve ratngs n general (response tendency), because some people are lkely to gve negatve ratngs about anythng, ncludng ther health and the medcal care they receve (correlated error), or because they get worse care, perhaps ther greater needs create more opportuntes for falure. There s the same ambguty wth the age assocaton. However, regardless of the reason, t s msleadng to rate an entty worse smply because of the knd of patents ts treats. Case-mx adjustment s ntended to mnmze the effects of dfferences between enttes n background characterstcs. The weghtng algorthm for compostes contrbutes one knd of case-mx adjustment, because t causes the tems of a composte to be weghted together n the same proportons, regardless of dfferences n the response rates to the dfferent tems at dfferent plans. Another knd of case-mx adjustment apples to all of the tems and potentally affects all reported results. Ths part of the adjustment uses a regresson methodology, also called covarance adjustment. The user of the software chooses the adjuster varables. For llustratve purposes and because ths has been a common choce n CAHPS so far, we assume n ths dscusson that the adjuster varables are age (AGE) and health status (GHR). If both adult and chld records are n the same analyss, there are three addtonal adjuster varables: chld ndcator (CHILD), age X chld nteracton, and health status X chld nteracton. The ncluson of these three addtonal varables has the effect of fttng separate regresson coeffcents for the adjuster varables among chldren and adults. If data are mssng for an adjuster varable, the program ether (at the opton of the user) deletes the case or mputes the entty mean for that varable. The latter procedure avods losng observatons because of mssng data; t s acceptable n ths settng because, typcally, both the sze of the adjustment and the amount of mssng data on adjusters are small. Let y pj represent the response to tem of respondent j from entty p (after recodng, f any, has been performed). The model for adjustment of a sngle tem s of the form; y pj = β x + µ + ε pj p pj Page 55

where β s a regresson coeffcent vector, xpj s a covarate vector consstng of two or fve adjuster covarates (as descrbed above), µ p s an ntercept parameter for entty p, and ε pj s the error term. The estmates are gven by the followng equaton: µ = µ, µ, 1 2 µ p s the vector of ntercepts, y s the vector of responses and the covarate matrx s where ( ) ( ˆ 1 β ) ( ) ˆ µ = X X X y X = ( X u u ) a 1 2 u p where the columns of and p X a are the vectors of values of each of the adjuster covarates, u s a vector of ndcators for membershp n entty p, p = 1, 2, P, wth entres equal to 1 for respondents n entty p and 0 for others. Fnally, the estmated ntercepts are shfted by a constant amount to force ther mean to equal the mean of the unadjusted entty means y p (to make t easer to compare adjusted and unadjusted means), gvng adjusted entty means ( 1/ P) yp ( 1/ P) a ˆ = ˆ µ + ˆ µ p p p p p For sngle-tem responses, these adjusted means are reported. For compostes, the several adjusted entty means are combned usng the weghted mean aˆ p = waˆ p Casemx adjustments for enttes that you do not want to affect the casemx model and adjustments Sometmes case-mx adjustments may be requred for an entty but for some reason t s not be desrable for the ratngs from that entty to affect the estmated casemx coeffcents or the recenterng of entty scores. An example would be where the purpose of the mplementaton s to make comparsons among HMO plans, but data were also collected for non-hmo unts and the sponsor wants to nclude them for comparson wthout affectng the HMO scores. A quck way to mplement case-mx adjustment n ths nstance s to use the case-weghtng opton. Data from the enttes Page 56

desgnated not to affect the model are retaned n the sample but assgned very small weghts (such as 0.0000001, or 0.0000001 tmes ther samplng weghts f the data are already weghted). The case-mx model s then appled as usual, usng the weghts. Ths trck works because (1) the weghts for the desgnated enttes are so small that the assocated data have essentally no nfluence on the ftted model and (2) case-mx adjustment s performed n full rrespectve of the weghts. Varance Estmaton An approach to varance estmaton s used whch s applcable to both the sngle-tem reports and the compostes. Varances are calculated for the mean for each entty, condtonal on the coeffcents for the adjuster varables. Condtonally these means are ndependent (gnorng the recenterng constant that s added to make the mean of the adjusted means equal to that of the unadjusted means for presentaton purposes). Condtonng on the regresson coeffcents s a standard procedure n varance estmaton n the analyss of surveys (see Cochran, Samplng Technques, 1977, Chapter 7). It s not dffcult to allow for the covarance of the adjusted means due to uncertanty about the regresson coeffcents n the case of sngle-tem reports, but t s dffcult to do ths n a general way for the mult-tem compostes, when the pattern of mssng data vares by tem. In the nterests of consstency, we use the same procedure for both classes of reports. We frst calculate resduals from the regresson model for every tem response, z pj = y pj β x pj where β s the regresson coeffcent vector for tem I and y pj s the response to tem from person j n entty p. The adjusted mean µˆ p for entty p, tem, s the mean (across nonmssng observatons) of z pj If we replace z pj wth 0 for all mssng responses and defne r pj = 1 f there s a nonmssng response and 0 otherwse, then we can wrte ths as µ p = ( zpj ) ( rpj ) j j and the composte score for the entty s µ p = w ( zpj ) ( rpj ) j j Page 57

Lnearzng ths expresson by takng dervatves wth respect to each of the sums z pj and j r pj, we obtan the followng approxmaton: j µ p ( 1 np ) w ( zpj rpjmp ) = j j d pj where n = r s the number of responses to tem from entty p, d pj s defned p j pj by the summand, and m p s the mean of z pj for the tem n entty p. We now apply the standard formula for the varance of an estmated sum, Vˆ = Var µ p ^ ( ˆ p ) = ( n p ( n p 1) ) j d 2 pj where n p s the number of respondents from entty p. Ths gves an estmate of a varance of the composte score for entty p. If the composte conssts of a sngle tem, or f there s no tem nonresponse, these results correspond to the standard varance formula. Note that we do not apply any fnte populaton correctons n ths varance calculaton. The fnte populaton correcton s approprate f the object of our nference s the mean ratng from the populaton of members or patents who are n entty p at the present tme. Our concern, however, s wth predctng the mean ratng that would represent the experences of a new set of subscrbers or patents jonng or seekng care at the entty, because we are attemptng to gve gudance to those who are consderng anew ther choce of nsurance or treatment ste. Conceptually, we regard the present members as a sample from a super-populaton of potental users of the entty. Combnaton of Strata Versons 1.5 and hgher of the CAHPS software permt stratfed analyses. For these analyses, the adjusted means µ s and varances V s are frst calculated separately for each stratum wthn each entty. These calculatons are dentcal to those descrbed above except that stratum wthn entty should be substtuted every tme there s a reference to entty. Then means are calculated by combnng all the stratum means wthn each entty. Suppose that the stratum weghts are Ws, where W = s p s 1. (Here the sum s over all strata s wthn entty p.) Typcally these stratum weghts are defned as the fracton of the entty s total enrollment that falls n each stratum. The entty mean s calculated as ˆ µ * p = W ˆ sµ s. The correspondng varance s calculated s Page 58

as V ˆ * 2 p = s W p s Vˆ solutons n place of Smoothng Varances s. The quanttes µˆ p and Vˆ p. * * ˆ µ p and V ˆp are then used n the remanng In some CAHPS mplementatons, enttes wth very small sample szes were reported as sgnfcantly above or below average, although ths clearly could not be establshed from ther small amount of data. Further nvestgaton revealed that ths occurred because the estmated standard error was mplausbly small (often 0, f all of a small number of respondents gave dentcal answers) whch made the reported score appear to be hghly precse and sgnfcantly dfferent from the mean. Ths fndng (typcally occurrng n tems wth low tem response rates, such as those pertanng to a servce used by only a few patents) rased concerns about the accuracy of drect samplng varance estmates when the number of responses s small. To overcome these problems arsng wth small but nonzero sample szes, we derved a method that combnes the sample varance for an entty wth an aggregated (pooled) varance estmate evaluated over all of the enttes. The procedure s motvated by the followng model for the sample varance S of an tem score or ndvdual-level ˆ composte score, computed for m respondents n the th of n enttes: S = S + δ + ε ˆ 0 where S0 + δ s the populaton varance for entty wth δδ [0, ττ 2 δδ ], and εε [0,2SS 2 2 /(mm = 1)] wth σ = S the sample varance for entty. The expresson for the samplng varance of S s mpled by the assumpton that the ˆ samplng dstrbuton of entty sample varances s ch-square; ths s the usual asymptotc assumpton and appears to hold approxmately n practce. Ths can be vewed as a small area estmaton problem n whch S s a suffcent statstc for the ˆ wthn area measurements and the objectve s to estmate S0 + δ. Under the normalnormal dstrbutonal assumpton (normalty of the error terms n the above equaton), the posteror mean estmate of S0 + δ s gven by ws ˆ (1 ) + w S, where 0 2 2 1 w = (1 + 2 S /{( m 1) τδ }). Ths can be re-expressed as m Sˆ S S 2 2 ( 1) + 2( / τδ ) 0 + δ = 2 2 m 1+ 2( S / τδ ) Est( S ) 0 (1) We take (1) as an approxmaton to the true estmate, a convenent expresson that s a lnear combnaton of the entty-specfc and pooled varances. The observatons whose sample varances are consdered here are smply ndvdual responses n an analyss of a sngle tem, but they are Taylor-lnearzed combnatons Page 59

across tems (as descrbed n the secton on varance estmaton) for analyses of multtem compostes. 2 We use the method of moments to estmate the between-entty varance τδ of the varance. Because ε and δ are ndependent, E ( Sˆ S) = E δ + E 2 2 2 ε 2 2 ( n 1) τδ 2 S / ( m 1) = + where S = ( m 1) Sˆ ( m 1) estmates S 0. The CAHPS macro output contans (n the varable VP) the value Vˆ ˆ macro, = S / m, the squared standard error as opposed to the sample varance. Hence, the between-entty component of the varance of the varance s estmated by ( ˆ ) τ = ( S S) 2 S / ( m 1) / ( n 1) 2 2 2 δ and the square of the coeffcent of varaton s gven by CV 2 2 2 τ δ / S =. 2 The square of the coeffcent of varaton of the ch-square dstrbuton s CV = 2/ A, where A s the degrees of freedom of the dstrbuton (whch can be thought of as the 2 nverse of a pror weght). Therefore, t makes sense that we use A = 2/ CV as the weght of the pooled varance across the enttes n the expresson for the usual precsonweghted estmator of the posteror mean of the varance of an ndvdual entty s ratngs. Substtutng nto (1), we obtan Sˆ smoothed, = AS + ( m 1) ˆ S A+ ( m 1). We express ths n terms of samplng varances (usng the relatonshp Vˆ ˆ = S / m ) to obtan: ˆ ˆ ˆ AS / m + ( m 1) Vmacro, Vsmoothed, = Ssmoothed, / m =. A+ ( m 1) Ths ensures that Vˆ smoothed, S / m when m s small (mplyng lttle nformaton about the varance) and Vˆ ˆ smoothed, V macro, when m (large amount of nformaton for Page 60

estmatng the varance) or when A 0 (that s, when the dfferences n sample varances across enttes are very small). In general V ˆ smoothed, les between these two extremes, smoothng the varances for small enttes a greater amount than the varance for larger enttes whose own estmate of varance s more precse. We have added an optonal module to the CAHPS macro that allows for smoothng of varance estmates n ths way, requrng the user to specfy only the weghtng factor A. Based on the 2010 Medcare CAHPS survey, the values obtaned for A were A = 25, A = 20, and A = 15 for the ratng, composte, and report tems respectvely. We recommend these (or larger) values to users who prefer to mnmze the devaton from prevous procedures whle obtanng adequate protecton from the unreasonable results obtaned wthout smoothng of varance estmates. Hypothess Tests and Assgnment of Fnal Ratngs Global F-test. The frst test calculated s ntended to determne whether there s evdence for dfferences among entty means. If ths test does not fnd sgnfcant dfferences, t s not necessarly approprate to report results by entty on the correspondng tem or composte. The weghted mean s calculated as ( ˆ Vˆ p ) ( 1 ˆ p ) ˆ µ = µ p p V p Then the F-statstc s calculated as F = ( 1 ( P 1) ) ( ˆ µ ˆ µ ) Ths statstc has an approxmate F dstrbuton wth (P-1, q) degrees of freedom; we have found n smulatons that q = n/p (the average sample sze per entty) makes the F-test at worst slghtly conservatve wth typcal sample szes and response dstrbutons. In other words, reported p-values from the test are slghtly larger than they should be, so sgnfcant dfferences are less lkely to be declared. t-tests for entty dfferences from mean. We compare each entty mean to the mean of the entty means usng a t-test. The correspondng contrast s p p 2 Vˆ p p = ˆ µ p ˆ µ p = P 1 P ˆ µ p 1 P ˆ p * ( ) ( ) ) ( ) 1 P µ p p * where Σ represents a sum over all enttes except entty p. Note that the last expresson s smply (P-1)/P tmes the dfference of µˆ from the mean of all enttes p Page 61

except entty p; therefore the two formulatons (mean vs. mean of all, or mean vs. mean of all others) are equvalent. The varance of p s ( ) = ( P 1) 2 [ P] Vˆ p + 1 P ˆ 2 p V p Vˆ p 1 and the t-statstc s calculated as p V ( ) 2 p ( 1) ˆ, and referred to a t dstrbuton wth n degrees of freedom, whch agan s usually slghtly conservatve. p Relablty of CAHPS measures. It s often of nterest to evaluate the precson wth whch CAHPS measures dstngush among the enttes beng compared n a gven mplementaton. The relablty statstc R summarzes the fracton (on a 0 to 1 scale) of the varaton among entty scores (based on samples) that s due to real varaton n qualty n the populaton At one extreme, R=0 means that there s no populaton varaton across enttes and all the observed varaton s due to samplng varaton, so the measure s essentally useless for dstngushng qualty among enttes. At the other extreme, R=1 means that all the entty scores are free of samplng error. Relablty wll be hgh when there s good agreement among respondents n the same entty, large dfferences among enttes, and large sample szes. There are two ways of computng relablty usng the CAHPS macro results, both requrng addtonal analyss outsde the current release of the macro. The frst bases relablty of CAHPS measurements about enttes on the F-statstc for testng dfferences among enttes on an tem or composte. The numerator of the F-statstc summarzes the amount of varaton among the means for dfferent enttes on the measure and thus measures between-entty varaton. The denomnator summarzes the amount of random varaton expected n these means due to samplng of ndvduals. If there were no real dfferences among enttes, so that all the dfferences were due to random varatons n the reports of the patents sampled for the survey, the F-statstc would be about 1. The greater the real dfferences among enttes, relatve to random varaton, the larger the F-statstc s expected to be. A summary measure of relablty s obtaned by the formula R=1 1/F. When F=1 (only random varaton), R = 0 (no relablty), whle for large F, R approaches 1 (best possble relablty). Because the CAHPS macro routnely outputs F, R s trval to compute. In CAHPS, R<0.70 s commonly consdered poor relablty, and R>0.90 s consdered hgh. 3 The above calculaton of R pools nformaton across all enttes nto a sngle surveywde scalar summary for each tem or composte. The number of respondents wll vary across enttes, however, gvng them dfferent samplng varances; hence an alternatve s to estmate relablty for each entty. Another possble objectve s to predct the relablty of measurements made on future enttes gven ther numbers of respondents. These calculatons use the adjusted entty mean and ts assocated 3 Keller S, O Malley AJ, Hays RD, Matthew RA, Zaslavsky AM, Hepner KA, Cleary PD. Methods Used to Streamlne the CAHPS Hosptal Survey. Health Servces Research, 2005, 40, 2057-2077. PMID: 16316438. Page 62

varance (squared standard error), whch are standard CAHPS macro outputs. Let the entty mean, standard error, and total number of responders to the tem or composte entty = 1,, D be denoted m, s and n respectvely. Then use the followng procedure: 1) Compute the total number of respondents across all enttes: N = n. 1 D m N nm 1 2) Compute the overall mean ratng: = = 3) Compute the sample varance estmate for each entty: v 4) Compute the wthn-entty varance: v N 1 D nv = 1 =. = ns. 2 5) Compute the between-entty varance: 1 2 b= max {( N(1 1 / D)) D n ( ), 0 1 m m v }. = 2 6) Relablty for a specfc entty s calculated as R = b/( b+ s ) 7) For projectons for a future survey wth r respondents per entty compute relablty as R= b/( b+ v / r). D = 1 Assgnment of star ratngs. An average entty s assgned two stars. One or three stars are assgned on the bass of smultaneously satsfyng a crteron of statstcal sgnfcance and one of substantve sgnfcance (f specfed). The dfference of an entty from the mean s deemed to be statstcally sgnfcant f the two-sded p-value of the t-test descrbed above s smaller than a predetermned level. The comparson value for determnaton of substantve sgnfcance s a mnmum dfference determned as ( ˆ µ µ, µ ˆ 0 = K mn low hgh µ ), where µˆ s the overall mean, µ low and µ hgh are the lowest and hghest possble values of the score and K s a fxed constant chosen by the user. A star s gven or removed (relatve to two stars) only f the dfference s statstcally sgnfcant and also ˆ µ p ˆ µ > 0. (If K = 0 then = 0 0 and the second part of the crteron has no effect.) Examnng Sample Sze Issues for CAHPS Surveys To examne the effect of small sample szes, the CAHPS Team looked at the data from the demonstraton stes for the CAHPS Health Plan Survey 3.0. In each ste the range of number of plans was 2 to 27, wth a mean of 10 plans. The mnmum sample sze per reportable measure ranged from 17 to 418 (mean = 155) and the average sample sze ranged from 46 to 468 (mean = 238). The percentage of two star plans per reportable measure ranged from 0 to 100 percent wth an average of 71 percent. 4 The observed ste data were used to estmate power for dfferent combnatons of number of plans and sample sze per plan. Table 3 provdes effect szes (dfference 4 Dchotomous tems were excluded n these calculatons because CAHPS Health Plan Survey 3.0 does not nclude them. Page 63

between the mean of one plan and the mean of all other plans/sqrt(mse) that can be detected wth 80 percent power and alpha of 0.05 (two-taled) for 2, 5 and 15 health plans wth health plan sample szes varyng from 40 to 300. Note that the effect sze that can be detected wth a sample sze per plan of 300 for two health plans s smlar to the effect that can be detected for a sample sze of 200 when there are 5 health plans (effect szes of 0.23 and 0.22, respectvely). Smlarly, the effect sze detectable s comparable for two plans wth n = 100 per plan versus fve plans wth n = 60 per plan (0.40 and 0.41, respectvely). Table 3. Effect sze detected wth 80 percent power (alpha = 0.05) by number of plans and sample sze per plan Sample sze per plan 2 plans 5 plans 15 plans 300 0.23 0.18 0.17 200 0.28 0.22 0.21 100 0.40 0.32 0.29 80 0.45 0.35 0.33 60 0.51 0.41 0.38 40 0.63 0.50 0.46 Table 4 provdes nformaton about the effect sze detectable for one plan when the sample sze of all plans except that one s fxed at 300. These effect szes are smlar to those reported n Table 5-1 wth a few exceptons (lower left corner of tables), revealng how small sample sze for one plan has a major mpact on the power to detect a dfference between the plan and the other plans. Table 4. Effect sze detected wth 80 percent power (alpha = 0.05) by number of plans and sample sze for one plan (n = 300 for all other plans) Sample sze for plan 2 plans 5 plans 15 plans 300 0.23 0.18 0.17 200 0.26 0.22 0.21 100 0.33 0.29 0.29 80 0.35 0.33 0.32 60 0.40 0.37 0.37 40 0.47 0.45 0.45 As an llustraton of how effect szes translate nto CAHPS scale ponts, adult data from one of the stes were examned for the ratng of specalst care. Sample szes ranged from 32 to 104 per plan (mean = 67). Seven of the 10 plans receved two stars, one plan receved a sngle star, and two plans receved three stars. The overall mean on the tem was 8.30 and the smallest plan dfference from the mean of other plans Page 64

that was statstcally sgnfcant was 0.24 (standard error of dfference = 0.12) for an entty wth 57 completes. The estmated square root of MSE (SE d /SQRT(1/n 1 + 1/(N-n 1 )) s 0.83. Thus, the observed effect sze for ths dfference was 0.28 (a relatvely small effect). Assumng a smlar MSE, we have 80 percent power (alpha =.05) to detect dfferences between one plan and the mean of the other plans of 0.33, 0.27 and 0.24 on the 0-10 scale for stes wth 2, 5 and 15 plans, respectvely, when the sample sze of each plan s 100. Note that these dfferences correspond to 0.19, 0.16 and 0.14 on the CAHPS meanngful dfferences scale (proporton of the dstance from the mean to the nearest extreme). Implcatons of ths Analyss The power to detect a dfference between one plan and the mean of the other plans depends upon the sample szes for all plans (although the sample sze of the plan n queston has an especally mportant nfluence) and the number of plans. Wth the current CAHPS recommendaton of a sample of 100 per health plan per reportable tem, there s adequate power to detect an effect sze of 0.29 (15 plans) to 0.40 (2 plans) f every plan has the mnmum sample sze. A smlar effect sze (0.38) can be detected for a sample sze of 60 per plan f there are a total of 15 plans. It s mportant to note, however, that these power calculatons pertan only to the determnaton of the number of stars an entty receves. The CAHPS bar charts provde an opportunty for par-wse comparsons and the sample sze requrements for a gven power are therefore larger. Page 65

Appendx Summary of Features Included n Each Verson of the CAHPS Analyss Program Verson 1.0 of the CAHPS SAS Analyss Program offered the followng features: An assessment of sgnfcance usng practcal and statstcal (p-value) crtera; An opton to analyze data based on outpatent utlzaton groupngs; An opton to analyze chld and adult data together or separately; Comparsons of health plan performance; and Case-mx adjustments. Verson 1.5 of the CAHPS SAS Analyss Program added the followng enhancements: Weghtng and stratfcaton. The SAS program performs the correct analyses for dsproportonate stratfed samplng desgns. One way such desgns mght appear s when two plans that were surveyed separately have subsequently merged ther operatons nto a sngle busness entty, and ther results wll be reported as a sngle plan. They also may appear when the sponsor decdes to collect addtonal surveys by usng larger sample szes for a certan subset of people (based on geographc area, gender, age groups, etc.) beyond what would appear there by proportonate allocaton. To use ths feature, the user must specfy whch strata are combned and the number of members n each stratum out of the entre populaton (the weghts). Plan name flexblty. Plan dentfers for programmng and output purposes are no longer requred to be numerc. Text or numerc names are allowed to facltate programmng and nterpretaton of results. Case-mx adjusters. The program no longer requres two case-mx adjusters (age and health status) to be used n the analyses. The user can now specfy an unlmted number of adjuster varables or choose not to adjust the data. Substantve dfferences. A new method of specfyng an absolute dfference that must be acheved before a dfference s meanngful has been added to the program. Whle the prevous method of determnng a meanngful dfference s stll avalable, the user can now smply choose an absolute dfference that must exst between means for a dfference to be flagged as sgnfcant. Page 66

Results tables. Verson 1.5 has an addtonal feature that creates SAS data sets of the results tables the program produces. Ths allows users to perform addtonal analyses on the aggregate results or to create summary reports. Lnear regresson coeffcents for the adjuster varables are now output as part of the results tables and reports. Mssng data for adjusters. In the ntal verson of the Analyss Program, mssng data for the case-mx adjustment varables was mputed at each tem s health plan mean. Verson 1.5 allows the user to specfy whether or not the analyss s conducted wth mputaton for the adjuster varables. Verson 2.0 and 2.1 of the CAHPS SAS Analyss Program added the followng enhancements and changes: The SAS code has been converted to requre only Base SAS and the SAS/STAT module, elmnatng the need for SAS/IML. If adjuster varables are excluded, then the REG procedure n the SAS/STAT module s not needed. The code has been modularzed nto macros to ad n mantanng the macro and understandng what the macro s dong. The macro now has two addtonal ways n whch to subset the data beng run through the Analyss Program wthout havng to create separate calls of the Analyss Program. Wth SUBSET = 2, the Analyss Program runs the case-mx model on the entre data set but does the plan/unt comparsons at the subset levels specfed n the fourth column of the plan detal fle created by the user. Wth SUBSET = 3, the Analyss Program does both the case-mx and the plan/unt comparsons at the subset levels. Data sets are now created for the output of the case-mx and hypothess test calculatons. Ths allows for easy export to Excel or other programs for report generaton. The compostes are no longer restrcted to the How Often (1-4) queston responses. The varable type s ndcated n the macro call and the macro runs a composte calculaton f the number of varables s greater than one. Ths change was made to accommodate the need to create compostes from questons wth dchotomous and trchotomous varables. The program can now create compostes usng all varable types used n the survey The weghtng of the composte tems now has the opton of dong equal weghtng across tems as well as weghtng based on the number of responses n each tem dvded by the total number of responses n all tems. The default opton for the macro s to use the equal weghtng. Page 67

An opton s avalable for recodng the global ratng scales from 0 10 to 1 3 and the How Often scales from 1 4 to 1 3 usng the new parameter RECODE. The prmary ratonale for the recodng nto three categores s to make the data enterng nto the hypothess tests entrely consstent wth the nformaton presented n the Bar Graph reports. A secondary ratonale for recodng s that t may mprove the statstcal propertes of the tests. On general statstcal prncples, t would not be surprsng f the analyss of very skewed data were mproved by a transformaton that reduced the skewness. In the CAHPS survey, t s plausble that the dfference between 0 and 2, both ndcatng strong dssatsfacton, carres wth t less nformaton than the dfference between 8 and 10, reflectng average and maxmum satsfacton, respectvely. Therefore, combnng categores at the low end of the scale may remove some meanngless varaton from the data. Statstcal mprovement would be reflected n larger values of the F-statstc n the recoded data compared to the orgnal data. The recodng s defned as: Ratng scale How often scale Response value Recode Response value Recode Opton 1: 0 6 1 1 2 1 7 8 2 3 2 9 10 3 4 3 Opton 2: 0 7 1 8 9 2 10 3 A new parameter, KP_RESID, has been added to the macro call to allow the resdual values from the regresson to be saved as a permanent SAS data set. By default, these values are only saved temporarly whle the macro s runnng. Page 68

Verson 3.0-3.3 of the CAHPS SAS Analyss Program adds the followng enhancements and changes: The plan detal fle, plandtal.dat, and the flename statement that assgns PLAN_DAT are now optonal. If the plan detal fle does not exst, then the macro uses the PLAN varable n the dataset called by the CAHPS macro. If used, the plan detal fle must have a unque record for each plan name or code. Only the frst column s requred; f the second column s mssng, then the macro creates dummy values for the new plan name equvalent to the frst column. If the thrd and fourth columns have mssng values, then they are all set to the value of 1. Each column must be separated by spaces. The Analyss Program now removes any plans that are to be analyzed that have only zero or one usable records. These changes were made n the submacro USABLE. The plans that are dropped by the macro are saved n a permanent SAS data set labeled dp&outname. The CHILD varable s now optonal. If t does not exst, then the macro creates the varable CHILD. If the ADULTKID parameter s set to 2, then the macro assumes all records n the analyss data set are chld records and sets CHILD = 1, otherwse CHILD wll be set to 0, ndcatng there are no chld records. If there s a mx of chld and adult records n the data set, the user must set up a varable named CHILD and set t equal to 1 for chld records and some other value, usually 0 for adult records. Verson 3.3 of the CAHPS macro corrects a logc error found n verson 3.2 of the macro. The EVEN_WGT parameter now can apply ndvdual level weghts to the composte tems. Ths thrd opton s actvated by settng EVEN_WGT=2 and uses the weght varable, referenced by WGTRESP. The varance of the mean varable, vp, has been added to the text output of the adjusted mean report. A CAHPS verson label has been added to the permanent data sets to ndcate whch verson of the CAHPS Analyss Program created the data set. The verson number has also been added to the text output. Users can now case-mx the trple-stacked bar frequences, usng the ADJ_BARS parameter, and nclude both the non-case-mxed frequences wth the case-mxed frequences n the fnal frequency output data set, n_*. For varables of type 5 (vartype = 5), these cannot have case-mxed bars snce the frequences for the response values are not aggregated nto three bars. To make ths work for nonstandard varable types, t s best to Page 69

do some recodng frst to make the three desred ranges and then run the new varable through as a vartype = 4. The followng parameters have been added: The parameter ID_RESP stores the orgnal respondent ID value, f one exsts, n the permanent data sets. If there s a unque varable n the data set that dentfes each respondent, then enter the varable name n ths parameter. The macro carres t through the ndvdual data sets and attaches t to the resdual data set f KP_RESID = 1 so the data set can be easly lnked to the orgnal f needed. If no ID varable s entered, then the ID_RESP varable n the macro s set to.z. The varable wll be a character and have a maxmum of 50 characters. The parameter flag OUTREGRE ndcates whether or not the regresson output should appear n the text output fle. If set to 0, the default, then the SAS prnted output from the regressons n the case-mx procedure s not prnted out nto the output fle. If set to 1, then the regresson output appears. The parameter WGTRESP accepts the varable name that contans the weghts for ndvdual respondents. Ths weght s used n the case-mx adjustment regresson procedure. The parameter WGTMEAN accepts a varable that contans the weghts to be appled to the means of the plans before the case-mx adjustments are appled. The parameter SPLITFLG allows the data set to be splt nto two groups for the purpose of centerng the means dfferently and runnng two case-mx models through the macro. Ths was done to deal wth the Medcare Managed Care and Fee-for-Servce analyss. By default, the parameter s 0 and s not used but, f set to 1, then the data set must contan a varable wth the name SPLIT and must have the values of 0 and 1. Any record wth a mssng value s dropped from the analyss. The parameter BAR_STAT stores the results of the case-mxed bars n permanent data sets wth the same format as the case-mxed survey queston results. The new data sets created have the format B#&outname and F#&outname where the B* fles hold the stars and statstcs by plan and the F* fles hold the overall means and statstcs. The # has the values 1-3 for a normal macro run, where 1 = the frst bar frequency, 2 = the second bar frequency, and 3 = the thrd bar frequency f t s not dchotomous. &outname s the value gven n the macro call parameter OUTNAME. If the data are stratfed and stratfcaton weghts are used by havng the macro parameter WGTDATA = 2, up to sx addtonal fles are Page 70

created wth # havng the values A-C, where A = the frst bar frequency of the combned strata, B = the second bar frequency of the combned strata, and C = the thrd bar frequency of the combned strata. Verson 3.3 corrects a logc error, contaned wthn verson 3.2, that occurred when the parameter SUBSET = 3, whch runs the macro multple tmes based on the subsettng varable n the plan detal fle referenced by the FILENAME PLANDTAL statement. The text output on the Warnngs and Parameter Info page contans more accurate nformaton about the adjusters when there are chld nteractons, when ADULTKID = 1. The number of adjusters wll reflect the orgnal adjuster varables tmes 2 plus 1, so f there are orgnally 2 adjusters, the total number of adjusters wth chld nteractons wll be 5, ADJ#1, ADJ#2, ADJ#1 * CHILD, ADJ#2 * CHILD, and CHILD. Added two flag lnes to the log fle that wll ndcate f the macro fnds the CHILD and PLAN varables n the orgnal analyss data set. If there s no chld varable, the flag wll ndcate how the macro created a new CHILD varable. Verson 3.4 (May-June 2003) of the CAHPS SAS Analyss Program adds the followng enhancements and changes: Added three addtonal varables to the sa* data set and the output text of the statstcal tests. The unweghted, unadjusted plan mean was added to help clarfy what the unadjusted mean actually s. Only when the wgtmean parameter s used wll the unweghted, unadjusted mean be dfferent from the weghted unadjusted mean. The other varable added s the 95% Confdence Lmts for the Dfference of the Mean. Ths s computed as 1.96 * the standard error of the dfference. When wgtplan = 1, then a thrd column contanng the summed weghts for each plan wll also be added to the sa* data set, the b* dataset f frequency bars are to be stored (bar_stat = 1) and the output text. Added n the weghted, unadjusted frequences to the frequency table n_* data set and the output text, when the frequency bars are also case mx adjusted. The wgtmean parameter purpose has been expanded to allow for the use of the sum of the weghts to the plan level to be used n the comparson of the plan means. If a varable exsts for the wgtmean parameter, then the ndvdual record level weght s used to compute the weghted, unadjusted plan means. In addton, f the new parameter wgtplan = 1, then the sum of the ndvdual weghts to the plan level wll be used n weghtng the plan mean comparsons. The wgtplan parameter can have Page 71

the value of 0, default, or 1. When 0, the macro wll use equal weghts when comparng the plan means. When 1, and the wgtmean parameter has a varable lsted, then the sum of the weghts to the plan level wll be used computng the overall and grand means whch are used n the statstcal comparsons of the plan means. Added checks on the DATASET parameter to make sure t exsts or that the value n the DATASET parameter s a vald SAS data set. If there s an error, the macro wll stop processng and prnt an error message to the log fle. Added error checkng on the mergng of the plan detal fle wth the analyss dataset. If there are no records matchng, then the macro wll prnt out the frequences of the unque PLAN values for both the plan detal fle and the analyss data set to the output text fle and also prnt out and error to the log fle. Verson 3.5 (September 2005) of the CAHPS SAS Analyss Program added the followng enhancements and changes: A dsclamer and copyrght statement were added. If weghts are beng used for the ndvdual or plan means, records wth weghts that are less than zero or mssng are removed. When macro converts the numerc plan n allcases to character, t left justfes and trms tralng blanks. The macro checks that there are plans n all subcodes after the usable data set s made. If some subcodes have all mssng plans, t recomputes how the subcodes are used n the loopng n the star macro. The log comment for when chld varable s not found n the orgnal data set was changed. A bug was dentfed n the CAHPS 3.4b macro: Two lnes that have length planname $ 20 when t should be $ 40 causng a merge problem wth the N_* data sets. $ 20 was changed to $ 40. Verson 3.6 (Aprl 2006) of the CAHPS SAS Analyss Program added the followng enhancements and changes: Ths new verson corrects an error n some prevous versons affectng calculaton of the varances for the comparson of an entty mean to the mean of all other plan means, when the plans were weghted. Ths error only affects analyses wth parameter wgtplan=1 usng CAHPS macro versons 3.4b (released May 2003) and 3.5 (released September 2005). By Page 72

default, the macro sets wgtplan=0 so the error does not affect unweghted plan analyss. The error caused sgnfcance tests to be calculated ncorrectly when determnng whether an entty's mean was sgnfcantly above or below the average. Ths could cause some plans to be declared 1- or 3-star plans when they were respectvely below or above average, but not by a statstcally sgnfcant amount. (July 2006) Modfed formula for specal case of usng only one plan unt and a dvson by zero error may occur. Ths case used to work n pror versons. Modfed code for checkng f SE may be mssng to set T=0 n that case. Also, VO can now have a zero denomnator, n the case where there s only one unt beng analyzed, modfed code to catch that error. (3.6b as of June 2007) Ths modfcaton to Verson 3.6 puts the _wgtmean varable n the strata data step n order to address a problem wth a mssng lne that was not keepng the _WGTPLAN varable n the data step that created wstemp. Because of the mssng lne, the use of wgtdata=2 for combnng strata generated a SAS error. Verson 4.0 (September 2011) and 4.1 (Aprl 2012) of the CAHPS SAS Analyss Program added the followng enhancements and changes: One part of the code that creates plandtal data set (t s n usable macro program) was modfed. Ths only affects when subset = 3. The calculaton of weghts for the composte tems was modfed. The sum of weghts based on the number of responses from each tem s used as the weght of the composte case. Also, the calculaton of tem weghts for even_wgt = 1 was modfed. For more detals about how the weghts are computed, please see the Explanaton of Statstcal Calculaton secton. A new warnng note was added n the macro output ( t s n. mkreport macro program). The note lsts plan IDs when they have zero responses n measured tems. A new opton of assgnng smoothng varances was added. Users can assgn a weght parameter called smoothng on the varances as opton. The default s smoothng = 0. Ths provdes the orgnal varances. If smoothng s greater than zero, the value that users nput wll be used as the weght for the varances. If smoothng s less than zero, the weght wll be computed nsde of the macro automatcally. For more detals about how that weght s computed nsde of the macro, please see the Explanaton of Statstcal Calculaton secton. Page 73

A SAS procedure PROC STANDARD was replaced wth PROC STDIZE. The macro centers all adjusters before t runs regresson procedure f adjusters are requred. PROC STANDARD was not applcable when some adjusters contan only the same values. As a result, t dd not standardze the value correctly. PROC STDIZE s able to handle the stuaton. (Aprl 2012) Modfed codes for computng adjusted composte means when composte even weght opton (even_wgt = 1) s selected. The macro computes the weghts for all enttes regardless of whether they get dropped out for the analyss due to a lack of the sample sze. In the pror verson, ths caused ncorrect adjusted means when some enttes dd not make t to the fnal analyss. Also, the macro dd not handle correctly on computng adjusted means when some enttes have dfferent weghts from the even weghts. Verson 4.1 s able to handle the case and provde approprate adjusted means. Page 74