Instructions for Analyzing Data from CAHPS Surveys:

Transcription

1 Instructons for Analyzng Data from CAHPS Surveys: Usng the CAHPS Analyss Program Verson 4.1 Purpose of ths Document...1 The CAHPS Analyss Program...1 Computng Requrements...1 Pre-Analyss Decsons...2 What Does the CAHPS Analyss Program Analyze?...6 Preparng Data for Analyss...8 SAS Data Set Requrements Usng the CAHPS Analyss Program Interpretng the Results Small Data Set Example Explanaton of Statstcal Calculatons Hypothess Tests and Assgnment of Fnal Ratngs Examnng Sample Sze Issues for CAHPS Surveys Appendx Summary of Features Included n Each Verson of the CAHPS Analyss Program Lst of Exhbts and Tables Table 1. Descrpton of test data set varables Table 2. Arguments for CAHPS 4.1 Macro Table 3. Effect sze detected wth 80 percent power (alpha = 0.05) by number of plans and sample sze per plan Table 4. Effect sze detected wth 80 percent power (alpha = 0.05) by number of plans and sample sze for one plan (n = 300 for all other plans) For addtonal gudance, please e-mal cahps1@ahrq.gov or call the Help Lne at (800)

2 Purpose of ths Document CAHPS Surveys and Instructons Ths document explans how the CAHPS Analyss Program works and how sponsors and vendors can use the program to nterpret the results of ther CAHPS survey. Whle the program was ntally desgned for the CAHPS Health Plan Survey, you can use t to analyze data from any of the CAHPS surveys. For most CAHPS surveys, the nstructons nclude a document wth analyss gudance specfc to that survey. The CAHPS Analyss Program The goal of the CAHPS Analyss Program often referred to as the CAHPS macro s to provde the user wth a flexble way to analyze CAHPS survey data n order to make vald comparsons of performance. Wrtten n SAS, the CAHPS Analyss Program s desgned to assst CAHPS survey users n mplementng two knds of statstcal adjustments. Comparng tems and compostes. CAHPS surveys collect consumers and patents reports and ratngs of a number of dmensons of health care. Comparng performance based on all the ndvdual CAHPS survey tems s a very complex task. Moreover, ndvdual survey tems are often less relable than multple tem combnatons. To smplfy the nterpretaton of the data and enhance the relablty of the results, questons that measure smlar topcs are grouped together. These groups of questons, called compostes, facltate comparsons of performance across your unt of analyss (e.g., health plan, medcal practce, clncan). (Note: The nstructons avalable for each survey nclude a document that lsts the tems n that survey s reportng measures,.e., composte measures and ratngs.) Adjustng for case mx. The CAHPS Team recommends that you adjust the survey data for respondent age, educaton, and general health status. Ths makes t more lkely that reported dfferences are due to real dfferences n performance, rather than dfferences n the characterstcs of enrollees or patents. Computng Requrements The CAHPS Analyss Program was developed usng SAS software. SAS s a data management, analyss, and presentaton product produced by the SAS Insttute, whch s headquartered n Cary, North Carolna. The operaton of SAS requres a Base system, but a number of ndvdual modules can be added to perform more complex analyses and data manpulaton. The CAHPS Analyss Program requres Base SAS and the SAS/STAT module. Base SAS, whch s requred to use any SAS product, provdes the data manpulaton, prnt commands, smple plottng capabltes, and procedures for descrptve statstcs. Base SAS ncludes the CORE module. The SAS/STAT module adds several statstcal procedures for use by SAS. The CAHPS Analyss Program uses the SAS regresson procedure, PROC REG, to do part of ts case-mx calculatons. If case-mx calculatons wll not be used, then the macro requres Base SAS only. Page 1

3 The CAHPS Analyss Program was wrtten n verson 6.12 of PC SAS and verson 6.12 of SAS/STAT. The program should work on all SAS platforms that have verson 6.0 or later. It has been extensvely tested on UNIX and Wndows SAS products and has performed well. Pre-Analyss Decsons The CAHPS Analyss Program offers the user a number of optons for analyzng the survey data. Before preparng to run the program, analysts should make sure that the project team has agreed upon answers to the followng questons. Ther mplcatons for the CAHPS Analyss Program are revewed below. Havng these questons answered early wll save tme when dong the analyses. What s the reportng unt (entty)? Any analyss of CAHPS data s ntended to assess, compare and report on some type of reportng unt. Examples of possble such unts nclude health plans, hosptals, provder groups, clncs, stes of care, and ndvdual physcans. To avod confuson, we use the neutral term entty n these nstructons to refer to the unt whose data wll be aggregated nto a summary measure. Users of the Analyss Program wll have to specfy whch varable dentfes the entty to whch each response wll be attached. Note that there mght be alternatve ways of analyzng the same data wth dfferent enttes, and f the data collecton desgn s sutable, more than one of them mght be vald. For example, a dataset mght be analyzed to compare provder groups and then, wth a dfferent entty varable, the same data mght be used to assess ndvdual doctors. The CAHPS Analyss Program was ntally wrtten for the CAHPS Health Plan Survey. Therefore, varable names, examples, and other references throughout the program often refer to health plans. Ths has no bearng on the sutablty of the program for analyzng data on other types of enttes. For specfc nstructons on adaptng the program for other surveys, please refer to Preparng and Analyzng Data ( SurveyDocuments/CG/12%20Month/Prep_Analyze/1035_Preparng_analyzng_data_ from_cg.pdf) n the nstructon documents avalable for the survey you are usng. Ths document s not avalable for all CAHPS surveys. Wll you analyze specfc populaton groups separately? If the project team has collected data for dfferent groups of people, the team needs to decde whether to analyze the data separately or together. Subgroups that you may be consderng for separate analyses can be defned by payer (e.g., Medcare, Medcad, prvately nsured), geographc regon (e.g., state, county, regon), or other factors. If the groups are to be analyzed together, no changes to the CAHPS Analyss Program are necessary. If a team decdes to analyze the groups separately and the data fle contans more than one group, t s mportant to set up selecton crtera n the CAHPS Analyss Program or splt the data set. Page 2

4 Wll adult and chld surveys be analyzed together or separately? The Analyss Program allows users to specfy how chld and adult surveys wll be analyzed. The project team needs to decde whether to analyze surveys about adults and chldren separately or together. If adult and chld survey data are to be analyzed together, the team must also decde whether to consder nteracton effects. Interacton effects are mportant to consder n an analyss when the mpact of age or health status on one of the reportng tems depends on whether an adult or chld survey s beng analyzed. See the secton called Explanaton of Statstcal Calculatons for a more detaled dscusson of nteracton effects. We recommend that you consder nteracton effects when combnng adult and chld data. If the team collected only adult or chld surveys, users stll need to specfy an opton lsted below n the CAHPS Analyss Program. There are four optons dependng on whch surveys are n the data set and how the user wants to analyze them: Opton value 0 Surveys n data set Adult and chld surveys, only adult surveys, or only chld surveys 1 Adult and chld surveys 2 3 Adult and chld surveys or only chld surveys Adult and chld surveys or only adult surveys Analyss method Combne adult and chld survey data; do not consder nteracton effects Combne adult and chld survey data; consder nteracton effects Chld surveys only Adult surveys only Wll hgh and low users of health care servces be analyzed together or separately? The CAHPS Analyss Program allows users to analyze the data for survey respondents wth a hgh or low number of outpatent vsts separately or combned. The crteron for establshng low and hgh users of outpatent servces s based on Queston 7 of the CAHPS Health Plan Survey Adult Commercal Questonnare or Queston 4 of the CAHPS Clncan & Group Surveys Adult 12-Month and Vst Questonnares. Hgh users are defned as respondents who ndcated that they have had three or more vsts for ther own care to a doctor s offce or clnc (not ncludng emergency room vsts). Low users ndcated that they have had fewer than three vsts to a doctor s offce or clnc. The CAHPS Consortum recommends reportng data for global ratngs (e.g., respondent s ratng of ther personal doctor or nurse, specalsts, health care n the last 12 months, and/or health plan) accordng to outpatent utlzaton. It s up to the project team to decde whether to analyze the survey data for low and hgh users Page 3

5 separately or together. You can then choose from the correspondng Opton Values lsted below for the CAHPS Analyss Program. Opton value Analyss decson 1 Combne low and hgh users. 2 Low users only (< 3 vsts) 3 Hgh users only ( 3 vsts) What level of sgnfcance (p-value) wll you use n the analyss? The CAHPS Consortum recommends a p-value of 0.05 to test for statstcally sgnfcant dfferences between the enttes beng compared. The p-value the team chooses must be specfed n the CAHPS Analyss Program. What, f any, level of substantve (practcal) sgnfcance wll you use to compare performance? Substantve sgnfcance refers to an absolute dfference between the enttes beng compared (e.g., health plans, provder groups, ndvdual physcans) that must be acheved before a change s consdered meanngful. If two health plans, for example, had sgnfcantly dfferent average scores based on the p-value crtera, the dfference between the plans average scores may stll not be large enough to be meanngful. The CAHPS Analyss Program has two optons that allow the user to specfy a dfference that s substantve. You can use these optons smultaneously or specfy only one. Frst method. The team decdes on a percentage of the dstance to the nearest bound that s meanngful. The example presented below explans ths concept. Assume the analyss of a global ratng queston (one that uses a 0-10 ratng scale) has the followng mean scores for a global ratng queston across all enttes: Global Ratng Queston overall mean (0-10 scale) = 6 To determne a level of dfference between enttes that s substantvely large: 1. Determne the dstance from the mean to the nearest bound: a) Compute the dstance from the lower bound Mean (6) - Lower bound (0) = 6 b) Compute the dstance from the mean to the upper bound Upper bound (10) - mean (6) = 4 Page 4

6 c) Determne the smaller dfference Mnmum (6,4) = 4 2. The dstance from the nearest bound s 4. Now the project team must decde what percentage of ths dstance s a meanngful dfference between enttes. Ths fracton s entered n the CAHPS Analyss Program. Second method. A much smpler method avalable n the Analyss Program s to specfy an absolute dfference that must exst between the entty mean and the mean for all enttes n the analyss for a dfference to be consdered sgnfcant. For ths method, the user needs only to specfy the absolute dfference consdered to be meanngful. Do you need to adjust the results for case mx? Case mx refers to the respondents health status and socodemographc characterstcs, such as age or educatonal level, that may affect the ratngs that the respondent provdes. Wthout an adjustment, dfferences between enttes could be due to case-mx dfferences rather than true dfferences n qualty. Each project team must determne f t s approprate to adjust ts data to account for case-mx dfferences. What to adjust for If the project team decdes to adjust the data for case-mx, t must choose the approprate adjusters. The CAHPS Consortum recommends usng general health status, age, and educaton. Indvduals n better health and older ndvduals tend to rate ther care, plans, and provders hgher. There s also evdence from a number of studes that educaton affects ratngs, wth more educated ndvduals gvng lower ratngs. Mssng data for case-mx adjusters If case-mx adjusted results wll be used, the project team must decde whether or not to mpute mssng data for the adjusters at each adjuster s entty-level mean. Rsk of out-of-range values for case-mxed means In the specal cases where there are very few records for an analyss varable or all respondents answered n only one or two response categores, there s the possblty that the case-mx adjusted values wll be out of range. For example, f all respondents to a Health Plan Survey answered Yes, where 0= No and 1= Yes to a yes/no queston, and the adjustment for that entty s up, the adjusted mean for that entty would be greater than 1 and the adjusted frequences would be less than zero for the No category and greater than 1 for the Yes category. Page 5

7 The macro does not force a change n these values, snce t would change the mean of the means on the adjusted scores but not on the unadjusted scores. It s recommended that, n reports of CAHPS results, you set these out-of-range values to the mnmum or maxmum value for that category. Then a manual adjustment could be made to the adjacent category f necessary. For example, n the case of three response categores, where the mnmum frequency should be zero and the maxmum value s 100, the case-mxed frequency results are as follows: category 1 = -2.0, category 2 = 25.0 and category 3 = 77.0 The results could be adjusted so that category 1 = 0.0, category 2 = 23.0 and category 3 = 77.0 Do results need to be analyzed usng weghtng and stratfcaton? As dscussed above, the survey samplng plan can be desgned to select dsproportonately potental respondents from certan geographc or demographc groups n the populaton. Alternatvely, stuatons can arse after samplng s complete that create the need to combne data for certan samplng unts. For example, ths can occur when two enttes merge ther operatons and the survey sponsor chooses to report ther results as a combned score. Whether enttes are mergng ther operatons or a dsproportonate stratfed samplng desgn was used, the CAHPS Analyss Program can perform the approprate analyses, provded the user specfes the correct strata to be combned and the number of members n each stratum out of the entre populaton. What Does the CAHPS Analyss Program Analyze? The CAHPS Analyss Program s desgned to analyze mult-tem compostes and sngle tems from the CAHPS survey data. The output from the program compares the performance of an entty to the overall performance of all enttes. The macro accepts fve varable types. For four of these, the macro cleans ther response values wthn the expected mnmum and maxmum range. For the ffth type, the mnmum and maxmum response values must be entered as an argument. Page 6

8 The followng are the varable types: Varable type Mn Max response values 1 Dchotomous Global ratng How often or other 4-pont 1 4 response scale 4 3-pont response scale Other mn_resp max_resp Global ratngs are based on survey tems that ask the respondents to rate dfferent aspects of health care on a scale from 0 to 10. For example, the global ratngs tems n the CAHPS Health Plan Survey 4.0 Adult Commercal Questonnare are: Health care n the last 12 months (Queston 8) Personal doctor (Queston 15) Specalst (Queston 19) Health plan (Queston 27) Overall Ratngs Response Format: 0-10 Q8 Q15 Q19 Q27 Usng any number from 0 to 10, where 0 s the worst health care possble and 10 s the best health care possble, what number would you use to rate all your health care n the last 12 months? Usng any number from 0 to 10, where 0 s the worst personal doctor possble and 10 s the best personal doctor possble, what number would you use to rate your personal doctor? We want to know your ratng of the specalst you saw most often n the last 12 months. Usng any number from 0 to 10, where 0 s the worst specalst possble and 10 s the best specalst possble, what number would you use to rate the specalst? Usng any number from 0 to 10, where 0 s the worst health plan possble and 10 s the best health plan possble, what number would you use to rate your health plan? Compostes are groupngs of two or more questons that measure the same dmensons of health care or health nsurance plan servces. Compostes usually are developed for survey tems that have the same response optons. Page 7

9 For example, the compostes n the Health Plan Survey represent the experences of respondents (adult enrollees or the parents/guardans of enrolled chldren) n the followng areas: Gettng needed care (2 questons for adults; 2 questons for chldren). Gettng care quckly (2 questons for adults; 2 questons for chldren). How well doctors communcate (4 questons for adults; 5 questons for chldren). Health plan nformaton and customer servce (2 questons for adults; 2 questons for chldren). The followng table llustrates how tems n the Health Plan Survey 4.0 are grouped nto the frst of these compostes. Gettng Needed Care Q17 Q21 In the last 12 months, how often was t easy to get appontments wth specalsts? In the last 12 months, how often was t easy to get the care, tests, or treatment you thought you needed through your health plan? Response Format Never Sometmes Usually Always A document lstng each survey s reportng measures compostes and ratngs s avalable n the nstructons provded for that survey. Preparng Data for Analyss Pror to applyng the CAHPS Analyss Program, you must perform several tasks to transform raw data from the completed questonnares nto data that the SAS analyss programs can use. (Gudance on determnng when a questonnare s complete s avalable for most CAHPS surveys n the appendx of the nstructons on feldng that survey.) Task 1: Identfy and exclude nelgble cases. Task 2: Code and enter the data. Task 3: Clean the data. Task 4: Conduct an audt. Page 8

10 Many nterm fles wll be created along the way. Before begnnng ths process, you must take steps to preserve the orgnal data fle created when the raw survey responses were entered. Any changes and correctons made durng the cleanng and data preparaton phase should be made on duplcate fles. There are three reasons for ths acton: 1. The orgnal data fle s an mportant component of the complete record of the project. 2. Havng an orgnal fle wll allow you to correct data errors that were made durng the cleanng process. 3. The exstence of an orgnal fle s crtcal f the vendor or sponsor wants to go back later and conduct other analyses or tests, such as extent of error tests or tests of skp patterns. Data Fle Specfcatons The data fle contans the raw data from responses to the survey. Short tem handles for the tems n each questonnare can be found n tables provded n the document called Overvew of the Questonnares at CG/~/meda/Fles/SurveyDocuments/CG/12%20Month/Get_Surveys/1350_cg_ overvew_of_questonnares.pdf. The responses to each queston must use the code numbers, or precodes, contaned n the questonnares. Users should construct a separate data fle for each verson of the survey. Do not nclude data from dfferent survey nstruments n the same data fle. For example, do not nclude responses to the Clncan & Group Adult 12-Month Survey and Chld 12-Month Survey n the same data fle. If you are nterested n submttng your data to the CAHPS Database, please refer to the data fle specfcatons for submsson at Database/Submttng-Data/CG-Data.aspx. The number and scope of the data preparaton tasks and the way they are carred out depend on the data collecton protocol and the way n whch the data were recorded. For example: If the vendor/sponsor collected data wth a self-admnstered maled questonnare, dd respondents record answers on optcal scan forms 1 or record them drectly on the CAHPS-formatted questonnares? If the vendor/sponsor collected data through telephone ntervews, dd the ntervewer use computer-asssted telephone ntervewng (CATI) or paper-and-pencl forms? 1 Optcal scan forms are answer sheets n whch respondents fll n the crcle that corresponds to ther answer choce. These forms are fed through an optcal scannng machne, and the data are automatcally captured by a computer. Standardzed tests for students, such as the SAT, generally use optcal scan forms. Page 9

11 Task 1: Identfy and Exclude Inelgble Cases Several stuatons render a case nelgble for analyss. One common scenaro that vendors must be prepared to handle occurs when the respondent reports he or she has not vsted the sampled entty (e.g., a physcan or medcal group). Ths mght be ndcated by a no response to Queston 1 (e.g., Our records show that you got care from the provder named below n the last 12 months. Is that rght? ). Other questonnares may be consdered ncomplete and excluded from analyss but are not excluded from the denomnator used to calculate the response rate. For example: If someone else asssted the respondent or answered the questons (as a proxy) or If at least half of the key tems on the questonnare were not flled n. (For most CAHPS surveys, a lst of key tems s avalable n the appendx of the nstructons on feldng the survey.) Task 2: Code and Enter the Data There are a varety of possble methods that can be used to enter data from CAHPS surveys. The exact level of codng requred wll depend on the method used to capture the data (e.g., questonnares that requre data entry versus questonnares that are scanned by a computer). Your codng specalst should revew each questonnare to see whether the responses are legble and whether any responses need to be coded. Each tem should have a correspondng code, even tems that were not answered. The table below shows examples of recommended codng. If you are nterested n submttng your data to the CAHPS Database, follow the data fle and codng specfcatons avalable at Data/CG-Data.aspx. After codng s completed, enter the data nto a computer fle. Response Code Actual response Use the correspondng survey precode ndcated besde the response opton. Precodes typcally begn wth 1 and number consecutvely for each response opton. Item was approprately skpped Code as 7 (or 77 f precodes exceed 7) Item showed more than one response Code as 8 (or 88 f precodes exceed 7) opton completed when only one s approprate (.e. multple marks) Item was left blank and should not have Code as 9 (or 99 f precodes exceed 7) been skpped Page 10

12 If you use optcal scan forms for your maled questonnares, the scannng equpment automatcally enters the data nto a computer-readable fle. If you do not use optcal scan forms, the mal questonnares are desgned for drect data entry wthout the need for codng most respondent answers. However, f t s unclear whch answer the respondent selected (e.g., the respondent s pencl mark does not neatly ft wthn a sngle answer category, or two responses are marked), then your codng specalst wll have to make a decson about whch response the respondent ntended. If t s not readly apparent what the respondent ntended, the codng specalst should ndcate that the answer be entered as mssng or multple marks, as approprate. If you use a CATI system for a telephone survey, data are entered drectly nto a data fle that has already been programmed to refuse unlkely and nvald responses. If you use paper questonnares to record answers gven n a telephone ntervew nstead, the process for codng and data entry s the same as for the standard paper verson of the maled questonnares. To ensure qualty, answers from paper-and-pencl questonnares should be keyentered by two separate data entry specalsts. The results from the two should be compared to dentfy and correct data entry errors. At the end of the codng and data entry process, you wll have an electronc data set of responses to all the questonnare tems. Addtonal codng and recodng may be necessary pror to usng the CAHPS Analyss Program. Refer to the SAS Data Set Requrements below for recodng varables for use wth the CAHPS Analyss Program. Task 3: Clean the Data In many cases, the data set you have created wll have mperfectons. You wll have to take several steps to fx these mperfectons before any results are reported. Check for out-of-range values. Out-of-range responses occur when respondents provde napproprate responses for a partcular queston. For example, f the vald response choces for a queston are 0 or 1, a value of 2 would be out of range. Smlarly, f a respondent crcled two categores when he or she was supposed to provde only one answer, the response s out of the acceptable range of the queston. To detect out-of-range values, you need to revew queston frequences. Ths can be done ether by vsually scannng a report showng the tem dstrbutons or frequences or by runnng the questonnare data through a computer program., Both are often used to mprove the qualty of the data. If a value s found that s mpossble (or unlkely) gven the response optons, then the questonnare should be revewed and revsons made to the data. These revsons often nvolve settng the out-of-range values to mssng, whch drops them from the data analyss for that partcular queston. Carefully document the results from ths revew process, ncludng any changes to the data set. Mantan an audt tral (electroncally and on paper) so t s possble to go back to the orgnal data fle. Check for skp pattern problems. Response nconsstences generally arse when a respondent msunderstands a queston or does not successfully follow nstructons to Page 11

13 skp questons. An example of a response nconsstency would be f a respondent answered that he or she had no doctor vsts n the past 6 months, but then answered followup questons about vsts n the past 6 months. If there are nconsstences between the response to the screener queston and the followng queston s response, assume the screener response s correct. Check agan for nelgble cases. Identfy any questonnares that are not elgble for analyss and remove them from the data set used for the CAHPS SAS program. Questonnares n whch fewer than half of key tems are answered should not be counted as completed surveys. (Gudance on determnng when a questonnare s complete, ncludng a lst of key tems, s avalable for most CAHPS surveys n the appendx of the nstructons on feldng that survey.) Check for duplcates. The number of records n the data fle should match the number of completes and partals n the sample fle. Duplcates can occur f the vendor conducts a followup phone ntervew, f the mal questonnare arrves at the same tme or soon after, and the case slps through the recept control system, or f there are errors n data entry. Your polcy should be to keep the frst questonnare that comes n. Task 4: Conduct an Audt Whether surveys are collected n standard paper format, as optcally scanned forms, or as paper telephone questonnares, a small random sample of the entered data should be audted by comparng hard-copy forms wth the results of data entry. Ths enables you to catch any systematc errors. For example, f the optcal scannng program was ncorrect, Queston 5 may be entered n the fle where Queston 6 was supposed to be. These types of systematc errors wll show up consstently across all questonnares. SAS Data Set Requrements Before runnng the CAHPS Analyss Program, make sure that the structure and propertes of the data fle meet the specfcatons lsted below. If the data set does not meet these requrements, the SAS program wll not work properly. Many of the varable codng and cleanng requrements are demonstrated n the next secton on usng the CONTROL.SAS Analyss Program. Data Fle Specfcatons Each row or case n the SAS data set represents the data for a unque questonnare. If data from dfferent CAHPS questonnares are n the same data set and are to be analyzed together, each questonnare s lsted on a separate row. If data from adult and chld questonnares are n the data set, the adult and chld questonnares are also lsted on separate rows. Page 12

14 If data from dfferent CAHPS questonnares are n the same data set, responses for equvalent questons are lsted under the same varable names. Sample Sze Requrements Number of enttes (.e., such as health plan or provders). The data set must have surveys from at least two enttes. If there s only one entty n the data beng analyzed, statstcal comparsons cannot be performed and some parts of the program wll not work properly. If the CAHPS macro s run wth data for one entty, a couple of warnng messages and notes wll appear n the log fle that would not be produced f two or more enttes were represented. All the reports wll stll be produced, though some of the results wll be of lmted value. Responses. At least two responses per entty are requred by the Analyss Program. We recommend analyss of at least 100 responses for each entty. The program flags enttes wth fewer than 100 responses for an ndvdual measure, but the analyss s performed on all enttes wth at least two records. Includng enttes wth very lttle data tends to reduce the precson of comparsons between ndvdual enttes or provders and the overall means. The user can consder removng enttes wth fewer than 100 responses from the data fle before analyss. Note: When analyzng unts of analyss such as medcal groups or ndvdual physcans, follow the mnmum response gudance n the nstructons for feldng the survey. Snce the program was ntally desgned for the CAHPS Health Plan Survey, you wll receve the program flags n your results when there are fewer than 100 responses even f the target number of completed responses s less for your survey. Varable Codng and Cleanng Requrements Numerc varables. All analytc varables used by the CAHPS Analyss Program must be numerc. Analytc varables nclude any questonnare tem used to compute CAHPS reportng tems, case-mx adjustment varables, the dchotomous varable used to dentfy chld and adult surveys, and the varable used to dentfy hgh and low users of outpatent servces. These varables are dscussed n more detal below. To ensure that an error does not occur n the SAS program, all varables created from survey questons should be coded numercally. If the user recodes character varables to numerc, there should be a mnmum length of 4. A length of 8 s recommended for the recode. WARNING: The varables PLAN, CHILD, VISITS, and SPLIT are varable names needed by the CAHPS macro. If the data set has other varables wth these names and they do not conform to the specfcatons below, the macro may produce errors n the log fle and the results may be erroneous. Varable PLAN. The varable PLAN must be ncluded n the data set. Note that ths varable represents a numerc code or text descrptor for each entty n the data set. Ths s the only varable that does not have to be coded numercally. The SAS program accepts alphanumerc, character, and numerc formats for ths varable. The maxmum varable length for PLAN s 40 characters. Page 13

15 Even f you are not analyzng health plan data, you must use the varable name PLAN to refer to your unt of analyss. Ths s because the SAS macros use that varable name. However, the varable can be any unt of analyss. For example, PLAN can represent the names of the medcal groups (Group A, Group B, etc.) or ndvdual physcans (Dr. A, Dr. B, etc.). Varable CHILD. The numerc varable CHILD needs to be n the data set f subsettng the data between adult and chld records. Ths varable s used to dstngush between adult and chld surveys n the SAS program. CHILD should be coded 0 for adult surveys and 1 for chld surveys. If ths varable s mssng from the data set, the CAHPS macro sets CHILD = 0 when ADULTKID has the values 0, 1, or 3, and sets CHILD = 1 when ADULTKID = 2. Varable VISITS. The varable VISITS needs to be n the data set f usng the VISITS parameter. Ths varable s used to dentfy hgh and low users of health care servces (e.g., refer to Queston 7 from the CAHPS Health Plan Survey Adult Commercal Questonnare). The table that follows shows the response values based on tem 7 n the CAHPS Health Plan Survey Adult Commercal Questonnare. We recommend that you use these values whle codng your questonnares. If the VISITS varable s mssng from the data set, the CAHPS macro wll work as long as the VISITS parameter s not equal to In the last 12 months, not countng tmes you went to an emergency room, how many tmes dd you go to a doctor s offce or clnc to get care for yourself? Response value Label/descrpton 0 None 1 1 tme 2 2 tmes 3 3 tmes 4 4 tmes 5 5 to 9 tmes 6 10 or more tmes All other values Not analyzed by the SAS program Varable SPLIT. The numerc varable SPLIT needs to be n the data set f you are dong separate case-mx adjustments on two dfferent populatons as ndcated by the macro parameter SPLITFLG = 1. For most cases, the default value 0 for SPLIT does not need to be modfed. An example of splttng the case-mx adjustments separately on two populatons s when comparng Medcad Fee-for-Servce populatons wth Medcad Managed Care populatons. Yes/No Varables. Varables wth yes/no response categores to be used n the analyss are coded as shown n the table below. Any varable wth dchotomous response optons should be coded n ths manner. For easer nterpretaton of the Page 14

16 results, the postve response should have the hghest value. Raw data for ths type of varables wll need to be recoded as the precodes typcally set the values of the responses to 1 and 2 rather than 0 and 1. Response value Label/descrpton 0 No 1 Yes All other values Not analyzed by the SAS program Three Response Optons. Any varable wth three response optons should be coded as shown n the table below. For easer nterpretaton of the results, the postve response should have the hghest value. Reverse codng may be necessary to ensure that the most postve response has the hghest value for example, where Yes, defntely s the most postve response. Response value Label/descrpton 1 Yes, defntely 2 Yes, somewhat 3 No All other values Not analyzed by the SAS program Four-Pont Frequency scale. Varables wth never to always response optons are coded as shown n the table below. Any varable wth four response optons should be coded n ths manner. For easer nterpretaton, the postve response should have the hghest value. Reverse codng may be necessary to ensure that the most postve response has the hghest value for example, where never s the most postve response. Response value Label/descrpton 1 Never 2 Sometmes 3 Usually 4 Always All other values Not analyzed by the SAS program Response value Label/descrpton 1 Defntely no 2 Somewhat no 3 Somewhat yes 4 Defntely yes All other values Not analyzed by the SAS program Page 15

17 Global Ratngs. Global ratng tems wth 0-10 response optons are coded as shown n the table below: Response value Label/descrpton 0 Worst Best All other values Not analyzed by the SAS program Codng for Adjuster Varables. If the project team decdes to case-mx adjust the CAHPS survey results, numerc varables must also be properly coded for each adjuster varable. The CAHPS Consortum recommends adjustng the data for age, educaton, and general health status; however, the program allows for a flexble number of adjuster varables. The user can choose the proper specfcaton for each varable used to adjust the data. Specfcatons used for age, educaton, and general health status n the CAHPS Analyss Program are descrbed below. Users may also specfy the varables as dchotomous wth reference categores (dummy varables). It s mportant to remember that the SAS program prevously cleaned out-of-range values for these varables. However, the added flexblty of a user-specfed number of varables and specfcaton of the varables makes t necessary to code out-of-range values to mssng before runnng data through the macro. The codng specfcaton for the numerc varable EDUCATION s ncluded n the data set. Educaton refers to the respondent s hghest level of school completed. Ths varable and ts response codes should be coded based on the responses to the educaton tem (such as Queston 35 n the CAHPS Health Plan Survey -- Adult Commercal Questonnare). Page 16

18 35. What s the hghest grade or level of school that you have completed? Response value Label/descrpton 1 8 th grade or less 2 Some hgh school, but dd not graduate 3 Hgh school graduate or GED 4 Some college or 2-year degree 5 4-year college graduate 6 More than 4-year college degree All other values Code to mssng The codng specfcaton for the numerc varable GENERAL HEALTH RATING (GHR) s ncluded n the data set. The GHR s a ratng of the survey respondent s overall health status. Ths varable and ts response codes should be based on the responses to the health status tem (for example, Queston 28 n the CAHPS Health Plan Survey -- Adult Commercal Questonnare). 28. In general, how would you rate your overall health now? Response value Label/descrpton 1 Excellent 2 Very good 3 Good 4 Far 5 Poor All other values Code to mssng Page 17

19 The numerc varable AGE s ncluded n the data set, representng age groupngs based on data from the CAHPS survey. Ths varable and ts response value codes should be based on questons about age, such as Queston 33 of the CAHPS Health Plan Adult Commercal Questonnare and Queston 32 of the CAHPS Health Plan Chld Commercal Questonnare. The response values for these questons need to match the values for AGE as follows: 33. What s your age? / 32. What s your chld s age? Response value Label/descrpton (years) For chld surveys: 0 < For adult surveys: 1 18 to to to to to to or older All other values Code to mssng Stratfed Data. If you want to combne data for reportng from dfferent samplng strata, you must create a text fle that dentfes the strata and ndcates whch ones are beng combned and the dentfer of the entty obtaned by combnng them. Some examples llustrate stuatons n whch ths feature mght be used: Two health plans are merged that were formerly separate and were treated as such n the survey. A hosptal decdes to sample 200 medcal and 200 surgcal patents, although ths s not proportonal to the numbers of dscharges n the two servces. A survey desgned to assess provders samples 80 patents from each regardless of the number of sessons each has, but the sponsor decdes to also use the data to assess provder groups. If no fle s specfed, the macro creates one usng the PLAN varable n the data set to set the Orgnal Plan and New Plan equal to the PLAN varable, the Populaton Sze equal to 1, and Subsettng Code equal to 1. If stratfcaton s part Page 18

20 of your survey desgn, an ASCII data set needs to be created wth columns separated by one or more spaces for these four varables: Orgnal Plan a unque dentfer of the unts or strata before they are combned. Ths varable can be coded as alphanumerc, but t cannot exceed 16 characters. Ths varable s the frst column of the data table. New Plan dentfer for the enttes that wll be created by combnaton of strata. Ths varable can be coded as alphanumerc, but t cannot exceed 16 characters. Ths varable s the second column of the data table. If no stratfcaton s beng done, ths column may look dentcal to the column for orgnal plan. Populaton Sze a numerc varable that ndcates the sze of the populaton for the unt or stratum. Ths varable s used to create the weghts for combnng the strata. The populatons for the combned strata should equal the total populaton of the new plan. Ths varable s the thrd column of the data table. If no stratfcaton s beng done, ths column may be set to 1s. Subsettng Code dentfer for the subset (.e., regon, state, county ) that the entty belongs n. Ths varable can be coded as alphanumerc. If no subsettng s to be done, ths column may be set to 1s. The ASCII fle for the plan detals should not contan any mssng data and each column of data should be separated by spaces. If tabs are used, the macro may not read n the data correctly. Also, be sure to not have any extra records at the bottom of the ASCII fle. If you want to make a quck sample plan detal fle from the CAHPS data set, use the program make_plandtal_dat.sas as a startng pont and change varable names and paths as needed. If the number of plans s small, t s probably easer to create the fle by hand. An example of the plan detal data set s provded for the test program (test.sas). The data fle s called plandtal.dat and looks lke the text below: HMO_A_URBAN HMO_A 5000 Northeast HMO_B_URBAN HMO_B 8000 Northeast HMO_C_URBAN HMO_C Atlantc HMO_B_RURAL HMO_B 2000 Northeast HMO_C_RURAL HMO_C 3000 Atlantc Page 19

21 The TEST data set provdes an example for three health plans (2 nd column): HMO_A, HMO_B, and HMO_C. The urban/rural strata for HMO_B and HMO_C are weghted together and the combned plans compared to HMO_A, whch had members only n urban areas. The frst column provdes a unque dentfer for each plan/regon combnaton (orgnal plan). The second column, the new plan varable, ndcates whch unts wll be combned. The thrd column, or the unt populaton sze, s used to compute the weghts for the plans. Unts wth greater populaton szes receve more weght than smaller unts n the combned plan. The fourth column s the regon (subset) of the country n whch each plan does busness. Adult and Chld Interactons (Macro Parameter ADULTKID) When the macro parameter ADULTKID equals 1, the macro creates adult and chld nteractons for the adjuster varables. The macro creates addtonal adjuster varables, wth the a set namng conventon, AC1, AC2,..., ACn, where n s the total number of adjusters orgnally submtted n the macro call parameter ADJUSTER. When there s an adult and chld nteracton, the macro creates the ACx varables by loopng through the lst of adjusters. For example: If &ADJUSTER = GHR AGE EDUCATION, then the followng addtonal nteracton adjuster varables are created: AC1 = GHR * CHILD AC2 = AGE * CHILD AC3 = EDUCATION * CHILD Usng the CAHPS Analyss Program The steps descrbed below assume a basc knowledge of how to use the SAS system. Step 1: Loadng the programs and test data Verson 4.1 of the CAHPS Analyss Program contans three core components: a SAS control program, a SAS macro, and a Plan Detal data fle. Page 20

22 All program and data sets needed for the CAHPS Analyss Program, Verson 4.1, are avalable for downloadng n the nstructons provded for every CAHPS survey. Below s a descrpton of the purpose of each fle. Each fle should be coped to a project folder related to the CAHPS data set that s to be analyzed. CONTROL.SAS CONTROL.SAS s a SAS program that contans examples of the macro call parameter arguments that need to be specfed to produce the recommended reportng measures for CAHPS surveys. For most surveys, specfc examples of the macro call are provded n separate nstructons on analyzng the results of that survey. The program also demonstrates the varable cleanng and codng steps needed to perform the analyses for entty-level comparsons. Modfcatons most lkely wll need to be made to ths program to reflect how varables are named, how varables are coded or formatted, whether or not entty stratfcaton s used, and whether or not the data set ncludes chld surveys, among the many possble combnatons. CAHPS41.SAS CAHPS40.SAS s the core SAS macro program that performs the analyses the user specfes n the control program. The macro fle should not be modfed. PLANDTAL.DAT Ths s a sample data set that s used by the CAHPS macro when runnng the TEST data set. It contans the unque plan names, combned strata names, strata weght, and subsettng code. If ths fle s not ncluded n your control programs, the macro wll create ths data set based on the PLAN varable n the nput data set. SMALLTEST.SAS Ths program creates a small data set wth ten records that can be used to better understand what the CAHPS macro s dong. More detals on how ths data progresses through the macro can be found n the Small Data Set Example secton n ths document. FORMAT.SAS FORMAT.SAS s the SAS program that creates formats helpful to vew the data wth Englsh words nstead of the data values assgned n the TEST data set. The formats have the essence of the tables descrbed n the secton SAS Data Set Requrements. The program creates the formats n the lbrary named LIBRARY as defned by the lbname LIBRARY statement n CONTROL.SAS and TEST.SAS programs. The fle, currently set up to work wth the test data programs, can be modfed for use wth other data. Modfcatons, such as changng the values of the formats, addng new formats or deletng formats, are the most common. Page 21

23 TEST.SAS TEST.SAS s a test control program for the CAHPS Analyss Program macros. It was desgned for use wth the test data sets descrbed below. Ths program was constructed to provde users wth a short program and data set that demonstrated the analyss optons and output for Verson 4.1 of the CAHPS Analyss Program. The hypothetcal example was desgned to ncorporate a dsproportonate samplng desgn of ndvduals n rural and urban areas for three health plans. One health plan (HMO_A) has members only n urban areas, whle the other two plans (HMO_B, HMO_C) have members n both urban and rural areas. To make comparsons across the three plans, the data for the plans wth members sampled from rural and urban regons need to be combned usng weghts. Varous optons are used n the test program to demonstrate the analyss features avalable to the user. Analyses are performed for all three types of reportng tems (sngle questons, global ratngs, and compostes). A varety of analyss features are also used, ncludng a varyng number of case-mx adjustment varables, turnng off the opton that creates output data sets, and the weghtng opton. TEST.SAS7BDAT, TEST.SSD01, TEST_wndows.SAS7BDAT, TEST.SD2 TEST*.* fles are SAS data sets that contan the same varables. TEST.SSD01 and TEST.SAS7BDAT are for use wth UNIX SAS programs, and TEST.SD2 and TEST_wndows.SAS7BDAT are desgned for use wth the Wndows verson of SAS. The table below descrbes the varables n the data sets and provdes value labels for each varable. (Note that the test data fles and sets were created for the CAHPS Health Plan Survey 3.0 and have not yet been updated.) Table 1. Descrpton of test data set varables Varable Descrpton Response optons ID Respondent dentfcaton number Unque numerc value PlanID Plan dentfcaton number 1 = HMO_A_URBAN ; 2 = HMO_B_URBAN ; 7 = HMO_C_URBAN ; 4 = HMO_B_RURAL ; 5 = HMO_C_RURAL ; 6 = HMO_BE_1 ;. = Mssng Q31 Global ratng of care 0 (worst) - 10 (best).=mssng 98=Inapplcable 99=No Answer Gven Page 22

24 Varable Descrpton Response optons Q38 Global ratng of plan 0 (worst) - 10 (best).=mssng 98=Inapplcable 99=No Answer Gven Q06 Q10 Q22 Q23 Q05 Q15 Problem to get doctor or nurse that you were happy wth Problem to get a referral to see a specalst that you needed Problem n gettng the care you or doctor beleved was necessary Problem wth delays n health care whle watng for approval from plan Was t easy to fnd a personal doctor or nurse How often got the help or advce you needed 1=A Bg Problem 2=A Small Problem 3=Not a Problem.=Mssng 98=Inapplcable 99=No Answer Gven 1=A Bg Problem 2=A Small Problem 3=Not a Problem.=Mssng 98=Inapplcable 99=No Answer Gven 1=A Bg Problem 2=A Small Problem 3=Not a Problem.=Mssng 98=Inapplcable 99=No Answer Gven 1=A Bg Problem 2=A Small Problem 3=Not a Problem.=Mssng 98=Inapplcable 99=No Answer Gven 1=Yes 2=No.=Mssng 98=Inapplcable 99=No Answer Gven 1=Never 2=Sometmes 3=Usually 4=Always.=Mssng 98=Inapplcable 99=No Answer Gven Page 23

25 Varable Descrpton Response optons Q17 How often got appontment as soon as you wanted 1=Never 2=Sometmes 3=Usually 4=Always.=Mssng 98=Inapplcable 99=No Answer Gven Q19 How often got needed care as soon as you wanted 1=Never 2=Sometmes 3=Usually 4=Always.=Mssng 98=Inapplcable 99=No Answer Gven Q24 How often wat n doctor s offce 1=Never 2=Sometmes 3=Usually 4=Always.=Mssng 98=Inapplcable 99=No Answer Gven Q39 General health ratng 1=Excellent 2=Very Good 3=Good 4=Far 5=Poor.=Mssng 98=Inapplcable 99=No Answer Gven Q40 Age of respondent 1=18 to 24 2=25 to 34 3=35 to 44 4=45 to 54 5=55 to 64 6=65 to 74 7=75 or older.=mssng 98=Inapplcable 99=No Answer Gven Q41 Gender 1=Male 2=Female.=Mssng 98=Inapplcable 99=No Answer Gven Page 24

26 Varable Descrpton Response optons Q21 Offce and clnc vsts n the past 6 months 1=None 2=1 tme 3=2 tmes 4=3 to 4 tmes 5=5 to 9 tmes 6=10 or more tmes.=mssng 98=Inapplcable 99=No Answer Gven Step 2: Modfyng CONTROL.SAS CONTROL.SAS s a SAS program that nvokes and executes the macro fle, CAHPS41.SAS, to perform basc analyses for the CAHPS surveys. Ths program can be modfed to perform the analyses that the team has decded to conduct. Statements from CONTROL.SAS are demonstrated below. The user can alter ths program to perform analyses on other data sets. The modfcatons demonstrated below apply to the test SAS program, TEST.SAS, as well. TEST.SAS demonstrates many of the key concepts for a lmted number of varables descrbed earler. (Note that the test data fles and sets were created for the CAHPS Health Plan Survey 3.0 and have not yet been updated.) Identfyng the Data Set, Macros, Program, and Output Fle Locatons The program statements below specfy the lbrary reference and fle names for the macros and data sets. These statements should be modfed based on the locaton of the fles. Note: The flename statements creatng logfle and outfle are not necessary unless the user wants to save the log nformaton to a fle named CONTROL.LOG and the prnted results to a fle named CONTROL.TXT. The lbname statement creatng out statement s requred to dentfy the locaton where the data sets of the summary results the program creates wll be placed. %let ProgramName = control ; %let root = /data/cahpsmmc/analyss_program/verson_4.1 ; lbname n &root./sasdata/ ; lbname out &root./sasdata/control/ ; lbname lbrary &root./sascatalog/ ; flename logfle &root./output/logs/&programname..log ; flename outfle &root./output/&programname..txt ; flename cahps &root./programs/cahps41.sas ; flename plan_dat &root./data_other/plandtal.dat ; Page 25