MA & PDP CAHPS Survey Vendor Training. November 2011

MA & PDP CAHPS Survey Vendor Training November 2011

Welcome! In today s MA & PDP CAHPS training, we will: Review Key Concepts and Protocols Provide Updates on the 2012 Survey Administration Review the 2012 Data Collection Schedule Administer the Post Training Certification Exercise 2

Post Training Certification Exercise Each survey vendor must complete and pass a Post Training Certification Exercise in order to be approved to administer the 2012 MA & PDP CAHPS Survey Certification Exercise must be completed immediately after training One Certification Exercise Form per organization We will notify survey vendors of their Post Training Certification results by November 22, 2011 3

Introductions CMS MA & PDP CAHPS Survey Project Team 4

Overview of the MA & PDP CAHPS Survey Background Goals 2012 Survey Administration Sampling Overview of the 2012 Survey 5

Background CMS has conducted a health plan version of the CAHPS Survey annually since 1998 FFS Medicare Survey added in 2000 Questions on PDP were added to MA CAHPS Survey in 2007 Stand Alone PDP Survey also added in 2007 Current survey captures Medicare beneficiaries experiences with and ratings of: Medicare Advantage Medicare Advantage Prescription Drug Medicare Advantage Preferred Provider Organization Stand Alone Prescription Drug Plans 6

Goals of the MA & PDP CAHPS Survey Provide Medicare beneficiaries and the general public information to make informed choices about Medicare coverage options Help plans identify problems and improve quality of care Enhance CMS ability to monitor quality of care and performance Provide data to evaluate plans for use in valuebased purchasing 7

Administration of 2012 MA & PDP CAHPS Survey All MA and PDP contracts with at least 600 enrollees as of July of the previous year are required to contract with approved survey vendors to administer survey In 2012, CMS is extending the survey requirement to Employer/Union Only Direct Contract Private Fee-for- Service contracts and Employer/Union Only Direct PDP contracts with at least 600 enrollees as of July 2011 8

2012 Survey Administration (cont.) Survey vendors hired by contracts have to be trained and approved by CMS Survey vendors must follow CMS data collection protocols, specifications, and timeline for conducting the survey CMS will provide survey vendors the survey and the sample for each Medicare contract Survey will be conducted at contract level for Medicare Advantage, Medicare Advantage Prescription Drug, Medicare Advantage PPO, and Stand Alone Prescription Drug Plans 9

Sample Selection Sample for the 2012 survey will be drawn by CMS Samples for the 2012 survey will be selected from MA and PDP contracts in effect on or before January 1, 2011 10

Minimum Sample Sizes Contracts with fewer than 600 enrollees are not required to conduct the survey MA contracts Those with 600-800 eligible enrollees will survey all eligible enrollees Those with more than 800 eligible enrollees will survey 800 eligible enrollees Stand Alone PDP contracts Those with 600-1,500 eligible enrollees will survey all eligible enrollees Those with more than 1,500 enrollees will survey 1,500 eligible enrollees 11

Oversampling In 2012, plans may request oversampling at the contract level Oversample requests were due November 7, 2011 Vendors must field all of the sample provided by CMS 12

2012 MA & PDP CAHPS Survey Available in English, Spanish, and Chinese Chinese translation targeted to speakers of Cantonese Use of Chinese-language materials is optional Four different versions: Medicare Advantage (MA Only) Medicare Advantage Prescription Drug Plan (MA-PD) Medicare Advantage Preferred Provider Organization (MA PPO) Stand Alone Medicare Prescription Drug Plan (PDP) 13

Overview of Survey Content All three MA questionnaires have nearly identical set of Core questions Shorter PDP questionnaire focuses on questions about Prescription Drug Plan Each questionnaire includes additional questions related to particular contract types Many items are preceded by screener questions so that only beneficiaries for whom item is relevant are asked to answer those questions Include reports of experience of care as well as overall ratings 14

Reporting Composites Getting Needed Care Getting Care Quickly Doctors Who Communicate Well Health Plan Customer Service Getting Needed Prescription Drugs (MA-PD, MA PPO and PDP) Getting Information From the Plan About Prescription Drug Coverage and Cost (MA-PD, MA PPO and PDP) 15

Overall Member Ratings Overall Rating of Health Plan Overall Rating of Care Received Overall Rating of Personal Doctor Overall Rating of Specialist Overall Rating of Drug Plan (MA-PD, MA PPO and PDP) 16

Single Item Measures Willingness to Recommend PDP Plan (MA-PD, MA PPO and PDP) Influenza Vaccination Pneumonia Vaccination Getting Medical Equipment 17

Potentially Reportable Measures The 2012 MA Only, MA-PD, and MA PPO Survey Versions include potentially reportable measures Coordination of Care Composite Stand Alone Items Satisfaction with help with coordination of care Experience with visit notes 18

QUESTIONS? 19

Survey Administration Protocols Roles and Responsibilities Communicating with Beneficiaries Monitoring and Quality Oversight Data Retention and Storage Data Collection Schedule Survey Administration 20

Roles and Responsibilities CMS provides: Survey instruments and sample files Quality Assurance Protocols & Technical Specifications V2.0 Training of survey vendors Technical assistance Analysis of data and official reports Public reporting of data Provide survey data files to NCQA Marketing guidelines to be used by health plans 21

Roles and Responsibilities (cont.) MA and PDP contracts will: Hire a CMS approved survey vendor Authorize the survey vendor to submit data on their behalf Preview the MA & PDP CAHPS Survey results from CMS prior to public reporting 22

Roles and Responsibilities (cont.) Survey vendors will adhere to all program requirements including: Participate in MA & PDP CAHPS Survey Vendor Training Successfully complete the Post Training Certification Exercise Execute a Data Use Agreement (DUA) with CMS Register as a user of the MA & PDP CAHPS Survey secure data warehouse (Extranet User Agreement) Data Administrator, Back-up Data Administrator and Program Manager Verify each contract s survey vendor authorization Perform checks of the beneficiary sample file for completeness 23

Roles and Responsibilities (cont.) Survey vendors will adhere to all program requirements including (cont.): Must not provide any beneficiary identifying information from the sample to plans Maintain confidential and secure data operations Must not violate HIPAA regulations Update their Quality Assurance Plan (QAP) Administer the survey and oversee staff Including subcontractor staff (if applicable) Submit data files to the secure data warehouse Review data submission reports Participate in all oversight activities 24

Communicating with Beneficiaries Plans and survey vendors may: Notify beneficiaries that they may be asked to participate in the MA & PDP CAHPS Survey Plans and survey vendors may NOT: Ask any MA & PDP CAHPS Survey questions of beneficiaries 4 weeks prior to and during survey administration (does not apply to CMS surveys) Attempt to influence or encourage beneficiaries to answer survey questions in a particular way Imply that the plan, its personnel or agents will be rewarded or gain benefits for positive feedback Offer incentives of any kind 25

Monitoring & Quality Oversight Survey vendors will: Perform interval checking of printed mail materials Conduct seeded mailings Perform address and telephone number verification Provide interviewer training Conduct silent monitoring of interviewers Provide oversight of staff and subcontractors (if applicable) 26

Data Retention/Storage Survey vendors will: Store returned paper questionnaires or scanned images of paper questionnaires in a secure and environmentally controlled location for a minimum of three years Must be easily retrievable Retain data collected via CATI interviewing system in a secure and environmentally controlled location for a minimum of three years Must be easily retrievable 27

Data Collection Schedule Task Date Contracts submit request for oversample 11/07/2011 Vendors provide toll-free customer support telephone numbers Plan must submit survey vendor authorization form Vendors must submit any supplemental questions (not approved for 2011 survey administration) 12/01/2011 01/06/2012 01/09/2012 Contract level sample files to vendors 02/03/2012 28

Data Collection Schedule (cont.) Task Date Mail out pre-notification letter 02/21/2012 Open customer support toll-free line 02/22/2012 Mail out first questionnaire 02/27-02/29/2012 Mail out second questionnaire 03/26-03/28/2012 Initiate telephone follow-up (first attempt) 04/16-04/23/2012 Conduct additional telephone attempts 04/24-05/29/2012 Submit interim data files by 05/11/2012 29

Data Collection Schedule (cont.) Task Date Complete the telephone follow-up sequence 05/29/2012 Cutoff date for returned mail surveys 05/29/2012 Customer support toll-free line closed 05/29/2012 Outbound telephone interviewing ends 05/29/2012 Submit final data to CMS by 06/12/2012 30

Survey Administration Sample Files Mail Materials and Protocol CATI and Telephone Protocol Spanish and Chinese Options 31

Sample Files Sample Files - CMS will: Select sample for each contract 18 years or older Continuously enrolled in a contract for at least 6 months Live in the United States, Puerto Rico, or U.S. Virgin Islands Ineligible beneficiaries include: Institutionalized members Beneficiaries sampled in another MA & PDP CAHPS Survey questionnaire Provide most complete and current addresses available for sampled beneficiaries Provide requested oversample if sufficient eligible beneficiaries available 32

Sample Files (cont.) The MA & PDP CAHPS Survey Data Coordination Team will: Create separate file for each survey vendor Two new fields for 2012: MARKETNAME and RESERVED Provide usernames and passwords for survey vendors to access secure data warehouse Upload encrypted sample files to the secure data warehouse at: http://extranet.rand.org/collaboration Survey vendor access is restricted to their own data files 33

Sample Files (cont.) Survey vendors will: Execute DUA with CMS Register as user of the MA & PDP CAHPS Survey secure data warehouse (Extranet User Agreement) Data Administrator, Back-up Data Administrator and Project Manager Perform checks of the sample files Update addresses using NCOA and/or other commercial tools Track sampled beneficiaries throughout the data collection process 34

Mail Materials and Protocol Survey vendors must be prepared to administer questionnaires in both English and Spanish Survey vendors have the option to administer questionnaires in Chinese Permissible to request entire member file from plans to obtain address, telephone number, and language preference Pre-notification letters Beneficiary full name and address used on all envelopes Contains the salutation Dear Medicare Beneficiary Include survey vendor toll-free customer support telephone number Include signature of CMS Privacy Officer on pre-notification letter 35

Mail Materials and Protocol (cont.) Pre-notification letters (cont.) CMS Logo must appear in the return address section with the survey vendor s return address Use of CMS Logo must conform to HHS guidelines (https://www.cms.gov/multimedia/02_communicationsguidelines.asp) Pre-notification letter envelope must be marked Return Service Requested or Address Service Requested Printed using a font size equal to or larger than Times New Roman 11 or Arial 11 point font Printed with English on one side and Spanish on the other, however if substantial number of Chinese-speakers in contract, option to print English-Chinese Pre-notification letter 36

Mail Materials and Protocol (cont.) Survey cover letters Printed on a separate sheet of paper not attached to questionnaire Salutation may be personalized with beneficiary name or Dear Medicare Beneficiary Include signature of senior employee of survey vendor on both cover letters Include survey vendor logo, the contract logo (or contract s parent organization s logo), or both Printed using a font size equal to or larger than Times New Roman 11 or Arial 11 point font Initial cover letter must contain Spanish text on how to request a Spanish translation of the questionnaire (Chinese text if applicable) Envelope must be printed with the survey vendor s return address Survey vendor or plan logo recommended 37

Mail Materials and Protocol (cont.) Questionnaires Window envelopes allowed Pre-paid business reply envelopes addressed to survey vendor (or approved subcontractor) must be included with questionnaire mailings Four approved questionnaires MA Only, MA-PD, MA PPO, PDP Must be printed as booklet in black and white Use of highlight color allowed OMB control number and statement must appear on first page OMB language may also appear on cover letter Full questionnaire titles including the year must be placed at the top of page one 38

Mail Materials and Protocol (cont.) Questionnaires (cont.) All survey instructions must be written verbatim at the top of the first page Plan name must be printed on the back page of questionnaire Return address of survey vendor must be included Form ID linked to the Unique Respondent Finder Number must be printed on back page Beneficiary name must NOT appear on the survey 39

Mail Materials and Protocol (cont.) Questionnaires (cont.) Format Minimum font size equal to or larger than Arial 11 point Question and answer category wording must not be changed Order of Core and About You questions must not be changed About You questions cannot be eliminated Question and answer categories must remain together in the same column and on same page Response choices must be listed individually for each question Vertical alignment of response items required May use precodes next to response options, except for the overall rating items Boxes or ovals are acceptable for response categories 40

Mail Materials and Protocol (cont.) Questionnaires (cont.) Supplemental Questions Must Be approved by CMS Submit request for new 2012 questions to MA & PDP CAHPS Survey Project Team via technical support email by 1/09/2012 Follow the Core questions Be preceded by transitional phrases Must NOT Pose a burden to the respondent Ask respondent why a specific response was chosen Affect the likelihood of response to the survey Cause respondent to terminate the survey Identify respondent 41

Mail Materials and Protocol (cont.) Data Receipt Track by date of receipt Key-entry or scanning technology If beneficiary returns more than one completed questionnaire, use the first completed questionnaire received Blank or partially completed questionnaires Send second questionnaire if first questionnaire is returned with all missing responses or incomplete Initiate telephone contact if second questionnaire is returned blank or incomplete Ambiguous responses Decision rules 42

Mail Materials and Protocol (cont.) Decision rules for data capture If a mark falls between two choices and is obviously closer to one choice than another, select the choice to which the mark is closest Example 1 (Mail) Never x Sometimes Usually Always Code as: Sometimes 43

Mail Materials and Protocol (cont.) Decision rules for data capture If a mark falls equidistant between two choices, code the value of the item as M Missing/Don t Know Do not impute Example 2 (Mail) Never x Sometimes Usually Always Code as: M - Missing/Don t Know 44

Mail Materials and Protocol (cont.) Decision rules for data capture If a value is missing, code it as M Missing/Don t Know Do not impute Note: dependent questions appropriately skipped should be coded as 8 Not Applicable Example 3 (Mail) Never Sometimes Usually Always Code as: M - Missing/Don t Know 45

Mail Materials and Protocol (cont.) Decision rules for data capture When more than one response choice is marked, code the value as M Missing/Don t Know Do not impute Exception: Several questions that have instructions to mark one or more may have multiple responses Example 4 (Mail) x x Never Sometimes Usually Always Code as: M - Missing/Don t Know 46

Mail Materials and Protocol (cont.) Quality control Conduct interval checking of printed mail pieces Conduct seeded mailings Perform address validation and updates Data retention and storage Paper questionnaires and/or scanned images must be stored In a secure and environmentally controlled location For minimum of three years 47

CATI and Telephone Protocol Survey vendors must be prepared to administer questionnaires in both English and Spanish Survey vendors have the option to administer questionnaires in Chinese Telephone interviews must not be conducted via inbound calls until after the telephone component of survey administration begins Permissible to request entire member file from plans to obtain telephone numbers and language preference 48

CATI and Telephone Protocol (cont.) CATI Program with official telephone scripts Program skip pattern questions appropriately Link electronically to survey management system Predictive dialing may be used Interviewer records respondent answers electronically Data must be stored: In a secure and environmentally controlled location For a minimum of three years 49

CATI and Telephone Protocol (cont.) Telephone Scripts Standardized telephone scripts provided by CMS Four approved telephone scripts MA Only, MA-PD, MA PPO, PDP Text must NOT be modified Scripts must be read verbatim 50

CATI and Telephone Protocol (cont.) Telephone Scripts (cont.) Supplemental questions Must Be approved by CMS Submit request for new 2012 questions to MA & PDP CAHPS Survey Project Team via technical support email by 1/09/2012 Follow the Core questions Be proceeded by transitional phrases Must NOT Pose a burden to the respondent Ask respondent why a specific response was chosen Affect the likelihood of response to the survey Cause respondent to terminate the survey Identify respondent 51

CATI and Telephone Protocol (cont.) Obtain telephone numbers for non-respondents to mail questionnaire Directly from contract Via a file that contains all contract enrollees Sample must not be shared with contract Commercial software Internet directories Directory assistance Other tested methods 52

CATI and Telephone Protocol (cont.) Telephone attempts Phone rings six times with no answer Beneficiary requests call back Phone answered by someone other than beneficiary who is unavailable Busy signal for each of three consecutive attempts (made approximately at 20 minute intervals) Answering machine reached Quality control Formal interviewer training Minimum of 10% silent monitoring of entire call Across all MA & PDP interviews Across all MA & PDP interviewers 53

CATI and Telephone Protocol (cont.) Telephone Interviewer Training Telephone scripts and CATI programs Survey introduction Guidelines for reaching respondents Identifying possible ineligible respondents Definition of telephone attempts Interviewing guidelines and conventions System conventions, e.g. CATI screens, interim disposition codes Avoiding refusals Probing for complete answers Customer Service FAQs 54

Spanish and Chinese Options Spanish translation Puerto Rico All sampled members administered questionnaires in Spanish unless member requests English translation Sampled members not residing in Puerto Rico Survey vendors have the option to mail Spanish translated materials in first mailing Double stuffing permitted Chinese translation Survey vendor has the option to mail Chinese translation of materials in first mailing Double stuffing permitted 55

QUESTIONS? 56

Data Coding and Preparation File Encryption File Specifications Decision Rules and Coding Guidelines Decision Rules for Screener and Dependent Questions Survey Disposition Codes 57

Survey Data File Encryption Data files must be encrypted prior to data submission Survey vendors required to use PGP Use Public Key encryption Data files submitted by survey vendors that are not encrypted will be rejected and must be resubmitted 58

File Specifications Survey vendors must use flat ASCII file format to submit survey data files Survey vendors will submit all contracts sampled member records in one file No substitutions for valid data elements are acceptable 59

File Specifications (cont.) Survey data will contain one record for each sampled member Each record will consist of two parts: Survey Status Section (Tracking) Beneficiary Survey Data Section 60

Survey Status Section (Tracking) Status section contains: Unique Finder Number for each sampled beneficiary Survey Type Contract Number Survey Completion Mode (mail vs. outbound CATI) Survey Language Final Disposition Code Contract Marketing Name Total Supplemental Items (# of supplemental items added to the survey) Each field in the Survey Status Section requires an entry for a valid submission 61

Beneficiary Survey Data Section Contains survey responses from every respondent who completed a survey Include only survey data where the final disposition is Complete (10) or Partially completed (31), leave blank for all others Survey vendors should submit only data from MA & PDP CAHPS Survey questions (data from supplemental items should not be submitted) 62

Beneficiary Survey Data Section (cont.) For survey records included, all response fields must have a valid value Valid values can include Missing or Not Applicable Format of the records in this section of the file will vary depending on type of questionnaire administered Survey results with fewer data fields should be left blank beyond last valid response 63

Decision Rules and Coding Guidelines Mail Surveys Except for questions where a respondent can select more than one response option, such as race, when more than one response option is marked, code as M Missing/Don t Know. 64

Decision Rules for Screener and Dependent Questions Some items can and should be skipped by certain beneficiaries Dependent questions that are appropriately skipped should be coded as 8 Not Applicable Screener questions that are left blank should be coded as M Missing/Don t Know 65

Decision Rules for Screener and Dependent Questions (cont.) If respondent made an error in the skip pattern, survey vendors must not clean or correct skip pattern errors Do not impute a response based on beneficiary s answers to dependent questions Enter value provided by beneficiary (even if incorrect) 66

Decision Rules for Beneficiary Survey Data Section Enter all survey responses provided by the beneficiary for each survey item All survey questions must have a valid code For completed/partially completed surveys, code missing answers as M Missing/Don t Know or 8 Not Applicable 67

Survey Disposition Codes Survey disposition codes are used to track and report whether a beneficiary has completed a questionnaire or requires further follow-up Survey vendors are required to assign and maintain up-to-date survey disposition codes for each beneficiary in the sample 68

Survey Disposition Codes (cont.) Interim disposition codes are for internal purposes only and should not be reported Final status codes only are provided to CMS Data files submitted to CMS must contain a final status code for each beneficiary in the file 69

Final Survey Status Disposition Codes Completed Survey Completed survey (Code 10): At least one reportable measure answered and 50% or more of the applicable to all (ATA) items answered Note that potentially reportable measures should be considered reportable for the purpose of determining a completed survey 70

Final Survey Status Disposition Codes Partially Completed Survey Partially completed survey (Code 31): At least one reportable measure answered but Less than 50% of the applicable to all (ATA) items answered Note that potentially reportable measures should be considered reportable for the purpose of determining a partially completed survey 71

Final Survey Status Disposition Codes for Non-Completes (1) Institutionalized Deceased Language Barrier: Unable to complete survey in English, Spanish, or Chinese Mentally or physically unable to respond Refusal: Refused to complete survey 72

Final Survey Status Disposition Codes for Non-Completes (2) Non-response: No response collected Blank Returned: Returned survey by mail, answered no questions Bad address and/or Bad Telephone number: Unable to obtain a viable address and/or telephone number Excluded from survey: Beneficiary excluded from survey process 73

QUESTIONS? 74

Data Submission Data Submission Process Data Submission Deadlines Preparing to Submit Data Files File Encryption Overview of Data Warehouse Guide to Data Submission Process Data Auditing and Validation Checks Data Submission Notification Technical Support 75

Data Submission Process MA & PDP CAHPS Project Team has developed a secure Data Warehouse using a web-based application hosted by the RAND Corporation Data Warehouse will operate as a secure file transfer system that survey vendors will use to pick up sample file and submit survey data files Use of the Data Warehouse does not require installation or special software or licensing fees for survey vendors (except PGP software) 76

Data Submission Deadlines Interim survey data file must be submitted by survey vendors by May 11, 2012 at 11:59 p.m. Eastern Time Survey vendors are encouraged to submit files early enough to allow them enough time to resubmit if they have to and still meet the deadline If vendors submit more than once, files must include all records in the re-submission Final survey data file must be submitted by survey vendors by June 12, 2012 at 11:59 p.m. Eastern Time It is the responsibility of the survey vendor to ensure that data are submitted on time 77

Preparing to Submit Data Files Survey vendors must have designated a Data Administrator and a Back-up Data Administrator Survey Vendor Authorization on file with MA & PDP CAHPS Data Coordination Team Completed Extranet User Agreement on file with MA & PDP CAHPS Data Coordination Team 78

File Encryption Sample files for survey vendors will be encrypted using PGP software (www.pgp.com) Survey vendors required to encrypt data files using PGP software (www.pgp.com) prior to submitting them to MA & PDP CAHPS Survey Data Warehouse 79

File Encryption (cont.) MA & PDP CAHPS Data Coordination Team will provide the survey vendors with a Public Key to receive sample data files Key will be communicated to vendor via the Outbox Survey vendors must create a Public Key to encrypt survey data files prior to submission Vendor must communicate the Key via the Inbox 80

Overview of Data Warehouse Available via the Internet Hosted on RAND Corporation s Web site Each survey vendor will have own folder on the site Survey vendor s folder will include an Outbox for downloading sample file and/or other project documentation as well as an Inbox for submitting survey data files 81

Overview of Data Warehouse (cont.) Quarantine folder where submitted data files that are not encrypted (don t have.pgp extension) or do not otherwise comply with the established naming standards are placed Quarantined files are deleted automatically After each data submission, survey vendor receives email letting them know that file was/was not successfully submitted 82

Guide to Data Submission Process Step-1 Login to RAND s Extranet: http://extranet.rand.org/collaboration Step-2 Enter your Username and Password 83

Guide to Data Submission Process (cont.) Step-3 Enter Username & Password. Click the checkbox Remember password. Note: When Remember password is checked, next time you login this page will not appear. 84

Guide to Data Submission Process (cont.) Step-4 Click on the MA & PDP CAHPS Survey Data Warehouse Tool link. 85

Guide to Data Submission Process (cont.) Step-5 Under Inbox click Add new document to upload a file. 86

Guide to Data Submission Process (cont.) Step-6 The following will appear. Click on Browse. 87

Guide to Data Submission Process (cont.) Step-7 The following will appear. Navigate to your file and select Open. 88

Guide to Data Submission Process (cont.) Step-8 If you have selected a file you will see the path of the file. Click OK to complete. 89

Guide to Data Submission Process (cont.) If you have selected a file with invalid characters you will see the image below. Please rename your file and remove the invalid characters. 90

Guide to Data Submission Process (cont.) Step-9 If you have successfully uploaded your file you will see your file appear under Inbox. PGP file check column may display In Progress. Refresh your page and if the file has the correct extension, the PGP File Check column will display Completed. 91

Data Auditing and Validation Checks MA & PDP CAHPS Data Coordination Team will audit data files as they are submitted for compliance with file layout specifications Data audit includes: Checking for.pgp file extension Logical record lengths, appropriate character set, naming conventions Presence of required data fields Range checks Verification of coding of Survey Disposition Code 92

Data Submission Notification Survey vendor (Data Administrator, Back-up Data Administrator and Project Manager) will receive two email notifications for each data submission First (automated) email contains notification that file was received Sent after encryption check 93

Data Submission Notification (cont.) Second email is sent after edit checks Indicates if file successfully passed checks Will go out by 8:00 p.m. Eastern Time on the next business day after submission If file fails any edit checks, email will: Instruct survey vendors that they must submit data files again Contain full detail of the edit check report including a list of involved records If file passes checks, email will say that no further action is necessary, and provide a summary of file contents for verification by the vendor 94

Data Submission Notification (cont.) Survey vendors are responsible for submitting corrected data file by deadline for submission Data files not received and accepted prior to 11:59 p.m. Eastern Time on the deadline date will result in the contract receiving 1 star for each of the CAHPS measures publicly reported on the Medicare Plan Finder tool 95

Technical Support Contact the MA & PDP CAHPS Data Coordination Team for technical support and/or assistance related to data submission at: MA-PDPCAHPSTECHSUPPORT@rand.org 96

QUESTIONS? 97

Survey Vendor Oversight Oversight activities ensure: Compliance with MA & PDP CAHPS Survey protocols Survey data collected and submitted are complete, valid and timely Standardization and transparency of publicly reported MA & PDP CAHPS Survey results 98

Oversight Activities Oversight activities include: Review of QAP Review of survey materials Conduct site visits and conference calls Analysis of submitted data 99

Oversight Activities (cont.) Quality Assurance Plan (QAP) Documents understanding, application and compliance with survey protocols Follows the Model QAP specifications Provides a guide for the site visit Submitted each year of survey administration Submit via the MA & PDP CAHPS Survey Technical Assistance email: ma-pdpcahps@azqio.sdps.org Due date of 12/16/2011 100

Oversight Activities (cont.) Review of Survey Materials Spanish and Chinese (if applicable) mail materials and CATI scripts to be submitted by 1/25/2012 101

Oversight Activities (cont.) Site visits and conference calls Review and observe systems and procedures Discussions with project staff Including subcontractors, if applicable All materials related to survey administration are subject to review Feedback report includes action items for follow-up Conference calls as needed 102

Oversight Activities (cont.) Analysis of submitted data Intended to detect errors in data submission Includes review of outliers, anomalies, unusual patterns, etc. Follow-up as appropriate 103

Discrepancy Reports Notify CMS of variation(s) from MA & PDP CAHPS Survey protocols during survey administration Complete and submit online report immediately upon discovery of issue Provide sufficient detail CMS contract number (i.e., Hxxxx) How issue was identified Number of eligible beneficiaries affected Corrective action plan Other details and information 104

Discrepancy Reports (cont.) CMS Review Process Acknowledgment of receipt Assessment of actual or potential impact on publicly reported results Additional information may be requested Notification of review outcome 105

Corrective Actions If survey vendors fail to adhere to MA & PDP CAHPS Survey protocols, they must develop and implement corrective actions If survey vendors do not fix persistent problems, they may lose approved status for conducting the MA & PDP CAHPS Survey Other sanctions may also be applied 106

QUESTIONS? 107

Reporting Public reporting of Medicare CAHPS data Additional reporting of Medicare CAHPS data to plans Potential reporting Coordination of care composite Satisfaction with help with coordination measure Experience with visit notes measure 108

Public Reporting of Medicare CAHPS Data CMS publicly reports Medicare CAHPS Survey data by contract (MA and PDP) and state (FFS) Medicare & You Handbook Medicare Plan Finder Purpose of public reporting Help beneficiaries choose coverage Inform plans quality improvement efforts Incentive for plans to improve performance Accountability and transparency Inform CMS and policymakers 109

Potential New Coordination of Care Composite Six reports about care items: Doctor had medical records or other information about your care? Someone from your personal doctor s office followed up to give you test results? Got test results as soon as you needed? Doctor talked about prescription medicines? Got help about managing care? Doctor informed and up-to-date about specialty care? 110

Potential New Coordination of Care Composite (cont.) One satisfaction item: How satisfied are you with the help you received from your personal doctor s office to manage your care in the last 6 months? Very dissatisfied Somewhat dissatisfied Neither dissatisfied nor satisfied Somewhat satisfied Very satisfied 111

Additional Reporting of Medicare CAHPS Data to Plans CMS provides more detailed reports for MA and PDP contracts Reports summarize contract s survey results and compare contract scores at state and national levels Measures that will be reported to plans Items within Composites Global ratings Individual items 112

CMS Analysis of Data Use of composite measures Case-mix adjustment and weighting Significance testing and star assignment 113

Use of Composite Measures Answers to questions about same topic are combined to form composites Items in a composite generally given equal weight Getting Needed Prescription Drugs composite is weighted within contract proportionately to the number of beneficiaries (within the contract) who report attempting to fill a prescription at a pharmacy or by mail Ease of using plan to get medicines doctor prescribed Ease of using plan to fill prescriptions at a local pharmacy Ease of using plan to fill prescriptions by mail 114

Case-mix Adjustment and Weighting Certain respondent characteristics affect responses but fall outside plan s control Age, education level, health status, etc. CMS adjusts for such respondent characteristics when comparing contracts Case-mix adjustment done via linear regression modeling Data from MA contracts weighted to represent each contract s proportion of PD enrollees 115

Case-mix Coefficients CMS contractors perform case-mix adjustment Case-mix coefficients are re-estimated each year based on data CMS receives Case-mix adjustment coefficients appear each year in reports to plans Item-level coefficients will be provided Reports to contracts will be delivered Fall 2012 116

Significance Testing and Star Assignment Significance testing used to compare mean scores with national mean scores Differences from national mean at p<0.05 level identified with up or down arrows Scores not reported if item or composite has fewer than 10 observations in plan reports For publicly reported measures, CMS converts mean scores into 100 scale and assigns star ratings Items with very low reliability for a given contract (due to small N) may be excluded from star calculations 117

Survey Vendor Analysis of Data CMS-calculated results Vendor analysis of Core survey items Vendor analysis of supplemental items 118

CMS-Calculated Results CMS-calculated results are official results Vendors should remind plans that vendor results are unofficial and for internal/qi purposes Only CMS official results can be used for marketing activities CMS will continue to provide reports to MA and PDP contracts These results include data from completed and partially completed surveys 119

Survey Vendor Analysis of Core Survey Items Survey Vendors may conduct own analyses of Core items for quality improvement purposes Cell sizes must not be less than 10 Intervention or follow-up with low scoring individuals is not permitted Survey Vendors cannot provide individual-level data to plans 120

Survey Vendor Analysis of Supplemental Items Survey Vendors can add small number of supplemental items CMS approval needed Supplemental items must meet certain conditions Must be placed after Core items Not included in data file vendors submit to CMS Results vendors send to plans must not allow identification of individuals 121

QUESTIONS? 122

Wrap Up and Next Steps Important Dates 12/16/2011 QAPs due to MA & PDP CAHPS Survey Project Team via Technical Assistance at: ma-pdpcahps@azqio.sdps.org 1/06/2012 Survey Vendor Authorization Forms due to Data Coordination Team: May be submitted via US Mail, Federal Express, UPS, or other overnight delivery service See Appendix B for address information 123

Wrap Up and Next Steps (cont.) Important Dates (cont.) 1/09/2012 Supplemental questions for 2012 are due to CMS via Technical Assistance email at: ma-pdpcahps@azqio.sdps.org 1/25/2012 Spanish and Chinese (if applicalbe) mail materials and CATI scripts due to MA & PDP CAHPS Survey Project Team via Technical Assistance at: ma-pdpcahps@azqio.sdps.org See 2012 Data Collection Schedule for key survey administration dates 124

Wrap Up and Next Steps (cont.) Post Training Survey Vendor Certification Exercise Immediately upon conclusion of training Accessible via webinar for 30 minutes Feedback on Training Follows certification exercise Accessible via webinar for 15 minutes Vendor Notification CMS follow-up regarding certification exercise by 11/22/2011 125

Contact Us MA & PDP CAHPS Survey Information and Technical Assistance Web site: www.ma-pdpcahps.org Email: ma-pdpcahps@azqio.sdps.org Telephone: 1-877-735-8882 126