CAHPS Survey for Accountable Care Organizations (ACOs) Participating in Medicare Initiatives Survey Vendor Training

Transcription

1 CAHPS Survey for Accountable Care Organizations (ACOs) Participating in Medicare Initiatives Survey Vendor Training August 2015

2 Welcome! In today s CAHPS for ACOs Survey training, we will: Review Key Concepts and Protocols Provide Updates on the 2015 Survey Administration Review the 2015 Data Collection Schedule Administer the Post Training Quiz 2

3 Post Training Quiz Each survey vendor must complete and pass a Post Training Quiz in order to be approved to administer the 2015 CAHPS for ACOs Survey Quiz must be completed online immediately after training One Quiz Form will be accepted per survey vendor We will notify survey vendors of their Post Training Quiz results by 8/21/2015 3

4 Overview and Background

5 Topics Overview of the Shared Savings Program and Pioneer ACO Model The Shared Savings Program Quality Standard The Patient/Caregiver Experience of Care Domain CAHPS for ACOs Survey ACO-9 and ACO-12 Survey Versions Other Key Changes for 2015 Survey Administration 5

6 Shared Savings Program Goals The Medicare Shared Savings Program is an approach to the delivery of health care aimed at reducing fragmentation, improving population health, and lowering overall growth in expenditures by: Promoting accountability for the care of Original Medicare beneficiaries Improving coordination of care for services provided under Medicare Parts A and B Encouraging investment in infrastructure and redesigned care processes 6

7 Shared Savings Program Vision ACOs promote the delivery of seamless, coordinated care that promotes better care, better health and lower growth in expenditures by: Putting the beneficiary and family at the center of care Remembering patients over time and place Attending carefully to care transitions Managing resources carefully and respectfully Evaluating data to improve care and patient outcomes Using innovation focused on the three-part aim Investing in care teams and their workforce 7

8 Pioneer Model and Advance Payment The Pioneer ACO Model is designed for health care organizations and providers that are already experienced in coordinating care for patients across care settings. It is designed to work in coordination with private payers by aligning provider incentives which will improve quality and health outcomes for patients across the ACO and achieve cost savings for Medicare, employers and patients. The Advance Payment Model is designed for physician-based and rural providers; through this model, selected participants will receive upfront and monthly payments which they can use to make important investments in their care coordination infrastructure. 8

9 Patient Population ACO accepts responsibility for an assigned patient population Assigned patient population is the basis for establishing and updating the financial benchmark, quality measurement and performance and focus of the ACO s efforts to improve care and reduce costs Assignment will not affect beneficiaries guaranteed benefits or choice of doctor or any other provider Shared Savings Program preliminary prospective assignment with a retrospective reconciliation Pioneer Model prospective assignment 9

10 Quality Measurement and Performance Quality measures are separated into the following four key domains that serve as the basis for assessing, benchmarking, rewarding and improving ACO quality performance: Better Care for Individuals 1. Patient/Caregiver Experience 2. Care Coordination/Patient Safety Better Health for Populations 3. Preventative Health 4. At-Risk Population Each of the 4 domains is equally weighted and is 25 percent of an ACO s quality score 10

11 CAHPS for ACOs: Two Survey Versions CMS will provide 2 versions of the CAHPS for ACOs Survey to assess patient experience of care received from ACOs for quality reporting period 2015: ACO-9: required content ACO-12: required content + optional content ACO-9 includes core CG-CAHPS Survey plus the Summary Survey Measures (SSMs) in the Shared Savings Program final rule and the 2015 Physician Fee Schedule rule ACO-12 includes 3 SSMs that are not part of the ACO quality score ACO-12 is the survey that was used to measure reporting periods 2012 and 2013, and has been optional since 2014 ACO data warehouse will accept data for both versions of survey 11

12 Summary Survey Measures: ACO-9 Measure SSM Content Source ACO 1-7 Getting Timely Care, Appointments, & Information How Well Your Providers Communicate Patient s Rating of Provider Access to Specialists Health Promotion and Education Shared Decision Making Health Status & Functional Status CG CAHPS Core, CG CAHPS Supplemental, Program Specific Scored ACO-34 Stewardship of Patient Resources CG-CAHPS CORE requirement Courteous & Helpful Office Staff CG CAHPS Supplemental Scored CG CAHPS Core Not part of ACO Quality Score 12

13 Summary Survey Measures: ACO-12 Measure SSM Content Source ACO 1-7 Getting Timely Care, Appointments, & Information How Well Your Providers Communicate Patient s Rating of Provider Access to Specialists Health Promotion and Education Shared Decision Making Health Status & Functional Status ACO-34 Stewardship of Patient Resources CG-CAHPS CORE requirement Additional SSMs Courteous & Helpful Office Staff Care Coordination Between Visit Communication Helping you Take Medications as Directed CG CAHPS Core, CG CAHPS Supplemental, Program Specific Scored CG CAHPS Supplemental Scored CG CAHPS Core Not part of ACO Quality Score CAHPS Supplemental, Program Specific items Not part of ACO Quality Score 13

14 Resources For More Information about the Medicare Shared Savings Program Fee-for-Service- Payment/sharedsavingsprogram/index.html?redirect=/shared savingsprogram/ For more information about the Pioneer ACO model For more information about CAHPS for ACOs survey implementation 14

15 Program Requirements

16 Objectives Communicating with Beneficiaries Roles and Responsibilities Minimum Survey Vendor Business Requirements Technical Assistance and Communication 16

17 Communicating with Beneficiaries ACOs may notify beneficiaries that they may be asked to participate in the CAHPS for ACOs Survey Must notify all beneficiaries ACOs and survey vendors are strongly encouraged not to: Ask any CAHPS for ACOs Survey questions of Original Medicare beneficiaries 4 weeks prior to, during or after survey administration ACOs and survey vendors must NOT: Attempt to influence or encourage beneficiaries to answer survey questions in a particular way Imply that the ACO, its personnel or agents will be rewarded or gain benefits for positive feedback Offer incentives of any kind 17

18 Roles and Responsibilities CMS provides: Survey administration protocols and timeline o Quality Assurance Guidelines Version 3 Training of survey vendors Technical assistance Tools, format and procedures for submitting the collected data Analyses of data and official reports o ACO reports and Excel files detailing survey scores 18

19 Roles and Responsibilities (cont d) ACOs will: Contract with a CMS approved survey vendor Select survey version and inform survey vendor Authorize the survey vendor to collect and submit data on their behalf o Authorize survey vendor no later than 9/22/

20 Roles and Responsibilities (cont d) Survey vendors will: Review and acknowledge agreement with the rules of participation Participate in: o CAHPS Survey for ACOs Participating in Medicare Initiatives Webinar Training Session Successfully complete a quiz measuring comprehension of CAHPS for ACOs Survey protocols o All CAHPS for ACOs Survey update training sessions, as scheduled Follow CAHPS for ACOs Survey administration requirements Complete and submit the Vendor Access to ACO Data Warehouse Form by 9/8/

21 Roles and Responsibilities (cont'd) Survey vendors will: Attest to the accuracy of their data collection processes Implement security procedures aligned with HIPAA and CMS Privacy Requirements Develop a Quality Assurance Plan (QAP) and submit by 9/11/2015 Submit Mail materials by 9/25/2015 Submit CATI screenshots by 10/23/2015 Participate in oversight activities 21

22 Roles and Responsibilities (cont'd) Survey vendors will: Execute a Data Use Agreement (DUA) with CMS by 9/30/2015 o DUA restricts use of sample and survey data o Data that are appended to the sample file or beneficiary survey data for reporting to clients must be approved in advance o the CAHPS for ACOs Survey Project Team to get approval Verify the ACO has authorized the survey vendor Receive and perform checks of each ACO s sample file Administer the CAHPS for ACOs Survey o According to protocols and procedures established by CMS o Following the required data collection schedule o Oversee the quality of work of staff and subcontractors Submit data on behalf of the ACO Review CAHPS for ACOs Survey data submission reports 22

23 Minimum Survey Vendor Business Requirements Organizations (vendor and subcontractors) must currently possess all required facilities and systems to implement the CAHPS for ACOs Survey Subcontractors will be subject to the same requirements as the vendor Subcontractors with key roles in the survey administration are required to attend all training sessions If approved, organizations must maintain a minimum of one active ACO client contract for at least one of two consecutive survey cycles Vendors must notify CMS if there are changes in key personnel 23

24 Minimum Survey Vendor Business Requirements (cont'd) Possess relevant business experience Demonstrate survey capability and capacity Participate in required training sessions Adhere to all protocols and specifications Participate in oversight activities 24

25 Technical Assistance and Communication For additional information and technical assistance: Phone (toll free): For ACO Data Warehouse or data submission issues: To learn more about the CAHPS for ACOs Survey and to see important announcements, visit the CAHPS for ACOs Survey website at acocahps.cms.gov. 25

26 Sample Design and Beneficiary Selection

27 Objectives Overview Sample Selection and Eligibility Criteria Sample File Layout Delivery of Sample File to Vendors 27

28 Overview Samples for the 2015 survey will be drawn by CMS 860 Original Medicare beneficiaries per ACO 28

29 Sample Selection and Eligibility Criteria CMS selects the sample for each ACO Original Medicare beneficiaries assigned to the ACO 18 years or older Live in the United States, Puerto Rico, or U.S. Virgin Islands Ineligible beneficiaries include individuals known to be institutionalized o Living in an institution or residential facility 29

30 Sample Files CMS will provide the most complete and current contact information available for sampled beneficiaries Addresses as of October 2015 Phone numbers as of October 2015 Oversample high users of care 25% of each ACO s sample Drawn from top 10% of beneficiaries based on visits 30

31 Sample File Layout File Record Layout for the Sample File RAND Field Name Starting Position in Record Field Length Valid Codes Field Contents FINDER 1 8 Numeric Unique beneficiary finder number assigned by CAHPS for ACOs Survey Data Coordination Team FNAME 9 30 Text CMS beneficiary first name MNAME Text CMS beneficiary middle name LNAME Text CMS beneficiary last name DOB_C 94 8 yyyymmdd Date of birth ZIP Char Mailing address zip code ADDR1FINAL Text Mailing address line 1 ADDR2FINAL Text Mailing address line 2 CITY Text Mailing address city name PR_CD Text Puerto Rican urbanization code STATE Char Mailing address USPS state code FIPS_STATE Char CMS state FIPS code, 2 numbers with leading zeros FIPS_CNTY Char CMS county FIPS code, 3 numbers with leading zeros 31

32 Sample File Layout (cont d) File Record Layout for the Sample File RAND Field Name Starting Position in Record Field Length Valid Codes Field Contents GENDER Gender code: 1 = male, 2 = female ACO_ID [A, P]nnnn Five-character ACO identifier: begins with a letter A or P followed by 4 numbers ACONAME Free text ACO name provided by CMS FOCALTYPE Numeric Provider type: 1 = primary care 2 = specialist PRTITLE Text Type of provider [physician, physician assistant, nurse practitioner, certified nurse midwife, certified clinical nurse specialist] PRFNAME Text Provider first name PRLNAME Text Provider last name TELEPHONE NUMBER Char Beneficiary phone number 32

33 Delivery of Sample File CAHPS for ACOs Survey Data Coordination Team will: Provide a separate file for each survey vendor Authorize survey vendors to access the secure ACO data warehouse Provide encrypted sample files to the secure ACO data warehouse at: o Survey vendor access is restricted to their own files 33

34 Data Collection Protocol

35 Objectives Overview 2015 Data Collection Schedule Mail Protocol Phone Protocol Languages 35

36 Overview Data collection procedures are the same regardless of whether ACO chooses to field the ACO-9 or the ACO-12 survey Mixed-mode survey administration Survey vendors must be prepared to administer questionnaires in English and one or more of the following languages: Spanish Cantonese Korean Mandarin Russian Vietnamese Portuguese May include an insert with the pre-notification letter and first survey mailing that includes a number to call to request a translation of the survey 36

37 Overview (cont'd) No supplemental questions permitted Proxy respondents are permitted Sampled beneficiaries who are unable to respond to the phone interview Permission must be received from the beneficiary 37

38 2015 Data Collection Schedule Task Vendors must complete and a Vendor Access to ACO Data Warehouse Form to aco-datasupport@rand.org Date 9/8/2015 Vendors must submit QAP 9/11/2015 ACO must submit Survey Vendor Authorization Form for 2015 CAHPS for ACOs Survey administration 9/22/2015 Vendors must submit mail survey materials 9/25/2015 Vendors must provide confirmation that a new or updated DUA has been executed with CMS 9/30/

39 2015 Data Collection Schedule (cont'd) Task Vendors provide toll-free customer support phone numbers 10/2/2015 ACO sample files become available to vendors 10/22/2015 Vendors must submit English CATI screenshots 10/23/2015 Date Mail-out pre-notification letter 11/13/ /16/2015 Open customer support toll-free line 11/16/2015 Mail-out first questionnaire 11/20/ /23/

40 2015 Data Collection Schedule (cont'd) Task Date Submit interim data 12/9/ /11/2015 Mail-out second questionnaire 12/15/ /16/2015 Initiate CATI follow-up (1 st attempt must occur during this time) 1/6/2016 1/13/2016 Conduct additional CATI follow-up 1/14/2016 2/3/2016 Complete the phone follow-up sequence 2/3/2016 Cutoff date for returned mail surveys 2/3/2016 Customer support toll-free line closed 2/3/2016 Submit final data to CMS 2/10/2016 2/12/

41 Mail Protocol Pre-notification letter Two mailings Survey cover letters Questionnaire Data receipt and processing Quality control guidelines 41

42 Mail Protocol (cont'd) Pre-notification letter Full name and address on envelopes Contains the salutation Dear Medicare Beneficiary Is dated November 16, 2015 Include survey vendor s customer service phone number o Provided to CMS in advance CMS logo in the return address section Envelope marked Return Service Requested or Address Service Requested or Change Service Requested Font equal to or larger than o Times New Roman 11 point o Arial 11 point 42

43 Mail Protocol (cont'd) Survey cover letters Are dated: o November 23, 2015 (1 st survey mailing) o December 16, 2015 (2 nd survey mailing) Printed on a separate sheet of paper not attached to questionnaire Salutation must be personalized with beneficiary name Signature of senior employee of survey vendor Survey vendor logo and return address Font size equal to or larger than o Times New Roman 11 point o Arial 11 point 43

44 Mail Protocol (cont'd) Questionnaire Full survey title must be placed at the top of page 1 Name of clinician provided in sample file printed in Question 1 Question and answer category wording must not be changed No changes in the order of the questions or answer categories About you questions must be included All instructions written at top of page 1 Return address for mail processing placed on bottom of the last page Print survey as booklet in black and white o May include a highlight color 44

45 Mail Protocol (cont'd) Questionnaire A form tracking ID linked to the Unique Respondent Finder Number must be printed on each survey o Best practice - form tracking ID linked to the Unique Respondent Finder Number printed on each page o The ID may be printed on the first and/or last page o An internal tracking barcode next to the tracking ID on the survey and other materials is acceptable May include a version code to indicate survey version (ACO-9 or ACO-12) Font size equal to or larger than Arial 11 point Optional Formatting o Two column format o Wide margins o Boxes or ovals are acceptable for response categories 45

46 Mail Protocol (cont'd) Mail packet Envelope must be printed with: o Survey vendor s logo May add CMS logo o Survey vendor s return address Use of window envelopes is permissible Include a prepaid business reply envelope addressed to the same address listed on the last page of the survey 46

47 Mail Protocol (cont'd) Data receipt and processing Track by date of receipt Key-entry or scanning technology If beneficiary returns more than one completed questionnaire, use the first completed questionnaire received Store returned paper surveys or scanned images of paper surveys o Secure and environmentally controlled location o 3 years Ambiguous responses o Decision rules 47

48 Mail Protocol (cont'd) Decision rules for data capture If a mark falls between two choices and is obviously closer to one choice than another, select the choice to which the mark is closest Example 1 (Mail) Never x Sometimes Usually Always Code as: Sometimes 48

49 Mail Protocol (cont'd) Decision rules for data capture If a mark falls equidistant between two choices, code the value of the item as M Missing Do not impute Example 2 (Mail) Never x Sometimes Usually Always Code as: M Missing 49

50 Mail Protocol (cont'd) Decision rules for data capture If a value is missing, code it as M Missing o Do not impute Note: Dependent questions appropriately skipped should be coded as 88 Not Applicable Example 3 (Mail) Never Sometimes Usually Always Code as: M Missing 50

51 Mail Protocol (cont'd) Decision rules for data capture When more than one response choice is marked, code the value as M Missing o Do not impute Exception: Several questions that have instructions to mark one or more may have multiple responses Example 4 (Mail) x Never x Sometimes Usually Always Code as: M Missing 51

52 Mail Protocol (cont'd) Decision rules for data capture When more than one response choice is marked and the beneficiary s intent is obvious, select the obvious response option Example 4 (Mail) x Never x Sometimes Usually Always Code as: Never 52

53 Mail Protocol (cont'd) The quality control guidelines contained in the QAG require: Performing address validation and updates Conducting interval checking of printed mail pieces Conducting seeded mailings Vendors are encouraged to perform additional quality control checks 53

54 Phone Protocol Same survey version (ACO-9 or ACO-12) used for mail must be used for phone interviews Phone interviews must not be conducted via inbound calls until after the phone component of survey administration begins CATI: Program with official phone script Program skip pattern questions appropriately Link electronically to survey management system Automated dialing may be used Interviewer records respondent answers electronically 54

55 Phone Protocol (cont'd) Eligible beneficiaries Did not respond to mail surveys Returned a blank or incomplete mail survey that does not fulfill the rules defining a completed or partially completed survey No valid address available after reasonable attempts to obtain one 55

56 Phone Protocol (cont'd) Where possible, CMS will provide phone numbers for beneficiaries as part of the sample file In addition, vendors must attempt to obtain phone numbers for beneficiaries using one of the following: Directly from the ACO o Via a file that contains all beneficiaries o Sample must not be shared with the ACO o Share no information with the ACO that might identify a beneficiary Commercial software Internet directories Directory assistance Other tested methods 56

57 Phone Protocol (cont'd) Survey vendors must attempt to reach every beneficiary identified for phone follow-up until the beneficiary is contacted, found ineligible or six attempts have been made No further attempts are to be made after attempting to reach the beneficiary by phone six times An attempt is defined as: Phone rings six times with no answer Beneficiary requests call back Phone answered by someone other than beneficiary who is unavailable Busy signal for each of three consecutive attempts (made approximately at 20 minute intervals, if possible) Answering machine/privacy manager reached Disconnect/out of service 57

58 Phone Protocol (cont'd) Phone scripts Standardized phone scripts provided by CMS for both survey versions Text must not be modified Scripts must be read verbatim o All questions and response choices must be read exactly as they appear in the CMS provided CATI script o Text that is underlined, bolded, highlighted, in uppercase lettering, or italicized must be emphasized 58

59 Phone Protocol (cont'd) Phone interviewer training: Phone scripts and CATI programs Guidelines for reaching beneficiaries Survey introduction Identification of possible ineligible beneficiaries Definition of phone attempts Interviewing guidelines and conventions o System conventions, e.g. CATI screens, interim disposition codes o Avoiding refusals o Probing for complete answers Customer support FAQs 59

60 Phone Protocol (cont'd) Monitoring and oversight 10% of all interviews through silent monitoring o Attempts and completed interviews o All interviewers o All times of day o Different days of the week Interviewers who consistently fail to follow the phone script verbatim, fail to employ proper probes, fail to remain neutral and courteous, have difficulty understanding or using the computer, must be identified and retrained or replaced, if necessary Monitor subcontractors, if applicable 60

61 Languages Both CAHPS for ACOs Survey versions will be administered in English and additional languages from the list below Spanish Cantonese Korean Mandarin Portuguese Russian Vietnamese ACOs in Puerto Rico must administer the survey in Spanish and offer English if requested 61

62 Languages (cont'd) Three options for implementing data collection in a language other than English 1. ACO provides language preference for the entire Original Medicare population to survey vendor to support language-specific survey mailings 2. Dual language survey mailings 3. Include an insert that contains instructions for the beneficiary to request a survey in the optional language Vendors must be prepared to conduct phone surveys in the same language offered for the mail survey Use of optional languages is at the request of the ACO 62

63 Data Specifications and Coding

64 Objectives File Encryption File Specifications Decision Rules and Coding Guidelines Mail Surveys Survey Disposition Codes Survey Completion Guidelines 64

65 File Encryption Data files must be encrypted prior to data submission Survey vendors required to use PGP Use Public Key encryption Data files submitted by survey vendors that are not encrypted will be rejected and must be resubmitted 65

66 File Specifications Survey vendors must use flat ASCII file format to submit survey data files Survey vendors will submit all ACO s sampled beneficiary records in one file, regardless of which survey version was administered No substitutions for valid data elements are acceptable 66

67 File Specifications (cont'd) Survey data will contain one record for each sampled beneficiary Each record will consist of two parts: Survey Status Section (found in Appendix J) Beneficiary Survey Data Section 67

68 Part 1: Survey Status Section Survey File Record Layout - Survey Status Section: RAND Field Name Field Contents Starting Position in Record Field Length Valid Codes FINDER Unique Beneficiary Finder Number Assigned by CAHPS for ACOs Survey Data Coordination Team ACO_ID Five character ACO identifier: begins with a letter (A or P), followed by 4 numbers DISPOSITN Final Disposition Code , 31, 11, 20, 22, 24, 32, 33, 34, 35, Numeric From sample file 9 5 [A, P]nnnn From sample file Coding Notes 10 = Completed survey 31 = Partially completed survey 11 = Institutionalized 20 = Deceased 22 = Language barrier 24 = Mentally or physically unable to respond 32 = Refusal 33 = Non-response when there is no indication of bad address or telephone number 34 = Blank survey or Incomplete survey returned 35 = Bad address and/or bad telephone number 40 = Excluded from survey 68

69 Part 1: Survey Status Section (cont'd) Survey File Record Layout - Survey Status Section (cont d): RAND Field Name Starting Position in Record Field Length Field Contents Coding Notes MODE Survey Completion Mode , 8 1 = Mail; 2 = Inbound CATI; 3 = Outbound CATI; 8 = Not applicable DISPO_LANG Survey Language Language survey was completed in or attempted to be administered: 1 = English; 2 = Spanish; 3 = Cantonese; 4 = Korean; 5 = Mandarin; 6 = Russian; 7= Vietnamese; 8 = Not applicable; 9 = Portuguese Valid Codes RECEIVED Date survey was received or completed: YYYYMMDD 18 8 yyyymmdd Date survey was received: YYYYMMDD, = Not applicable FOCALTYPE Provider type: 1= Primary care, 2 = Specialist 26 1 Numeric From sample file PRTITLE Type of provider (physician, physician assistant, Text From sample file nurse practitioner, certified nurse midwife, certified clinical nurse specialist) PRFNAME Provider first name Text From sample file PRLNAME Provider last name Text From sample file VERSION Which version of the survey was administered: 09 = ACO-9; 12 = ACO Char Version: 09 = ACO-9; 12 = ACO-12; 88 = beneficiary excluded from sample. To be provided by vendor to indicate which version of the survey, ACO-9 or ACO-12, was fielded for the beneficiary. 69

70 Survey Status Field Requirements Each field in the Survey Status Section requires an entry for a valid submission Use code 8 Not Applicable if appropriate Survey Language for a blank mail survey Survey Completion Mode for a mail survey that was not returned AND no phone number was obtained 70

71 Part 2: Beneficiary Survey Data Section Beneficiary Survey Data Section Contains survey responses from every beneficiary who has a final disposition of Completed (10), Partially completed (31) or Blank or Incomplete survey returned (34) o Leave survey data section blank for all other dispositions 71

72 Part 2: Beneficiary Survey Data Section (cont'd) Beneficiary Survey Data Section (cont d) For survey records included, all response fields must have a valid value Valid values can include: o 88 - Not Applicable o 98 - Don t Know o 99 - Refused o M - Missing 72

73 Decision Rules and Coding Guidelines Mail Surveys Except for questions where a respondent can select more than one response option, such as the race or proxy question (if applicable), when more than one response option is marked, code as M Missing 73

74 Decision Rules and Coding Guidelines Mail Surveys (cont'd) Decision rules for screener and dependent questions Some items can and should be skipped by certain beneficiaries Dependent questions that are appropriately skipped should be coded as 88 Not Applicable Screener questions that are left blank should be coded as M Missing 74

75 Decision Rules and Coding Guidelines Mail Surveys (cont'd) Decision rules for screener and dependent questions If respondent made an error in the skip pattern, survey vendors must not clean or correct skip pattern errors o Enter the value provided by beneficiary Do not impute a response based on beneficiary s answers to dependent questions An error in the skip pattern will occur if a respondent left a screener question missing and then skipped subsequent dependent questions 75

76 Decision Rules and Coding Guidelines Mail Surveys (cont'd) Dependent questions answered in violation of skip patterns are not counted toward the number of applicable to all (ATA) or SSM items to determine a complete or partially complete A screener question left blank does not trigger a skip, so subsequent responses to dependent questions should be included in count of answered survey items 76

77 Decision Rules and Coding Guidelines Mail Surveys (cont'd) Decision rules for beneficiary survey data section Enter all survey responses provided by the beneficiary for each survey item For Completed (10), Partially completed (31) or Blank or Incomplete survey returned (34) surveys, code missing answers as M Missing or 88 Not Applicable o All survey questions must have a valid code 77

78 Survey Disposition Codes Survey disposition codes are used to track and report whether a beneficiary has completed a questionnaire or requires follow-up Survey vendors are required to assign and maintain up-to-date survey disposition codes for each beneficiary in the sample 78

79 Survey Disposition Codes (cont'd) Vendor s interim disposition codes are for internal purposes only and should not be reported Only final disposition codes are reported Submitted data files must contain a final disposition code for each beneficiary in the file 79

80 Survey Disposition Codes (cont'd) Final Disposition Code Description Criteria Completed survey 10 A completed survey includes a response to at least one question in the 8 SSMs and >50% of the ATA items Partially completed survey 31 A partially completed survey includes a response to at least one question from the 8 SSMs and <50% of the ATA items A completed survey includes a response for at least one question from the 8 SSMs and greater than or equal to 50% of the ATA items. Appropriately skipped questions don t count against the required 50 percent. There must be no evidence that the beneficiary is ineligible. A partially completed survey includes a response to at least one question from the 8 SSMs and less than 50% of the ATA items. There must be no evidence that the beneficiary is ineligible. Institutionalized 11 Institutionalized Institutionalized or residing in a group home or institution (hospice, nursing home, etc.). Deceased 20 Deceased Deceased at the time of survey administration. Language barrier 22 Unable to complete the survey in English and any offered optional language Unable to complete the survey in English and any offered optional language. 80

81 Survey Disposition Codes (cont'd) Final Disposition Code Description Criteria Mentally or physically unable to respond 24 Mentally or physically unable to respond to either mail or phone portion of the survey Mentally or physically unable to respond either to mail or phone portion of the survey. Refusal 32 Refused to complete the survey Refused to complete the survey. Non-response 33 No response collected No response collected either by mail or by phone when there is no indication of bad address or bad phone number. Blank or Incomplete survey returned Bad address and/or Bad phone number 34 Responded by mail or initiated CATI interview, no answers to any question from the 8 SSMs 35 Unable to obtain a viable address or phone number for the beneficiary Excluded from survey 40 Was excluded from all survey processes Responded by mail or CATI, with no answers to any question from the 8 SSMs. There must be no evidence that the beneficiary is ineligible. Unable to obtain a viable address and/or phone number. Beneficiary was determined to be ineligible after sample selection but before data collection was initiated (see sampling section of manual). 81

82 Survey Data Section Required Following codes require submission of the survey data section: Completed survey (Code 10) Partially completed survey (Code 31) Blank or Incomplete survey returned (Code 34) 82

83 Survey Data Section Required (cont d) Completed survey (Code 10): A response to at least one question from the 8 SSMs A response to at least 50% of the ATA items Questions answered in violation of a skip pattern do not count as a response 83

84 Survey Data Section Required (cont d) Partially completed survey (Code 31): A response to at least one question from the 8 SSMs A response to fewer than 50% of the ATA items Questions answered in violation of a skip pattern do not count as a response 84

85 Survey Data Section Required (cont d) Blank or Incomplete survey returned (Code 34): No responses to any question from the 8 SSMs Questions answered in violation of a skip pattern do not count as a response 85

86 Survey Data Section Not Required Final survey disposition codes that do not require submission of survey data section DISPOSITION CODE Non-response: No response collected 33 Bad address and/or Bad phone number: Unable to obtain a viable address and/or phone number 35 Excluded from survey: Beneficiary excluded from survey processes 40 Institutionalized 11 Deceased 20 Language Barrier: Unable to complete survey in English and any offered translation 22 Mentally or physically unable to respond 24 Refusal: Refused to complete survey 32 86

87 Break

88 Data Preparation and Submission

89 Objectives Data Warehouse Access Data Submission Processes Data Submission Deadlines Data File Submission File Encryption Overview of ACO data warehouse Guide to Data Submission Process Data Auditing and Validation Checks Data Submission Notification 89

90 Data Warehouse Access Survey vendors must: Submit a Vendor Access to ACO Data Warehouse Form (Appendix D) by 9/8/2015 Designate a Data Administrator, Back-up Data Administrator and a Project Manager 90

91 Data Submission Processes CAHPS for ACOs Survey Project Team has developed a secure ACO data warehouse using a web-based application hosted by the RAND Corporation Operates as a secure file transfer system that survey vendors will use to pick up sample files and submit survey data files Does not require special software or licensing fees for survey vendors with the exception of PGP 91

92 Data Submission Deadlines Interim survey data file must be submitted by survey vendors no later than 12/11/2015 at 11:59 p.m. Eastern Time Survey vendors must submit files early to allow them enough time to resubmit if they have to and still meet the deadline If vendors submit more than once, files must include all records in the re-submission Final survey data file must be submitted by survey vendors by 2/12/2016 at 11:59 p.m. Eastern Time It is the responsibility of the survey vendor to ensure that data are submitted on time 92

93 Data File Submission Survey vendors must use the following file naming convention: Vendorname.submissionN.mmddyy.txt.pgp N = number of the submission sent in that day mm = number of month of submission (justify leading zero) dd = day of the month of submission (justify leading zero) yy = 2 digit year of submission Example: XYZResearch.submission txt.pgp 93

94 File Encryption Data files from survey vendors must be encrypted using PGP software prior to submitting them to ACO data warehouse All prior versions of PGP acceptable If necessary, install the latest version Symantec File Share Encryption 94

95 File Encryption (cont'd) Survey vendors must create a Public Key that the CAHPS for ACOs Survey Data Coordination Team will use to encrypt vendor sample files CAHPS for ACOs Survey Data Coordination Team will provide the survey vendors with a Public Key to receive survey data files Public Keys will be exchanged using the vendor s folder in the data warehouse 95

96 Overview of ACO Data Warehouse Available via the Internet Hosted on RAND Corporation s website Survey vendor s folder will contain controls for submitting survey data files as well as for downloading sample file and/or other project documentation 96

97 Overview of ACO Data Warehouse (cont'd) Submitted data files that are not encrypted (don t have.pgp extension) or don t otherwise comply with the established naming standards are deleted After each data submission, survey vendor staff receive letting them know that the file was/was not successfully submitted 97

98 Guide to Data Submission Process Once you have completed the Vendor Access to ACO Data Warehouse Form, you will receive an from a member of the RAND Corporation Data Team with an invitation to the ACO data warehouse Step-1: Click on the CAHPS for ACOs Secure File Sharing link. You will be directed to the data warehouse login page. 98

99 Guide to Data Submission Process (cont'd) Step-2: Enter your address and temporary password from your invitation mail Step-3: Click the Login button 99

100 Guide to Data Submission Process (cont'd) Step-4: The first time you login, you will be prompted to choose a new password 100

101 Guide to Data Submission Process (cont'd) Step-5: Re-enter your temporary password in the Verify Current Password box Enter your new password in both the Change Password To and Retype New Password boxes Click Update Password. You will see the confirmation screen. 101

102 Guide to Data Submission Process (cont'd) Step-6: Click OK, you will be transferred to the warehouse (File Manager tab) from where you can access your secure folder 102

103 Guide to Data Submission Process (cont'd) Step-7: Click your folder name to open the folder and enable action buttons 103

104 Guide to Data Submission Process (cont'd) Step-8: To send a file to your workspace folder within the ACO data warehouse, click the Add File button to start the Add Files dialog 104

105 Guide to Data Submission Process (cont'd) Step-9: Click Choose File, navigate to the folder on your local system where your file is located Step-10: Select the file then click Open 105

106 Guide to Data Submission Process (cont'd) Step-11: The file name will appear in the Add Files window Step-12: Click Add to submit the file to the Secure Workspace of the ACO data warehouse. To remove the file without submitting, click the red X. 106

107 Guide to Data Submission Process (cont'd) Step-13: During the secure transfer you will see a progress bar 107

108 Guide to Data Submission Process (cont'd) Step-14: When the upload is complete, the file will be listed in your secure workspace folder 108

109 Data Auditing and Validation Checks CAHPS for ACOs Survey Data Coordination Team will audit data files as they are submitted for compliance with file layout specifications File audit includes: Checking for.pgp file extension Logical record lengths, appropriate character set, naming conventions Presence of required data fields Range checks Verification of coding of Survey Disposition 109

110 Data Submission Notification Survey vendor will receive two notifications for each data submission First (automated) , from contains notification that file was received Second , from the CAHPS for ACOs Survey Data Coordination Team, is sent after audit checks Indicates if file successfully passed checks Will go out no later than 8:00 p.m. Eastern Time one business day after submission 110

111 Data Submission Notification (cont'd) If file fails any audit checks, will: Instruct survey vendors that they must submit data files again Contain full detail of the audit check report including a list of involved records If file passes checks, will say that no further action is necessary, and provide a summary of file contents for verification by the vendor 111

112 Technical Support Contact the CAHPS for ACOs Survey Data Coordination Team for technical support and/or assistance related to data submission at: 112

113 Data Analysis and Public Reporting

114 Objectives Overview of SSMs Data Transformation Benchmarks and Scoring Data Analyses Conducted by CMS Public Reporting of ACO Data How Data are Reported to ACOs Data Analyses Conducted by Survey Vendor 114

115 What is an SSM? SSMs are a roll-up of multiple questions on a similar aspect of experience (e.g. care coordination) Individual questions are scored on an ordinal scale, the range of the scale varies SSM scores are the average of case-mix adjusted scores on their contributing questions The score is converted to a scale using a linear transformation 115

116 Transformation to scale For reporting, individual questions and SSMs are converted to a scale For all scores, 0 is the lowest and 100 is the best performance When items within a measure use a different scoring scale*, the conversion occurs before the SSM is calculated Rescaling does not change ACO performance rankings (i.e. the best performers remain the best, the poorest performers remain the poorest) * Health Status and Functional Status and Helping You to Take Medications as Directed 116

117 Formula for Transformation Let X = the CAHPS score on its original scale Let a = the minimum possible score on the original scale Let b = the maximum possible score on the original scale The score is calculated as: Y = (X a) x 100 (b a) 117

118 Example: Provider Rating Provider rating is a singleitem SSM The original scale for this measure is 0 to 10 Item Question Q. Using any number from 0 to 10, where 0 is the worst provider possible and 10 is the best provider possible, what number would you use to rate this provider? 118

119 Example: Provider Rating (cont d) The table below shows case-mix adjusted means for several hypothetical ACOs and the converted score for each mean ACO Original Mean Score Calculation of Score Converted Score ACO A 7.5 (7.5 0)/(10 0) x ACO B 8.0 (8.0 0)/(10 0) x ACO C 9.0 (9.0 0)/(10 0) x

120 Example: Access to Specialists Access to Specialists is composed of two survey items Each item has a 1 to 4 ordinal scale: 1 = Never 2 = Sometimes 3 = Usually 4 = Always Item Questions Q1. In the last 6 months, how often was it easy to get appointments with specialists? Q2. In the last 6 months, how often did the specialist you saw most seem to know the important information about your medical history? 120

121 Example: Access to Specialists (cont d) The table below shows case-mix adjusted means for several hypothetical ACOs and the converted score for each mean ACO Original Mean Score Q1 Q2 Average of Adjusted Scores Calculation of Score Converted Score ACO A (1.5 1)/(4 1) x ACO B (3.5 1)/(4 1) x ACO C (2.5 1)/(4 1) x

122 Point Value Assignment for Scored Measures 8 SSMs contribute to the final CAHPS quality score Each measure is worth a maximum of 2 points All ACOs receive an automatic 2 points for two pay for reporting measures Health and Functional Status Stewardship of Patient Resources The point value for the remaining measures is determined by comparing the scores against a set of established benchmarks 122

123 Benchmarks Measure benchmarks are established by CMS prior to the reporting period to which they are applied The benchmarks correspond to either percentiles of the score distribution or flat percentage points The groups corresponding to the flat percentage points are: 0-30, 31-40,, The percentile values are derived from surveys of the Medicare FFS population and do not include the current survey results 123

124 Example: Health Promotion and Education As an illustration, we consider the actual RP14 benchmarks for Health Promotion and Education* Based on these benchmarks, an ACO with a score of 70 for this measure would receive the full 2 points, while an ACO with a score of 56 would receive 1.25 points * Payment/sharedsavingsprogram/Downloads/MSSP-QM- Benchmarks.pdf Score Points Earned or less 0 124

125 Scoring ACO-9 Survey Data ACO-9 Survey Data 6 SSMs Score Getting Timely Care, Appointments, and Information 0-2 Points How Well Your Providers Communicate 0-2 Points Patient s Rating of Provider 0-2 Points Access to Specialists 0-2 Points Health Promotion and Education 0-2 Points Shared Decision Making 0-2 Points 2 SSMs Score Health Status and Functional Status 2 Points Stewardship of Patient Resources 2 Points 1 SSM Score Courteous and Helpful Office Staff Not Scored Total Possible Score for 8 SSMs 4-16 Possible Points 125

126 Scoring ACO-12 Survey Data All ACO-9 Survey Data Plus 3 SSMs Score Care Coordination Not Scored Between Visit Communications Not Scored Helping You to Take Medications as Directed Not Scored 126

127 Data Analyses Conducted by CMS In addition to scoring and benchmarking, CMS conducts the following analyses Reliability Reliability is assessed for each question and SSM Reliability summarizes the extent to which variation in patient experience responses reflect true differences between organizations Case-Mix Adjustment and Weighting Ensures a level playing field and that data represent ACOs fairly 127

128 Public Reporting of ACO Data Beginning with 2015 quality reporting period, CAHPS for ACOs Survey data for 7 scored SSMs will be reported on the Physician Compare website ( The Health Status and Functional Status SSM will not be reported on Physician Compare However, SSM scores with low or very low reliability (<0.75) are suppressed from reporting Regardless of reliability, scores for all SSMs for all ACOs are reported here: Accountable-Care-O/yuq5-65xt 128

129 How Data are Reported to ACOs Every ACO receives a detailed report SSMs o Reported as case-mix adjusted mean scores o SSM scores with low ( ) or very low (<0.60) reliability are reported and flagged in individual reports to ACOs Multi-question measures Individual questions Response rate information Shows data for each ACO as well as: Selected data for all ACOs Original Medicare data for the geographic area in which the ACO operates 129

130 Data Analyses Conducted by Survey Vendor Survey Vendors may conduct own analyses of data for quality improvement purposes, however, per your CMS DUA: Cell sizes must not be less than 11 o No information based on fewer than 11 sampled members can be released, meaning no cell sizes under 11 can be displayed in any cross tabulations, frequency distributions, tables, Excel files, or other reporting mechanisms o No number smaller than 11 should appear in any material provided to your client For example, if a certain response option is chosen fewer than 11 times, data for that response option must not be displayed, even if 11 or more responses were received for the corresponding question as a whole Intervention or follow-up with low or high scoring individuals is not permitted Survey vendors cannot provide individual-level data to ACOs 130

131 Data Analyses Conducted by Survey Vendor (cont'd) CMS-calculated results are official results Vendors will not have sufficient information to replicate CMS analyses All reports provided to the ACOs must include a statement that vendor results are unofficial and are for ACOs internal quality improvement purposes only CMS-calculated results include data from completed and partially completed surveys 131

132 Vendor Oversight

133 Objectives Oversight Activities Non-compliance and Sanctions Discrepancy Reports 133

134 Oversight Activities Ensure: Compliance with CAHPS for ACOs Survey protocols Survey data collected and submitted are complete, valid and timely Standardization and transparency of CAHPS for ACOs Survey results Data security 134

135 Oversight Activities (cont'd) Oversight activities include: Review of Quality Assurance Plan (QAP) and survey materials Conducting site visits and conference calls Analysis of submitted data 135

136 Oversight Activities (cont'd) QAP Documents understanding, application and compliance with survey protocols Follows the Model QAP specifications Provides a guide for the site visit Vendor submits each year Submit via the CAHPS for ACOs Survey Technical Assistance acocahps@hcqis.org Due date of 9/11/

137 Oversight Activities (cont'd) Review of survey materials Review for compliance with CAHPS for ACOs Survey protocols and guidelines Submitted each year of survey administration Only survey vendors with a contracted ACO client(s) need to submit survey materials Submit via the CAHPS for ACOs Survey Technical Assistance acocahps@hcqis.org English mail survey materials due 9/25/2015 English CATI screenshots due 10/23/

138 Oversight Activities (cont'd) Site visits and conference calls Review and observe systems, procedures, facilities, and resources Discussions with project staff o Including subcontractors, if applicable All materials related to survey administration are subject to review Feedback report includes action items for follow-up Conference calls as needed 138

139 Oversight Activities (cont'd) Analysis of submitted data Intended to detect errors in data submission Includes review of outliers, anomalies, unusual patterns, etc. Follow-up as appropriate 139

140 Non-compliance and Sanctions If survey vendors fail to adhere to CAHPS for ACOs Survey protocols, including missing deadlines/due dates, they will be required to develop and implement corrective actions If survey vendors do not fix problems, they may lose approved status for conducting the CAHPS for ACOs Survey Other sanctions may also be applied such as increased oversight or a special monitoring plan 140

141 Discrepancy Reports Report any variations from CAHPS for ACOs Survey protocols during survey administration Complete and submit online report within 1 business day at acocahps.cms.gov A second updated report may be submitted once root cause, scope of issue and/or corrective action has been identified Vendors must not wait until the discrepancy has been resolved to submit an initial Discrepancy Report 141