EMERGENCY DEPARTMENT SURVEY PSYCHOMETRICS

Transcription

1 EMERGENCY DEPARTMENT SURVEY PSYCHOMETRICS VALIDATION PAPER One of the hallmarks of Press Ganey s surveys is their scientific basis: our products incorporate the best characteristics of survey design. Surveys are developed by conducting focus groups of patients, providers and/or administrators, reviewing surveys from health care facilities across the country, reviewing current professional and scientific publications on health care delivery, and utilising the latest research on survey statistics and design. Testing the Questionnaire Effective questionnaires have three important attributes: focus, brevity, and clarity. Questions should focus directly on a specific issue or topic, be as brief as possible while still conveying the intended meaning, and be expressed as simply and as clearly as possible. When we were confident that the set of questions developed for the Emergency Department questionnaire met these criteria and that their face, content, and consensus validities had been established, we tested the questionnaire with the assistance of 69 test sites. Patient satisfaction questionnaires were mailed to patients within 3-5 days of their visit to the emergency department. A mail-out methodology was chosen over hand distribution to eliminate selection and acquiescence biases. A total of 12,749 surveys were mailed to patients (405 surveys were returned as undeliverable). The test concluded with the receipt of 5,003 questionnaires, resulting in an average response rate of 40% (range = 20% - 69%). Perceptions of emergency department care and service were measured by asking patients to rate 39 specific aspects of their visit. The 39 test items were divided into eight sections, designed to index care dimensions of the patient s experience during an emergency department visit: Arrival, Nurses, Doctors, Tests, Family or Friends, Personal Information, Personal Issues, Overall Assessment. As with other Press Ganey questionnaires, a Likert-type response scale was used with the following categories: very poor, poor, fair, good, and very good. Questionnaire Psychometrics The accuracy of a questionnaire is assessed by measuring its validity and reliability. Validity is the degree to which a questionnaire measures what it was designed to measure. Reliability is the degree to which survey data are consistent and reproducible across respondents or across surveys. The Emergency Department instrument was found to be psychometrically sound across a wide variety of tests of reliability and validity as described below. Emergency Department Survey 2010 Press Ganey Associates Pty Ltd Page 1

2 Response Patterns and Variance. Measures of central tendency (i.e., mean, median, and mode) and variability (standard deviation, standard error) were examined for all questions. Response frequencies were evaluated for patterns of missing data and question-wording or ambiguity problems. These steps are important for ensuring the clarity of questions and the absence of instrument bias and error. Measure Redundancy. As discussed above, it also is important for a survey instrument to be concise. A correlation matrix of all test questions was examined to find questions that were too highly associated with one another. If two questions are highly intercorrelated, it suggests that they may be measuring the same issue or concept and are, therefore, redundant. Removing one of the two questions removes this redundancy and leaves a more parsimonious scale or questionnaire. Several items were found to be highly correlated with other items in their respective sections, and were, therefore, dropped as standard questions: Degree to which the doctor took the time to listen to you Degree to which you received satisfactory answers to your questions from the doctor who treated you Friendliness/Courtesy of the person who took your blood Friendliness/Courtesy of the radiology staff Likelihood of returning to our Emergency Department if you had a choice. Construct Validity. Factor analysis is a technique used to identify factors that statistically explain the variation among responses to a questionnaire. In other words, factor analysis helps to identify which questions belong together, confirming a questionnaire s construct validity or structure. A factor analysis was completed on the test data for the 31 questions. Because it was likely that factors would not be independent of one another (i.e., a patient s response to one question may be correlated with his or her response to a question in another section), a principle component extraction was performed with Promax oblique rotation. The 3 overall assessment questions were omitted from the factor analysis due to their (predicted) high intercorrelation with other items. Factor analysis is used to divide a multi-question survey into meaningful subscales. Questions that are highly correlated with one another typically represent a common dimension or concept. For example, Nurses attention to your needs and Nurses concern for your privacy are more likely to define a Nurses dimension than Cleanliness of the treatment area or Waiting time for radiology test. The factor analysis did not produce a clean factor structure until Ease in finding the Emergency Department and Overall rating of the parking were dropped as standard questions. A principal component factor analysis with Promax oblique rotation was re-run using the remaining 29 questions. The factor analysis identified seven dimensions of care and service in addition to patients overall assessment of their visit. Factor loadings in Table 1 represent the correlations between factors and variables uncontaminated by factor overlap. Table 2 shows the correlations between the seven factors. Together the seven factors accounted for 72% of the variance in patient responses. Emergency Department Survey 2010 Press Ganey Associates Pty Ltd Page 2

3 Table 1. Item Content and Primary Factor Loadings Factors and Loadings Arrival Nurses Doctors Tests Family/Friends Personal Info Personal Issues Questionnaire Item Helpfulness of the nurse who first asked you about your condition Comfort of the waiting area Waiting time before you were brought to the treatment area Waiting time in the treatment area, before you were seen by a doctor.706 Cleanliness of the treatment area.584 Friendliness/Courtesy of the nurses Degree to which the nurses took the time to listen to you Nurses attention to your needs Nurses concern to keep you informed about your treatment Nurses concern for your privacy Friendliness/Courtesy of the doctor Doctor s concern to keep you informed about your treatment Doctor s concern for your comfort while treating you Degree to which doctor explained your diagnosis Thoroughness of care received from the doctor Concern shown for your comfort when your blood was drawn Waiting time for radiology test Concern shown for your comfort during your radiology test Friendliness/Courtesy with which family or friends were treated Staff concern to keep family or friends informed about your status during your course of treatment Staff concern to let a family member or friend be with you while you were being treated Friendliness/Courtesy of the person who took your personal information Privacy you felt when asked about your personal information Ease of giving your personal information How well you were kept informed about delays Extent to which staff cared about you as a person How well your pain was controlled Extent to which you have a better understanding of your medical problem than when you came Information you or your family/friends were given about caring for yourself at home (e.g., taking medications, follow-up medical care) Emergency Department Survey 2010 Press Ganey Associates Pty Ltd Page 3

4 Table 2. Factor Correlation Matrix Scale Arrival Nurses Doctors Tests Family Info Issues Overall Arrival 1 Nurses Doctors Tests Family Personal Information Personal Issues Overall Assessment All correlations are significant at the.01 level (2-tailed). Convergent and Discriminant Validities. Although factor analysis is a popular method of establishing the construct validity of a questionnaire, other methods are available. For example, one of the assumptions of questionnaire and scale construction is that an individual item from a scale should be well correlated with the other items in that scale. Researchers suggest a minimum item-to-scale correlation of.30. Table 3 shows the average and range of correlations between each question and its parent section. These correlations are corrected in the sense that the item of interest is omitted from its section score when correlations between the two are calculated. Corrections are performed to avoid inflated or spuriously positive correlations. An item should be correlated with its own scale (convergent validity) as well as correlated more with its own scale than with other scales (discriminant validity). That is, item-to-scale correlations should be higher than item-to-non-scale correlations. For example, a Doctors question should be more highly correlated to its section than to the Nurses section. As Table 3 confirms, the Emergency Department questionnaire demonstrates both convergent validity and discriminant validity. Because these are subclasses of construct validity, the questionnaire s effectiveness at measuring perceptions of emergency department care is confirmed across multiple tests. Table 3. Item Analyses and Reliability Estimates Scale Alpha Average Corrected Item-Scale Range of Corrected Item-Scale Average Item-Non- Scale Range of Item-Non- Scale Arrival Nurses Doctors Tests Family or Friends Personal Information Personal Issues Overall Assessment Emergency Department Survey 2010 Press Ganey Associates Pty Ltd Page 4

5 Criterion, or Predictive, Validity. Predictive validity is defined as the ability of an instrument to predict outcomes that theoretically should be tied to the construct measured by the instrument. In consumer satisfaction, the perception of a satisfying experience is expected to be linked to two types of behavioural outcomes in an individual: re-purchase behavior (coming back at a future date to purchase another product or service) and positive word of mouth (recommending a product to family and friends). Within the health care industry, limitations on patients choice of providers and health services make it difficult to assess the relationship between satisfaction and re-purchase behavior. Patients and families may be very satisfied but not return to a provider or institution because their insurance plans either don t cover the service or require that they receive the medical care from a different provider. An estimate of the relationship between satisfaction and positive word of mouth can be obtained by asking patients about their intent to recommend a provider or service. The predictive validity of a satisfaction instrument then can be estimated by the degree to which individual items on the instrument predict the patient s intentions to recommend. In this regard the questionnaire shows high levels of predictive validity. A series of simple regression analyses revealed that each of the individual items is a significant predictor (at the.001 level) of patients reported likelihood to recommend the emergency department (beta s ranged between.25 and.77). Additionally, multiple regression analysis revealed that, collectively, all items are significant predictors of patients reported likelihood to recommend the emergency department, F(31, 4971) = , p <.001, R 2 =.65 (adjusted R 2 =.65). In other words, the instrument explains approximately 65% of the variance in patients likelihood to recommend the emergency department. According to Carey, 1 A good survey will account for 50% or more of the variance in global evaluations. The Emergency Department Survey has surpassed this high standard. Reliability. Reliability testing is a method of evaluating the internal consistency of a questionnaire. The traditional statistic used to illustrate the degree of consistency among the items of a scale or questionnaire is Cronbach s alpha. A set of questions with no internal consistency no reliability has an alpha of 0.0 indicating that the questions within the scale may not be measuring the same issues. A set of questions with perfect internal consistency has an alpha value of 1.0. All eight of the subscales meet or exceed the stringent.7 standard for reliable measures: Reliability estimates range from.70 to.94 (see Table 3). The Cronbach alpha for the entire questionnaire is.97, confirming the instrument s high internal consistency and reliability. Readability. According to the Flesch-Kincaid Index, which is based on the average number of syllables per word and words per question, the final questionnaire tests at a 7 th grade reading level. 1 Carey, R. G. (1999). How to choose a patient survey system. The Joint Commission Journal of Quality Improvement, 25, Emergency Department Survey 2010 Press Ganey Associates Pty Ltd Page 5

6 Descriptive Statistics. The means and standard deviations of all survey questions are listed in Table 4. The means and standard deviations of the subscales are listed in Table 5. The overall database mean is (SD=16.74). Table 4. Question Means and Standard Deviations Mean SD ARRIVAL Helpfulness of the nurse who first asked you about your condition Comfort of the waiting area Waiting time before you were brought to the treatment area Waiting time in the treatment area, before you were seen by a doctor Cleanliness of the treatment area NURSES Friendliness/Courtesy of the nurses Degree to which the nurses took the time to listen to you Nurses attention to your needs Nurses concern to keep you informed about your treatment Nurses concern for your privacy DOCTORS Friendliness/Courtesy of the doctor Doctor s concern to keep you informed about your treatment Doctor s concern for your comfort while treating you Degree to which the doctor explained your diagnosis Thoroughness of care received from the doctor TESTS Concern shown for your comfort when your blood was drawn Waiting time for radiology test Concern shown for your comfort during your radiology test FAMILY/FRIENDS Friendliness/Courtesy with which family or friends were treated Staff concern to keep family or friends informed about your status during your course of treatment Staff concern to let a family member or friend be with you while you were being treated PERSONAL INFORMATION Friendliness/Courtesy of the person who took your personal information Privacy you felt when asked about your personal information Ease of giving your personal information Emergency Department Survey 2010 Press Ganey Associates Pty Ltd Page 6

7 Mean SD PERSONAL ISSUES How well you were kept informed about delays Extent to which staff cared about you as a person How well your pain was controlled Extent to which you have a better understanding of your medical problem than when you came Information you or your family/friends were given about caring for yourself at home (e.g., taking medications, follow-up medical care) OVERALL ASSESSMENT Overall rating of the care received during your visit Extent to which your condition has improved at least as much as expected Likelihood of recommending our Emergency Department to others Table 5. Subscale Means and Standard Deviations Scale Mean SD Arrival Nurses Doctors Tests Family/Friends Personal Information Personal Issues Overall Assessment Emergency Department Survey 2010 Press Ganey Associates Pty Ltd Page 7