Responding to the Naysayers. news, views & ideas from the leader in healthcare satisfaction measurement

Transcription

1 news, views & ideas from the leader in healthcare satisfaction measurement Satisfaction Survey Research The Satisfaction Snapshot is a monthly electronic bulletin freely available to all those involved or interested in improving the patient/ client experience. Each month the Snapshot showcases issues and ideas which relate to improving patient satisfaction and customer service, improving workplace culture and improving the way we go about our work in the healthcare industry. The Satisfaction Snapshot features: «relevant articles from healthcare industry experts «case study success stories «tips and tools for quality improvement «patient satisfaction and other industry research findings «articles with ideas to help achieve success in your role Responding to the Naysayers Robert Wolosin, Ph.D., Research Manager Press Ganey Associates Edited by Terry Grundy, Director Press Ganey Associates Satisfaction measurement has been the fundamental driving force for every successful industry, as it provides the platform for guiding the strategic direction of a business. In health care, consumer input and feedback into the design and delivery of our health system is growing in significance. Quality professionals working in health care understand the importance of satisfaction improvement and its link to organisational efficiency, however have found it difficult to win over the naysayers who, rather than embrace and act on the perceptions, continually question the data and challenge the validity of results. In some instances (i.e., poorly developed surveys and methodologies), this can be justified. Quality professionals need the ammunition to counter the naysayers. The December 2005 satisfaction snapshot provides this ammunition, however it requires that you take an audit of your current systems to ensure that your survey processes meet appropriate standards. Just how valid are the questions on your satisfaction surveys? Do they really get to the heart of the issues and problems? Can acting on the results improve the perception of the organisation? Here are the answers behind the issues and questions you should be addressing when looking at the surveys you use. If you would like your colleagues to receive the Satisfaction Snapshot please send us their names and addresses. The Satisfaction Snapshot is published by Press Ganey Associates Pty Ltd. All material is copyright protected. Quotation is permitted with attribution. Subscribers are permitted and encouraged to distribute copies within their organisations. Subscription to the Satisfaction Snapshot is FREE! Please direct any comments, suggestions or article submissions to: Manager of Client Relations snapshot@pressganey.com.au P: F:

2 page 2 Is the data from your satisfaction survey reports valid? This is a legitimate question from anyone who is confronted with a set of numbers that is supposed to tell them something about the experiences or opinions of people who are important to them. Surveys should be designed to measure satisfaction. By definition, they are valid if they really do this. The problem is that we have no direct way of determining the correspondence between a measure of satisfaction and the true state of satisfaction. The measure is designed because no direct determination of satisfaction is available. There are however, other ways we know that satisfaction data is valid. Over the years, organisations that have used valid measurement tools and services to improve their service quality have reported dramatically increased levels of satisfaction. So here s the scenario: You are the quality improvement specialist in your organisation. You know that the key to your organisation s success is the creation of experiences that exceed expectations, and that in order to do so, you need to know how consumers currently evaluate what you are providing so that you can celebrate your successes and correct service deficiencies. You have just seen your latest survey results and are about to post them on your intranet Objection 1: The Survey Questions are not Appropriate. Survey questions must represent the topic being measured. Choosing the right content means defining what you are interested in measuring, then selecting specific areas that need to be evaluated, and finally, judging how well the questions probing the areas relate to the definitions and cover all areas. Health care providers want to know- How do our consumers react to the experiences we create for them? - so that experiences can be designed closer to the ideal. One of the important design criteria is the customercentredness of the questions. Customer-centredness means looking at the health care experience through the eyes of the consumer, and evaluating experiences as to how well they meet or fail to meet the unique wants and needs of the consumer. This is in contrast to provider- or facility-centred measures, which look at the environment through the eyes of those providing the service. An example may be: Suppose as an administrator in a nursing home facility, you want to know how well your managers handle residents complaints. You might ask if a complaint was addressed within a certain amount of time, say, within two business days of its receipt. And you might find out that the vast majority of complaints are in fact, addressed within that time span. By asking how complaints were handled favourably or poorly you will learn whether or not the complaint response mechanism actually meets individual, unique resident needs. BUT NO! Here comes the Big Naysayer, the Doubting Thomas, the person in your organisation who simply doesn t buy it as far as satisfaction data is concerned. And it looks like you are in for another lecture on wasting the organisation s precious resources. You silently scream Help! and wish you were somewhere else. The typical survey naysayer will attack the data on one (or more) of several grounds. Some may be based on experiences with previous satisfaction surveys and some on a posture of maintaining the status quo by attacking what seems to threaten it. Regardless of the motivation behind naysaying, it is important to be able to debunk the most common complaints lodged against good satisfaction data so you can get beyond them and on to the real task quality improvement. To take another context, that of employee satisfaction, you might ask about whether employees received information about the benefits program in your institution. But to be employee-centred, you need to know NOT whether your employees got information, but if they understood what they received. To develop an instrument that is used to assess service quality from the consumer s point of view, several additional content considerations come into play. First, instrument content must adequately sample all of the relevant service areas it was designed to measure. Prior to designing your questionnaires, you need to conduct research to learn how consumers define satisfaction. You will find that various consumers define satisfaction across differing but important dimensions.

3 page 3 In the context of measuring doctor satisfaction with a hospital as a place to practice medicine, for example, the domains that Press Ganey determined as contributing to satisfaction are (1) quality of care delivered in the hospital, (2) ease of practice at the hospital, and (3) relationships with the leadership of the hospital. These domains were determined through literature review and subsequent discussion with experts who agreed that they were important contributors to a doctor s sense of satisfaction as a member of medical staff. Other examples for the development of a satisfaction survey include seeking input from hospitals or agencies representing different geographic regions and serving both large and small patient bases. Structured focus groups should also be conducted with patients or clients. These processes assure that the resulting instruments represent the full range of salient experiences in the health care environment being assessed. After the relevant content domains within a setting are defined, questions that measure them must be formulated. It is here that most amateur surveymakers go astray. Writing good questions is an art! The basic attributes of good questions are focus, brevity, content, clarity, consistency, avoidance of bias and appropriate reading level. These attributes are briefly discussed below. Focus: Each question should focus on a single, specific issue or topic. When this attribute is violated, that is, when more than one concept is being rated, a respondent can legitimately answer in one way to one part of the question and in a different way to another part. Consequently, answers are ambiguous because the survey user cannot be sure which part of the question was actually rated by a respondent or if both parts were. Brevity: Beyond the minimum number of words needed to adequately describe the concept to be rated, the longer the question, the more it is open to differing interpretations since each word brings different associations for different respondents. Content: Once you have decided upon the salient characteristics of the domain of interest, make sure each questionnaire item is relevant to one of those characteristics. This means excluding questions that might be interesting to know about but are not directly relevant. For example, every item on an employee satisfaction survey should tap an issue relevant to how satisfied employees are with the workplace, not how satisfied they are with other aspects of their lives. Clarity: Word questions carefully to avoid ambiguity. It is very difficult to express ideas in a way that creates the same stimulus for every respondent since we all bring different experiences to bear when we encounter the printed word. Even such a simple idea as the quality of hospital food referring to physical things will have different meanings to someone used to gourmet meals at upscale restaurants and someone who predominantly has takeaway food! It is much harder to create a standard stimulus for a concept such as how much respect is shown by employees to the residents at a nursing home facility. Yet by keeping language as clear and concise as possible, and by checking during the validation process for too many missing responses, for respondent comments indicating confusion, or for too much response variability, the inevitable ambiguity of written questions can be kept in bounds. Consistency: Word items so as to be consistent with the response scale. If your response scale runs from very poor to very good, formulate questions to be answerable in that format, rather than requiring a Yes or No, or a statement of agreement. Avoidance of Bias: It is easy to phrase a question in such a way as to get responses that are not accurate reflections of what the consumer actually thinks. A leading question pushes respondents to answer in a way that differs from what they would have answered if the question was worded neutrally. Leading questions indicate that there is a desired answer, and most respondents don t want to portray themselves in an undesirable light. Reading Level: Surveys should be designed that are easily read and understood by respondents. The most widely used measure for assessing reading level is the Flesch-Kincaid Grade Level analysis, which is based on the average number of syllables per word and words per question. A grade level of 6-8 is desirable for patient satisfaction surveys. Good survey questions represent the domain of interest and are focused, brief, relevant, clear, consistent with the response scale, unbiased and readable. Objection 2: Inappropriate Response Scales. Most surveys, including Press Ganey s, use a Likerttype response scale throughout. The consistent use of just one type of response scale reduces the cognitive burden on the respondent; he or she does not have to focus on what type of judgment to make, just on the service issue at hand. Use of consistent response scales also means that results from different items, survey sections, and different respondents can be easily compared.

4 page 4 The statistical basis for the use of the very poor to very good scale (as used by Press Ganey) is the work done by survey research pioneer Rensis Likert in the 1930s. Likert-type questions typically use a 5- point scale in which two ends of a response continuum are balanced by a middle category. The response scale used in surveys should generate desirable variability, diminish positivity bias, and allow the computation of statistics that are useful for quality improvement purposes. Objection 3: Satisfaction Data is Unreliable! One of the hallmarks of good satisfaction surveys is their scientific basis, which incorporates the best characteristics of survey design. After a survey instrument has been created in accordance with the principles stated above, it should be field-tested and the resulting responses subjected to a standard set of statistical tests known collectively as psychometric analysis. This assesses the degree to which the instrument measures what it is supposed to measure. This type of scale has several advantages. First is the variability of scores that they generate. Without sufficient variability, a health care facility will not get data that is rich enough to guide quality improvement efforts. A 5-point scale allows respondents to make reasonable discriminations among their evaluations, which translates into useful variability. Second, the use of a balanced scale, with equal numbers of responses on the positive as on the negative side, avoids a potential bias toward positivity of response. When there are more response categories on the positive end of the scale than there are at the negative, such as with a poorfair-good-very good-excellent scale, the message is subtly communicated that service quality should be rated in one of the good categories. This can lead to inaccurate and misleading data since negative evaluations would tend to be suppressed. And third is the fact that Likert-type scales are interval scales in which the distance between adjacent response categories is identical. The use of numerical anchors (e.g., 1, 2, 3, 4, 5) in the rating scale reinforces the notion of equidistance for the respondent, and verbal labels on the scale should be present to direct respondents to the end of the continuum to use for high or for low marks. A response scale of 1-5 also allows easy conversion into a 100-point scale for reporting and to generate useful statistics such as mean scores and correlations. One of the outcomes of a psychometric analysis is a concise question set. A well-designed questionnaire should provide a full assessment of the relevant aspects of the health care environment, but should do so succinctly. This shows respect for the respondent s time and mental energy. During psychometric analysis, a correlation matrix of all questions should be examined to find questions that are essentially measuring the same underlying experience. If two questions are highly intercorrelated, they measure the same concept and are redundant. Removing one of the two questions eliminates the redundancy and results in a more concise scale. A second outcome of psychometric analysis is the determination of each survey s reliability. A measure any measure is reliable if it gives consistent, dependable results over and over again, assuming that what you are measuring isn t changing. Reliability reflects the amount of error inherent in any measurement. The more an instrument contains measurement error, the less reliable it is. Survey researchers should use an internal consistency test to assess the reliability of a survey (e.g., Cronbach s alpha). Internal consistency considers the average correlation between all of the survey questions and the number of survey questions. Cronbach s alpha is a statistic that shows the internal consistency of a survey. It ranges from 0.00 to 1.00, with scores increasing as the survey becomes more reliable.

5 page 5 Mathematically, reliability is defined as the proportion of variability in responses to the survey that is the result of differences in respondents. A reliable survey will differ because respondents have different opinions, not because the survey has multiple interpretations. A high reliability coefficient reflects the high degree to which the instrument can discriminate among patients with different levels of satisfaction. Reliability is an essential survey characteristic that is often overlooked. If the survey you use cannot prove that it has been tested for reliability, the inferences that you derive from it can be misleading. A survey cannot be valid if it is not reliable. Surveys should be tested, modified, and re-tested until excellent reliability is achieved. Objection 4: Satisfaction Data is Invalid! There are two types of survey validity internal and external. Internal validity means that a survey measures what it is intended to measure and reflects the characteristics of the instrument. External validity measures how well you can make accurate inferences about a particular population on the basis of sample results and reflects the characteristics of respondents. 1. Internal Validity Of the many aspects of internal validity, the most relevant to patient satisfaction surveys are content, construct, convergent, discriminant, and predictive validity. The process for establishing content validity was discussed previously under the first objection. Factor analysis is a technique used to infer the dimensions used by health care consumers to sort their experiences. The basic idea is that ratings along the same mental dimension should be more highly related to each other than ratings coming from different dimensions. Factor analysis helps to identify the questions that belong together because they measure the same dimension. In other words, a factor analysis is the means of dividing a multi-question scale into meaningful subscales. The factor analysis should group questions in a way that is similar to the sections of the questionnaire. This establishes a questionnaire s construct validity or structure. Thus, factor analysis should be used in the testing of a satisfaction survey to confirm the placement of questions on the instrument and the underlying dimensions or constructs being measured. Good satisfaction surveys should have empirically validated factor structures that reflect a theoretical framework for understanding the dimensions of health care consumer experiences. The testing framework and results should be documented and made available for reference. 2. Convergent and Discriminant Validity If questions within each survey section are truly measuring the same mental dimension, and each mental dimension is a distinct aspect of consumer satisfaction, then they should correlate highly with other questions in their section (convergent validity) and correlate more highly with other questions in their own section than with questions in other sections (discriminant validity). Construct Validity Construct validity is defined as the extent to which the measurement corresponds to theoretical concepts (constructs) concerning the phenomenon under study. For example, if on theoretical grounds, the phenomenon should change with age, a measurement with construct validity would reflect such a change. Construct validity thus encompasses both the measuring instrument and the theory behind it. Psychometrically sound surveys in health care are based on the theory that health care consumers, such as patients, employees, and medical staff, mentally sort their experiences along distinct dimensions, and, when asked to reflect on experiences (e.g., inpatient visits), review them along these dimensions (e.g., nursing care, doctor care). It s like family resemblance: children from a particular family should resemble each other, and should look different from children from an unrelated family. For example, in an Employee Satisfaction survey, a Senior Leadership question should be more highly correlated to other Senior Leadership questions than to questions from a Recognition section.

6 page 6 Good satisfaction surveys should demonstrate convergent and discriminant validity. Thus, their effectiveness at measuring perceptions of care can be confirmed across multiple tests of construct validity. 3. Predictive Validity Predictive validity is the relationship between satisfaction scores and other scores that theoretically should be linked to satisfaction. Customer service research indicates that satisfaction is related to two important behaviours: 1. The tendency to return to the provider for additional services when needed ( repurchase behaviour ). 2. The tendency to recommend the provider to others ( positive word of mouth ). In health care, there are limitations on consumers (such as location and/or health insurance) that interfere with a straightforward relationship between satisfaction and repurchase behaviour. However, a reasonable proxy for repurchase behaviour is the respondent s stated intent to recommend the provider to others. Therefore, evidence for predictive validity is a high correlation between measured satisfaction and intent to recommend. Good satisfaction surveys should show high levels of predictive validity, as assessed by examining the relationship of each survey question and the entire set of survey questions to intent to recommend. For example, overall satisfaction in an emergency department should predict over 60% of the variation in the patient s likelihood of recommending the emergency department. Objection 5: Improper Data Collection. Once a valid and reliable survey has been constructed, the problem of actually collecting data still exists. The format used to distribute blank surveys, collect completed surveys, and select respondents has enormous influence over the validity of the entire survey process. Method of Distribution: Mail vs In Person or Telephone A defining feature of any good satisfaction vendor is the ability to provide clean, comparative databases. In order for comparative data to be accepted as fair, data must be collected through comparable methods. Comparable collection methods insure that differences in facilities scores are due to actual differences in patient satisfaction, rather than to the way the data was collected. Mailed surveys are preferable to faceto-face or telephone surveying since they reduce the possibility of acquiescence and social desirability biases inherent in a face-to-face interview situation. Acquiescence refers to a tendency to agree with a statement regardless of its content, while social desirability refers to a respondent s desire to give answers that put him or herself in the best possible light. Social forces such as power differentials that operate in face-to-face data collection especially on the provider s turf (e.g., by the bedside) are more likely to engage such biases than mail settings. Sampling Methodology You want to know how your consumers patients, employees, and doctors feel about their experiences. Ideally, you would like to hear from everyone because that is the only way you can be certain that you have not missed anything important. But to conduct a census of the population (as it is called) is very expensive and, as it turns out, not necessary from a statistical standpoint. Using an appropriate sampling strategy, you can approximate the population values with a high degree of certainty and actually estimate how far off from them your sample numbers are. If all respondents were identical in all respects, there would be no need for careful sampling, or choosing some respondents to be in the survey while passing over others. But because there are differences in the population of respondents, we need more controlled sampling procedures. The rule is, for a sample to be representative of the population from which it is drawn, it must contain the same variation as exists in the population. For example, if 80% of the population is over 65 years of age, 80% of the sample should be over 65 years of age. Random sampling provides a way for selecting a sample of patients that reflects the total patient population and guarantees that any differences between the sample and the population are due to chance, not selection bias. In a random sample, every person has an equal chance of being chosen to receive a survey.

7 page 7 It s like putting all patients names in a bowl, mixing them thoroughly, and pulling out a handful. The accuracy ( margin of error ) of a random sample should be calculated so that you can be assured that the data is within so many points of the population mean. Stratified Random Sampling From a quality improvement action planning viewpoint it is important to analyse data for key groups of consumers, employers or doctors. An overall facility-wide satisfaction result provides good global feedback, however this may mask under performing areas such as certain wards or units. A high performing ward/unit will balance a poor performing unit, thus the facility loses the ability to acknowledge and reward the high performer and take corrective action on the poor performer. Objection 6: Only Dissatisfied Patients Respond to Surveys! Figure 1 shows it is not the case that only dissatisfied patients respond to surveys. The bar graph contrasts the distribution of responses for poorly performing hospitals (those in the 10 th percentile of the Press Ganey database) and well performing hospitals (in the 90 th percentile). Note that most of the responses are either Good or Very Good, not Very Poor or Poor. Moreover, the major difference between the good performers and poor performers is in the number of Good and Very Good responses they garner. A stratified random sampling technique should be employed to ensure sufficient data is received for each area or sub group. In these instances a minimum number of 30 returned surveys is desirable to ensure adequate confidence in the data at a sub group level. What is so special about having 30 or more responses per key area? Traditionally, statisticians have used thirty as the threshold between small and large samples. When a sample contains fewer than 30 members, the standard deviation is less reliable in estimating the standard deviation of the population, which is critical to the calculation of standard error and confidence intervals. In small samples, the standard deviation underestimates the population standard deviation more than half the time. Objection 7: With so Few Surveys, Our Scores Can t Possibly be Valid! Instead of Z values, small samples must use t values. The t values approximate Z values for large samples but begin to depart from Z values when samples have fewer than 30 cases. Stated somewhat differently, the larger the sample, the more closely the t distribution approaches the normal distribution. According to the Central Limit Theorem, as samples exceed 30 responses, the distribution of sample means will be normally distributed regardless of the distribution of the underlying population. This strategy is recommended to ensure that the data reported is reliable and accurate. You can of course report mean scores for areas having fewer than 30 responses, however appropriate caution is recommended in interpreting the results. It is true that the smaller the sample, the more likely it is that a couple of extremely unhappy respondents could pull the mean score down, just as a few completely satisfied respondents could push it up. You need to apply statistical tests to the data t-tests and control charts, for example take into account the sample size when determining statistical significance. They know that larger differences in mean scores are required for a score change to be flagged as significant when the size of the sample is small. Response Rates and Representativeness Response rate is important because it is one indication of the representativeness of a sample. However, the major threat to representativeness is response bias. Response bias occurs when certain kinds of individuals are more or less likely to respond to the survey than others, which leads, respectively, to their over- or under-representation in the total sample.

8 page 8 If respondents were chosen randomly, then the higher the overall response rate, the less likely it is that the resulting sample is affected by response bias. However, high response rates do not necessarily preclude response bias and low response rates do not necessarily result in response bias. Response bias is a concern when the scores differ by type of patient and the under-represented groups comprise a sizable proportion of the entire patient base. The possibility then exists that ratings given by the individuals who responded to the survey are different than the ratings that would have been received if everyone had responded. In order to investigate the possibility that response bias exists within your data, you should look at the demographic characteristics of respondents and ask whether they match your population. Summary If your data is appropriately collected using a psychometrically tested satisfaction survey, the data is valid because it can be used to truly evaluate the issues that you wish to assess. When the scores are low, problems in service delivery or perceptions of the organisation are discovered and when quality improves, the scores go up. The scenario, continued: You have just seen your latest survey results and are about to post them on your intranet, and your Naysayer comes along. You smile inwardly as you proceed with your work, knowing that what you are about to share with the organisation truly represents the experiences of your consumers. Objection 8: How Can we be Compared to a Benchmark When Our Patients Are Different? Health care organisations generally want to know how their facility compares with others in both large and small peer groups. When making such comparisons, it is important to consider the effect that facility characteristics (location, size of population, urban-rural, part of the country, etc.) have on patient satisfaction scores. Facilities should have the ability to select meaningful peer comparisons through the creation of custom peer groups.