Workbook 6 Cient Satisfaction Evauations Workbook 6 Cent Satisfaction Evauations 1
c Word Heath Organization, 2000 WHO Word Heath Organization UNDCP United Nations Internationa Drug Contro Programme EMCDDA European Monitoring Center on Drugs and Drug Addiction This document is not a forma pubication of the Word Heath Organization (WHO) and a rights are reserved by the Organization. The document may, however, be freey reviewed, abstracted, reproduced and transated, in part or in whoe but not for sae nor for use in conjunction with commercia purposes. The views expressed in documents by named authors are soey the responsibiity of those authors. 2 Evauation of Psychoactive Substance Use Disorder Treatment
Acknowedgements The Word Heath Organization gratefuy acknowedges the contributions of the numerous individuas invoved in the preparation of this workbook series, incuding the experts who provided usefu comments throughout its preparation for the Substance Abuse Department, directed by Dr. Mary Jansen. Financia assistance was provided by UNDCP/EMCDDA/Swiss Federa Office of Pubic Heath. Cam Wid (Canada) wrote the origina text for this workbook and Brian Rush (Canada) edited the workbook series in earier stages. JoAnne Epping- Jordan (Switzerand) wrote further text modifications and edited the workbook series in ater stages. Munira Laji (WHO, Substance Abuse Department) and Jennifer Hiebrand (WHO, Substance Abuse Department) aso edited the workbook series in ater stages. Maristea Monteiro (WHO, Substance Abuse Department) provided editoria input throughout the deveopment of this workbook. Workbook 6 Cent Satisfaction Evauations 3
4 Evauation of Psychoactive Substance Use Disorder Treatment
Tabe of contents Overview of workbook series 6 What is a cient satisfaction evauation? 7 Why do a cient satisfaction evauation? 7 How to do a cient satisfaction evauation? 8 Concusion and practica recomendation 17 Comments about case exampes 19 Case exampes of cient satisfaction evauation Part A: An evauation of satisfaction with a state drinker driver treatment program 21 Part B: Cient satisfaction with residentia substance treatment programmes 26 Part C: The case of community methadons treatment programs 32 Workbook 6 Cent Satisfaction Evauations 5
Overview of workbook series This workbook is part of a series intended to educate programme panners, managers, staff and other decision-makers about the evauation of services and systems for the treatment of psychoactive substance use disorders. The objective of the series is to enhance their capacity for carrying out evauation activities. The broader goa of the workbooks is to enhance treatment efficiency and cost-effectiveness using the information that comes from these evauation activities. This workbook discusses the assessment of cient satisfaction. It focuses on: reasons for assessing cient satisfaction the use of cient satisfaction measures for programme improvement measures of cient satisfaction Introductory Workbook Framework Workbook Foundation Workbooks Workbook 1: Panning Evauations Workbook 2: Impementing Evauations Speciaised Workbooks Workbook 3: Needs Assessment Evauations Workbook 4: Process Evauations Workbook 5: Cost Evauations Workbook 6: Cient Satisfaction Evauations Workbook 7: Outcome Evauations Workbook 8: Economic Evauations 6 Evauation of Psychoactive Substance Use Disorder Treatment
What is a cient WHO/MSD/MSB 00.2g satisfaction evauation? Cient satisfaction evauations are an exceent opportunity to invove cients or patients in the process of evauating your programme. Cient satisfaction evauations can address 1. the reiabiity of services, or the assurance that services are provided in a consistent and dependabe manner; waiting times for service components frequency of appointments time spent with counseor 2. the responsiveness of services or the wiingness of providers to meet cients/customer needs; 3. the courtesy of providers; and 4. the security of services, incuding the security of records. Specific questions may assess cients views about : the physica setting of services the hepfuness of support staff information resources the competence of counseors the costs of service the reevance of services to their needs the accessibiity of services the humanness of services the effectiveness of services in ameiorating their probems Cient satisfaction occupies an intermediate step in estabishing a heathy cuture for evauation within a programme or a setting. It often foows process evauation and cost anaysis, and precedes outcome and economic evauations. Accordingy, measures of cient satisfaction ie somewhere between process and outcome measures. When the concern is with the extent to which cients are satisfied with the context, processes, and perhaps the costs of a treatment service or network, the reevant measures of satisfaction can be viewed as process measures. However, when the concern is with the extent to which cients view the programme as having been hepfu in resoving their probems, cient satisfaction becomes a proxy outcome measure. Workbook 6 Cent Satisfaction Evauations 7
It is worth keeping in mind that satisfaction with the treatment processes, treatment compiance, and positive treatment outcomes are inter-reated. Cient satisfaction with treatment processes may both infuence, and be infuenced by, treatment outcomes. Cients who are not satisfied with a service may have worse outcomes than others because they miss more appointments, eave against advice or fai to foow through on treatment pans. On the other hand, cients who do not do we after treatment may have ess than favourabe attitudes towards a treatment service, even if it was of high quaity by other criteria. In practice, these mutua infuences may be difficut to disentange. It is worth keeping in mind that satisfaction with the treatment processes, treatment compiance, and positive treatment outcomes are inter-reated. Ratings of different dimensions of satisfaction have been highy correated in some studies, and scores on these dimensions have been added to yied overa satisfaction ratings. However, responses to specific items are of interest to service providers who want to find out how a particuar aspect of the service coud be improved. Cient satisfaction surveys may provide the ony means for cients to express concerns about the services received.... evidence of positive cient satisfaction is not, in itsef, sufficient to estabish the effectiveness of treatment. Why do a cient satisfaction evauation? The assessment of cient satisfaction adds an important consumer perspective to evauations of PSU treatment services and systems. Cient satisfaction evauations can be viewed as an opportunity to consut with cients about their experiences in your programme. Cient satisfaction surveys may provide the ony means for cients to express concerns about the services received, and to express their views about new services that are needed. Cient satisfaction ratings have been criticised as indicators of the quaity of human services because they may refect unreaistic expectations. Whie this criticism may be vaid in some instances, re- search with cients of menta heath services suggests that they can effectivey discriminate between services that are different in quaity (Lebour, 1983; Sheppard, 1993). It is, however, important to recognise that evidence of positive cient satisfaction is not, in itsef, sufficient to estabish the effectiveness or accessibiity of treatment. Cients with no base for comparison may be satisfied with services that are ineffective as determined by more objective outcome evauations. On the other hand, cients may be dispeased with services that achieve the objective of reducing their PSU but empoy rigid or authoritarian approaches. 8 Evauation of Psychoactive Substance Use Disorder Treatment
How to do a cient satisfaction evauation? The most common method for assessing cient satisfaction is with sef-administered questionnaires. These may be given to cients as they enter or eave services, or at various times in between. They can aso be administered at some point after treatment has been competed, when the outcomes of treatment are more cear to the cient. Cient satisfaction questionnaires can be competed at the time they are distributed, or at a ater date seected by the cient or program personne. Stamped, return enveopes can be provided if questionnaires are to be returned by mai. Satisfaction questionnaires aso can be maied to former cients with stamped, return enveopes. A cover etter shoud expain why the questions are being asked and how the information wi be used. The cover etter shoud aso indicate if individua repies wi be considered confidentia or anonymous, and what steps wi be taken to ensure that this is the case. For ethica reasons, risks to cients shoud be made cear. It shoud be stated that their responses wi not in any way affect present or future treatment. Programme managers typicay want the questionnaire to identify the respondent so that they can foow-up with these individuas who express concerns about the services received. If this is the case, cear provisions for confidentiaity must be made, incuding, for exampe, remova of the information identifying the cient prior to data anaysis by computer or other tabuar means. For more information about these ethica issues, see Workbook 2 of this series, Step 1A, entited Manage Ethica Issues. Cient satisfaction aso can be assessed in face-to-face or teephone interviews or focus groups. These strategies are more expensive than sef-competed questionnaires. If interviews or focus groups are used, it is preferabe to have them conducted by someone who is not connected directy with the service. This may be an independent evauator, vounteers or former cients themseves trained to take on this roe. If interviews or focus groups must be done by a manager or staff member, it is best not to have the individua s principa therapist ask about cient satisfaction because cients may be reuctant to comment negativey about their treatment directy to their therapist. Interviews may be highy structured, perhaps guiding the cient through the same type of questionnaire used on a sef-administered basis in other situations. Other interviews, and certainy focus groups, wi be much ess structured and the resuting information wi be anaysed quaitativey. Workbook 1 provides guidance for conducting focus group and semi-structured/unstructured interviews. Workbook 2 offers advice on anaysing the resuting information. The design and conduct of cient satisfaction surveys Cient satisfaction surveys are most usefu when they are designed to meet specific objectives and when they use appropriate methods and measures. This Workbook 6 Cent Satisfaction Evauations invoves choices of samping procedures, timing, cutura acceptabiity, and sensitivity of the questions to various eves of satisfaction. 9
There are no right or wrong ways to choose sampes in cient satisfaction surveys. However, it is important that your sampe be consistent with the evauation objectives. Choosing sampes of cients Your strategy for seecting cients for a satisfaction survey can infuence the kinds of resuts you obtain. If the surveys are imited to cients who compete treatment, the resuts wi probaby differ from those obtained in surveys that incude peope who have dropped out of the programme. There are no right or wrong ways to choose sampes in cient satisfaction surveys. However, it is important that your sampe be consistent with the evauation objectives. If the objective is to earn about cient satisfaction among those who compete treatment then there wi be no need to invove treatment drop-outs. However, if the aim is to find how, in genera, cients fee about the programme, a representative sampe of a cients competing the intake process woud be more appropriate. Regardess of the sampe chosen, you must be sure to ceary describe the sampe in subsequent reports. Limitations to the generaizabiity of resuts must be stated. For exampe, are your resuts biased due to the excusion of eary drop outs? Once you have decided which types of cients wi be invoved in satisfaction surveys, you have a number of options for choosing particuar cients, incuding a random or systematic sampe. These and other options for samping are discussed in Workbook 2. There is no best timing for these surveays, except to ensure consistency with the objectives of the evauation. Choosing sampes of cients The timing of cient satisfaction surveys can infuence your resuts. Cients with positive views during or immediatey foowing treatment may change their minds if they ater reapse. On the other hand, cients may gain a greater appreciation of services as their vaue becomes evident in an increasing number of rea ife situations. There is no best timing for these surveys, except to ensure consistency with the objectives of the evauation. If the objective is to find out what cients fee at the time of discharge, then ask cients to compete a satisfaction questionnaire as they are about to eave. However, if the aim is to find out if cients are satisfied as part of an outcome evauation, wait unti some period of time has passed before asking former cients to compete a satisfaction questionnaire. The timing of surveys shoud be ceary indicated in reports and any associated biases shoud be discussed. If, for exampe, cients compete satisfaction questionnaires foowing an emotiona graduation ceremony this coud bias attitudes in favour of the programme. Cuture sensituvity Cutures differ with respect to expectations of feedback on pubic and private services. In jurisdictions where consumerism is firmy estabished, frank verba or written feedback may be freey given. However, direct negative feedback in some cutures may be considered impoite and compaints may ony be shared with intimate acquaintances. Direct and chaenging questions aso may be cutur- ay inappropriate (NIDA, 1993). Experiences with (and attitudes toward) the use of questionnaires, interviews, focus groups and other methods of inquiry aso differ between cutures. Methods for soiciting cient feedback must take into account the prevaiing cutura norms and seek to ensure the use of appropriate methods that assess cient beiefs and opinions. 10 Evauation of Psychoactive Substance Use Disorder Treatment
Measures vaidated in one cuture may not be appropriate in others. Simpe transation of questionnaire items does not guarantee that the items wi have the same meaning across cutures (Attkisson and Greenfied, 1994). Considerabe effort may be required to generate new, cuturay appropriate questions. Cients, or peope advocating on their behaf, shoud be invoved in this process of questionnaire design to ensure that the measures wi provide a vaid indication of cient satisfaction. Some groups of cients may aso find particuar methods for assessing cient satisfaction more acceptabe than others. For exampe, those with poor cognitive or reading skis may prefer persona interviews over a written questionnaire. However, cients who are shy or have ow sef-esteem may prefer questionnaires over interviews. Sensitivity to different eves of satisfaction...if satisfaction is rated on a five-point scae, the proportions of cients who are very satisfied, somewhat satisfied or neutra can be better discriminated... Many satisfaction surveys of cients of heath and socia services have shown high eves of satisfaction party because they have used insensitive measures (Ruggeri, 1994). An exampe woud be using questionnaire items that ony have two response options (satisfied/not satisfied). Such items tend to invite a satis- fied response, even from those who are neutra of even midy dissatisfied. However, if satisfaction is rated on a five-point scae, the proportions of cients who are very satisfied, somewhat satisfied or neutra can be better discriminated, as can the proportions who are somewhat or very unsatisfied. Seeking out expressions of dissatisfaction Cients... may fee especiay obiged to show that they are gratefu and satisfied with the services provided. Cients of human service agencies have a tendency to be gratefu for the attention they receive, and to be reuctant to criticise in the event that this eads to negative consequences. Cients with ow sef-esteem, or who are conscious of status differences between themseves and service providers, may fee especiay obiged to show that they are gratefu and satisfied with the services provided. These tendencies can be overcome if cients are assured that their honest feedback is being sought and that there wi be no consequences for those who criticise the services in question. This can be made cear in verba or written instructions for competing satisfaction questionnaires or participating in interviews or focus groups. Confidentiaity of the resuts shoud be assured as strongy as possibe. It is desirabe to activey seek out sources of discontentment by asking the foowing kinds of questions: Are there any parts of the programme that you iked more than others? Have you any suggestions for ways in which the programme can be improved? Aso, ook for behavioura indicators of dissatisfaction, for exampe, high drop-out or no-show rates within specific programmes, or for specific counseors. Whie many factors may contribute to ow participation, ow cient satisfaction may be invoved. Workbook 6 Cent Satisfaction Evauations 11
Estabished questionnaires for assessing cient satisfaction When choosing a questionnaire for your evauation, you first need to consider whether a dimensions of cient satisfaction are reevant to the service components being evauated. It is often the case that one treatment agency provides different types of services and activities. You wi have to decide whether your cient satisfaction questionnaire wi provide feedback about individua service components, or whether you wi focus on a more goba eve of programme participation. This wi be an issue to resove in the assessment of satisfaction with services received across a arge network of agencies. If the intention is to use the resuting information to suggest highy specific areas for service or system enhancement, you may need to customise your seection of cient satisfaction measures to fit particuar service or system components. This may utimatey invove a choice between a standardised, goba measure of satisfaction avaiabe from pubished iterature (see beow), and questionnaires taiored to your specific information needs. If you are going to use a structured, sefadministered questionnaire, you may seect one from the pubished iterature. Such measures in the pubic domain wi ikey have data avaiabe on reiabiity and vaidity in a particuar setting. This is a big advantage, but must be considered in ight of cutura variations between the cuture in which the questionnaire was vaidated and the cuture in which you intend to use it now. In addition, standardised questionnaires may be too genera to give you the kind of detaied feedback you need for making improvements to specific parts of the program. Feedback unique to your program can be derived from a speciay-taiored questionnaire, athough issues of reiabiity and vaidity wi be of concern. Openended questions can aso be added to a sef- administered questionnaire and then anaysed quaitativey. A questionnaire which can be used to assess cient satisfaction is the Cient Satisfaction Questionnaire (CSQ-8). This is a widey used instrument with pubished data on reiabiity and vaidity (Greenfied and Attkisson, 1989). The instrument is avaiabe in severa anguages, incuding Engish, Spanish, Dutch and French (de Brey, 1983; Roberts et a., 1984; Sabourin et a., 1987). Case exampes of evauations that used the CSQ-8 aso are reported at the end of this workbook. Workbook 1, Appendix 2 aso contains four other exampes of questionnaires that can be used to assess cient satisfaction. There are no data on the reiabiity and vaidity of these other instruments. However, they may be hepfu in your situation or stimuate ideas for the deveopment of a questionnaire unique to your needs. A report from the Nationa Institute on Drug Abuse (1993) entited How Good is Your Drug Abuse Treatment Program? contains a series of cient satisfaction questions used in an AIDS Risk Reduction Project. Two other measures of cient satisfaction appropriate for PSU services are the Service Satisfaction Scae (SSS-30) (Attkisson and Greenfied, 1984), and the Verona Service Satisfaction Scae (VSSS) (Tansea, 1991). The SSS-30 is a 30-item muti-dimensiona scae deveoped on the basis of experience with the Cient Satisfaction Scae. The first case exampe at the end of this workbook (Part A: by Thomas Greenfied) describes the SSS-30 in greater detai. The VSSS is an 82-item scae which covers seven dimensions overa satisfaction, professiona skis and behaviours, information, access, efficacy of interventions and reative improvement. 12 Evauation of Psychoactive Substance Use Disorder Treatment
Deveoping your own cient satisfaction questionnaires A good starting pace for the deveopment of a new cient satisfaction questionnaire that is taiored to your individua service or treatment system wi be your programme ogic mode and accompanying written descriptions of your programme (see Workbook 1). These wi identify the main components, activities and treatment processes for which cient satisfaction ratings coud be deveoped. In addition, cients coud be asked to rate their satisfaction with the staff, comprehensiveness of the services provided and aspects of the physica environment. It woud aso be usefu to convene sma groups of current and former cients to expore issues most reevant to their needs. These groups may be hepfu in testing ideas for questionnaire items and response options. The instruments contained in Workbook 1, Appendix 2 aso wi be usefu. To hep vaidate measures of cient satisfaction, the ratings can be compared with verba reports or satisfaction ratings from famiy members or others that are famiiar with the services received. You can aso compare the resuts using your new questionnaire with the resuts of an instrument ike the CSQ-8 competed by the same peope. Cient satisfaction ratings can aso be compared with actua behaviours that signify satisfaction with services. Comparisons coud be made, for exampe, between cient satisfaction ratings and their record of keeping appointments, competing treatment, or returning for further treatment foowing a reapse. Using cient satisfaction measures during times of change in service deivery Once reiabe cient satisfaction measures are avaiabe, they can be used for routine or periodic check-ups on the quaity of services from the cients perspective. They aso can be used to assess cient reactions to changes in service deivery being impemented. For exampe, changes may be panned to increase the efficiency of a service but there are concerns that these coud ead to decreased cient satisfaction. Measures of satisfaction taken before or after the changes are introduced wi show if this has been the case. [Cients] coud be asked if they fee satisfied with the information that is avaiabe on the range of services in the community. Measuring cient satisfaction in evauations across two or more agencies It is possibe to assess cient satisfaction with services received across a network of programmes, rather than focusing on the cient s experience with ony one service provider. Not a cients wi have experience with other services in the treatment network. However, cients of a services may have usefu perspectives on system-wide issues. For exampe, they coud be asked if they fee satisfied with the information that is avaiabe on the range of services in the community. They coud aso be asked to rate their satisfaction with recommendations for referra given the options that were presented. Did they ike this referra? Was it too far away for them? Do they fee satisfied with being referred to a residentia service when they Workbook 6 Cent Satisfaction Evauations 13
might have gone to a day treatment or outpatient service (and vice versa)? A review of a ogic mode for the treatment system may suggest other topics to be incuded in cient satisfaction surveys. For exampe, waiting times for moving from one service component to another may be of particuar concern. The issue of dupication of services is aso important to expore, for exampe, whether agencies in the system dupicate the coection of assessment information when the cient moves from one service to another. Cients who have experienced two or more services in a network may have vauabe perspectives on the degree to which these services are co-ordinated. Sampe questions concerning cient satisfaction with inter-service co-ordination are: How satisfied are you with the way that (name of both services) exchanged treatment information about your probems? How satisfied are you with the information that (name of both services) provided to you about each other s treatment programmes? How satisfied are you with the ways the treatment staff of (name of both services) worked together to hep you with your probems? Based on your experience, how we do (name of both services) work together? It s your turn Put the information from this workbook to use for your own organisation or treatment network. Compete these exercises beow. Remember to use the information from Workbooks 1 and 2 to hep you compete an evauation pan. Review that information now, if you have not aready done so. Exercise 1 Think about your treatment programme or oca treatment network. List five genera areas in which you want to know the views of cients or patients. Exercise 2 Using the information provided in this workbook about how to design and conduct a cient satisfaction evauation, make the foowing decisions: Exampe: What do cients think about the hepfuness of our cinicians? 1) 2) 3) 4) 5) Decide what modaity you wi use to coect the data (questionnaires, interviews, focus groups) Choose a samping procedure for choosing cients to survey Decide the timing of the evauation 14 Evauation of Psychoactive Substance Use Disorder Treatment
Deveop a procedure for ensuring cients confidentiaity and promoting their honesty in answering questions Decide who wi hep you administer the questionnaires/interviews/focus groups. Exampe (from above): Data wi be coected using a sef-report questionnaire. Pease hep us improve our programme by answering some questions about the services you have received. We are interested in your honest opinion, whether it is positive or negative. To ensure your confidentiaity, pease do not write your name on this form. When you are finished, pace the form in the enveope (provided) and sea it cosed, then pace it in the coection box in the waiting area. A cients checking in for appointments during the week of December 10th wi be handed the survey to compete whie waiting for their appointments. Data wi be coected over a one week period of time ony, from 10-15 December. Cients wi be given enveopes in which to pace their competed questionnaires before returning them to the coection box. The foowing statement wi appear at the top of the questionnaire: Because the questionnaire assesses cient satisfaction with staff, it is not feasibe for staff to be invoved with distributing or coecting questionnaires. A outside research assistant wi be hired to hand out the introductory etters, consent forms, and questionnaires. The assistant aso wi remove the questionnaires from the coection box and keep them in a safe pace to ensure their confidentiaity from the staff. Now it s your turn. Foow the same procedure for your evauation questions. Exercise 3 You wi need to prepare an introductory etter and consent form that expains the purpose of your study. Review Section 1A of Workbook 2, entited Manage Ethica Issues, for more information about the important topic of participants rights in evauation research. describes the purpose and methods of the study expains what they wi need to do if they participate expains that participation is vountary In genera, a participants shoud be asked permission ahead of time before being enroed in the study. When you do this, your shoud expain the purpose, nature, and time invoved in their participation. No person shoud be forced or coerced to participate in the study. A standard practice is to have each participant sign a consent form, which: Note that an ethica committee may waive the requirement of a signed consent form if the research contains minima risk. In these cases, researchers sti need to provide fu information to participants. A consent form is incuded in the foowing exampe for the sake of competeness. Exampe (from above): Workbook 6 Cent Satisfaction Evauations 15
Introductory Letter: We are asking your hep in improving our programme by fiing out a 2 page questionnaire about the services you have received here. The questions wi ask about your views regarding our staff members. They wi take about 10 minutes to compete. A information that you provide us wi remain stricty private and confidentia. If you agree to participate, pease read and sign the consent form (attached) and return it to the research assistant who gave you this packet when you arrived. Thank you for your time. Sincerey, Dr. X Director, Treatment Programme Consent Form You agree to participate in a cient survey of satisfaction with our staff. You wi compete a 2 page questionnaire today, which wi take about 10 minutes to compete. Your participation is competey vountary. You can refuse to answer any questions and/or withdraw from the study at any time without a probem to you or your treatment here. A your responses wi remain stricty confidentia: programme staff members wi not have access to your responses, your name wi not appear on your questionnaire, and your responses wi not be inked to your identity at any time. I have read the information above and agree to participate. Signature: Date: Now it s your turn. Using the exampe above, and the additiona information provided in Workbook 2, section 1A, write your own introductory etter and consent form. Exercise 4 Run a piot test of your evauation measurement and procedures to ensure that everything runs smoothy. Review section IC of Workbook 2 entited Conduct a Piot Test for specific information about how to do this. In genera, piot tests assess these questions: Do the questions provide usefu information? Can the questions be administered propery? For exampe, is it too ong or too compicated to be fied out propery? Exampe (from above): A piot test wi be run during one cinic day: 3 November. During this day, 10-15 patients checking in wi be given the questionnaire. Afterwards, their responses wi be examined to determine whether they seemed to understand the questions and were answering honesty. A persons invoved with distributing the forms and compiing the data wi be interviewed to determine their views on any improvements that coud be made in the process and/or to the forms. Can the information be easiy managed by peope responsibe for compiing the data? Now it s your turn. Write down how you wi piot test your evauation study. Don t forget to review Workbook 2 first! Does other information need to be coected? 16 Evauation of Psychoactive Substance Use Disorder Treatment
Concusion and practica recommendation In this workbook, we have outined the basic principes and practices in the evauation of cient satisfaction with PSU services and systems. After competing your evauation, you want to ensure that your resuts are put to practica use. One way is to report your resuts in written form (described in Workbook 2, Step 4). It is equay important, however, to expore what the resuts mean for your programme. Do changes need to happen? If so, what is the best way to accompish this? Return to the expected user(s) of the evauation with specific recommendations based on your resuts. List your recommendations, ink them ogicay to your resuts, and suggest a period for impementation of changes. The exampes beow iustrate how to manage two different kinds of resuts using this technique. Unfavourabe findings Based on the finding that over 1/4 of cients were very dissatisfied or somewhat dissatisfied with the friendiness of the cinica staff, we recommend that the programme institute a 2 hour cient satisfaction training workshop for a cinicians to attend. The workshop coud happen in March, which traditionay is a ow-census month for the programme, and be run by Dr. Z, who is we-iked and respected by the staff. Favourabe findings The resuts indicate that cients are very satisfied overa with the hepfuness of the cinica staff. Therefore, we recommend that the composition of the cinica staff remain unchanged, and that these favourabe findings are pubicy acknowedged at the next programme-wide staff meeting. Remember, cients provide an invauabe perspective on the success of your programme. It is important to use the information that they provide to improve treatment services. Through carefu examination of your resuts, you can deveop hepfu recommendations for your programme. In this way, you can take important steps to create a heathy cuture for evauation within your organisation. Workbook 6 Cent Satisfaction Evauations 17
References Addiction Research Foundation. A Directory of Outcome Measures. Toronto: Addiction Research Foundation, undated. Attkisson, C.C., & Greenfied, T.K. The Cient Satisfaction Questionnaire-8 and the Service Satisfaction Questionnaire-30. In: Maruish, M. (ed.). Psychoogica testing: treatment panning and outcome assessment. Lawrence Erbaum Associates: San Francisco, 1994: 402-420. de Brey, H. A cross-nationa vaidation of the Cient Satisfaction Questionnaire: The Dutch experience. Evauation and Programme Panning, 1983, 6: 395-400. Greenfied, T.K., & Attkisson, C.C. Steps toward a mutifactoria satisfaction scae for primary care and menta heath services. Evauation and Programme Panning, 1989, 12:271-278. Kurtz, L.F. Measuring member satisfaction with a sef-hep association. Evauation and Programme Panning, 1990, 13: 119-124. Larsen, D.L. Enhancing cient utiization of community heath services. Dissertation Abstracts Internationa, 39, 4041B. (University Microfims No. 7904220), 1979. Larsen, D.L., Attkisson, C.C., Hargreaves, W.A., & Nguyen, T.D. Assessment of cient/ patient satisfaction: Deveopment of a genera scae. Evauation and Programme Panning, 1979, 2: 197-207. Lebour, J.L. Simiarities and differences between menta heath and heath care evauation studies assessing consumer satisfaction. Evauation and Programme Panning, 1983, 6: 237-245. Lettieri, D.J., Neson, J.E., & Sayers, M.A. (Eds.) NIAAA treatment handbook series: Acohoism treatment assessment research instruments. Rockvie, MA: Nationa Institute on Acoho Abuse and Acohoism, 1985. Levois, M., Nguyen, T.D., & Attkisson, C.C. Artifact in cient satisfaction assessment: Experience in community menta heath settings.evauation and Program- me Panning, 1981, 4: 139-150. Moos, R.H., & Finney, J.W. Acohoism programme evauations: The treatment domain. In D.J. Lettieri (Ed.), Research strategies in acohoism treatment assessment (pp.31-51). New York: Haworth Press, 1988. Nationa Institute on Drug Abuse. How good is your drug abuse program? Resource Manua. Nationa Institute on Drug Abuse, Rockvie, MD, 1993. Nguyen, T.D., Attkisson, C.C., & Stegner, B.L. Assessment of patient satisfaction: Deveopment and refinement of a Service Evauation Questionnaire. Evauation and Programme Panning, 1983, 6: 299-314. Roberts, R., Attkisson, C., & Mendias, R.M. Assessing the Cient Satisfaction Questionnaire in Engish and Spanish. Hispanic Journa of Behavioura Science, 1984, 6: 385-396. Ruggeri, M. Patients and reatives satisfaction with psychiatric services; the state of the art of its measurement. Socia Psychiatry Psychiatric Epidemioogy, 1994, 29: 212-227. Sabourin, S., Gendreau, P., & Frerette, L. Le neveau de satisfaction des cas d abandon dans un service universitaire de psychoogie. Canadian Journa of Behavioura Science, 1987, 19: 314-323. Sheppard, M. Cient satisfaction, extended intervention and interpersona skis in community menta heath. Journa of Advanced Nursing, 1993, 18: 246-259. Tansea, M. (ed.) Community-based psychiatry: Long-term patterns of care in South Verona. Psychoogica Medicine Suppement, 19, Cambridge University Press, Cambridge, 1993. Zwick, R.J. The effect of pretherapy orientation on cient knowedge about therapy, improvement in therapy, attendance patterns, and satisfaction with services. Masters Abstracts, 1982, 20: 307. (University Microfims No. 13-18082). 18 Evauation of Psychoactive Substance Use Disorder Treatment
Comments about case exampes Each of the foowing case exampes describes evauations comparing cient satisfaction across sites. The first evauation (Part A) presents a cient satisfaction evauation of a state-sponsored treatment, primariy for peope convicted of drunk driving. Satisfaction was measured using two scaes discussed in this workbook: the CSQ-8 and SSS-30. Whereas the CSQ-8 provides a singe satisfaction score, the SSS-30 assesses severa aspects of cient satisfaction. Resuts were used to guide site procedura improvements. The second evauation (Part B) is a good exampe of how cient satisfaction evauations can be competed with imited resources. In this case, a menta heath intern wanted to examine cient satisfaction across severa residentia PSU treatment programmes. With imited assistance, he was abe to pan and successfuy impement his evauation. Differences in satisfaction across sites were detected by the mutidimensiona SSS-30. The third evauation examined cient satisfaction across three community methadone treatment sites in Austraia. Evauators used the CSQ-8 and two quaitative, open-ended questions to assess satisfaction. They found significant differences in satisfaction across sites. These differences were given to cinic managers to make procedura improvements. Whie each of these cases generated usefu information about cient satisfaction, it is noteworthy that none attempted to provide information about cient outcome or treatment effectiveness. As described earier in this workbook, measurement of cient satisfaction is usefu yet distinct from measurement of cient outcome or treatment effectiveness. On occasion, cients can be satisfied with treatment that is ineffective in reducing PSU. On the other hand, certain treatments can be effective but unpopuar with cients. Evauators must remember that cient satisfaction and cient outcome are distinct evauation concepts. Each of the foowing case exampes describes evauations comparing cient satisfaction across sites. The first evauation (Part A) presents a cient satisfaction evauation of a state-sponsored treatment, primariy for peope convicted of drunk driving. Satisfaction was measured using two scaes discussed in this workbook: the CSQ-8 and SSS-30. Whereas the CSQ-8 provides a singe satisfaction score, the Workbook 6 Cent Satisfaction Evauations 19
SSS-30 assesses severa aspects of cient satisfaction. Resuts were used to guide site procedura improvements. The second evauation (Part B) is a good exampe of how cient satisfaction evauations can be competed with imited resources. In this case, a menta heath intern wanted to examine cient satisfaction across severa residentia PSU treatment programmes. With imited assistance, he was abe to pan and successfuy impement his evauation. Differences in satisfaction across sites were detected by the mutidimensiona SSS-30. The third evauation examined cient satisfaction across three community methadone treatment sites in Austraia. Evauators used the CSQ-8 and two quaitative, open-ended questions to assess satisfaction. They found significant differences in satisfaction across sites. These differences were given to cinic managers to make procedura improvements. Whie each of these cases generated usefu information about cient satisfaction, it is noteworthy that none attempted to provide information about cient outcome or treatment effectiveness. As described earier in this workbook, measurement of cient satisfaction is usefu yet distinct from measurement of cient outcome or treatment effectiveness. On occasion, cients can be satisfied with treatment that is ineffective in reducing PSU. On the other hand, certain treatments can be effective but unpopuar with cients. Evauators must remember that cient satisfaction and cient outcome are distinct evauation concepts. 20 Evauation of Psychoactive Substance Use Disorder Treatment
Case exampes of cient satisfaction evauations Part A: An evauation of satisfaction with a state drinker driver treatment program The author aone is responsibe for the views expressed in this case exampe. by Thomas K. Greenfied, Ph.D. Senior Scientist and Area Director for Popuation Surveys NIAAA Nationa Acoho Research Center Acoho Research Group 2000 Hearst Ave., Suite 300 Berkeey, Caifornia 94709-2130 USA Workbook 6 Cent Satisfaction Evauations Who is asking the questions and why do they want this information? Purposes for the cient satisfaction evauation There are a number of reasons why cients satisfaction with services is such a critica variabe in an overa substance abuse treatment outcome evauation effort. First, substance abuse treatment programme directors and managers are often required to justify their programs. They find consumer satisfaction is a concept readiy understood by their cients, the pubic, government bodies, or other funding agencies (Greenfied, 1983). This was the case for a private provider in a sma Eastern U.S. state icensed to provide a brief outpatient counseing service to the chemicay dependent cient. The programme is icensed by the state to provide its services in response to the Driving Under the Infuence (DUI) probem. It of course interacts cosey with courts, corrections, and the Department of Motor Vehices (DMV). Its managers knew that the state and invoved agencies woud need a positive response from its cientee, in addition to objective outcomes, for continued referras and reicensing. In addition, the managers wanted data on specific aspects of their programme so that improvement efforts might target areas of greatest concern to cients. The mutidimensiona SSS-30 questionnaire was seected for this reason based on prior experience in giving usefu feedback to student services, primary care, and EAP managers (Attkisson & Greenfied, 1994; Greenfied & Attkisson, 1989a). 21
Specific programme In the programme, referras are primariy (but not excusivey) individuas guity of a second DUI, and required by aw to receive treatment. The programme operates soey on cient fees totaing U.S. $495 at the time of the study. After two individua sessions, a treatment pan is deveoped with suitabe, eigibe cients. Eigibiity requirements incude (a) wiingness to expore drinking and drug taking, aowing for a norma degree of denia, (b) agreement to participate and remain sober and drug free, (c) commitment to be invoved and work toward reasonabe treatment goas, and (d) no overt psychiatric difficuties impying a primary menta heath probem. The programme invoves approximatey 25 contact hours, incuding four individua sessions, six educationay oriented group sessions, and eight 90 minute group sessions (under 16 members in each group). Additiona individua and famiy counseing may be incuded if needed. Conditiona driving privieges may be restored by the court after 16 hours. The programme has some coercive eements: cients unwiing to participate meaningfuy are deemed noncompiant, with paperwork indicating this sent back to the referring agency. The court must act to cear this up before the cient may again be enroed in the programme. Upon competion of the treatment, an aftercare pan is deveoped in the discharge session. The programme s offices are ocated in various counties and the evauation focused on three sites to assess their cients satisfaction. For a more compete description of the programme and the basis for cient seection, see Greenfied (1989) and Greenfied (1994). What resources were needed to coect and interpret the information? Questionnaires were handed out to cients upon arriva for their fina session by programme staff. Competed forms were coected daiy during the survey period. Thus, itte additiona effort was required for administration. A copier was used to dupicate the questionnaires so that the main resource needed was for data entry, accompished by office staff using the existing dbase software. This was the software package used for maintaining cient records (questionnaires were not identified, so no inkage to other cient data was possibe or attempted). Data entry required approximatey two minutes per questionnaire and was done by the office staff responsibe for the cient information system. Because the programme did not have anaysis software or capacity, the dbase fies were sent on diskette to the scaes authors for anaysis inspss. (Scoring keys for inagency use, and SPSS syntax for reading data from common spredsheet or reationa database fie formats, are avaiabe from the scae s firs author). At each of the two phases, anaysis time invoved about a day of work with an additiona day needed for report writing. How were the data coected? The two case studies use direct cient satisfaction measures, providing exampes of each of the two measurement strategies. Both questionnaires were designed to be broad enough to assess satisfaction with a range of human services incuding substance abuse treatment. The two are the Cient Satisfaction Questionnaire-8 (Nguyen, Attkisson & Stegner, 1983; 22 Evauation of Psychoactive Substance Use Disorder Treatment
Attkisson & Greenfied, 1994; Attkisson & Greenfied, 1995b), a widey-used, brief (8 item), genera satisfaction measure, used in the first case study, and the Service Satisfaction Scae-30 (Attkisson & Greenfied, 1994; Attkisson & Greenfied, 1995a, 1995b; Greenfied & Attkisson, 1989a) a 30-item mutidimensiona version with derivative forms avaiabe for case management and residentia settings (Greenfied et a., 1996) incuding a famiy member version (Greenfied & Attkisson, 1989b). Itaian (Ruggeri & Greenfied, 1995) and Spanish transations are avaiabe. There is evidence that these human service measures are suitabe for use in substance abuse treatment. The research done suggests good psychometric performance of these two measures (SSS-30 and CSQ-8) (Attkisson & Greenfied, 1994; Greenfied, 1989; Greenfied, 1994; Greenfied & Attkisson, 1989a). The scaes are both products of the University of Caifornia, San Francisco (UCSF), Department of Psychiatry s research programme on cient satisfaction which has extended over a quarter of a century (Attkisson & Greenfied, 1995b). Permission to use these copyright scaes may be obtained from Dr. C.C. Attkisson (CSQ-8) at the UCSF Graduate Division, 200 West Miberry Union, 513 Parnassus Avenue, San Francisco, Caifornia 94143-0404 USA (FAX+1-415-476-9690) and Dr. Greenfied (SSS-30) at the Acoho Research Group, 2000 Hearst Ave., Berkeey, Caifornia, 94709 USA (FAX+1-510-642-7175). Both scaes can be scored and simpy anaysed using common statistica programmes such as SPSS or EPI INFO, or spreadsheet or nationa data base management software such as Lotus 1-2-3, Exce, dbase, or Paradox, among other software programmes. If programming capabiity is not avaiabe in-hours, or more sophisticated anayses are needed, anaysis can be done by an evauator or anayst. SPSS scoring keys and syntax are avaiabe from the scae s first author (Thomas Greenfied, INTERNET: tgreenfied@arg.org). The CSQ has been incuded in a compendium of instruments assembed by the U.S. Nationa Institute on Acoho Abuse and Acohoism (NIAAA), the NIAAA Treatment Handbook Series 2: Acohoism Treatment Assessment Research Instruments (Attkisson, et a., 1985). Both the CSQ and the SSS have been incuded in Loyd Sederer and Barbara Dickey s Outcomes Assessment in Cinica Practice (Attkisson & Greenfied, 1995a) which incudes the scaes as appendices. Norms and psychometric resuts for the SSS-30 and CSQ-8 are avaiabe in a chapter in Mark Maruish s usefu book Psychoogica Testing: Treatment Panning and Outcome Assessment (Attkisson & Greenfied, 1994). A second edition of this book contains updated chapters on the CSQ-8 (Attkisson & Greenfied, in press) and the SSS-30 (Greenfied & Attkisson, in press). Because this was a reativey new appication to substance abuse treatment, the project invoved two phases. In the first, it was decided to conduct factor anayses of the SSS-30 data to confirm the factorbased scaes previousy deveoped in menta heath and primary care programmes (Attkisson & Greenfied, 1994; Greenfied & Attkisson, 1989a). In the second phase, data were coected from three programme sites in different ocations. In both phases, for vaidation purposes, the CSQ-8 scae was administered at the same time as the SSS-30. The dimensiona anayses and comparison of resuts with the two measures may be considered to be the methodoogica aims of the study. For practica reasons, it was ony possibe to obtain data for peope competing the programme. The two questionnaires were to be competed during the ast session and eft in a box at the door prior to departure. Questionnaires were fied out anonymousy. In Workbook 6 Cent Satisfaction Evauations 23
Tabe 1: Interna reiabiity of the four SSS-30 subscaes. SSS-30 Subscae Reiabiity coefficient (Cronbach Apha) Nº Items Substance Abuse Program Pubished norm groups* Practicioner manner and ski 9.83.89 Perceived outcome 8.83.83 Office procedures 5.74.74 Accessibiity 4.60.67 * Based on 3 Norm groups - Four heath Cinics, a Menta Heath Service, and an Empoyee Assistance Programme (see Attkisson & Greenfied, 1994). the first phase, demographics were not coected due to an oversight. In the second phase, the SSS-30 s standard demographics section was added incuding gender, age, income, ethnicity, distance from programme, and number of sessions attended. How were the data anaysed? Anayses were done using SPSS PC and SPSS for Windows7'. Data were read in from dbase7' fies provide by the programme (SPSS can read such fies directy). Reversed items were recoded 5=1, 4=2, 3=3, 2=4, 1=5 prior to anaysis and scoring, based on the pubished SSS-30 subscaes (scoring key and code avaiabe from T. K. Greenfied at the Acoho Research Group, 2000 Hearst Ave., Berkeey, Caifornia 94709, USA). In the first psychometric and confirmatory phase (n=1027), item descriptive anayses, factor anayses and reiabiity anayses (using SPSS factor and reiabiity routines) of the SSS-30 items were done, comparing simiarity of factor soutions using Harmon s (1970) coefficient of congruence cacuated using a simpe spreadsheet. The SSS-30 Tota Scae score was correated (SPSS correate) with the CSQ-8 score. These preiminary anayses heped assure that the scae functioned we in a substance abuse programme. In the second evauation phase (n=720), demographic profies of cients at each of the three programme sites were first compared. Overa satisfaction was then assessed by examining item and subscae distributions using the SPSS frequencies routine. Subscaes were again scored and these scores compared across the three programme sites using SPSS anova (Anaysis of Variance) routine. This aowed satisfaction across sites to be compared whie controing for gender differences, since men tend to be more wiing to indicate ower satisfaction than women (or are actuay ess satisfied). Finay, the CSQ-8 and SSS-30 tota scores were again correated. What did they find out? Phase 1 The two major SSS-30 factors found earier in menta heath and primary care sampes (Attkisson & Greenfied,1994) were confirmed. Practitioner Manner and Ski and Perceived Outcome factors were highy congruent with equivaent ones from earier studies (Harmon coefficients.88 -.93). These two standard factor-based subscaes were, therefore, usefu for assessing cient satisfaction in this substance 24 Evauation of Psychoactive Substance Use Disorder Treatment