Client Satisfaction Evaluations

Workbook 6 Cient Satisfaction Evauations Workbook 6 Cent Satisfaction Evauations 1

c Word Heath Organization, 2000 WHO Word Heath Organization UNDCP United Nations Internationa Drug Contro Programme EMCDDA European Monitoring Center on Drugs and Drug Addiction This document is not a forma pubication of the Word Heath Organization (WHO) and a rights are reserved by the Organization. The document may, however, be freey reviewed, abstracted, reproduced and transated, in part or in whoe but not for sae nor for use in conjunction with commercia purposes. The views expressed in documents by named authors are soey the responsibiity of those authors. 2 Evauation of Psychoactive Substance Use Disorder Treatment

Acknowedgements The Word Heath Organization gratefuy acknowedges the contributions of the numerous individuas invoved in the preparation of this workbook series, incuding the experts who provided usefu comments throughout its preparation for the Substance Abuse Department, directed by Dr. Mary Jansen. Financia assistance was provided by UNDCP/EMCDDA/Swiss Federa Office of Pubic Heath. Cam Wid (Canada) wrote the origina text for this workbook and Brian Rush (Canada) edited the workbook series in earier stages. JoAnne Epping- Jordan (Switzerand) wrote further text modifications and edited the workbook series in ater stages. Munira Laji (WHO, Substance Abuse Department) and Jennifer Hiebrand (WHO, Substance Abuse Department) aso edited the workbook series in ater stages. Maristea Monteiro (WHO, Substance Abuse Department) provided editoria input throughout the deveopment of this workbook. Workbook 6 Cent Satisfaction Evauations 3

4 Evauation of Psychoactive Substance Use Disorder Treatment

Tabe of contents Overview of workbook series 6 What is a cient satisfaction evauation? 7 Why do a cient satisfaction evauation? 7 How to do a cient satisfaction evauation? 8 Concusion and practica recomendation 17 Comments about case exampes 19 Case exampes of cient satisfaction evauation Part A: An evauation of satisfaction with a state drinker driver treatment program 21 Part B: Cient satisfaction with residentia substance treatment programmes 26 Part C: The case of community methadons treatment programs 32 Workbook 6 Cent Satisfaction Evauations 5

Overview of workbook series This workbook is part of a series intended to educate programme panners, managers, staff and other decision-makers about the evauation of services and systems for the treatment of psychoactive substance use disorders. The objective of the series is to enhance their capacity for carrying out evauation activities. The broader goa of the workbooks is to enhance treatment efficiency and cost-effectiveness using the information that comes from these evauation activities. This workbook discusses the assessment of cient satisfaction. It focuses on: reasons for assessing cient satisfaction the use of cient satisfaction measures for programme improvement measures of cient satisfaction Introductory Workbook Framework Workbook Foundation Workbooks Workbook 1: Panning Evauations Workbook 2: Impementing Evauations Speciaised Workbooks Workbook 3: Needs Assessment Evauations Workbook 4: Process Evauations Workbook 5: Cost Evauations Workbook 6: Cient Satisfaction Evauations Workbook 7: Outcome Evauations Workbook 8: Economic Evauations 6 Evauation of Psychoactive Substance Use Disorder Treatment

What is a cient WHO/MSD/MSB 00.2g satisfaction evauation? Cient satisfaction evauations are an exceent opportunity to invove cients or patients in the process of evauating your programme. Cient satisfaction evauations can address 1. the reiabiity of services, or the assurance that services are provided in a consistent and dependabe manner; waiting times for service components frequency of appointments time spent with counseor 2. the responsiveness of services or the wiingness of providers to meet cients/customer needs; 3. the courtesy of providers; and 4. the security of services, incuding the security of records. Specific questions may assess cients views about : the physica setting of services the hepfuness of support staff information resources the competence of counseors the costs of service the reevance of services to their needs the accessibiity of services the humanness of services the effectiveness of services in ameiorating their probems Cient satisfaction occupies an intermediate step in estabishing a heathy cuture for evauation within a programme or a setting. It often foows process evauation and cost anaysis, and precedes outcome and economic evauations. Accordingy, measures of cient satisfaction ie somewhere between process and outcome measures. When the concern is with the extent to which cients are satisfied with the context, processes, and perhaps the costs of a treatment service or network, the reevant measures of satisfaction can be viewed as process measures. However, when the concern is with the extent to which cients view the programme as having been hepfu in resoving their probems, cient satisfaction becomes a proxy outcome measure. Workbook 6 Cent Satisfaction Evauations 7

It is worth keeping in mind that satisfaction with the treatment processes, treatment compiance, and positive treatment outcomes are inter-reated. Cient satisfaction with treatment processes may both infuence, and be infuenced by, treatment outcomes. Cients who are not satisfied with a service may have worse outcomes than others because they miss more appointments, eave against advice or fai to foow through on treatment pans. On the other hand, cients who do not do we after treatment may have ess than favourabe attitudes towards a treatment service, even if it was of high quaity by other criteria. In practice, these mutua infuences may be difficut to disentange. It is worth keeping in mind that satisfaction with the treatment processes, treatment compiance, and positive treatment outcomes are inter-reated. Ratings of different dimensions of satisfaction have been highy correated in some studies, and scores on these dimensions have been added to yied overa satisfaction ratings. However, responses to specific items are of interest to service providers who want to find out how a particuar aspect of the service coud be improved. Cient satisfaction surveys may provide the ony means for cients to express concerns about the services received.... evidence of positive cient satisfaction is not, in itsef, sufficient to estabish the effectiveness of treatment. Why do a cient satisfaction evauation? The assessment of cient satisfaction adds an important consumer perspective to evauations of PSU treatment services and systems. Cient satisfaction evauations can be viewed as an opportunity to consut with cients about their experiences in your programme. Cient satisfaction surveys may provide the ony means for cients to express concerns about the services received, and to express their views about new services that are needed. Cient satisfaction ratings have been criticised as indicators of the quaity of human services because they may refect unreaistic expectations. Whie this criticism may be vaid in some instances, re- search with cients of menta heath services suggests that they can effectivey discriminate between services that are different in quaity (Lebour, 1983; Sheppard, 1993). It is, however, important to recognise that evidence of positive cient satisfaction is not, in itsef, sufficient to estabish the effectiveness or accessibiity of treatment. Cients with no base for comparison may be satisfied with services that are ineffective as determined by more objective outcome evauations. On the other hand, cients may be dispeased with services that achieve the objective of reducing their PSU but empoy rigid or authoritarian approaches. 8 Evauation of Psychoactive Substance Use Disorder Treatment

How to do a cient satisfaction evauation? The most common method for assessing cient satisfaction is with sef-administered questionnaires. These may be given to cients as they enter or eave services, or at various times in between. They can aso be administered at some point after treatment has been competed, when the outcomes of treatment are more cear to the cient. Cient satisfaction questionnaires can be competed at the time they are distributed, or at a ater date seected by the cient or program personne. Stamped, return enveopes can be provided if questionnaires are to be returned by mai. Satisfaction questionnaires aso can be maied to former cients with stamped, return enveopes. A cover etter shoud expain why the questions are being asked and how the information wi be used. The cover etter shoud aso indicate if individua repies wi be considered confidentia or anonymous, and what steps wi be taken to ensure that this is the case. For ethica reasons, risks to cients shoud be made cear. It shoud be stated that their responses wi not in any way affect present or future treatment. Programme managers typicay want the questionnaire to identify the respondent so that they can foow-up with these individuas who express concerns about the services received. If this is the case, cear provisions for confidentiaity must be made, incuding, for exampe, remova of the information identifying the cient prior to data anaysis by computer or other tabuar means. For more information about these ethica issues, see Workbook 2 of this series, Step 1A, entited Manage Ethica Issues. Cient satisfaction aso can be assessed in face-to-face or teephone interviews or focus groups. These strategies are more expensive than sef-competed questionnaires. If interviews or focus groups are used, it is preferabe to have them conducted by someone who is not connected directy with the service. This may be an independent evauator, vounteers or former cients themseves trained to take on this roe. If interviews or focus groups must be done by a manager or staff member, it is best not to have the individua s principa therapist ask about cient satisfaction because cients may be reuctant to comment negativey about their treatment directy to their therapist. Interviews may be highy structured, perhaps guiding the cient through the same type of questionnaire used on a sef-administered basis in other situations. Other interviews, and certainy focus groups, wi be much ess structured and the resuting information wi be anaysed quaitativey. Workbook 1 provides guidance for conducting focus group and semi-structured/unstructured interviews. Workbook 2 offers advice on anaysing the resuting information. The design and conduct of cient satisfaction surveys Cient satisfaction surveys are most usefu when they are designed to meet specific objectives and when they use appropriate methods and measures. This Workbook 6 Cent Satisfaction Evauations invoves choices of samping procedures, timing, cutura acceptabiity, and sensitivity of the questions to various eves of satisfaction. 9

There are no right or wrong ways to choose sampes in cient satisfaction surveys. However, it is important that your sampe be consistent with the evauation objectives. Choosing sampes of cients Your strategy for seecting cients for a satisfaction survey can infuence the kinds of resuts you obtain. If the surveys are imited to cients who compete treatment, the resuts wi probaby differ from those obtained in surveys that incude peope who have dropped out of the programme. There are no right or wrong ways to choose sampes in cient satisfaction surveys. However, it is important that your sampe be consistent with the evauation objectives. If the objective is to earn about cient satisfaction among those who compete treatment then there wi be no need to invove treatment drop-outs. However, if the aim is to find how, in genera, cients fee about the programme, a representative sampe of a cients competing the intake process woud be more appropriate. Regardess of the sampe chosen, you must be sure to ceary describe the sampe in subsequent reports. Limitations to the generaizabiity of resuts must be stated. For exampe, are your resuts biased due to the excusion of eary drop outs? Once you have decided which types of cients wi be invoved in satisfaction surveys, you have a number of options for choosing particuar cients, incuding a random or systematic sampe. These and other options for samping are discussed in Workbook 2. There is no best timing for these surveays, except to ensure consistency with the objectives of the evauation. Choosing sampes of cients The timing of cient satisfaction surveys can infuence your resuts. Cients with positive views during or immediatey foowing treatment may change their minds if they ater reapse. On the other hand, cients may gain a greater appreciation of services as their vaue becomes evident in an increasing number of rea ife situations. There is no best timing for these surveys, except to ensure consistency with the objectives of the evauation. If the objective is to find out what cients fee at the time of discharge, then ask cients to compete a satisfaction questionnaire as they are about to eave. However, if the aim is to find out if cients are satisfied as part of an outcome evauation, wait unti some period of time has passed before asking former cients to compete a satisfaction questionnaire. The timing of surveys shoud be ceary indicated in reports and any associated biases shoud be discussed. If, for exampe, cients compete satisfaction questionnaires foowing an emotiona graduation ceremony this coud bias attitudes in favour of the programme. Cuture sensituvity Cutures differ with respect to expectations of feedback on pubic and private services. In jurisdictions where consumerism is firmy estabished, frank verba or written feedback may be freey given. However, direct negative feedback in some cutures may be considered impoite and compaints may ony be shared with intimate acquaintances. Direct and chaenging questions aso may be cutur- ay inappropriate (NIDA, 1993). Experiences with (and attitudes toward) the use of questionnaires, interviews, focus groups and other methods of inquiry aso differ between cutures. Methods for soiciting cient feedback must take into account the prevaiing cutura norms and seek to ensure the use of appropriate methods that assess cient beiefs and opinions. 10 Evauation of Psychoactive Substance Use Disorder Treatment

Measures vaidated in one cuture may not be appropriate in others. Simpe transation of questionnaire items does not guarantee that the items wi have the same meaning across cutures (Attkisson and Greenfied, 1994). Considerabe effort may be required to generate new, cuturay appropriate questions. Cients, or peope advocating on their behaf, shoud be invoved in this process of questionnaire design to ensure that the measures wi provide a vaid indication of cient satisfaction. Some groups of cients may aso find particuar methods for assessing cient satisfaction more acceptabe than others. For exampe, those with poor cognitive or reading skis may prefer persona interviews over a written questionnaire. However, cients who are shy or have ow sef-esteem may prefer questionnaires over interviews. Sensitivity to different eves of satisfaction...if satisfaction is rated on a five-point scae, the proportions of cients who are very satisfied, somewhat satisfied or neutra can be better discriminated... Many satisfaction surveys of cients of heath and socia services have shown high eves of satisfaction party because they have used insensitive measures (Ruggeri, 1994). An exampe woud be using questionnaire items that ony have two response options (satisfied/not satisfied). Such items tend to invite a satis- fied response, even from those who are neutra of even midy dissatisfied. However, if satisfaction is rated on a five-point scae, the proportions of cients who are very satisfied, somewhat satisfied or neutra can be better discriminated, as can the proportions who are somewhat or very unsatisfied. Seeking out expressions of dissatisfaction Cients... may fee especiay obiged to show that they are gratefu and satisfied with the services provided. Cients of human service agencies have a tendency to be gratefu for the attention they receive, and to be reuctant to criticise in the event that this eads to negative consequences. Cients with ow sef-esteem, or who are conscious of status differences between themseves and service providers, may fee especiay obiged to show that they are gratefu and satisfied with the services provided. These tendencies can be overcome if cients are assured that their honest feedback is being sought and that there wi be no consequences for those who criticise the services in question. This can be made cear in verba or written instructions for competing satisfaction questionnaires or participating in interviews or focus groups. Confidentiaity of the resuts shoud be assured as strongy as possibe. It is desirabe to activey seek out sources of discontentment by asking the foowing kinds of questions: Are there any parts of the programme that you iked more than others? Have you any suggestions for ways in which the programme can be improved? Aso, ook for behavioura indicators of dissatisfaction, for exampe, high drop-out or no-show rates within specific programmes, or for specific counseors. Whie many factors may contribute to ow participation, ow cient satisfaction may be invoved. Workbook 6 Cent Satisfaction Evauations 11

Estabished questionnaires for assessing cient satisfaction When choosing a questionnaire for your evauation, you first need to consider whether a dimensions of cient satisfaction are reevant to the service components being evauated. It is often the case that one treatment agency provides different types of services and activities. You wi have to decide whether your cient satisfaction questionnaire wi provide feedback about individua service components, or whether you wi focus on a more goba eve of programme participation. This wi be an issue to resove in the assessment of satisfaction with services received across a arge network of agencies. If the intention is to use the resuting information to suggest highy specific areas for service or system enhancement, you may need to customise your seection of cient satisfaction measures to fit particuar service or system components. This may utimatey invove a choice between a standardised, goba measure of satisfaction avaiabe from pubished iterature (see beow), and questionnaires taiored to your specific information needs. If you are going to use a structured, sefadministered questionnaire, you may seect one from the pubished iterature. Such measures in the pubic domain wi ikey have data avaiabe on reiabiity and vaidity in a particuar setting. This is a big advantage, but must be considered in ight of cutura variations between the cuture in which the questionnaire was vaidated and the cuture in which you intend to use it now. In addition, standardised questionnaires may be too genera to give you the kind of detaied feedback you need for making improvements to specific parts of the program. Feedback unique to your program can be derived from a speciay-taiored questionnaire, athough issues of reiabiity and vaidity wi be of concern. Openended questions can aso be added to a sef- administered questionnaire and then anaysed quaitativey. A questionnaire which can be used to assess cient satisfaction is the Cient Satisfaction Questionnaire (CSQ-8). This is a widey used instrument with pubished data on reiabiity and vaidity (Greenfied and Attkisson, 1989). The instrument is avaiabe in severa anguages, incuding Engish, Spanish, Dutch and French (de Brey, 1983; Roberts et a., 1984; Sabourin et a., 1987). Case exampes of evauations that used the CSQ-8 aso are reported at the end of this workbook. Workbook 1, Appendix 2 aso contains four other exampes of questionnaires that can be used to assess cient satisfaction. There are no data on the reiabiity and vaidity of these other instruments. However, they may be hepfu in your situation or stimuate ideas for the deveopment of a questionnaire unique to your needs. A report from the Nationa Institute on Drug Abuse (1993) entited How Good is Your Drug Abuse Treatment Program? contains a series of cient satisfaction questions used in an AIDS Risk Reduction Project. Two other measures of cient satisfaction appropriate for PSU services are the Service Satisfaction Scae (SSS-30) (Attkisson and Greenfied, 1984), and the Verona Service Satisfaction Scae (VSSS) (Tansea, 1991). The SSS-30 is a 30-item muti-dimensiona scae deveoped on the basis of experience with the Cient Satisfaction Scae. The first case exampe at the end of this workbook (Part A: by Thomas Greenfied) describes the SSS-30 in greater detai. The VSSS is an 82-item scae which covers seven dimensions overa satisfaction, professiona skis and behaviours, information, access, efficacy of interventions and reative improvement. 12 Evauation of Psychoactive Substance Use Disorder Treatment

Deveoping your own cient satisfaction questionnaires A good starting pace for the deveopment of a new cient satisfaction questionnaire that is taiored to your individua service or treatment system wi be your programme ogic mode and accompanying written descriptions of your programme (see Workbook 1). These wi identify the main components, activities and treatment processes for which cient satisfaction ratings coud be deveoped. In addition, cients coud be asked to rate their satisfaction with the staff, comprehensiveness of the services provided and aspects of the physica environment. It woud aso be usefu to convene sma groups of current and former cients to expore issues most reevant to their needs. These groups may be hepfu in testing ideas for questionnaire items and response options. The instruments contained in Workbook 1, Appendix 2 aso wi be usefu. To hep vaidate measures of cient satisfaction, the ratings can be compared with verba reports or satisfaction ratings from famiy members or others that are famiiar with the services received. You can aso compare the resuts using your new questionnaire with the resuts of an instrument ike the CSQ-8 competed by the same peope. Cient satisfaction ratings can aso be compared with actua behaviours that signify satisfaction with services. Comparisons coud be made, for exampe, between cient satisfaction ratings and their record of keeping appointments, competing treatment, or returning for further treatment foowing a reapse. Using cient satisfaction measures during times of change in service deivery Once reiabe cient satisfaction measures are avaiabe, they can be used for routine or periodic check-ups on the quaity of services from the cients perspective. They aso can be used to assess cient reactions to changes in service deivery being impemented. For exampe, changes may be panned to increase the efficiency of a service but there are concerns that these coud ead to decreased cient satisfaction. Measures of satisfaction taken before or after the changes are introduced wi show if this has been the case. [Cients] coud be asked if they fee satisfied with the information that is avaiabe on the range of services in the community. Measuring cient satisfaction in evauations across two or more agencies It is possibe to assess cient satisfaction with services received across a network of programmes, rather than focusing on the cient s experience with ony one service provider. Not a cients wi have experience with other services in the treatment network. However, cients of a services may have usefu perspectives on system-wide issues. For exampe, they coud be asked if they fee satisfied with the information that is avaiabe on the range of services in the community. They coud aso be asked to rate their satisfaction with recommendations for referra given the options that were presented. Did they ike this referra? Was it too far away for them? Do they fee satisfied with being referred to a residentia service when they Workbook 6 Cent Satisfaction Evauations 13

might have gone to a day treatment or outpatient service (and vice versa)? A review of a ogic mode for the treatment system may suggest other topics to be incuded in cient satisfaction surveys. For exampe, waiting times for moving from one service component to another may be of particuar concern. The issue of dupication of services is aso important to expore, for exampe, whether agencies in the system dupicate the coection of assessment information when the cient moves from one service to another. Cients who have experienced two or more services in a network may have vauabe perspectives on the degree to which these services are co-ordinated. Sampe questions concerning cient satisfaction with inter-service co-ordination are: How satisfied are you with the way that (name of both services) exchanged treatment information about your probems? How satisfied are you with the information that (name of both services) provided to you about each other s treatment programmes? How satisfied are you with the ways the treatment staff of (name of both services) worked together to hep you with your probems? Based on your experience, how we do (name of both services) work together? It s your turn Put the information from this workbook to use for your own organisation or treatment network. Compete these exercises beow. Remember to use the information from Workbooks 1 and 2 to hep you compete an evauation pan. Review that information now, if you have not aready done so. Exercise 1 Think about your treatment programme or oca treatment network. List five genera areas in which you want to know the views of cients or patients. Exercise 2 Using the information provided in this workbook about how to design and conduct a cient satisfaction evauation, make the foowing decisions: Exampe: What do cients think about the hepfuness of our cinicians? 1) 2) 3) 4) 5) Decide what modaity you wi use to coect the data (questionnaires, interviews, focus groups) Choose a samping procedure for choosing cients to survey Decide the timing of the evauation 14 Evauation of Psychoactive Substance Use Disorder Treatment

Deveop a procedure for ensuring cients confidentiaity and promoting their honesty in answering questions Decide who wi hep you administer the questionnaires/interviews/focus groups. Exampe (from above): Data wi be coected using a sef-report questionnaire. Pease hep us improve our programme by answering some questions about the services you have received. We are interested in your honest opinion, whether it is positive or negative. To ensure your confidentiaity, pease do not write your name on this form. When you are finished, pace the form in the enveope (provided) and sea it cosed, then pace it in the coection box in the waiting area. A cients checking in for appointments during the week of December 10th wi be handed the survey to compete whie waiting for their appointments. Data wi be coected over a one week period of time ony, from 10-15 December. Cients wi be given enveopes in which to pace their competed questionnaires before returning them to the coection box. The foowing statement wi appear at the top of the questionnaire: Because the questionnaire assesses cient satisfaction with staff, it is not feasibe for staff to be invoved with distributing or coecting questionnaires. A outside research assistant wi be hired to hand out the introductory etters, consent forms, and questionnaires. The assistant aso wi remove the questionnaires from the coection box and keep them in a safe pace to ensure their confidentiaity from the staff. Now it s your turn. Foow the same procedure for your evauation questions. Exercise 3 You wi need to prepare an introductory etter and consent form that expains the purpose of your study. Review Section 1A of Workbook 2, entited Manage Ethica Issues, for more information about the important topic of participants rights in evauation research. describes the purpose and methods of the study expains what they wi need to do if they participate expains that participation is vountary In genera, a participants shoud be asked permission ahead of time before being enroed in the study. When you do this, your shoud expain the purpose, nature, and time invoved in their participation. No person shoud be forced or coerced to participate in the study. A standard practice is to have each participant sign a consent form, which: Note that an ethica committee may waive the requirement of a signed consent form if the research contains minima risk. In these cases, researchers sti need to provide fu information to participants. A consent form is incuded in the foowing exampe for the sake of competeness. Exampe (from above): Workbook 6 Cent Satisfaction Evauations 15

Introductory Letter: We are asking your hep in improving our programme by fiing out a 2 page questionnaire about the services you have received here. The questions wi ask about your views regarding our staff members. They wi take about 10 minutes to compete. A information that you provide us wi remain stricty private and confidentia. If you agree to participate, pease read and sign the consent form (attached) and return it to the research assistant who gave you this packet when you arrived. Thank you for your time. Sincerey, Dr. X Director, Treatment Programme Consent Form You agree to participate in a cient survey of satisfaction with our staff. You wi compete a 2 page questionnaire today, which wi take about 10 minutes to compete. Your participation is competey vountary. You can refuse to answer any questions and/or withdraw from the study at any time without a probem to you or your treatment here. A your responses wi remain stricty confidentia: programme staff members wi not have access to your responses, your name wi not appear on your questionnaire, and your responses wi not be inked to your identity at any time. I have read the information above and agree to participate. Signature: Date: Now it s your turn. Using the exampe above, and the additiona information provided in Workbook 2, section 1A, write your own introductory etter and consent form. Exercise 4 Run a piot test of your evauation measurement and procedures to ensure that everything runs smoothy. Review section IC of Workbook 2 entited Conduct a Piot Test for specific information about how to do this. In genera, piot tests assess these questions: Do the questions provide usefu information? Can the questions be administered propery? For exampe, is it too ong or too compicated to be fied out propery? Exampe (from above): A piot test wi be run during one cinic day: 3 November. During this day, 10-15 patients checking in wi be given the questionnaire. Afterwards, their responses wi be examined to determine whether they seemed to understand the questions and were answering honesty. A persons invoved with distributing the forms and compiing the data wi be interviewed to determine their views on any improvements that coud be made in the process and/or to the forms. Can the information be easiy managed by peope responsibe for compiing the data? Now it s your turn. Write down how you wi piot test your evauation study. Don t forget to review Workbook 2 first! Does other information need to be coected? 16 Evauation of Psychoactive Substance Use Disorder Treatment

Concusion and practica recommendation In this workbook, we have outined the basic principes and practices in the evauation of cient satisfaction with PSU services and systems. After competing your evauation, you want to ensure that your resuts are put to practica use. One way is to report your resuts in written form (described in Workbook 2, Step 4). It is equay important, however, to expore what the resuts mean for your programme. Do changes need to happen? If so, what is the best way to accompish this? Return to the expected user(s) of the evauation with specific recommendations based on your resuts. List your recommendations, ink them ogicay to your resuts, and suggest a period for impementation of changes. The exampes beow iustrate how to manage two different kinds of resuts using this technique. Unfavourabe findings Based on the finding that over 1/4 of cients were very dissatisfied or somewhat dissatisfied with the friendiness of the cinica staff, we recommend that the programme institute a 2 hour cient satisfaction training workshop for a cinicians to attend. The workshop coud happen in March, which traditionay is a ow-census month for the programme, and be run by Dr. Z, who is we-iked and respected by the staff. Favourabe findings The resuts indicate that cients are very satisfied overa with the hepfuness of the cinica staff. Therefore, we recommend that the composition of the cinica staff remain unchanged, and that these favourabe findings are pubicy acknowedged at the next programme-wide staff meeting. Remember, cients provide an invauabe perspective on the success of your programme. It is important to use the information that they provide to improve treatment services. Through carefu examination of your resuts, you can deveop hepfu recommendations for your programme. In this way, you can take important steps to create a heathy cuture for evauation within your organisation. Workbook 6 Cent Satisfaction Evauations 17

References Addiction Research Foundation. A Directory of Outcome Measures. Toronto: Addiction Research Foundation, undated. Attkisson, C.C., & Greenfied, T.K. The Cient Satisfaction Questionnaire-8 and the Service Satisfaction Questionnaire-30. In: Maruish, M. (ed.). Psychoogica testing: treatment panning and outcome assessment. Lawrence Erbaum Associates: San Francisco, 1994: 402-420. de Brey, H. A cross-nationa vaidation of the Cient Satisfaction Questionnaire: The Dutch experience. Evauation and Programme Panning, 1983, 6: 395-400. Greenfied, T.K., & Attkisson, C.C. Steps toward a mutifactoria satisfaction scae for primary care and menta heath services. Evauation and Programme Panning, 1989, 12:271-278. Kurtz, L.F. Measuring member satisfaction with a sef-hep association. Evauation and Programme Panning, 1990, 13: 119-124. Larsen, D.L. Enhancing cient utiization of community heath services. Dissertation Abstracts Internationa, 39, 4041B. (University Microfims No. 7904220), 1979. Larsen, D.L., Attkisson, C.C., Hargreaves, W.A., & Nguyen, T.D. Assessment of cient/ patient satisfaction: Deveopment of a genera scae. Evauation and Programme Panning, 1979, 2: 197-207. Lebour, J.L. Simiarities and differences between menta heath and heath care evauation studies assessing consumer satisfaction. Evauation and Programme Panning, 1983, 6: 237-245. Lettieri, D.J., Neson, J.E., & Sayers, M.A. (Eds.) NIAAA treatment handbook series: Acohoism treatment assessment research instruments. Rockvie, MA: Nationa Institute on Acoho Abuse and Acohoism, 1985. Levois, M., Nguyen, T.D., & Attkisson, C.C. Artifact in cient satisfaction assessment: Experience in community menta heath settings.evauation and Program- me Panning, 1981, 4: 139-150. Moos, R.H., & Finney, J.W. Acohoism programme evauations: The treatment domain. In D.J. Lettieri (Ed.), Research strategies in acohoism treatment assessment (pp.31-51). New York: Haworth Press, 1988. Nationa Institute on Drug Abuse. How good is your drug abuse program? Resource Manua. Nationa Institute on Drug Abuse, Rockvie, MD, 1993. Nguyen, T.D., Attkisson, C.C., & Stegner, B.L. Assessment of patient satisfaction: Deveopment and refinement of a Service Evauation Questionnaire. Evauation and Programme Panning, 1983, 6: 299-314. Roberts, R., Attkisson, C., & Mendias, R.M. Assessing the Cient Satisfaction Questionnaire in Engish and Spanish. Hispanic Journa of Behavioura Science, 1984, 6: 385-396. Ruggeri, M. Patients and reatives satisfaction with psychiatric services; the state of the art of its measurement. Socia Psychiatry Psychiatric Epidemioogy, 1994, 29: 212-227. Sabourin, S., Gendreau, P., & Frerette, L. Le neveau de satisfaction des cas d abandon dans un service universitaire de psychoogie. Canadian Journa of Behavioura Science, 1987, 19: 314-323. Sheppard, M. Cient satisfaction, extended intervention and interpersona skis in community menta heath. Journa of Advanced Nursing, 1993, 18: 246-259. Tansea, M. (ed.) Community-based psychiatry: Long-term patterns of care in South Verona. Psychoogica Medicine Suppement, 19, Cambridge University Press, Cambridge, 1993. Zwick, R.J. The effect of pretherapy orientation on cient knowedge about therapy, improvement in therapy, attendance patterns, and satisfaction with services. Masters Abstracts, 1982, 20: 307. (University Microfims No. 13-18082). 18 Evauation of Psychoactive Substance Use Disorder Treatment

Comments about case exampes Each of the foowing case exampes describes evauations comparing cient satisfaction across sites. The first evauation (Part A) presents a cient satisfaction evauation of a state-sponsored treatment, primariy for peope convicted of drunk driving. Satisfaction was measured using two scaes discussed in this workbook: the CSQ-8 and SSS-30. Whereas the CSQ-8 provides a singe satisfaction score, the SSS-30 assesses severa aspects of cient satisfaction. Resuts were used to guide site procedura improvements. The second evauation (Part B) is a good exampe of how cient satisfaction evauations can be competed with imited resources. In this case, a menta heath intern wanted to examine cient satisfaction across severa residentia PSU treatment programmes. With imited assistance, he was abe to pan and successfuy impement his evauation. Differences in satisfaction across sites were detected by the mutidimensiona SSS-30. The third evauation examined cient satisfaction across three community methadone treatment sites in Austraia. Evauators used the CSQ-8 and two quaitative, open-ended questions to assess satisfaction. They found significant differences in satisfaction across sites. These differences were given to cinic managers to make procedura improvements. Whie each of these cases generated usefu information about cient satisfaction, it is noteworthy that none attempted to provide information about cient outcome or treatment effectiveness. As described earier in this workbook, measurement of cient satisfaction is usefu yet distinct from measurement of cient outcome or treatment effectiveness. On occasion, cients can be satisfied with treatment that is ineffective in reducing PSU. On the other hand, certain treatments can be effective but unpopuar with cients. Evauators must remember that cient satisfaction and cient outcome are distinct evauation concepts. Each of the foowing case exampes describes evauations comparing cient satisfaction across sites. The first evauation (Part A) presents a cient satisfaction evauation of a state-sponsored treatment, primariy for peope convicted of drunk driving. Satisfaction was measured using two scaes discussed in this workbook: the CSQ-8 and SSS-30. Whereas the CSQ-8 provides a singe satisfaction score, the Workbook 6 Cent Satisfaction Evauations 19

SSS-30 assesses severa aspects of cient satisfaction. Resuts were used to guide site procedura improvements. The second evauation (Part B) is a good exampe of how cient satisfaction evauations can be competed with imited resources. In this case, a menta heath intern wanted to examine cient satisfaction across severa residentia PSU treatment programmes. With imited assistance, he was abe to pan and successfuy impement his evauation. Differences in satisfaction across sites were detected by the mutidimensiona SSS-30. The third evauation examined cient satisfaction across three community methadone treatment sites in Austraia. Evauators used the CSQ-8 and two quaitative, open-ended questions to assess satisfaction. They found significant differences in satisfaction across sites. These differences were given to cinic managers to make procedura improvements. Whie each of these cases generated usefu information about cient satisfaction, it is noteworthy that none attempted to provide information about cient outcome or treatment effectiveness. As described earier in this workbook, measurement of cient satisfaction is usefu yet distinct from measurement of cient outcome or treatment effectiveness. On occasion, cients can be satisfied with treatment that is ineffective in reducing PSU. On the other hand, certain treatments can be effective but unpopuar with cients. Evauators must remember that cient satisfaction and cient outcome are distinct evauation concepts. 20 Evauation of Psychoactive Substance Use Disorder Treatment

Case exampes of cient satisfaction evauations Part A: An evauation of satisfaction with a state drinker driver treatment program The author aone is responsibe for the views expressed in this case exampe. by Thomas K. Greenfied, Ph.D. Senior Scientist and Area Director for Popuation Surveys NIAAA Nationa Acoho Research Center Acoho Research Group 2000 Hearst Ave., Suite 300 Berkeey, Caifornia 94709-2130 USA Workbook 6 Cent Satisfaction Evauations Who is asking the questions and why do they want this information? Purposes for the cient satisfaction evauation There are a number of reasons why cients satisfaction with services is such a critica variabe in an overa substance abuse treatment outcome evauation effort. First, substance abuse treatment programme directors and managers are often required to justify their programs. They find consumer satisfaction is a concept readiy understood by their cients, the pubic, government bodies, or other funding agencies (Greenfied, 1983). This was the case for a private provider in a sma Eastern U.S. state icensed to provide a brief outpatient counseing service to the chemicay dependent cient. The programme is icensed by the state to provide its services in response to the Driving Under the Infuence (DUI) probem. It of course interacts cosey with courts, corrections, and the Department of Motor Vehices (DMV). Its managers knew that the state and invoved agencies woud need a positive response from its cientee, in addition to objective outcomes, for continued referras and reicensing. In addition, the managers wanted data on specific aspects of their programme so that improvement efforts might target areas of greatest concern to cients. The mutidimensiona SSS-30 questionnaire was seected for this reason based on prior experience in giving usefu feedback to student services, primary care, and EAP managers (Attkisson & Greenfied, 1994; Greenfied & Attkisson, 1989a). 21

Specific programme In the programme, referras are primariy (but not excusivey) individuas guity of a second DUI, and required by aw to receive treatment. The programme operates soey on cient fees totaing U.S. $495 at the time of the study. After two individua sessions, a treatment pan is deveoped with suitabe, eigibe cients. Eigibiity requirements incude (a) wiingness to expore drinking and drug taking, aowing for a norma degree of denia, (b) agreement to participate and remain sober and drug free, (c) commitment to be invoved and work toward reasonabe treatment goas, and (d) no overt psychiatric difficuties impying a primary menta heath probem. The programme invoves approximatey 25 contact hours, incuding four individua sessions, six educationay oriented group sessions, and eight 90 minute group sessions (under 16 members in each group). Additiona individua and famiy counseing may be incuded if needed. Conditiona driving privieges may be restored by the court after 16 hours. The programme has some coercive eements: cients unwiing to participate meaningfuy are deemed noncompiant, with paperwork indicating this sent back to the referring agency. The court must act to cear this up before the cient may again be enroed in the programme. Upon competion of the treatment, an aftercare pan is deveoped in the discharge session. The programme s offices are ocated in various counties and the evauation focused on three sites to assess their cients satisfaction. For a more compete description of the programme and the basis for cient seection, see Greenfied (1989) and Greenfied (1994). What resources were needed to coect and interpret the information? Questionnaires were handed out to cients upon arriva for their fina session by programme staff. Competed forms were coected daiy during the survey period. Thus, itte additiona effort was required for administration. A copier was used to dupicate the questionnaires so that the main resource needed was for data entry, accompished by office staff using the existing dbase software. This was the software package used for maintaining cient records (questionnaires were not identified, so no inkage to other cient data was possibe or attempted). Data entry required approximatey two minutes per questionnaire and was done by the office staff responsibe for the cient information system. Because the programme did not have anaysis software or capacity, the dbase fies were sent on diskette to the scaes authors for anaysis inspss. (Scoring keys for inagency use, and SPSS syntax for reading data from common spredsheet or reationa database fie formats, are avaiabe from the scae s firs author). At each of the two phases, anaysis time invoved about a day of work with an additiona day needed for report writing. How were the data coected? The two case studies use direct cient satisfaction measures, providing exampes of each of the two measurement strategies. Both questionnaires were designed to be broad enough to assess satisfaction with a range of human services incuding substance abuse treatment. The two are the Cient Satisfaction Questionnaire-8 (Nguyen, Attkisson & Stegner, 1983; 22 Evauation of Psychoactive Substance Use Disorder Treatment

Attkisson & Greenfied, 1994; Attkisson & Greenfied, 1995b), a widey-used, brief (8 item), genera satisfaction measure, used in the first case study, and the Service Satisfaction Scae-30 (Attkisson & Greenfied, 1994; Attkisson & Greenfied, 1995a, 1995b; Greenfied & Attkisson, 1989a) a 30-item mutidimensiona version with derivative forms avaiabe for case management and residentia settings (Greenfied et a., 1996) incuding a famiy member version (Greenfied & Attkisson, 1989b). Itaian (Ruggeri & Greenfied, 1995) and Spanish transations are avaiabe. There is evidence that these human service measures are suitabe for use in substance abuse treatment. The research done suggests good psychometric performance of these two measures (SSS-30 and CSQ-8) (Attkisson & Greenfied, 1994; Greenfied, 1989; Greenfied, 1994; Greenfied & Attkisson, 1989a). The scaes are both products of the University of Caifornia, San Francisco (UCSF), Department of Psychiatry s research programme on cient satisfaction which has extended over a quarter of a century (Attkisson & Greenfied, 1995b). Permission to use these copyright scaes may be obtained from Dr. C.C. Attkisson (CSQ-8) at the UCSF Graduate Division, 200 West Miberry Union, 513 Parnassus Avenue, San Francisco, Caifornia 94143-0404 USA (FAX+1-415-476-9690) and Dr. Greenfied (SSS-30) at the Acoho Research Group, 2000 Hearst Ave., Berkeey, Caifornia, 94709 USA (FAX+1-510-642-7175). Both scaes can be scored and simpy anaysed using common statistica programmes such as SPSS or EPI INFO, or spreadsheet or nationa data base management software such as Lotus 1-2-3, Exce, dbase, or Paradox, among other software programmes. If programming capabiity is not avaiabe in-hours, or more sophisticated anayses are needed, anaysis can be done by an evauator or anayst. SPSS scoring keys and syntax are avaiabe from the scae s first author (Thomas Greenfied, INTERNET: tgreenfied@arg.org). The CSQ has been incuded in a compendium of instruments assembed by the U.S. Nationa Institute on Acoho Abuse and Acohoism (NIAAA), the NIAAA Treatment Handbook Series 2: Acohoism Treatment Assessment Research Instruments (Attkisson, et a., 1985). Both the CSQ and the SSS have been incuded in Loyd Sederer and Barbara Dickey s Outcomes Assessment in Cinica Practice (Attkisson & Greenfied, 1995a) which incudes the scaes as appendices. Norms and psychometric resuts for the SSS-30 and CSQ-8 are avaiabe in a chapter in Mark Maruish s usefu book Psychoogica Testing: Treatment Panning and Outcome Assessment (Attkisson & Greenfied, 1994). A second edition of this book contains updated chapters on the CSQ-8 (Attkisson & Greenfied, in press) and the SSS-30 (Greenfied & Attkisson, in press). Because this was a reativey new appication to substance abuse treatment, the project invoved two phases. In the first, it was decided to conduct factor anayses of the SSS-30 data to confirm the factorbased scaes previousy deveoped in menta heath and primary care programmes (Attkisson & Greenfied, 1994; Greenfied & Attkisson, 1989a). In the second phase, data were coected from three programme sites in different ocations. In both phases, for vaidation purposes, the CSQ-8 scae was administered at the same time as the SSS-30. The dimensiona anayses and comparison of resuts with the two measures may be considered to be the methodoogica aims of the study. For practica reasons, it was ony possibe to obtain data for peope competing the programme. The two questionnaires were to be competed during the ast session and eft in a box at the door prior to departure. Questionnaires were fied out anonymousy. In Workbook 6 Cent Satisfaction Evauations 23

Tabe 1: Interna reiabiity of the four SSS-30 subscaes. SSS-30 Subscae Reiabiity coefficient (Cronbach Apha) Nº Items Substance Abuse Program Pubished norm groups* Practicioner manner and ski 9.83.89 Perceived outcome 8.83.83 Office procedures 5.74.74 Accessibiity 4.60.67 * Based on 3 Norm groups - Four heath Cinics, a Menta Heath Service, and an Empoyee Assistance Programme (see Attkisson & Greenfied, 1994). the first phase, demographics were not coected due to an oversight. In the second phase, the SSS-30 s standard demographics section was added incuding gender, age, income, ethnicity, distance from programme, and number of sessions attended. How were the data anaysed? Anayses were done using SPSS PC and SPSS for Windows7'. Data were read in from dbase7' fies provide by the programme (SPSS can read such fies directy). Reversed items were recoded 5=1, 4=2, 3=3, 2=4, 1=5 prior to anaysis and scoring, based on the pubished SSS-30 subscaes (scoring key and code avaiabe from T. K. Greenfied at the Acoho Research Group, 2000 Hearst Ave., Berkeey, Caifornia 94709, USA). In the first psychometric and confirmatory phase (n=1027), item descriptive anayses, factor anayses and reiabiity anayses (using SPSS factor and reiabiity routines) of the SSS-30 items were done, comparing simiarity of factor soutions using Harmon s (1970) coefficient of congruence cacuated using a simpe spreadsheet. The SSS-30 Tota Scae score was correated (SPSS correate) with the CSQ-8 score. These preiminary anayses heped assure that the scae functioned we in a substance abuse programme. In the second evauation phase (n=720), demographic profies of cients at each of the three programme sites were first compared. Overa satisfaction was then assessed by examining item and subscae distributions using the SPSS frequencies routine. Subscaes were again scored and these scores compared across the three programme sites using SPSS anova (Anaysis of Variance) routine. This aowed satisfaction across sites to be compared whie controing for gender differences, since men tend to be more wiing to indicate ower satisfaction than women (or are actuay ess satisfied). Finay, the CSQ-8 and SSS-30 tota scores were again correated. What did they find out? Phase 1 The two major SSS-30 factors found earier in menta heath and primary care sampes (Attkisson & Greenfied,1994) were confirmed. Practitioner Manner and Ski and Perceived Outcome factors were highy congruent with equivaent ones from earier studies (Harmon coefficients.88 -.93). These two standard factor-based subscaes were, therefore, usefu for assessing cient satisfaction in this substance 24 Evauation of Psychoactive Substance Use Disorder Treatment

abuse programme. There was aso some confirmation for the earier estabished Accessibiity and Office Procedures factors, so these subscaes too were constructed. Interna reiabiities for the four scaes were acceptabe (see Tabe 1). Finay, the SSS-30 Tota score and the CSQ-8 score correated.70, which provides some added vaidity to the newer scae. When a items are combined to assess genera satisfaction, the tota scae score may be used as a genera satisfaction measure. Substantive findings indicated that these substance abuse programme s court mandated treatment cients were quite satisfied with both their counseor s Manner and Ski (programme eve = 38.0 + 5.4 versus the norm of 38.4 + 5.0 in menta heath counseing) and a bit ess so for Perceived outcome (program eve = 29.6 + 5.6 versus the norm of 32.4 + 4.0 in menta heath counseing). Athough ower by haf a standard deviation, the mean satisfaction was high (mean-itemmean=4.1), equivaent to a Mosty Satisfied response. Item-eve resuts showed that many programme competers (41%) were dissatisfied ( Mosty Dissatisfied or Terribe responses) with cost. It wi be recaed that this coerced group of peope charged with driving whie intoxicated were required by the courts to pay for their substance abuse treatment programme. Faciity ocation and accessibiity were aso sources of dissatisfaction to 17% and it wi be recaed that many had their drivers icenses suspended. Otherwise few cients (under 10%) were dissatisfied with remaining programme aspects athough somewhat more were Mixed in their responses. It was important to demonstrate congruence between the factors found for substance abuse treatment and those in earier primary care and menta heath norm groups. Athough not unexpected, this comparative dimensiona anaysis confirmed the appropriateness of retaining the origina subscae composition, making comparison with findings and norms from other human services reaistic and appropriate. In addition, the correation between the widey used and we vaidated CSQ-8 genera satisfaction measure and the SSS- 30 composite scae ends construct vaidity to the newer instrument as a measure of cient satisfaction. The SSS-30 being a mutidimensiona measure adds to its vaue to programme managers who find its subscaes provide reevant feedback on programme strengths and weaknesses. In addition, its scaing resuts in ess skewed item distributions, eading to more normay distributed scae scores than with the CSQ- 8 (Greenfied & Attkisson, in press). It aso makes the use of this outcome measure as a dependent variabe in mutivariate anayses controing for demographics and other variabes more appropriate, increasing its sensitivity to differences in programme performance. Phase 2 Cientee were mosty mae (84%) with moda age 26-35 years od (46%). They were predominanty of Caucasian origin (77%) with African Americans making up 10%, typicay high schoo graduates (48%) with moda income US $20-40,000 (39%) and tended to ive 6-10 mies from the programme site. Some cientee differences were seen across sites with women under-represented at one of them. In the cross-site anayses, controing for gender (which was a significant predictor), Manner and Ski satisfaction did vary significanty (p<.05), though not strongy, between sites (together, gender and site accounted for ony 3% of the variance, so programme managers were cautioned not to over interpret the statisticay significant difference). For Perceived Outcome gender was again significant, but ony a trend toward significance (p=.06) was seen by site. Contrasting these minima differences, stronger differences were found for Workbook 6 Cent Satisfaction Evauations 25

Office Procedures subscae satisfaction scores. One site had markedy higher satisfaction with office personne, procedures, referras, coaboration between staff, and record handing, suggesting support staff were functioning we from the service consumers viewpoint. Accessibiity satisfaction, though not significanty different, favoured the same site. However, one of the other two sites had cients who ived further away. How were the resuts used? It is important to assure that data provided by cients are actuay used to improve services, so that the commitment to the cient to use her or his input for this purpose is carried through, justifying the sma burden of competing the measure. Resuts were provided to programme managers in a graphic form, showing overa subscae and item means, as we as cross site comparisons on the subscaes for men and women cient separatey. Subscae score distributions, shown as averaged item means, were given so that the reative number of dissatisfied of mixed responses coud readiy be seen. Managers can easiy share such resuts with staff. In this case, most of the feedback was positive, aowing for reinforcement of a job we done, much needed in substance abuse services where staff burnout tends to be high. In addition, positive office personne and procedures in the one site coud be identified and emuated across sites via cross-site training and seective procedura tune-ups. Resuts were aso used in presenting findings to referring agencies, the courts, Department of Motor Vehices and accreditation bodies. The methodoogica resuts from phase 1 were used to assure the managers and evauators that the Service Satisfaction Scae was a reiabe and vaid too for assessing satisfaction with substance abuse services. 26 Evauation of Psychoactive Substance Use Disorder Treatment

Part B: Cient satisfaction with residentia substance treatment programmes The author aone is responsibe for the views expressed in this case exampe. by Thomas K. Greenfied, Ph.D. Senior Scientist and Area Director for Popuation Surveys NIAAA Nationa Acoho Research Center Acoho Research Group 2000 Hearst Ave., Suite 300 Berkeey, Caifornia 94709-2130 USA Workbook 6 Cent Satisfaction Evauations Who is asking the questions and why do they want this information? An intern in a county s umbrea organisation responsibe for funding and supervising the management of pubicy funded drug and acoho treatment programmes had concerns about how to measure cient satisfaction given an earier experience with a genera satisfaction scae in the county s community menta heath centers (CMHCs). In the earier study, the vast majority of cients in a range of CMHCs, both those thought by the county to be exceent and those deemed weaker, had indicated mosty satisfied. The cient satisfaction measure was seen as insensitive. In fact, the Research Director had given up on satisfaction questionnaires as a means of obtaining vaid cient feedback (Nebeker, 1992, p.2). The intern knew of the newy deveoped, mutidimensiona scae, the SSS-30 (Greenfied & Attkisson, 1989a) and approached its first author to consut with him on its use in assessing cient satisfaction in residentia substance abuse probems. He had hopes that the mutidimensiona measure, unike goba measures that the county had given up on, might produce sufficient variation in the data to be abe to differentiate one faciity from another Nebeker (1992, p.1). The intern wanted to find a way to reduce Areactivity@ which Lebow (1983, 1983a; 1983b) has discussed as biasing satisfaction responses upward when therapists coect data, or questions are read to cients rather than answered by paper-and-penci. He wanted to achieve high response rates and potentiay study the effect of response rate on satisfaction eves. Substantivey, he wished to see if there were measurabe differences in satisfaction between sampes of active residents in four different residentia programmes. What resources were needed to coect and interpret the information? The intern reproduced the SSS-30 instrument himsef and served as the administrator. First, he secured county agreement for the piot study. Then he obtained agreement from programme directors to administer the scae himsef. He used the samping and administration method described in the next section which required a medium sized 27

cardboard box. Once obtained, he entered the data in a word processor, saving the fina fie as an ASCII text fie, which can be read by SPSS (or other statistica programmes). Lasty, he secured the services of the evauator to assist him with data anaysis using an IBM compatibe microcomputer version SPSS and wrote up the resuts as a Masters Thesis (Nebeker, 1992). Considerabe independent effort was required of him over the course of a year to accompish this research project. How were the data coected? The intern deveoped what he caed a Agroup momentum@ method of data coection which he described as foows: AThrough tria and error [in piot work] the foowing method emerged: (1) the staff caed a meeting [of residents] for the expicit purpose of fiing out the questionnaire; (2) I expained that participation in the study was anonymous and vountary; (3) the cients were instructed to pace the competed forms in a cardboard box with a sit cut into the top of the box; (4) I eft the room so that the cients fied out the questionnaire without the presence of staff or a test administrator; (5) I removed the box as soon as the ast cient had competed the questionnaire (Nebeker, 1992, pp.27-28). The intern comments further that AInstances where the cients fied out the questionnaire individuay, outside the group, produced a ower response rate@ (p.28). How were the data anaysed? Data were anaysed in the same way as described in Phase 2 of the previous case study. The standard subscae scores were computed and used to compare the mean scores for each subscae across the four residentia programmes, using ANOVA, with post-hoc tests to indicate the source of any difference if found. What did they find? Cient satisfaction with the programmes differed between faciities on some but not other subscaes. One dimension Office Procedures and Personne showed a significant overa difference between sites (F(3m 147)= 4.79; p<.01). In posthoc comparisons, one specific programme was found to differ from another on this dimension (Mean-itemmean 3.8 vs 3.3). Perceived Outcome showed an overa trend toward a difference (p=.11) and again post-hoc anaysis showed the same two residentia programmes differed (M = 3.8 vs 3.4) significanty (p<.05). In terms of Accessibiity, the same eve of trend toward overa difference was observed. This time the post-hoc pairwise comparisons showed a different programme provided highest satisfaction, statisticay (p<.05) higher (Mean-item-mean = 3.7) than the residence that had shown ower satisfaction on the other two subscaes as we (M = 3.3). Ony Coun- 28 Evauation of Psychoactive Substance Use Disorder Treatment

seor Manner and Ski showed no significant differences between the faciities. Sampes obtained during the piot phase at two faciities had ower response rates than those obtained at each using the Agroup momentum@ approach, which achieved 90-97% response rates. As hypothesised, the sampes invoving ow response rates (20-41%) showed higher satisfaction eves, significanty so for two subscaes: Counseor Manner and Ski (p<.01) and Accessibiity (p<.05). At one of the faciities the ow-responserate sampe gave a Manner and Ski mean vaue of 4.1, or.5 higher than the mean from the more compete sampe. An important finding from this sma study was the iustration of the fact that when insufficient efforts are made to assure the highest possibe response rates in a nonreactive cimate, or when data coection is haphazard and possiby eft in the hands of staff, resuts are ikey to be seriousy biased toward greater satisfaction. From an ethica view point, it is essentia for evauators, whether interna or externa, to hep assure a neutra setting for competion of questionnaires where there is minima programme staff invovement and infuence. In other studies that have achieved exceent response rates, a vounteer has served as the individua approaching waiting room cients in an open, friendy manner, expaining the purpose of the study and encouraging candid feedback as most usefu for improving the programme (Attkisson & Greenfied, 1994, Greenfied & Attkisson, 1989a). It is aso important to seect a scae that has the sensitivity needed to assess rea difference in degree of cient satisfaction (Greenfied & Attkisson, in press). Goba scaes in widespread use are brief and attractive to programme administrators, but sedom have the requisite psychometric quaities to aow genuine differences in satisfaction to be detected, except with extremey arge sampes, sometimes gathered in years of routine monitoring (Greenfied, 1983). For sma sampe studies a sensitive instrument is absoutey essentia. How were the resuts used? The resuts were provided to programme managers and to the county umbrea group s staff. Anecdota evidence suggests the county personne were not surprised by the differences observed which confirmed informa observations of the programme sites. However, the main resut of the study was to demonstrate the feasibiity of using a mutidimensiona satisfaction scae designed to have greater sensitivity than goba scaes in more widespread use, and to show the importance of obtaining high response rates for unbiased estimates of cient satisfaction, especiay so if the intent is crossprogramme comparison. Workbook 6 Cent Satisfaction Evauations 29

Strengths of the case study 1 It s your turn What are the strengths and the weaknesses of the presented case exampe? List three positive aspect and three negative aspects: 2 3 Weaknesses of the case study 1 2 3 30 Evauation of Psychoactive Substance Use Disorder Treatment

References for case exampe Parts A and B Workbook 6 Cent Satisfaction Evauations Attkisson, C.C., & Greenfied, T.K. The Cient Satisfaction Questionnaire-8 and the Service Satisfaction Scae-30. In: M.A. Maruish (Ed.). Psychoogica testing: Treatment panning and outcome Assessment. Hisdae, N.J.: Lawrence Erbaum Associates, 1994. Attkisson, C.C., & Greenfied, T.K. The Cient Satisfaction Questionnaire (CSQ) Scaes and the Service Satisfaction Scae- 30 (SSS-30). In: L.I a. D. Sederer (Eds.). Outcomes Assessment in Cinica Practice (pp. 120-127). Batimore, MD.: Wiiams & Wikins, 1995a. Attkisson, C.C., & Greenfied, T.K. The Cient Satisfaction Questionnaire (CSQ) Scaes: A history of scae deveopment and a guide for users. (Unpubished Report) Department of Psychiatry, University of Caifornia at San Francisco, 1995b. Attkisson, C.C., & Greenfied, T.K. The UCSF cient satisfaction scaes: I. The Cient Satisfaction Questionnaire-8. In: M.A. Maruish (Ed.). Psychoogica testing: Treatment panning and outcome Assessment (2nd ed.). Mahwah, N.J.: Lawrence Erbaum Associates, in press. Greenfied, T.K. The roe of cient satisfaction in evauating university counseing services. Evauation and Program Panning, 1983, 6: 315-327. Greenfied, T.K. Consumer satisfaction with the Deaware Drinking Driver Program in 1987-1988. Department of Psychiatry, University of Caifornia, 1989. Greenfied, T.K. Consumer satisfaction with the Deaware Drinking Driver Program in 1993. Berkeey, CA, Acoho Research Group, 1994. Greenfied, T.K., & Attkisson, C.C. Steps toward a mutifactoria satisfaction scae for primary care and menta heath services. Evauation and Program Panning, 1989, 12: 271-278. Greenfied, T.K., & Attkisson, C.C. Famiy satisfaction with services (Report to Northwest Residentia services, Inc.). University Caifornia, San Francisco, Department of Psychiatry, 1989b. Greenfied, T.K., & Attkisson, C.C. The UCSF Cient cient satisfaction scaes: II. The Service Satisfaction Scae-30. In: M.A. Maruish (Ed.). Psychoogica testing: Treatment panning and outcome Assessment (2nd ed.) Mahwah, N.J.: Lawrence Erbaum Associates, in press. Greenfied, T.K., Stoneking, B.C., & Sundby, E. Two community support program research demonstrations in Sacramento: Experiences of consumer staff as service providers. The Community Psychoogist, 1996, 29: 17-21. Harmon, H.H. Modern factor anaysis. Chicago: University of Chicago Press, 1970. Lebow, J.L. Research assessing consumer satisfaction with menta heath treatment: A review of findings. Evauation and Program Panning, 1983a, 6: 211-236. Lebow, J.L. Cient satisfaction with menta heath treatment. Evauation Review, 1983b, 7: 729-752. Nebeker, H. Cient satisfaction in four acoho and drug treatment programs. masters Thesis, John F. Kennedy University, 1992. Nguyen, T.D., Attkisson, C.C., & Stegner, B.L. Assessment of patient satisfaction: Deveopment and refinement of a Service Evauation Questionnaire. Evauation and Program Panning, 1983, 6: 299-313. Ruggeri, M., & Greenfied, T. The Itaian version of the Service Satisfaction Scae (SSS-30) adapted for community-based psychiatric patients: Deveopment, factor anaysis and appication. Evauation and Program Panning, 1995, 18: 191-202. 31

Part C: The case of community methadons treatment programs The author aone is responsibe for the views expressed in this case exampe. by Jeff Ward Division of Psychoogy Austraian Nationa University Canberra, ACT 0200 Austraia Who is asking the questions and why do they want this information? Three pubicy funded methadone maintenance cinics ocated in Sydney, Austraia participated in an evauation conducted by the Austraian Nationa Drug and Acoho Research Center (NDARC). The purpose of the study was to examine reationships between treatment received, cient characteristics, cient satisfaction and treatment outcome in terms of heroin use, crime and HIV risk-taking behaviour. In this case study, the outcome of interest is how satisfied cients are with the treatment they have been receiving and what, if any, variabes are reated to their eve of satisfaction. The questions were being asked by researchers at NDARC as part of a arger research effort into the cinica aspects of methadone maintenance treatment. Cient satisfaction with treatment was investigated, because it has become an important outcome of interest to poicy makers and treatment providers (Staard, 1996). Furthermore, previous research has found cient satisfaction to be associated with cient characteristics, service utiisation and better treatment outcome in other areas of heath care (Pascoe, 1983; Tanner 1981). What resources were needed to coect and interpret the data? As the data coected were to be used in statistica anayses, an interview questionnaire was required that preferaby woud provide a singe quantitative estimate of cient satisfaction. It was aso desirabe that the questionnaire be quick to administer and have estabished vaidity and reiabiity (i.e. that it has been demonstrated that the questionnaire measures what it caims to measure and that it does so consistenty with different popuations and over different situations). Such a questionnaire is the 8- item version of the Cient Satisfaction Questionnaire (CSQ-8; (Attkisson & Zwick, 1982). The questionnaire was incorporated in a much arger interview schedue that incuded questions and questionnaires assessing a range of other variabes reated to cients histories, current functioning and recent treatment experiences. Approximatey 350 interview schedues were printed and 348 cients attending the three methadone cinics were interviewed by trained interviewers from NDARC. 32 Evauation of Psychoactive Substance Use Disorder Treatment

How were the data coected? Trained interviewers from NDARC visited each of the three cinics and interviewed cients on-site for the evauation project. Cients were tod that the information coected woud not be communicated to cinica staff in any way that woud identify individuas. This was to ensure that they woud not modify their answers to questions in order to either pease staff or avoid retribution. The CSQ-8 was fied out by the cients themseves and took approximatey 5 minutes to compete. Each of the eight items that make up the CSQ-8 has five responses to choose from ranging from being very dissatisfied to very satisfied to which a score of between 0 and 4 is attached. A higher score indicates more satisfaction with treatment, and the eight scores are summed to yied a singe overa measure of satisfaction. The CSQ-8 score for each cient interviewed was entered into a computer using SPSS for Windows software. Scoring and data entry for the CSQ-8 took approximatey 6 hours. As we as the 8 individua items that make up the CSQ-8, there are two questions that aow for a more openended response. These two items ask respondents to compete the sentences: The thing I ike best about this agency is.. and If I coud change one thing about this agency, it woud be.. These items were scanned for consistent responses specific to the cinic concerned and recorded for feedback to the cinic staff. This took an additiona 10 hours. How were the data anaysed? The data was entered and anaysed using SPSS for Windows Version 6.0. When the questionnaires were scored, it was found that 6 of the cients had not competed the forms propery. The responses of these 6 cients were not incuded in the anaysis; this eft 342 scores on the CSQ-8 for the anaysis. The data anaysis proceeded through four steps, each designed to answer a different question. The four basic questions that informed this anaysis were: What form does the distribution of CSQ-8 scores take? Is there a difference in cient satisfaction at each of the three cinics? If the three cinics differ in eve of cient satisfaction, why and how do they differ? What aspects of the treatment program were identified as being in need of change in the cients responses to the open-ended questions? What form does the distribution of scores take? A preiminary step in anaysing the data was to inspect the distribution of the CSQ-8 scores to see if the pattern foowed that observed in other studies, where a majority of study participants indicate more, rather than ess, satisfaction with the treatment they receive (Staard, 1996). The distribution of the CSQ-8 scores for the study of the pubic methadone cinics is set out beow in Figure 1. Figure 1 is a histogram which represents graphicay the number of cients who returned each of the scores on the CSQ-8. As Figure 1 (on next page) shows, the shape of the distribution suggests that, as in previous studies, cients tended to express more, rather than ess, satisfaction with the treatment they received. Workbook 6 Cent Satisfaction Evauations 33

Figure 1 Distribution of CSQ-8 scores (N=342) t n u o C 60 50 40 30 20 10 0 9 1011121314151617181920212223242526272829303132 Do cients attending the three methadone cinics differ in their eve of satisfaction with treatment? Having determined that cients attending the three methadone cinics tended to be more rather than ess satisfied with the treatment they were receiving, an important subsequent question was whether there were any differences in satisfaction with treatment across the three cinics. In order to answer this question the means on the CSQ-8 for each of the three cinics were cacuated and can be found in Tabe 1. Tabe 1: Mean CSQ-8 scores for the three methadone cinics Cinic Mean CSQ-8 Score A 22.8 B 24.9 C 26.3 As can be seen from Tabe 1, the mean CSQ-8 scores for the cients attending the three methadone cinics were different from each other, with the cinic indicated by A having the owest score and the cinic indicated by C having the highest. To determine whether these differences were simpy due to chance or not, the next step was to subject them to a statistica test. The appropriate statistica test for assessing whether the differences in the scores on the CSQ-8 were due to chance or not is a oneway anaysis of variance (ANOVA). The F-ratio, which assesses the statistica significance of the ANOVA was found to be equa to 15.18, which was statisticay significant with p set at 0.05 (F = 15.18; df =2,340; p =.000). This means that the differences observed were not simpy due to chance. To determine which cinics differed from each other, the east significant difference test was empoyed with adjustment for mutipe tests. This reveaed that cients attending cinic C were more satisfied than those attending cinics B and A and that those attending cinic B were more satisfied than those attending cinic A. 34 Evauation of Psychoactive Substance Use Disorder Treatment

If the three cinics differ in eve of cient satisfaction, why and how do they differ? Having found out that there were statisticay significant differences between the three cinics in satisfaction with treatment, as measured by the CSQ-8, the question arose as to why these differences were observed. In order to answer this question and to examine whether the previousy observed reationships between age, service utiisation, treatment outcome and satisfaction woud be found with the methadone cients, a mutipe inear regression mode was deveoped. In a regression mode, an outcome (in this case the scores on the CSQ-8) is predicted by a set of variabes often referred to as predictor variabes. The mode aows us to estimate to what extent any given predictor variabe in the mode is reated to the outcome after taking into account the contribution of a of the other variabes in the mode. In this case study, a simpe mode is deveoped as an exampe. The procedure foowed in deveoping the mode is the one recommended by Keinbaum (Keinbaum, Kupper, & Muer, 1988). As noted at the beginning, cient characteristics and treatment outcome have been found in previous research to be reated to cient satisfaction. In this case study, we wi use gender and reported crime in the past month as exampes. Typicay, previous research has shown women to be more satisfied with heath care services than men (Pascoe, 1983; Tanner, 1981), whie one of the major outcomes expected of methadone maintenance treatment is that it wi reduce crime (Ward, Mattick, & Ha, 1994). The regression mode is set out beow in Tabe 2. In Tabe 2, variabes marked Cinic B and Cinic C are known as dummy variabes and indicate the extent of the reationship between these two cinics and the CSQ-8 when compared with Cinic A which is used as a reference category. The F statistic at the bottom of the tabe indicates that the mode, as a whoe, is associated with cient satisfaction as measured by the CSQ-8. Looking at the variabes in the mode, it can be seen that there appears to be no difference between men and women in their eve of satisfaction. Simiary, the outcome from treatment indicated by whether the cient reported committing a crime in the month prior to interview, is aso unreated to satisfaction with treatment. However, in the case of crime, the p vaue (.059) is cose to the significance eve (.05) and suggests a coser ook at this reationship Tabe 2: Mutipe regression mode for predicting cient satisfaction with methadone treatment Variabes in mode Regression coefficient Standard error t P (Constant) 22.83.58 39.36.000 Cinic B 3.59 3.59 5.60.000 Cinic C 2.24 2.24 3.45.001 Crime reported in past month -0.99-0.99-1.90.059 Gender (1=mae) 0.37 0.37 0.75.455 F=8.663, p=.000 Workbook 6 Cent Satisfaction Evauations 35

might be warranted. After adjusting for gender and crime, we find that there are sti significant differences between the three cinics. The meaning of the statisticay significant reationships for the cinics is that when compared with Cinic A as a reference category, cients attending both Cinic B and Cinic C are more satisfied with the treatment they have been receiving. What aspects of the treatment program were identified as being in need of change in the cients responses to the open-ended questions? The cients responses to the questions concerning what they iked and what they thought needed changing concerning their treatment were read and sorted into major thematic groups. There is insufficient space in this context to eaborate fuy on these responses. However, an exampe wi suffice. One of the main cient concerns were the restricted hours that the cinics were open for methadone dosing. By far the most common thing that cients woud change if they coud was the times at which the cinics were open. What did the study find out? In terms of the variabes seected for investigation in this case study, it has been shown that whie cients attending three pubic methadone cinics in Sydney, Austraia tend to be satisfied with the treatment they have been receiving, there are statisticay significant differences between the eve of satisfaction at these three cinics. Unike studies in other areas of heath care, gender and treatment outcome were not found to be reated to eve of satisfaction, athough the treatment outcome seected (crime) was very cose to being statisticay significant, suggesting that further investigation may be warranted to determine if and under what circumstances it may be reated to satisfaction with treatment. It is important to note, however, that this anaysis has intentionay been restricted to a sma number of variabes for the purposes of this cases study and that the reationships investigated may be different when paced in the context of the arger number of variabes incuded in the study. How where the resuts used? In the first instance, the resuts were communicated to staff at the participating cinics. As an exampe of the way in which the resuts were used by the cinic managers, one aspect of the survey s use by the manager of Cinic A wi be discussed. The manager of Cinic A, which had the east satisfied cientee was not surprised by the resuts. A recent change in the cinic s ocation and changes to staffing eves were thought to be the cause of the cient s unhappiness. This was refected in the cients answers to the open-ended questions at the end of the CSQ-8. The manager requested a copy of the data set which was made avaiabe so that future surveys conducted by cinic staff coud be compared with the resuts of this first survey. In this way, the manager woud be abe to assess whether changes that were being panned woud improve cients satisfaction with their methadone treatment. The resuts of the study wi aso be pubished in an appropriate journa and wi contribute to the scientific iterature on methadone maintenance cinics and on cient satisfaction in genera. 36 Evauation of Psychoactive Substance Use Disorder Treatment

Strengths of the case study 1 It s your turn What are the strengths and the weaknesses of the presented case exampe? List three positive aspect and three negative aspects: 2 3 Weaknesses of the case study 1 2 3 Workbook 6 Cent Satisfaction Evauations 37

References for case exampe Part C Attkisson, C. C., & Zwick, R. (1982). The cient satisfaction questionnaire: Psychometric properties and correations with service utiization and psychotherapy outcome. Evauation and Program Panning, 5, 233-237. Keinbaum, D. G., Kupper, L. L., & Muer, K. E. (1988). Appied regression anaysis and other mutivariabe methods. (Second ed.). Boston: PWS-Kent. Pascoe, G. C. (1983). Patient satisfaction in primary heath care: A iterature review and anaysis. Evauation and Program Panning, 6, 185-210. Staard, P. (1996). The roe and use of consumer satisfaction surveys in menta heath services. Journa of Menta Heath, 5, 333-348. Tanner, B. A. (1981). Factors infuencing cient satisfaction with menta heath services. Evauation and Program Panning, 4, 279-286. Ward, J., Mattick, R. P., & Ha, W. (1994). The effectiveness of methadone maintenance treatment: An overview. Drug and Acoho Review, 13, 327-336. 38 Evauation of Psychoactive Substance Use Disorder Treatment