Test retest reliability of self-reported drug treatment variables

Transcription

1 Journal of Substance Abuse Treatment 33 (2007) 7 11 Regular article Test retest reliability of self-reported drug treatment variables Jordan W. Edwards, (M.A.) a, Dennis G. Fisher, (Ph.D.) a, 4, Mark E. Johnson, (Ph.D.) b, Grace L. Reynolds, (D.P.A.) a, David P. Redpath, (M.A.) a a Center for Behavioral Research and Services, California State University, Long Beach, CA 90813, USA b Behavioral Health Research and Services, University of Alaska, Anchorage, AK, USA Received 15 July 2006; received in revised form 20 October 2006; accepted 24 November 2006 Abstract We conducted two studies to assess the test retest reliability of drug treatment questions in the Risk Behavior Assessment (RBA) and Risk Behavior Follow-Up Assessment (RBFA). In Study 1, 218 active drug users participated in an RBA 48-hour test retest reliability assessment; in Study 2, 257 active drug users participated in an RBFA 48-hour test retest reliability assessment. Results revealed acceptable reliability for the main drug treatment question in both the RBA (.85) and RBFA (.85): b a drug treatment or detoxification center?q Other more specific drug treatment questions yielded mixed results. Twenty-four of the 35 RBA and RBFA treatment variables (68.6%) that could be calculated displayed acceptable test retest reliability coefficients. Test retest reliability coefficients for the RBFA continuous variables could not be calculated owing to a lack of participants in drug treatment during the previous 6 months. Future research establishing greater delineation of this problem may aid in the construction of assessment instruments. D 2007 Elsevier Inc. All rights reserved. Keywords: Drug treatment; Self-report; Test retest reliability; Risk Behavior Assessment 1. Introduction Evaluating drug treatment programs remains to be an important and challenging process (Booth, Crowley, & Zhang, 1996; Hubbard et al., 1989; Zanis, McLellan, Belding, & Moyer, 1997). To remain cost-effective, many research studies rely on self-reported data. However, the test retest reliability of self-reported drug treatment experience has not been widely tested psychometrically. The design of most self-report instruments necessitates evaluating the reliability and validity of self-reported data on the variable level because such instruments are designed to capture self-reported discrete behaviors (Fisher, Reynolds, Jaffe, & Johnson, 2005). 4 Corresponding author. Center for Behavioral Research and Services, California State University, 1090 Atlantic Ave., Long Beach, CA 90813, USA. Tel.: x125; fax: address: dfisher@csulb.edu (D.G. Fisher). One notable exception is the study by McLellan, Alterman, Cacciola, Metzger, and O Brien (1992), which assessed the test retest reliability of the Treatment Services Review (TSR). The TSR is a 5-minute technician-administered interview that provides a quantitative profile of the number and types of treatment services received by patients during their alcohol and drug abuse rehabilitation. The TSR has been successfully used in gathering information from both drug and alcohol rehabilitation clients to describe the breadth and scope of treatment services available to them (Alterman & McLellan, 1993; Alterman et al., 2000). Test retest analyses using 20 participants over a 24-hour period indicated satisfactory reliability when the TSR was administered by phone or in person (French, Roebuck, McLellan, & Sindelar, 2000; McLellan et al., 1992). A similar psychometric study was conducted on the Teen TSR (Kaminer, Blitz, Burleson, & Sussman, 1998). The researchers conducted 24-hour test retest analyses on two groups of 20 participants in a partial hospital program and 24 participants from an outpatient clinic. Their /07/$ see front matter D 2007 Elsevier Inc. All rights reserved. doi: /j.jsat

2 8 J.W. Edwards et al. / Journal of Substance Abuse Treatment 33 (2007) 7 11 results revealed a high level of consistency in test retest reliability overall for the treatment variables. In addition, their research showed higher reliability coefficients for the outpatient group as compared with the partial hospital program group. Two instruments that have been widely used to collect information from drug-using individuals, including information on drug treatment history, are the Risk Behavior Assessment (RBA; National Institute on Drug Abuse [NIDA], 1993) and the Risk Behavior Follow-Up Assessment (RBFA; NIDA, 1992). Booth et al. (1996) conducted a 15-site study on client characteristics and community interventions on treatment entry. This led to the design and testing of free treatment coupons on methadone maintenance treatment entry (Booth, Corsi, & Mikulich, 2003). The RBA and the RBFA have been proven to be invaluable in these and other studies for collecting demographic, drug use, sexual, criminal, drug treatment, and work history data in community-based health education and research clinics for many years (Dowling-Guyer et al., 1994; Fisher et al., 1993; Fisher, Reynolds, Wood, & Johnson, 2004; Johnson et al., 2000; Johnson, Fisher, & Reynolds, 1999; Needle et al., 1995; NIDA, 1993; Weatherby et al., 1994). Many of these two instruments questions have been tested for their reliability and validity, including validity tests on the recent use of drugs and HIV status. Studies have found many of the RBA variables to be psychometrically sound (Dowling-Guyer et al., 1994; Fisher et al., in press; Weatherby et al., 1994). Using two samples of active street drug users, we sought to examine RBA and RBFA self-reported drug treatment variables for their test retest reliability. This analysis may establish which drug treatment questions are reliable enough to be used in future research. 2. Methods 2.1. Study Participants Participants were 219 out-of-treatment drug users recruited from 11 of the 23 sites of the NIDA Cooperative Agreement for AIDS Community-Based Outreach/Intervention Research Program. Recruitment for the test retest study occurred at the following sites: Anchorage, AK; Denver, CO; Detroit, MI; Houston, TX; Long Beach, CA; Miami, FL; New York, NY; Philadelphia, PA; Portland, OR; San Francisco, CA; and Tucson, AZ. Eligibility for participation in the study required that participants be at least 18 years old, understand English, have used illegal drugs in the last 30 days, and not be intoxicated or mentally impaired to the point that they cannot give informed consent and/ or respond to the interview. The demographic characteristics of the sample have been previously reported (Johnson et al., 1999) Measure The RBA (NIDA, 1991, 1993) was developed by the Community Research Branch of the NIDA as a collaborative effort with the AIDS cooperative agreement program grantees. It gathers demographic information, work and income information, drug use history and behavior data, HIV risk behavior data (including sexual and needle sharing information), incarceration history, and past drug treatment information. Most of the items on the instrument have been shown to have good reliability, and the self-reports of recent drug use and HIV status have been shown to have good validity (Dowling-Guyer et al., 1994; Fisher et al., 1993; Fisher et al., in press; Fisher et al., 2004; Johnson et al., 1999; Needle et al., 1995; Weatherby et al., 1994). The present research focused on the drug treatment variables. The initial question asks, bhave you ever in your lifetime been in a drug treatment or detoxification center?q For affirmative responses, the participants subsequently responded to each of the following types of drug treatment programs: methadone detoxification; methadone maintenance; outpatient drug-free treatment; residential treatment; prison/jail treatment; or other drug treatment/detoxification program. For each of these treatment programs, the RBA ascertains how many weeks in the participants lifetime they spent in each specific program, the most recent year they were enrolled in the program, and the number of days they had been in any program in the last 30 days. The next RBA item asks, bduring the last year, have you ever tried but been unable to get into a drug treatment or detoxification program?q For affirmative responses, participants subsequently responded to the question, bhow many times in the last year were you unable to get into a drug treatment or detoxification program?q; they were also asked to identify which of seven possible reasons explains why they were unable to receive treatment. The possible reasons were (1) did not qualify, (2) not enough money, (3) program did not have room, (4) program does not take women or those with children, (5) set appointment but did not follow through, (6) went to jail or other correctional facilities before program started, and (7) other reason Procedures Participants were recruited through targeted sampling plans constructed by each site (Watters & Biernacki, 1989). Outreach workers, flyers, recruiting agencies, and word of mouth were all tapped in recruiting participants. Participants were informed of the purpose of the study, conditions of eligibility, time of the retest 48 hours later, and assurance of confidentiality. After data about the research project were provided, the participants provided informed consent and all procedures were implemented in compliance with the institutional review boards at each data collection site. At Time 1, the participants were administered the RBA and their urine and blood samples were collected for drug and HIV testing. At Time 2, 48 hours later, a different staff readministered the RBA. Standardization of the interview

3 J.W. Edwards et al. / Journal of Substance Abuse Treatment 33 (2007) Table 1 Test retest reliability coefficients for RBA categorical drug treatment variables among the Study 1 sample (n = 218) Cohen s j (95% CI) a.85 (.78 to.91) drug treatment center? (EDT).82 (.71 to.92) methadone detoxification? (EDTA).76 (.62 to.89) methadone maintenance? (EDTB).76 (.61 to.91) outpatient drug-free treatment? (EDTC).82 (.72 to.92) a residential treatment center? (EDTD).71 (.54 to.88) a prison or jail treatment center? (EDTE).59 (.38 to.81) any other drug treatment center? (EDTF) During the last year, have you ever tried.67 (.54 to.81) but been unable to get into drug treatment or detoxification program? (ENODT) Did not qualify for drug treatment.78 (.37 to 1.0) program? (ENODTA) Unable to get treatment because of not.34 (.23 to.91) enough money? (ENODTB) Unable to get treatment because you did.66 (.30 to 1.0) not follow through with your program appointment? (ENODTE) Unable to get treatment for other.79 (.51 to 1.0) reasons? (ENODTG) technique and procedures of the RBA should have minimized differences between staff and sites. Data were analyzed to determine the concordance of responses from Time 1 to Time 2. Pearson correlation coefficients were used to assess continuous variables, and Cohen s j coefficients and 95% confidence intervals (CIs) were used to assess categorical variables (Cohen, 1960). All analyses were analyzed on SAS for Windows 9.1 (SAS Institute, Cary, NC) Study Participants Participants were 257 out-of-treatment drug users recruited from five NIDA Cooperative Agreement for AIDS Community-Based Outreach/Intervention Research Program sites: Anchorage, AK; Denver, CO; Detroit, MI; Houston, TX; and Long Beach, CA. The eligibility criteria used for this study were identical to those used in Study 1. The demographic characteristics of the sample have been previously reported (Johnson et al., 2000) Measure The RBFA (NIDA, 1993) was developed in the same manner as was the RBA as described for Study 1 and is used as an RBA follow-up assessment tool. Most of the non-drug treatment variables have previously been shown to have good reliability and validity (Johnson et al., 2000). The drug treatment variables on the RBFA are very similar to those on the RBA as described for Study 1. The main difference between the RBA and the RBFA is the period during which the treatment takes place. The RBA elicits the number of weeks of lifetime treatment, whereas the RBA elicits the number of days in treatment in the last 6 months. Another difference is that the RBFA asks the following additional question: bwhen was the last time (month and year) you were unable to get into a drug treatment or detoxification program?q Procedures Procedures and statistical analyses as described for Study 1 were used, with the RBFA being readministered within 48 hours instead of the RBA. 3. Results 3.1. Study 1: RBA Cohen s j coefficient for the first RBA drug treatment variable (b a drug treatment or detoxification center?q) was calculated to be.85, well above the.70 criterion that indicates acceptable reliability for research purposes (Nunnally & Bernstein, 1994). On the remaining categorical variables, 7 of 11 questions had at least met the.70 criterion for acceptable reliability. Table 1 shows Cohen s j coefficients and 95% Table 2 Test retest reliability coefficients for continuous RBA drug treatment variables among the Study 1 sample (n = 218) Pearson r (95% CI).33 (.00 to.59) methadone detoxification? (EDTWA).89 (.76 to.95) methadone maintenance? (EDTWB).84 (.61 to.93) outpatient drug-free treatment? (EDTWC).88 (.79 to.92) residential drug treatment? (EDTWD).96 (.86 to.98) prison or jail treatment centers? (EDTWE) What year was your last in methadone.93 (.86 to.96) detoxification? (EDTYRA) What year was your last in methadone.83 (.64 to.92) maintenance? (EDTYRB) What year was your last in outpatient.94 (.84 to.98) drug-free treatment? (EDTYRC) What year was your last in a prison or.92 (.76 to.97) jail treatment center? (EDTYRE) How many days were you in an outpatient.89 (.70 to.96) drug-free program during the last 30 days? (EDT30C) How many times have you tried but been.85 (.68 to.93) unable to get drug treatment during the last year? (ENODTT)

4 10 J.W. Edwards et al. / Journal of Substance Abuse Treatment 33 (2007) 7 11 Table 3 Test retest reliability coefficients for RBFA categorical treatment variables among the Study 2 sample (n = 257) Cohen s j (95% CI) a.85 (.79 to.92) drug treatment or detoxification center? (FEDT) a.69 (.56 to.82) methadone detoxification center? (FEDTA).73 (.60 to.86) methadone maintenance? (FEDTB) an.59 (.42 to.75) outpatient drug treatment center? (FEDTC).74 (.63 to.86) a residential treatment center? (FEDTD) a.30 (.05 to.66) prison or jail treatment center? (FEDTE).76 (.61 to.92) any other drug treatment or detoxification center? (FEDTF) Since we last interviewed you, have you ever.47 (.30 to.65) tried but been unable to get into a drug treatment or detoxification program? (FENODT) Have you been in methadone.84 (.62 to 1.0) detoxification in the last 6 months? (FEDT6MA) Have you been in methadone maintenance.91 (.74 to 1.0) in the last 6 months? (FEDT6MB) Have you been in an outpatient drug-free.69 (.29 to 1.0) program in the last 6 months? (FEDT6MC) Have you been in a residential treatment.69 (.44 to.94) center in the last 6 months? (FEDT6MD) CIs. An item was omitted if it had insufficient variance to calculate meaningful reliability results. Also included in the tables are the acronyms for each RBA variable to assist researchers interested in looking into the national cooperative agreement data set that is available from the University of Michigan Substance Abuse and Mental Health Data Archive ( Pearson correlation coefficients for most of the continuous drug treatment variables illustrated good reliability: 10 of the 11 calculated variables had reliability coefficients higher than.83, well above the.70 criterion (Nunnally & Bernstein, 1994). Table 2 shows the Pearson correlation coefficients for the continuous drug treatment variables Study 2: RBFA Cohen s j coefficient for the first RBFA drug treatment variable (b a drug treatment or detoxification center?q) was calculated to be.85, well above the criterion of.70. Six of the 12 categorical RBFA drug treatment variables were above the.70 criterion for good reliability, with only 3 of the nonsignificant variables missing the.70 cutoff with a j coefficient of at least.68. Table 3 shows Cohen s j coefficients and 95% CIs associated with the RBFA categorical drug treatment variables. Test retest reliability coefficients for the RBFA continuous variables could not be calculated owing to a lack of participants in drug treatment during the previous 6 months. 4. Discussion Drug treatment data are of interest to scientists for a variety of reasons, including program evaluation, clinical matching of best patient program fit, and assessment of the need for additional services (Adair, Craddock, Miller, & Turner, 1996). Self-reporting methods have obvious advantages in that they are economical, although the reliability of selfreporting has been criticized by some researchers (Ball, 1967). More recent research studies, including the current study, have found that closed-ended questions (e.g., bhave you ever in your lifetime been in drug treatment?q) have much better reliability as compared with open-ended questions (e.g., bhow many days have you been in methadone detoxification in the last 6 months?q). This finding is supported by Adair et al. (1996), who analyzed self-reported drug treatment variables and found similar results with regard to closed-ended versus open-ended questions. When constructing a reliable assessment tool, researchers must try to balance creating a period small enough to obtain detailed information and obtaining respondent variability enough to provide an adequate amount of information. This discrepancy was poignantly demonstrated in the RBA and the RBFA. The continuous questions in the RBA ascertained the number of weeks during the individuals lifetime. This provided information enough to obtain adequate test retest reliability. However, lifetime treatment may not present enough fine-grained information for researchers to provide informative findings. In contrast, the continuous questions in the RBFA ascertained the number of days in treatment in the last 6 months and in the last 30 days. Although these data provided useful current drug treatment information, not enough respondents had participated in drug treatment in those periods to establish test retest reliability. Because of this lack of respondents, no continuous RBFA question test retest reliability coefficient could be calculated. Future research could in fact establish minimum periods to aid in developing reliable and informative assessment tools. The data provided a large discrepancy between the reliability coefficients in the number of weeks in methadone detoxification and the number of weeks in methadone maintenance in the last 6 months. We hypothesize that this discrepancy is a result of confusion concerning what constitutes detoxification. The definition, prevalence, length, and intensity of detoxification have all changed over time. Respondents may have been able to remember whether a health worker had informed them that they were undergoing detoxification but unable to accurately understand how many days were actually part of the detoxification. This is supported by the fact that the test retest reliability for detoxification and that for maintenance were above acceptable levels when respondents were asked

5 J.W. Edwards et al. / Journal of Substance Abuse Treatment 33 (2007) whether they had ever/never had treatment, but the test retest correlation dropped dramatically when the number of weeks of detoxification was elicited. The current research provides mixed results concerning drug treatment reliability. It also provides psychometric support for using sound variables from which conclusions can be drawn. Overall, drug treatment variables within the RBA and the RBFA tended to show good test retest reliability. However, we suggest using only those variables that were demonstrated to be reliable. In general, our advice to clinicians is to ask questions to which clients know the answers; this means keeping the questions simple as well as unambiguous and not assuming that clients have knowledge of treatment program classifications or technical names for modalities used. There were also problems with respondents reporting on their attempts to enter treatment. It is not clear whether the problems represent respondents (1) remembering when they tried, (2) knowing why they were rejected, or (3) knowing what counts as an attempt to enter treatment. There may also be some ambiguity concerning what constitutes a prison/jail treatment program or what the term outpatient means. Future RBA and RBFA revisions may be able to use the present data to increase low reliability through clarification and rewording or through changing the length of the recall period. Acknowledgments This research was supported in part by grants from the NIDA (Grant Nos. 1-U01-DA 07290, 1-U01-DA-06903, 1-U01-DA-06908, 1-U01-DA-06906, 1-U01-DA-06919, 1- U01-DA-06912, 1-U01-DA-07474, U01-DA-07694, 1- U01-DA-07295, 1-U01-DA-07302, and 1-U01-DA-07286). References Adair, E. B. G., Craddock, S. G., Miller, H. G., & Turner, C. F. (1996). Quality of treatment data: Reliability of self-reports given by clients in treatment for substance abuse. Journal of Substance Abuse Treatment, 13, Alterman, A. I., & McLellan, A. T. (1993). Inpatient and day hospital treatment services for cocaine and alcohol dependence. Journal of Substance Abuse Treatment, 10, Alterman, A. I., Shen, Q., Merrill, J. C., McLellan, A. T., Durell, J., & McKay, J. R. (2000). Treatment services received by Supplemental Security Income drug and alcoholic clients. Journal of Substance Abuse Treatment, 18, Ball, J. C. (1967). The reliability and validity of interview data obtained from 59 narcotic drug addicts. American Journal of Sociology, 72, Booth, R. E., Corsi, K. F., & Mikulich, S. K. (2003). Improving entry to methadone maintenance among out-of-treatment injection drug users. Journal of Substance Abuse Treatment, 24, Booth, R. E., Crowley, T. J., & Zhang, Y. (1996). Substance abuse treatment entry, retention and effectiveness: Out-of-treatment opiate injection drug users. Drug and Alcohol Dependence, 42, Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, Dowling-Guyer, S., Johnson, M. E., Fisher, D. G., Needle, R., Watters, J., Andersen, M., et al. (1994). Reliability of drug users self-reported HIV risk behaviors and validity of self-reported recent drug use. Assessment, 1, Fisher, D. G., Needle, R., Weatherby, N., Brown, B., Booth, R., Williams, M. L., et al. (1993, June, 2003). Reliability of drug user self-report. Paper presented at the IXth International Conference on AIDS, Berlin, Germany. Fisher, D. G., Reynolds, G. L., Jaffe, A., Johnson, M. E. (in press) Reliability, sensitivity, and specificity of self-report of HIV test results. AIDS Care. Fisher, D. G., Reynolds, G. L., Wood, M. M., & Johnson, M. E. (2004). Reliability of arrest and incarceration questions on the Risk Behavior Assessment. Crime and Delinquency, 50, French, M. T., Roebuck, M. C., McLellan, A. T., & Sindelar, J. L. (2000). Can the Treatment Services Review be used to estimate the costs of addiction and ancillary services? Journal of Substance Abuse, 12, Hubbard, R. L., Marsden, M. E., Rachal, J. V., Harwood, H. J., Cavanaugh, E. R., & Ginzburg, H. M. (1989). Drug abuse treatment: A national study of effectiveness. Chapel Hill7 University of North Carolina Press. Johnson, M. E., Fisher, D. G., Montoya, I., Booth, R., Rhodes, F., Andersen, M., et al. (2000). Reliability and validity of not-in-treatment drug users follow-up self-reports. AIDS and Behavior, 4, Johnson, M. E., Fisher, D. G., & Reynolds, G. (1999). Reliability of drug users self-report of economic variables. Addiction Research, 7, Kaminer, Y., Blitz, C., Burleson, J., & Sussman, J. (1998). The Teen Treatment Services Review (T-TSR). Journal of Substance Abuse Treatment, 15, McLellan, A. T., Alterman, A. I., Cacciola, J., Metzger, D., & O Brien, C. P. (1992). A new measure of substance abuse treatment: Initial studies of the Treatment Services Review. Journal of Nervous and Mental Disease, 180, Needle, R., Fisher, D. G., Weatherby, N., Chitwood, D., Brown, B., Cesari, H., et al. (1995). Reliability of self-reported HIV risk behaviors of drug users. Psychology of Addictive Behaviors, 9, NIDA (1993). Risk Behavior Assessment. (First ICPSR ed.). Ann Arbor, MI: Inter-University Consortium for Political and Social Research. Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory. (3th ed.). New York7 McGraw-Hill. Watters, J. K., & Biernacki, P. (1989). Targeted sampling: Options for the study of hidden populations. Social Problems, 36, Weatherby, N. L., Needle, R., Cesari, H., Booth, R., McCoy, C. B., Watters, J. K., et al. (1994). Validity of the self-reported drug use among injection drug users and crack cocaine users recruited through street outreach. Evaluation and Program Planning, 17, Zanis, D. A., McLellan, T. A., Belding, M. A., & Moyer, G. (1997). A comparison of three methods of measuring the type and quantity of services provided during substance abuse treatment. Drug and Alcohol Dependence, 49,