A Major Dilemma in Psychotherapy Outcome Research:

Transcription

1 A Major Dilemma in Psychotherapy Outcome Research: Disentangling Therapists From Therapies Irene Elkin, University of Chicago In interpreting results of psychotherapy outcome stud- ies, particularly those of comparative studies, it is often dmcult to disentangle the dfo& due to the treat- ments per se from those that may be due to dfirential competence and other characte~istics of the therapists carrying out the treatments. This article attempts to address this general dilemma in psychotherapy out- come research, drawing on the experience and findings of the NlMH Treatment of Depression Collaborative Research Program. Issues discussed include the nature of the overall therapist umple, variability in therapist ef- ficacy, therapist characteristics, training and supervision of therapists, and adherence and competence in u r- wing out a treatment. Implications discussed include the need to provide, both in preunt.tions of the re- sults of individual outcome studios and in disumina- tion of information ngarding "empiricaliy validated treatments," critical infomution about the therapists uying out the treatment. Key wods: therapist effects, supenfirion, compe- tence, outcome, empidully validated treatments. [Ch Psycho1 Sci Pmc 6:t0-32, The major purpose of most psychotherapy outcome studies, particularly in recent years, has been to investigate the efficacy of a specific form of psychotherapy for treating a specific disorder and/or to compare the efficacy of different forms of psychotherapy. Although the role of therapist characteristics or aspects of the therapeutic relationship Address correspondence to I. Elkin. School of Social Service Administration, University ofchicago, 969 E. 60th Street, Chicago, IL may also be explored, they are generally given secondary status. Such findings are, for example, often reported in articles appearing some time after the initial report of outcome findings (e.g., Henry, Schacht, & Strupp, 1986; Krupnick et al., 1996; Luborsky, McLellan, Woody, O'Brien, & Auerbach, 1985; Shapiro, Firth-Cozens, & Stiles, 1989). On the other hand, studies that focus primarily on the therapist and his or her role in the process and outcome of treatment are generally not camed out within the context of major outcome studies. When psychotherapy research findings are reported or when reviews are carried out, they tend to focus either on therapeutic approaches or on therapist, patient, or process variables. Partly as a result of this separation, both between studies and within studies, in reporting on results of outcome analyses and on investigations of the role of the therapist, conclusions about the efficacy of various treatments are generally not qualified on the basis of therapist-related findings. One rarely, if ever, sees a clear statement that a particular treatment is efficacious only in the hands of therapists with specified training, certain personal characteristics, or even with a specified level of competence in carrying out the therapy. Yet clinicians and researchers alike feel that they "know" that the therapist carrying out the treatment may be as important a factor in its success as the treatment approach itself. Clinicians, for example, do not simply refer a patient to any cognitive therapist or any pspzhodynamically or experientially oriented therapist, but rather recommend a specific therapist whom they consider especially competent with a particular approach or whom they consider particularly gifted in working with certain types of patients. Reflecting their belief in the importance of the therapist, researchen have devoted many thousands of hours to AMERICAN PSYCHOLOGICAL ASSOCIATION Dl2 10

2 the investigation of the role of various therapist characteristics and relationship qualities in psychotherapy. They have studied such factors as therapist personality, adjustment, discipline, demographic variables, experience level, values, expectations, therapeutic styles, and specific therapist interventions (for a recent review, see Beutler, Machado, & Neufeldt, 1994). These variables have generally been studied in order to better understand the role of the therapist in the process and outcome of treatment or to predict outcome, and their results have generally not been used to qualify conclusions about outcome findings. From the early days of psychotherapy research, a few researchers have focused on the importance of considering variability among therapists in outcome studies. For example, in Kiesler s classic (1 966) article, he characterized the assumption of therapist uniformity as one of the myths of psychotherapy research. Going back even hrther, Meehl, in his 1955 Annual Review OfPsychology chapter on Psychotherapy, recommended that research be carried out designed (among other things) to yield information about the (no doubt tremendous) individual differences among therapists officially homogeneous in views (p. 376). Bergin, in his chapter on The Evaluation of Therapeutic Outcomes in the first edition of the Handbook of Psychotherapy and Behavior Change (1971), addressed the possibility of therapist-caused deterioration of patients. In recent years, there has been a growing recognition of the need to consider the role ofthe therapist in research focused on the outcome of psychotherapy. Researchers have begun to study the role of therapist effects in contributing to outcome variance (e.g., Crits-Christoph et al., 1991; Crits-Christoph & Mintz, 1991; Luborsky et al., 1986) and the determinants of differences in efficacy of different therapists (e.g., Blatt, Sanislow. Zuroff, & Pilkonis, 1996; Lafferty, Beutler, & Crago, 1989; Luborsky et al., 1985; Najavits & Strupp, 1994). Leading researchers have emphasized the critical importance of our including the neglected variable of the therapist in psychotherapy outcome research (see, e.g., Kazdin, 1997). This recent focus will undoubtedly lead to an increase in our knowledge about the contribution of the therapist (and the interaction of the therapist and patient) to patient improvement in therapy. At the present time, however, researchers and consumers of psychotherapy outcome research findings are lefi with a basic dilemma when interpreting the findings of studies focusing on the efficacy of specific treatments: how to disentangle the efiects due to the therapeutic approach from those due to the particular therapists who have carried out the approach. It is a particularly pressing issue when different therapists carry out each of the treatments in a comparative outcome study. The purpose of the present article is to address this dilemma of disentangling therapists from therapies, to discuss possible approaches to dealing with this issue, and to concider its implications for psychotherapy research and for the dissemination of psychotherapy outconie findings. The discussion is developed within the context of questions raised regarding the outcome findings of the National Institute of Mental Health Treatment of Depression Collaborative Research Program (TDCKP). Although this study was focused mainly on the efficacy of specific treatments for outpatient depression, the dilemma of disentangling effects due to therapists from those due to therapies became particularly salient during the course of implementing this project and especially during the course of analyzing and interpreting the outcome findings. It is, I believe, an underlying issue in some of the debates in the field about the TDCRP findings (e.g., Elkin, Gibbons, Shea, & Shaw, 1996; Jacobson ei Hollon, 1996a). The central question to be asked, in regard to the TDCRP, is whether the outcome findings for each of the treatments, and especially for differences between them, might be partly attributable to the particular therapists participating in the study. We have not yet dealt, in a forthright and comprehensive fashion, with this possibility. It should be possible to do so, at least to some extent, within the context of the current article. The purpose of this article is much broader in intent, however. The problem of disentangling the effects on outcome due to the treatment approach from the role played by the therapists has relevance for any outcome study that focuses on specific treatments and for the interpretability and generalizability of its findings. Although some of the discussion may be particularly relevant for research on the treatment of depression, it should have broader applicability as well. The subject is also particularly timely, given current attempts to identify and disseminate information about empirically validated treatments (Chambless et al., 1996; Task Force on Promotion and Dissemination of Psychological Procedures, 1995). These efforts are also discussed in the final section of this article. DISENTANGLING THERAPISTS FROM THERAPIES ELKIN 11

3 This is, of course, not a new issue. As indicated above, many researchers have studied the role of therapist variables (for reviews, see Beutler et a]., 1994; Gurman & Razin, 1977; Lambert, 1989); some have focused on the intertwined nature of the therapeutic relationship and specific techniques (e.g., Butler & Strupp, 1986); some have, even from fairly early on, studied differential efficacy of individual therapists (e.g., Orlinksy & Howard, 1980; Ricks, 1974); and there has been considerable recent recognition of the therapist as a neglected variable (Garfield, 1997) in psychotherapy outcome research. It is hoped that, by embedding the current discussion in the context ofa particular study and the questions it has raised, it will be possible to highlight, to clarifi, and to contribute to our thinking about the many problems involved in attempting to disentangle therapists from therapies. I do not propose to provide solutions to the dilemma; I am not even sure there really are any. I do believe, however, that it is a. critical issue for the field to address. There are many related issues that are beyond the scope of the present article. One of these is the contribution to outconie of patient variables, a topic that has received a great deal of attention in the psychotherapy research literature. Especially relevant for the current discussion is the importance of patient-therapist interactions and the differential efficacy of particular therapists with different types of patients. Although I will touch on this subject, my major focus is the more narrow issue of the role of the therapist in general, and especially on the role of therapist competence in carrying out a specific form of treatment (which may, of course, include flexibility in responding to the needs of different patients). Another restriction in the scope of the current discussion is the focus on psychotherapy, even though the TDCRP included two pharniacotherapy conditions. Although the therapist may actually play an important role in the efficacy of drug treatments, the focus here is on the dilemma as it is faced in psychotherapy research. Finally, while it is important to recognize the niovenient toward eclecticism and/or integration among theorists and practitioners (Garfield, 1994; Goldfried & Newnian, 1992), the additional problems that researchers must address in assessing the efficacy of integrative approaches is beyond the scope of this article. The discussion focuses on issues arising and needing to be addressed in the current empirical outcome literature, which is still largely focused on specific therapy approaches. Before proceeding to the substantive discussion, two semantic clarifications are necessary. First, the term psychotherapy outcome studies is used throughout the article, in preference to the recently more popular clinical trials. The concept of clinical trials is borrowed from research in psychopharmacology and, more broadly, from the general field of medicine (Klerman et al., 1994). A basic assumption in such research is that the only active ingredient is the medication (or other medical procedure), and the possibility that the provider of the medication may play a role in treatment outcome is generally not considered. With the exception of a flurry of studies in the 1960s (Rickels, 1968), there has been very little research in psychopharmacology that considers the possible role of the therapist (see discussion in Elkin, Pilkonis, Docherty, & Sotsky, 1988). Since the focus of this article is precisely on the intertwined nature of effects due to the treatment approach and those due to the therapist, it seems appropriate to use the more traditional, and more accurate, label of psychotherapy outcome studies. The second choice to be made was between the terms efficacy and effectiveness. Since most of the discussion focuses on experimentally controlled outcome studies, efficacy is the more appropriate term. The term effectiveness is reserved for the outcome of treatments in actual clinical practice and service provision (Clarke, 1995). FOCUS ON THE THERAPIES IN THE TDCRP Four treatment conditions were examined in the TDCRP: the two psychotherapies of primary interest in the study, cognitive behavior therapy (CBT) and interpersonal psychotherapy (IPT), and two pharmacotherapy standard reference (imipraniine plus clinical management [IMI-CMJ) and control (placebo plus clinical management [PLA-CM]) conditions. Different groups of therapists carried out each of the treatments, with the exception of the two pharmacotherapy conditions, which were carried out double-blind by the same pharmacotherapists. (For extensive discussions of the background and design of the TDCRP and for a review of findings through 1993, see Elkin, 1994; Elkin, Parloff, Hadley, & Autry, 1985; Elkin et al., 1989.) In designing the TDCRP, we had made a comprehensive effort to ensure that therapists would conduct their respective treatments in a consistent, fairly standardized, and competent fashion,(rounsaville, Chevron, & Weiss- CLINICAL PSYCHOLOGY: SCIENCE AND PRACTICE * V6 N1. SPRING

4 man, 1984; Shaw, 1984; Waskow, 1984). Thus, therapists were to be carehlly selected to have training, experience, and interests relevant to their respective treatments, were to use guidelines provided by treatment manuals (Beck, Rush, Shaw, & Emery, 1979, for CBT; Fawcett, Epstein, Fiester, Elkin, & Autry, 1987, for pharmacotherapy; Klerman, Weissman, Rounsaville, & Chevron, 1984, for IPT), and were to receive intensive supervision during a training/pilot phase of the study, meet competence criteria in order to take part in the outcome study, and receive additional monitoring and consultation during the outcome study. Major proponents of each of the treatments were asked to design their respective training programs, develop competence criteria, and provide training across the three research sites. In addition, they were to include other experts in their approaches in the evaluation of therapist competence. The focus in the TDCRP on specification or standardization of the treatments rather than on the efficacy of individual therapists was quite deliberate. Although a major purpose of this research program was to test the feasibility of carrying out collaborative outcome studies in the field of psychotherapy, we also wished to investigate, in the context of the collaborative model, the efficacy of CBT and IPT for the treatment of outpatient depression. We were acutely aware of the problems in drawing conclusions and making generalizations fiom many earlier psychotherapy outcome studies due to the vagueness with which treatments were described (see Waskow, 1984). Sometimes there was only one sentence devoted to the description of a particular approach; at other times there was no mention of the specific approach(es) used. Even if the psychotherapy being studied proved to be eficacious, it was ofien not possible to communicate to other researchers, to practicing clinicians, or to prospective patients just what the treatment consisted of. Such inadequacies in the definition of treatments clearly limited the applicability of the findings from some of these earlier studies. In an effort to avoid this problem, a great deal of energy was devoted to the clear definition or specification of the treatment intervention in the TDCRP. It was hoped that, as a result of this effort, it would be possible to say: 771is is the treatment that had such and such effects. In planning the TDCRP, we did not completely ignore the issue of therapist variability. A Therapist Information form and a Therapist Attitudes and Expectations form provided information on therapist characteristics and attitudes that could later be used as predctor variables, although it was recognized that the relatively small number of therapists (28 in the outcome study) would restrict the generalizability of any findings. Further, in terms of the process of therapy, ratings were made throughout the study of therapists competence in their respective treatment approaches and, later, of therapist adherence and therapeutic alliance. In addition, an analysis of therapist effects was carried out prior to the major outcome analyses, to determine whether differences in outcome for individual therapists made a significant contribution to outcome variance. The lack of significant findings (generally corroborated in a later metaanalysis by Crits-Christoph et al., 1991) resulted in our not including therapist as a variable in the major analyses. We did not report these analyses, since we considered them a necessary, but preliminary, step for any outcome analyses. It is worth noting, however, that the lack of significant findings seemed due not so much to small mean differences among the therapists, as to the large variability within therapists, suggesting differential therapist efficacy with different patients. Most relevant for the present article, the difficulty of separating the effects of the treatments in the TDCRP fiom the particular therapists conducting them was acknowledged fiom the beginning. In the article describing the background of the TDCRP (Elkin et al., 1985), we wrote: The treatment conditions being compared... are, in actuality, packages of particular therapeutic approaches and the therapists who choose to and are chosen to administer them (p. 308). Thus, early on, we gave lip service to the difficulty of disentangling therapists from therapies. Further, we had early plans for exploratory analyses of therapist variables such as age, experience, and discipline, in addition to plans for studying the therapeutic alliance and therapist adherence to and conipetence in conducting the treatments, as well as other process variables. Yet, once the data were available, my colleagues and I followed the usual pattern of focusing first on the efficacy of the specific treatments being studied (e.g., Elkin et al., 1989; Gibbons et al., 1993; Imber et al., 1990; Shea et al., 1992; Watkins et al., 1993) and on patient predictors of outcome (Shea et al., 1990; Sotsky et al., 1991). We did not address the subject of possible variability in outcome associated with therapist factors. (This was probably due not only to the interest in the field regarding the compara- DISENTANGLING THERAPISTS FROM THERAPIES - ELKIN 13

5 tive outcotiie findings of the TDCRP but also, at least in part, to our initial focus on analyses based on the entire sample of 2.39 patients, which seemed to take precedence over analyses based on the small N of 28 therapists.) The closest we came was in our exploratory analyses suggesting possible differences in outcome due to research sites, in the subsatiiple of more severely depressed and functionally impaired patients. Only later did we and others begin to report on fiictors directly related to the therapist (e.g., Blatt et al., 1996; Hill, O Grady, & Elkin, 1992; Krupnick et al., 1994; Krupnick et al., 1996: Shaw et al ). and even then, we generally did not qualify our interpretation of major outcome findings in terms of thcse fiictors. Although there was. thus, an early recognition of the probable contribution of the therapist to outcome and of the ditficitlty of separating the effrcts due to the therapies from those due to the therapists, we reported the outcome tindings as if we were truly describing only the effects of the treatment approaches per se. As time has gone by, however, in the process of coordinating the TDCRP and in attempting to interpret our outcome findings, I have become inore and more concerned about the need to grapple with the issue of the inherent intertwining and possible confounding of the effects due to the treatment approach with those due to the particular therapists carrying out the treatment. EMERGENCE OF THE DILEMMA IN THE TDCRP As mentioned above, the training of therapists in the TDCRP and their certification as competent were carried out by experts in each of the treatment approaches. Certain requirements were imposed by the study design for the selection of therapists in all treatments and for the components to be included across training programs (for details, see Rounsaville et al., 1984; Shaw, 1984; Waskow, 1984). These requirements were intended to maximize consistency in training both within treatments (across sites) and acrds treatments, to ensure an adequate and comparable level of therapist expertise, and to minimize differences among treatments in the final composition of therapist pools. In addition, NlMH staff monitored the implementation of the training programs through site visits and, for a period of time, through sitting in on supervision calls, in order to ensure consistency in the nature and intensity of supervision. By and large, with some minor exceptions, the various requirements were met. Despite all of these precautions, however, there were some indications of possible differences in the level of conipetence acquired and/or maintained by the therapists in the two psychotherapy conditions. The first hint of such differences appeared in the way that trainers discussed the performance of the therapists in their respective treatment conditions. The IPT trainers spoke with considerable enthusiasm about the competence level of most of the IPT therapists. In contrast, the CBT trainers focused more on the fairly wide range of their therapists performance, although they were able (with the concurrence of four other experts in cognitive therapy) to certify eight therapists as sufficiently competent to take part in the outcome study. The second hint was more quantitative. Trainers for each of the treatments had agreed to identify a red-line (Rounsaville et al., 1984; Shaw, 1984) to indicate a level below which a therapist s performance was not considered adequate. During the outcome study, in addition to regular monthly consultation calls, phone calls were to be made to therapists following any session designated as falling below this red-line, in order to provide targeted consultation regarding the therapist s performance. During the course of the outcome study, 137 monitored tapes led to red-line calls (33% of the 416 tapes regularly monitored) to CBT therapists, while only 12 tapes (3% of the 390 tapes regularly monitored) led to calls to IPT therapists. The third hint came from the trend in the average competence ratings made by the trainers for therapists across successive patients seen in the training/pilot phase and the beginning of the outcome study. IPT therapists attained a high level of competence by their second training case, and this level was maintained for their remaining training cases as well as for their first outcome study patient, while CBT therapists gradually improved in competence throughout the entire training/pilot phase, but their level of competence dropped, on average, for their first outcome study patient (see Rounsaville, O Malley, Foley, & Weissman, 1988). It is difficult to know, of course, whether the above differences were due to actual differences in competence between therapists in the two psychotherapy conditions, or whether they were due instead to differences in the stringency of the competence criteria developed and applied by the trainers in the two treatments. (The CBT trainers, e.g., had a specific numerical score to define their red-line, a score of 39 or less on the Cognitive Therapy CLINICAL PSYCHOLOGY: SCIENCE AND PRACTICE V6 Nl, SPRING

6 Scale [CTS; Young & Beck, 19801, while the IPT trainers used a more qualitative criterion.) This quandary leads to the general, and very critical, question of how or, in fact, whether one can establish the comparability of competence of therapists in two different treatments, an issue that is dealt with at some length later in this article. Once the initial TDCRP outcome findings were presented and later published (Elkin et al., 1989), questions were raised, especially by advocates of one or another of the treatment approaches, about the interpretation of the results. Although the most heated of the controversies that ensued was between a psychopharmacology researcher and the TDCRP collaborators (Elkin et al., 1996; Elkin et al., 1990; Klein, 1990, 1996; Klein & Ross, 1993), the focus here continues to be on the psychotherapies and particularly on questions raised about the findings for the CBT treatment condition. The most relevant findings, in brief, were the following: CBT was not significantly superior to the PLA-CM treatment condition (a combination of a pill placebo and clinical management, or what could be considered minimal supportive therapy ); for the more severely depressed patients, there was evidence that IMI-CM and, to a lesser extent, IPT were significantly superior to CBT and PLA- CM. For those who were both severely depressed and functionally impaired, IMI-CM was significantly superior to all of the other treatments. For the less severely depressed patients, none of the active treatments, including IMI-CM, was superior to PLA-CM. There were also initial suggestions of possible differences between the results in different research sites for a subsample of more severely depressed and functionally impaired patients, although later analyses suggest that we cannot actually have much confidence in these exploratory findings. (For a review of the early findings, see Elkin, 1994, and for later relevant analyses, Elkin et al., 1996; Elkin et al., 1995.) Cognitive and behavioral researchers raised a number of questions regarding these findings (see, e.g., Hollon, Shelton, & Loosen, 1991; Ilardi & Craighead, 1994). Basically, the various issues raised all reduce to one basic question: Was cognitive therapy in the TDCRP being delivered in an adequate fashion? And its corollary, Does the CBT condition in this study provide an adequate test of the efficacy of cognitive therapy in the treatment of outpatient depression? Critics of the TDCRP, or at least of the TDCRP collaborators interpretation of the findings, question whether the CBT therapists at one or more research sites may have been insufficiently competent in carrying out the treatment and/or whether therapists received sufficient supervision once the outcome study began. The purpose of the present article is not to revisit these issues. They have been discussed at length and from different vantage points in a series of articles in the_lorrntal 4 Cottsultirg and Clinical Psychology (Elkin et al., 1996; Jacobson & Hollon, 1996a, 1996b). Regardless of one s own conclusions about the answers to these questions, it seems incumbent upon us, as psychotherapy researchers, to address the basic issue that they point to: the difficulty in making definitive statements about the efficacy of a treatment approach in a particular study without considering the role played by the therapists who participate in the study. The issue of the role played by the therapists becomes particularly critical when a comparison is made of the relative efficacy of two or more treatments. Is a difference found between them due to the superiority of the particular treatment approach or to the characteristics and/or relative competence of the therapists conducting it? And if there is no difference, might this, too, reflect the relative competence of the therapists, for example, with a hypothetically superior treatment approach being carried out by less competent therapists? When we turn specifically to the outcome findings of the TDCRP, we must question whether the superiority of IPT over CBT for the more severely depressed patients (Elkin et al., 1995) might have as much to do with the relative competence of the therapists in conducting the two treatments (or at least to their competence in treating severely depressed patients) as to the treatment approaches themselves. It is this question, in fact, that has been most troubling to me, particularly in light of some of the issues raised during implementation of the study and referred to above, and has resulted in some of the thoughts articulated in this article. It is important to point out, however, that although I am focusing here on differences in outcome for the IPT and CBT patients, we should be somewhat cautious in our thinking about the magnitude of these differences. Although IPT was sometimes significantly superior to PLA-CM whereas CBT was not, the actual differences between the two psychotherapies were not uniformly large or necessarily significant. For the more severely depressed patients, differences were marginal on one measure (p <.10 on the Hamilton Rating Scale for Depression [HRSD]), though significant on another ( p < DISENTANGLING THERAPISTS FROM THERAPIES ELKIN 15

7 .01 on the Beck Depression Inventory [BDI]), but for the patients who were more functionally impaired, IMI-CM was superior to both of the psychotherapies and there was no difference between them. Thus, the focus in this article should not be construed simply as an attempt to understand differences between treatments in the TDCRP. Rather, my primary intent is to utilize these differences and the issues they raise as a starting point for a general discussion of the difficulties of disentangling therapists from therapies. In the following sections, I consider some ways in which the problems related to disentangling the therapist from the therapy, particularly in the context of comparative outcome studies, might be addressed. As indicated earlier, I do not propose a solution to the dilemma, but simply suggest some ways that we can begin to thmk about it. Again, I draw on the experience in the TDCRP as a general context for this discussion, as well as other literature in the area of psychotherapy outcome research. While recognizing the underlying complexity of the determinants of therapy outcome, I will, for ease of presentation, consider the issues to be addressed under the following discrete headings: overall therapist sample, variability in therapist efficacy, therapist characteristics, training and supervision of therapists, and adherence and competence in carrying out a treatment. Special attention is given to the conceptualization and the measurement of competence and the problem of establishing comparable levels of competence across treatments. ADDRESSING THE DILEMMA Overall Therapist Sample As we stated originally (Elkin et al., 1985), a specific treatment condition in a study may be considered a package of the therapy and the therapists who choose to and are chosen to administer it. Yet we know very little about the therapists who actually do choose to and are chosen to take part in our studies, and how they may dffer from the more general population of therapists. Nor do we know the impact that this may have on our findings. In regard to who chose to take part in the TDCRP, for example, several of the IPT therapists were psychoanalysts; it is very likely that those analysts who, agree to take part in a research study in which they practice a time-limited interpersonally oriented therapy may not be representative of the population of psychoanalysts in the real world. One way of addressing the issue of generalizability of findings to other therapists practicing a particular approach could be to use the model suggested by Clarke (1 995) in his article on bridging efficacy and effectiveness research. He suggests that we study different levels of a variable within the context of one study; for example, in regard to this issue, one could include in the same study both therapists willing to take part in a controlled clinical trial and therapists practicing the same treatment approach in uncontrolled settings. In the absence of empirical findings from such studies, it is important for researchers to provide more information than we now do about the characteristics of therapists in a particular study, so that it is possible to have some idea of the limits in generalizability of a study s findings. In regard to who is chosen to take part in a study, it would seem important to provide information on the number of therapists screened, the number chosen for training, the number who are then chosen to take part in a study, and the number for whom outcome findings are reported, just as we provide parallel information about the initial and final patient pools. It would also be valuable to know what screening criteria result in a diminishing therapist pool at each of these stages. In the TDCRP, of the therapists who were trained in the program, 85% met competence criteria and were certified to take part in the outcome study (Elkin et al., 1985); two CBT therapists, one IPT therapist, and two pharmacotherapists did not meet their respective treatments competence criteria. This is a quite different percentage from that reported, for example, by Beutler (1997), where only 5 of 19 (or 26%) of the therapists trained to carry out cognitive therapy with older adults reached criterion level. Problems of generalizability obviously increase when there is such selectivity in the therapist pool in a study, and this needs to be clearly considered in interpreting and considering the broader applicability of a study s findings. Variability in Therapist Efficacy When a single treatment is compared to a control group, such as a wait-list condition, the problems related to the role of the therapist are somewhat easier to deal with, because only one group of therapists is involved. (Of course, if the control group is some type of attentioncontrol and involves the participation of therapists, the resulting issues become more similar to those discussed below for comparative outcome studies.) Nonetheless, we cannot disregard the fact that the efficacy of the single CLINICAL PSYCHOLOGY: SCIENCE AND PRACTICE V6 N1, SPRING

8 treatment may be affected by the therapists administering it. It is thus very important for the consumer of the research to know what role the individual therapists may have played. We should, therefore, be regularly reporting, along with major outcome$ndings, and regardless of whether one or more treatments are being studied, the contribution to outcome variance due to therapists. There is some empirical evidence to support what clinicians generally take for granted that rather than there being good and bad therapists, some therapists may be more or less effective for particular types of patients. This differential efficacy may be related to such variables as degree of patient disturbance (Ricks, 1974), diagnosis, or life status (Orlinksy & Howard, 1980). Especially in studies where attention is directed at particular patient variables, it would be very informative for researchers to also indicate whether there are any interactions between the therapist variable and these patient characteristics. In addition to reporting therapist variance, it may often be important, as pointed out by Crits-Christoph and Mintz (1991), to actually include the therapist as a random variable in data analyses of treatment effects. As mentioned earlier, in our preliminary analyses of TDCRP outcome data, we determined that there were no sign& cant effects due to therapist and, therefore, did not include the therapist as a variable in hrther data analyses. Crits- Christoph and Mintz recommend, however, that for ruling out therapist effects in a preliminary analysis... p values greater than.2 or.3 should be employed, not the 25). They point out that, where differences may exist between therapists, yet the therapist is not treated as a random variable in outcome analyses, one is likely to get overly liberal F ratios and an increased risk of Type I errors. We are currently conducting more complex analyses of therapist effects in the CBT and IPT conditions in the TDCRP, utilizing random regression models to evaluate the contribution of the therapist to patient trends on dependent variables over the course of treatment, and we are including the interaction of therapist with patient initial level of severity of depression, as well as treatment effects, in the models. Preliminary results suggest that there are no significant therapist effects or therapist X initial severity interactions. Further explorations of the interaction of therapists with other patient variables will also be carried out. In evaluating the significance of therapist variability, it is of course important to ascertain that the characteristics of patients in different therapist case loads were basically similar. In the TDCRP, although with the small case loads in the study there was some variation among therapists on patient clinical and demographic variables, with the exception of pretreatment Global Assessment Scale scores among CBT therapists, there were no significant differences. As indcated above, our current analyses oftherapist effects include patient pretreatment severity measures in the model. Crits-Christoph and Mintz (1991) also recommend that, for studies in which sizable therapist effects are expected, a large number of therapists be employed, in order to have adequate statistical power for testing treatment effects. Although this is a reasonable recommendation in terms of generalizability as well as statistical concerns, there may be an upper limit to the number of therapists that can realistically be included in studies in which therapists receive intensive training, and more patients per therapist may be necessary in studies exploring the interaction of therapists with Merent types of patients. While acknowledging the importance of the statistical concerns raised by Crits-Christoph and Mintz and others, the focus on the role of the therapist in the present article is more on the conceptual issues involved in evaluating the contribution to outcome of a treatment and the therapists conducting it. It is important to recognize here that, even where there is not a significant amount of variance in outcome due to therapists (using the recommended liberal probability levels), there may still be an impact of the therapists on the outcome results. For example, if dfferent groups of therapists carry out different treatments in a comparative study and if therapists in one of the treatment conditions are, in general, more competent than those in another, treatment differences in patient outcome may be due as much to the particular therapists as to their respective treatment approaches. Thus, while paying attention to therapist variance IS important, one must also consider the likelihood of confounds between treatments and the characteristics and competence of the therapists who conduct them. The issue of the therapist s role in treatment efficacy clearly becomes more salient, and the related problems more dificult to deal with, when more than one treatment is involved in a study. A suggestion that has often been made over the years for dealing with this problem is to have the same therapists carry out the different treat- DISENTANGLING THERAPISTS FROM THERAPIES * ELKIN 17

9 ments being studied (i.e., crossing therapists with treatments rather than having therapists nested within treatments), in this way controlling for the effects of the therapist per se. This approach has been used, for example, in studies of behavior therapy and cognitive behavior therapy (e.g., Jacobson et al., 1996), where the different treatments or treatment components being compared are quite closely related to one another and have all been included in the therapists training. Similarly, this approach has been used in recent studies comparing process-experiential therapy with its baseline component of client-centered therapy (Watson & Greenberg, 1996). In studies such as the TDCRP that compare markedly different approaches, however, researchers tend to consider it unlikely that therapists could both be adequately trained and have an adequate degree of commitment to and competence in these different approaches. For this reason, researchers carrying out such comparative studies have generally used different groups of therapists, each of whom has interest and training in one particular approach. The aim is to control for the level of therapist training and competence in a specific approach rather than for the person of the therapist per se. An exception, here, are the studies by Shapiro and colleagues (Shapiro et al., 1994; Shapiro & Firth, 1987), in which the same therapists carried out two quite different treatments, exploratory (psychodynamic-interpersonal) and prescriptive (cognitive behavioral) therapy. It is of more than passing interest that follow-up analyses of the first study s data (Shapiro et al., 1989) revealed that the difference reported in favor of prescriptive therapy was primarily due to patients seen by one of the two main therapists, again highlighting the potential role played by therapists in comparative outcome findings. One must clearly consider whether the therapist responsible for the differential outcome might have been more committed to and/or more competent in carrying out the prescriptive treatment than the exploratory treatment. Thus, crossing therapists with treatments does not, necessarily resolve the dilemma of disentangling the effects of therapists from treatments. There has been a recent interest in not only determining whether there are significant therapist effects in a study, but also in trying to uncover the determinants of therapist differences in efficacy (e.g., Blatt et al., 1996; Lafferty et al., 1989; Luborsky et al., 1985; Najavits & Strupp, 1994). Although it is not possible to draw defini- tive conclusions from the studies to date, accumulated findings of this type should eventually make it possible to qualify statements about the efficacy of particular therapies in outcome studies; for example, it should be possible to say that a therapy is most likely to be efficacious when it is delivered by individuals with those characteristics found associated with highly efficacious therapists in a number ofstudies. In the meantime, we may note that some ofthe reported findings (e.g., of the importance of the helping alliance in Luborsky et al. and of positive vs. negative therapist behaviors in Najavits and Strupp) are consistent with those of studies directly relating therapist characteristics or relationship quahies to outcome. Therapist Characteristics In addition to information about therapist variability and anything that is known about variables related to differences in therapist efficacy, it is of course important, in reporting results of outcome studies, to provide basic information about the characteristics of the therapists, including demographics, past training and experience, and, if at all possible, some indication of their competence in carrying out the treatment. This information should give consumers of the research findings some indication of the generalizability of the results. Of course, where available, the past track record of the therapists performance would be invaluable. It is of interest, in this regard, to note the recent report by Luborsky, McLellan, Diguer, Woody, and Seligman (1997) that two of the therapists in their research had superior results across outcome studies with different patient populations. In interpreting the results of comparative outcome studies involving different groups of therapists nested within treatments, there is even more reason for concern about therapist characteristics and especially about the comparability of therapists in the different treatment conditions. As stated by Kazdin (1994), it is important to make implausible the possibdity that treatment outcome differences or the absence of differences can be attributed to therapist characteristics, competence, or execution of treatment (p. 39). In regard to therapist characteristics, Kazdin recommends that therapists in the different treatment conditions be similar on such factors as age, gender, and professional experience. In this way, he suggests, one can try to rule out, in a study of treatment effects, the alternative hypothesis that treatment differences are CLINICAL PSYCHOLOGY: SCIENCE AND PRACTICE V6 N1, SPRING

10 actually due to differences between therapists. In accord with this recommendation, Szapocznik and colleagues (1989), in their study comparing structural family therapy and psychodynamic child therapy for problematic Hispanic boys, matched therapists on ethnicity, gender, professional discipline, and years of clinical experience. Although equating therapists on such factors is a laudable goal when it can be met, it may sometimes not be a realistic one. There may, for example, be intrinsic differences in the treatments being compared that result in different characteristics of the therapists who choose to practice them. Of special relevance here is what Jeanne Phillips, one of the advisers to the TDCRP, referred to as the old therapy/new therapy problem. One aspect of this problem is that those who choose to practice an older and more established treatment may, themselves, be older and more clinically experienced, whereas younger clinicians may be drawn to a more recently developed, cutting edge treatment approach. This was, in fact, the case in the TDCRP. IPT, which is quite similar to older psychodynamically oriented therapies, drew therapists who were older on the average than did the much more recently developed CBT (45.7 and 39.0 years old respectively, t = 2.13, &= 16, p <.05), and IPT therapists had correspondingly more years of clinical experience following their professional training than did the CBT therapists (15.3 and 10.0, respectively, t = 2.01, df = 16, p <.lo). Had the sites chosen, from among the applicants, groups of IPT and CBT therapists ofsimilar ages, they would not have adequately represented the therapists who choose to carry out each of these treatments, at least those who volunteered to do so in the context of a controlled research st~dy.~ There were no significant differences between therapies on other therapist variables, including number of depressed patients previously treated and number of hours a week treating depressed patients. These latter variables may, in fact, be more adequate reflections of therapist experience relevant for this study, for, as Beutler (1997) has pointed out, years since professional training may not actually say very much about skills gained, exposure to particular types of patients, or actual expertise. There were also fairly striking (though not significant, probably due to the low Ns) differences in the distribution of professional disciplines of the therapists: in IPT there were six psychiatrists and four psychologists (one of whom saw only one patient), whereas in CBT, there were two psychiatrists and six psychologists. The critical question, here, is whether outcome of treatment might be related to the age, years of clinical experience, and professional discipline of the therapists in the different treatment conditions. For this reason, both therapist discipline and experience level (which is highly correlated with age) were included in our most recent analyses of therapist effects, using random regression models. Preliminary results indicate that there were, in hct, no significant effects for either discipline or years of experience, although there was a significant experience X treatment interaction. Inspection of the observed means over time suggests that this was due largely to a greater reduction of symptoms by 4 weeks (though no difference in final symptom level) in CBT patients seen by the more experienced therapists. Where therapists in different treatment conditions do differ on demographic or experience variables, it would seem important to ascertain what effect this might have on outcome. In the case of the TDCRP, the experience X treatment interaction suggests that, had there been more experienced CBT therapists, CBT patients might have shown a more rapid initial response to treatment, though not necessarily a difference in final level. This is ofspecial interest in light ofthe report by Ilardi and Craighead (1994) of more rapid initial reduction of symptoms in other studies of CBT. Therapist Training and Supervision In addition to indicating therapists past training and experience, it is necessary to address the issue of training and supervision of therapists in the study being reported. One of the earliest questions raised about the TDCRP findings (see Elkin et al., 1989; Hollon et al., 1991) was whether performance in the CBT condition might have been less than optimal because the therapists did not receive the intensive supervision during the outcome study that had been provided in other studies comparing CBT with medication (e.g., Hollon et al., 1992; Murphy, Simons, Wetzel, & Lustman, 1984; Rush, Beck, Kovacs, & Hollon, 1977): We had deliberately reduced the amount of supervision in the TDCRP (going fiom weekly supervision to once a month consultation plus monitoring and red-line calls, as compared to the once or twice weekly supervision in the other studies) once therapists were certified as competent to take part in the outcome study. We DISENTANGLING THERAPISTS FROM THERAPIES ELKIN 19

11 hoped in this way to render the treatments in the TDCRP more like practice out in the real world. Even this amount of consultation is, of course, more than many therapists would receive after having been trained in a particular treatment approach. Jacobson and Hollon (1996b) apparently believe that in order to provide the best possible test of a treatment, intensive supervision should be in place throughout an outcome study. This raises an interesting quandary for psychotherapy researchers: how to balance the desire to establish the best possible representation of the treatment with the wish to generalize to practice in less controlled settings. Clearly, there is no simple answer to this dilemma; both aims are reasonable ones to pursue, and one or the other may be given more weight, depending on the purpose of a particular study. What is vitally important, however, in regard to continued supervision during an outcome study is that, in reporting the results of the study, the presence and intensity of supervision be indicated (as well as the length and intensity of the original training) and that its relevance for interpreting and generalizing from results be acknowledged. In addition to supervision by an expert, it is also important to consider the role that may be played by other types of support therapists may be receiving, such as peer supervision, group consultation, and so on. Support from other therapists as well as from consultants may be especially helpful, and sometimes even crucial, when therapists are seeing patients who are particularly difficult to treat. Linehan (1 993), in her description of dialectical behavior therapy (DBT) for patients diagnosed with borderline personality disorders who have engaged in repeated selfinjurious behaviors, includes an unusually explicit description of the role of supervision/consultation team meetings in providing support for therapists. These meetings serve not only to help therapists problem solve (within a DBT framework) regarding treatment strategies, but also to cheerlead when therapists become demoralized. Since she considers the support from other therapists and consultants extremely important in working with this challenging patient population, Linehan states that supervision/consultation with therapists is integral, rather than ancillary, to DBT (p. 423). Although not always considered so central to a treatment approach, therapist team meetings or more informal peer support probably play an important role in other treatments as well. This is certainly true for the team approach in Assert- ive Community Treatment programs for persons with serious and persistent mental illness, which not only provides occasional respite for individual workers from especially difficult clients but also maximizes the chances for preventing staff burnout and nurturing staff morale (Witheridge, 1991, p. 57). I have previously suggested that supervision and/or consultation during an outcome study be considered as part of the definition of the treatment intervention. It could also be seen, in the context of the issues being discussed in this article, as an extension of the definition of the therapist; that is, the therapy is being provided by a combination of the therapist and his or her supervisors/ consultants (or therapist peer support group). This may be especially important where supervisors are widely acknowledged as the experts in an approach, as, for example, in the study of cognitive therapy by Rush -et al. (1977), where Aaron T. Beck was one of the supervisors. The importance of the person and style of the trainer or supervisor has been highlighted by the recent training effects reported in the Vanderbilt I1 Study (Henry, Schacht, Strupp, Butler, & Binder, 1993); changes found in therapist performance from pre- to posttraining were largely due to groups run by one of the two trainers (reminiscent ofthe treatment effects due to one ofthe therapists in Shapiro et al., 1989). Not to acknowledge the role played by supervision and consultation during an outcome study could result in erroneous conclusions about the expected effects of the treatment in settings where such supervision is not available. This is discussed further below in the section on dissemination of outcome research findings. Therapist Adherence and Competence Rather than focusing on therapist variables related to outcome of treatment, many researchers in recent years have focused instead on trying to minimize variance due to individual therapists, in an effort to study the effects of the treatment, as we did in the TDCRP. One of the mechanisms for attaining a reduction in therapist variability has been the use of treatment manuals for defining the treatment; therapists are trained, or at least instructed, to conduct therapy in accord with a manual and, frequently, checks are made of therapists adherence to the manual. A great deal has already been written about the use ofmanuals in psychotherapy research, in terms of their advantages (e.g., Luborsky & DeRubeis, 1984), their drawbacks (e.g., CLINICAL PSYCHOLOGk SCIENCE AND PRACTICE V6 N1. SPRING

12 Strupp & Anderson, 1997), and suggestions for their improvement (Addis, 1997). Further, there is some empirical evidence (Crits-Christoph et al., 1991) that the use of manuals does indeed decrease variance due to therapists. The relevance of manuals for the current discussion is that they often serve as the basis for defining a therapist s adherence to a particular treatment approach. The assumption is that, if therapists are adhering to the approach, as outlined in the manual, we can be relatively confident that treatment effects are due to the treatment approach itself and that variance due to therapists will be minimized. Recent years have seen an increase in attempts to document that therapists are carrying out a particular treatment in an adherent fashion. In the TDCRP, Hill et al. (1992) determined that the therapists in the different treatment conditions exhibited more behaviors specific to their respective modalities than to other modalities (p. 77) and that the treatments could be discriminated almost perfectly from one another. In an initial analysis of the relationship of adherence to outcome (Elkin, 1988), adherence (as measured by the Collaborative Study Psychotherapy Rating Scale [CSPRS]; Hollon, Waskow, Evans, & Lowery, 1984) was not related to outcome of either CBT or IPT, although the CSPRS did adequately differentiate the two treatments. Whether strict adherence to specified techniques is the most appropriate measure of the adequacy with which a therapy is being carried out is, however, open to question. There are data, in fact, that suggest that strict adherence to an approach may sometimes be accompanied by problems in the therapist s handling of the therapeutic relationship in both psychodynamic (Henry, Strupp, Butler, Schacht, & Binder, 1993) and cognitive (Castonguay, Goldfried, Wiser, Raue, & Hayes, 1996) therapies. Rather than simply relying on measures of adherence to a manualized treatment to indicate whether a treatment is being carried out as intended, efforts have begun to focus more on the competence with which a treatment is being provided. In a study based on TDCRP training phase data, O Malley et al. (1988) found that trainers ratings of therapist competence in the IPT treatment condition were related to some outcome measures. Unfortunately, reliability problems with the IPT competence ratings in the outcome phase of the TDCRP interfered with the possibility of trying to replicate these findings. Further work would be needed to either train new raters in using the IPT competence measures or in developing new scales. Frank, Kupfer, Wagner, McEachran, and Cornes (1991), in a study of IPT-maintenance treatment for patients with recurrent depression, have reported positive findings relating outcome to their own measure of the quality of IPT-related therapy process. In a recent analysis of the relationship to outcome of competence in the TDCRP CBT condition (Shaw et al., 1998), removing the variance attributable to adherence actually enhanced the relationship of competence to outcome. Even so, competence, as measured by trainers ratings on the Cognitive Therapy Scale (CTS; Young & Beck, 1980), accounted for only a modest amount ofvariance in outcome and results were not very consistent across measures. We have joined others, however (e.g., Jacobson & Hollon, 1996b; Whisman, 1993), in questioning the adequacy of the CTS for capturing the cornpetence oftherapist provision ofcbt (Shaw et al., 1998). I also agree with Waltz, Addis, Koerner, and Jacobson (1 993) in their call for the development of better measures of competence in general, measures that take into account contextual issues, particularly with regard to the appropriateness and timing of interventions. A fair amount has been written over the years (e.g., Schaffer, 1982, 1983; Shaw & Dobson, 1988; Waltz et al., 1993) about the importance of measuring therapist competence or skill in conducting treatment, ways in which this might be defined, and suggestions for method development. Yet, there is not yet a great deal of agreement about what is meant by competence or how best to measure it. It is hoped that, with the current increased attention to this issue and recommendations or guidelines for the composition of instruments (e.g., Waltz et al., 1993), more adequate instruments will be developed. Where competence in a single treatment is being evaluated, application across studies of an instrument that is generally accepted by proponents of the approach might provide us with valuable information about the relative adequacy of the implementation of that treatment in different settings. This would, of course, have to be evaluated within the context of the particular patient sample, since it is undoubtedly true that samples differ from site to site. For example, Hollon s sample included a higher proportion of severely depressed patients than did the TDCRP, while the TDCRP s requirement of rescreening 1-2 weeks after initial screening assured a sample with stable depressive symptomatology. Of particular importance would be pos- DISENTANGLING THERAPISTS FROM THERAPIES - ELKlN I1

13 sible differences in difficulty level of the patient sample, since this has been shown to have an inverse relationship with nieasures of competence (Foley, O Malley, Rounsaville, I rusoff, & Weissman, 1987). Reporting results on the saiiie competence measures across studies would, of course, not really solve our problems of disentangling the therapists from the therapy but, along with reporting the amount of variance attributable to competence (and to therapists per se), it might at least help us to always keep in mind the potential importance of the therapist in our reports of the efficacy of a treatment. Once again, special problems emerge in comparative studies with different therapists in each treatment condition. The critical question then becomes: How can we establish whether therapists in two different treatments have an equal level of competence? This question has been touched on repeatedly throughout this article, because it would seetii to be the most pressing concern when trying to discern whether differential treatment findings are in Eict due to the treatment approaches or to the therapists who carry thein out. In fact, the other issues considered here-whether therapists differ in terms of demographic or experience variables, whether they have received dieerent amounts of training or different types and intensity of supervision-derive their importance largely &om the possibility that they may impact on the competence with which the treatment is being provided. In that sense, competence may be the final comnion pathway through which other variables may affect outcome. Although a comprehensive discussion of the conceptualization and nieasurenient of therapist competence is beyond the scope of this article, some specific issues related to comparing competence across different treatments are addressed, again largely within the context of the TDCRP. The major problem faced by the researcher attempting to unravel effects due to possible differences in conipetence of therapists in different treatment conditions lies in the intrinsic differences in the ways in which proponents of the different approaches view competence. As Schaffer (1982) has pointed out, In order to develop a measure of skillhlness, one s theory of psychopathology and psychotherapy must be used to define the qualities of skillful therapist behavior (p. 678). In addition to conceptual differences, proponents of different treatments may also differ in the way they approach the development of instruments to tap competence. This was certainly true in the TDCRP, in which the competence measures developed by the CBT and IPT experts were very different, in structure as well as in content. The CTS (Young & Beck, 1980), developed to measure competence in CBT, consists of l l items, each rated on a 7-point Likert scale. Items include both general therapeutic skills (eg, appropriate setting of an agenda, interpersonal effectiveness) and more specific CBT skills (e.g., quality of strategy for change and skill in application of cognitive behavioral techniques).. A detailed manual provides rating guidelines. The summed score for all of the items is used to represent the overall level of competence. In addition, two major factors have been identified in the CTS, an overall Cognitive Therapy Skill factor, which includes general as well as technical skills, and a Structuring Skills Factor. A fair amount of psychometric work has been carried out on this instrument (e.g., Vallis, Shaw, & Dobson, 1986). The IPT measure of competence is a more complex instrument, with two major components, a Therapy Strategy Rating Form (TSRF) and a Process Rating Form (O Malley et al., 1988). The TSRF first asks the rater to identify the patient s major IPT problem type (e.g., role disputes, griefand loss) and then to indicate whether each of nine strategies for addressing these problems is present and, on a 7-point scale, to rate the skill with which the strategies are implemented. Similarly, the Process Rating Form asks the rater to indicate which of nine IPT techniques (e.g., exploratory techniques, decision analysis) are used as well as the quality with which they are applied. A composite index of overall therapist skill is derived &om the means of the item ratings on each of these forms as well as four genera1,ratings of IPT skills and a rating of overall quality of the session. Given these differences in the nature of the CBT and IPT competence scales, there is no way that we could judge from their respective trainers ratings whether the therapists in the two treatments were equally competent. This problem is probably inherent in the actual differences in what is focused on in the training of therapists for each of the treatments, as well as differences in approach to the evaluation of performance. In designing the TDCRP, we had emphasized the need to maintain the integrity ofeach of the treatment conditions, with their being carried out according to the specification of the experts in that approach. This included our encouraging each of the training sites to develop and use instruments that they CLINICAL PSYCHOLOGY: SCIENCE AND PRACTICE V6 N1, SPRING

14 themselves would consider adequate for evaluating their therapists competence. No attempt was made to equate these measures, since we considered it important to rely on the experts in each of the treatments to determine, based on their own judgments and evaluation methods, whether therapists were sufficiently competent in their approach to represent it in the outcome study. Only in the process of carrying out the study did we recognize, as previously indicated, the possible iniportance of being able to compare directly the competence levels of the therapists in the dfferent treatments. In light of the difficulty of comparing ratings based on the CBT and IPT competence measures, NIMH staff asked trainers and their expert raters to complete an additional rating for each therapist. This was a global rating of how close the therapist came to the rater s concept of the ideal therapist for his or her respective treatment condition, a measure that was intended to provide a common metric across treatments. This ideal therapist rating was completed by the trainers on the basis of all of their information about a therapist s performance during the course of the study and by the outside experts on the basis of viewing several videotapes of treatment sessions for each therapist. A comparison of mean ratings on this measure for therapists in CBT and IPT corroborated some of the impressions described earlier in this article, in that IPT therapists were rated as being significantly closer to an ideal standard for IPT than were CBT therapists to an ideal standard for CBT (means of 5.3 and 3.7, respectively, on a 7-point scale [I = completely unlike ideal to 7 = completely matches ideal], t = 3.02, df = 15, p <.Ol). This finding was also consistent with the larger number of red-line calls made by the CBT trainers when sessions were rated at below a level of adequate performance. These findings, coupled with the more informal comments by the trainers, do suggest that the IPT therapists may have been more competent in carrying out IPT than the CBT therapists were in carrying out CBT. Although efforts had been made to equate the competence of the therapists, in terms of training standards and the use of competence criteria developed by the proponents of each approach, the possibility of differential levels of competence should, perhaps, not be too surprising. This goes back to the old therapy/new therapy problem referred to above. IPT therapists generally reported that the IPT approach was quite similar to their usual approach to treatment of acutely depressed patients (with the excep- tion of the 16-week time limit and some specific strategies), while many of the CBT therapists, even if they had some previous experience with the approach, had a great deal of new material to learn in their training, both in terms of specific techniques and especially in terms of overall conceptualization of cases. As Dobson and Shaw (1988) have pointed out, it takes much longer to teach formulations than to teach specific techniques. If their past experience and training had been largely in another treatment approach, CBT therapists probably also had a great deal to unlearn. In this regard, it is of interest to note the therapists responses, prior to their participation in the study, to a question on the Therapist Information form: What kind of therapy have you used with depressed patients? All of the IPT therapists had used either long- or short-term psychodynamically oriented therapy or both, and none had used cognitive or behavioral approaches. On the other hand, while all of the CUT therapists had used either cognitive or behavioral approaches, or both, seven ofthe eight had also used some psychodynamic treatment. Given these issues, it may well be that the training of CBT therapists, although fairly intensive, was not suficient to produce a level ofcompetence that would later be consistently maintained when supervision was markedly reduced. This view is in accord with that of Thase (1994) when he questions the relative exportability of the two treatments in the TDCRF! Thase makes the interesting suggestion, relevant also to some of the earlier discussion regarding training and supervision, that in future studes of CBT therapists demonstrate their capacity to both achieve and maintain competence before they are approved for research 49). It should be stressed, however, that the findings in regard to a differential level of competence of the IPT and CBT therapists cannot at this time be considered definitive, since they are based on rather crude measures, and since we cannot rule out the possibility that CBT trainers and experts used more stringent standards and IPT trainers more liberal standards in judging therapist performance. The total context of the TDCRP should also be taken into account, in that average outcome for the CBT patients was in the same general ballpark as in most previous studies, IMI-CM and IPT were very efficacious treatments, especially for the more severely depressed patients, and PLA-CM was a stringent control condition (see discussions in Elkin, 1994; Elkin et al., 1996; Elkin et al., DISENTANGLING THERAPISTS FROM THERAPIES ELKIN 23

15 1995). The fact that what data we have, however, point in the direction of a possibly higher level of competence for the IPT as compared to the CBT therapists does underline a major theme of the present article: the difficulty of determining whether a difference in outcome between two therapies is truly due to differences in the treatments or whether it may be due to differences between the groups of therapists carrying out those treatments. This, then, is the issue that we must still confront. Across-Treatment Measures of Competence Given the difficulty in comparing competence of therapists across treatment approaches with instruments developed separately within each of the approaches, a reasonable question to ask is: Would it be possible to measure at least some aspects of therapeutic competence with instruments that cut across therapies? The ideal therapist rating was only a very crude beginning in this direction. What would be needed are instruments that could measure characteristics or dimensions of therapist behavior that would be considered indices of competent therapist performance in a variety of therapeutic modalities. One obvious candidate is the therapist s ability to foster a good or productive relationship, since the therapeutic alliance is currently considered an important component of many different therapeutic approaches. Therapeutic alliance measures have, in fact, been applied across different treatments in several outcome studies (e.g., Krupnick et al., 1994; Krupnick et al., 1996; Luborsky et al., 1985; Marmar, Gaston, Gallagher, & Thompson, 1989). In the TDCRP, an instrument originally developed within the context of psychodynamically oriented therapy, the Vanderbilt Therapeutic Alliance Scale (VTAS; Hartley & Strupp, 1983), was modified to apply to all of the treatment conditions (Krupnick et al., 1996). Mean scores on this measure were nearly identical for the CBT and IPT treatment conditions. It is of interest here, however, that it was only the patient factor of the modified VTAS that was significantly related to outcome. Not only was the therapist factor not related to outcome, but it could not, in fact, be rated with adequate reliability. Thus, these data do not provide us with much admtional information about the equivalence of this important aspect of the therapists performance in the two treatment conditions. This speaks, again, to the need for hrther development of methods to rate therapist performance across treatments. We must keep in mind in this regard, however, that it is the patient sperception of the therapeutic conditions offered by the therapist that have been found to be most strongly related to outcome (Horvath & Symonds, 1991). One measure that may prove especially useful here is the Working Alliance Inventory (WAI; Horvath & Greenberg, 1986), which was developed as a pantheoretical measure and which has been rated from therapist, patient, and observer perspectives. In hoping to find a single measure of the alliance that is equally applicable across different treatment conditions, however, we must be aware of the possibility that, although the strength of the alliance is probably important in all treatments, there may be differences in the type of alliance that is necessary for progress in different treatments as well as with different patient populations (Bordin, 1979). The task of developing across-treatment measures becomes much more difficult, both conceptually and methodologically, when we consider other aspects of therapist competence, such as the implementation of therapeutic strategies and use of specific techniques. * Here the intrinsic differences in the treatment approaches would tend to argue against the possibility of developing across-treatment measures. Yet the current psychotherapy research literature does offer some ideas about possible sources of input for the development of such instruments. Several of these are listed below. 1. Goldfiied s (1 980) conceptualization of pantheoretical principles of therapeutic change focuses on the midlevel construct of therapeutic strategies (rather than theoretical formulations or specific techniques) that span different forms of therapy. Although the specific techniques used to implement the strategies may differ, it may be possible to develop measures of competence with which such strategies are implemented that could be applied across treatments. 2. Related to Goldfiied s conceptualization of pantheoretical strategies, one might try to develop a competence measure based on the adequacy of therapists interventions for accomplishing goals shared by different treatment approaches at specific phases of therapy, for example, the goals of adequate assessment withm the context of the treatment approach, adequate case formulation, adequate strategic approaches and interventions consistent with the formulation, and adequate handling of termination issue^.'^ 3. Measures of competence developed within a specific treatment approach might usefully be extended to other treatments. For example, Crits-Christoph s method CLINICAL PSYCHOLOGY: SCIENCE AND PRACTICE V6 N1. SPRING

16 for measuring accuracy of interpretation (Crits-Christoph, Cooper, & Luborsky, 1988), developed in a psychodynamic therapy setting, is now being applied to tapes of IPT and CBT in the TDCRP. 4. Measures of behaviors found to differentiate therapists who are more or less efficacious could serve as the basis for competence measures, especially if any of the behaviors are found to be consistent across treatments. 5. Findings regarding therapist behaviors that contribute to positive patient change in intensive studies of therapy process could be drawn upon, for example, findings of Henry and colleagues (Henry et al., 1986; Henry, Schacht, & Strupp, 1990) regarding negative effects of multiple or complex therapist communications (essentially mixed messages) and communications that contain hostile elements; findmgs by Rice and colleagues (e.g., Rice & Saperia, 1984) regarding the negative impact of a therapist s responding to a patient s secondary, rather than primary, emotional reaction. Although developed in particular therapeutic frameworks, these probably have relevance to other approaches as well. 6. It might be fruitful to revisit other measures already developed, especially of negative contributions to outcome, for example, the Negative Indicators Scale (Suh, Strupp, & O Malley, 1986), to see if they could be modified to be relevant across therapeutic approaches. This list is simply a sampling of a few potentially useful sources of input for the development of across-treatment competence measures. Such measures could draw not only on theoretically based conceptualizations of competence, within or across treatment approaches, but also on what we have already learned about the relationship of certain therapist behaviors to patient change.14 In the actual development of instruments, we might also avail ourselves of newer approaches in psychometrics, such as Item Response Theory (IRT) methods (Hambleton, Swaminathan, & Rogers, 1991), which may make it possible to develop comparable instruments for different therapeutic approaches (see, e.g., Chang & Cella, 1997). Such instruments could include both shared (generic) items and some that are idiosyncratic for each treatment.i5 Perhaps it is unrealistic or at least premature to foresee the possibility of the development of pan-theoretical measures of therapist competence. We may not be at a point where either our general theories of therapeutic change processes or our empirical knowledge about therapist behavior related to patient change is sufficiently developed to make the creation of such an instrument possible. And, in fact, some important components of competence in individual treatments may have no counterparts in others. To the extent that it would be possible, however, the development of across-treatment measures of at least some components of therapist competence could greatly facilitate the identification of treatment effects that are at least in part due to differences in the competence with which therapists conduct the treatment. In the absence of such measures at the present time, we have had to content ourselves, in the TDCRP, with the global ideal therapist ratings and other more indirect indices of competence. Until better across-treatment methods are developed, it may in fact be helpful to at least use a measure such as the ideal therapist rating, perhaps refining it by including various dimensions of idealness. Another possibility is to operationahze the ideal for a particular treatment approach by using the ratings assigned to a recognized expert in that approach on a treatmentspecific competence measure as a criterion for the evaluation of competence of therapists using that approach. D IS S EM I N All 0 N 0 F PSYCH OT H E R APY 0 UTCOM E STUDY FINDINGS At the same time that I was struggling with the dilemma of disentanglmg therapists &om therapies, stimulated by the questions raised by the TDCRP findings, this topic was made even more salient by the increased emphasis on the dissemination of information regarding effective treatments for specific disorders and the publication of sets of related guidelines and recommendations (e.g., American Psychiatric Association, 1993; Depression Guideline Panel, 1993). Most important for psychologists was the publication of the reports of the American Psychological Association s Division 12 Task Force on Promotion and Dissemination of Psychological Procedures (Task Force, 1995) and the succeeding Task Force on Psychological Interventions (Chambless et al., 1996). The first Task Force s mandate was to consider methods for educating clinical psychologists, third party payors, and the public about effective psychotherapies (Task Force, 1995, p. 3). The reasons for this mandate are easily understood, particularly in the current context of concerns regarding insurance coverage and managed care. The publication of the Task Force report led to a great deal of discussion and some controversy (Kazdin, 1996) about the criteria for empirically validated treatments, and the list of examples of treatments meeting the criteria. Questions were also raised about the role of therapist and patient variabil- DISENTANGLING THERAPISTS FROM THERAPIES ELKIN 25

17 ity and the relevance of controlled efficacy studies for clinical practice (e.g., Garfield, 1996). Of greatest relevance for the current discussion are questions about the identification of and dissemination of information about empirically validated treatments, without considering the role of the therapists conducting the treatments. As Garfield (1 996) has pointed out, Once training [in a manualized treatment] is completed and perforniance monitored, outcome is appraised in terms of the specific therapies evaluated, hence the reference to validated therapies and the implication that all therapists who perform that form of therapy will secure positive results. One can reasonably ask whether all therapists perform at the same or comparable levels of efficacy even if they supposedly have had similar manualized 221). In fairness to the Task Force members, the first report (Task Force, 1995) does focus heavily on the training of therapists in approaches that have been empirically supported, and training needs are, in fact, a major concern of the entire endeavor. In addtion, there is attention paid to the need to evaluate therapist competence. Nonetheless, in the listing of treatments that have been empirically validated or supported, there are no qualifications included regarding therapist experience, training, supervision, competence, or other therapist attributes required for the efficacious practice of specific treatments. It would seem important to include, in any listing of efficacious or empirically validated treatments, just as in the reports of any individual outcome study, critical information about the therapists carrying out the treatments. Of foremost importance is information about the variability of therapist performance and efficacy and, if it is known, about the relationship of particular therapist characteristics to outcome. (Here, again, it is important to check on comparability of patients in the case loads of different therapists.) The nature of the therapist samples in the studies forming the basis for the judgment that a treatment is empirically supported should be clearly described, including such variables as therapists past experience with the treatment they are carrying out in the study and with other treatments. Therapist training, supervision during the course of the study, and level of competence in conducting the treatment are all crucial pieces of information. In the absence of information about the therapists in the studies that form the basis for judgments about empirical support for particular treatments, there is the implica- tion that the specific techniques of the designated therapies are the necessary and sufficient factors for positive change in psychotherapy (Garfield, 1996, p. 221) and that therapists are all equally efficacious. (There is also the assumption that the therapies are equally efficacious for all patients meeting a certain diagnostic criterion; this is also open to question, although it is not the main focus here.) If there is considerable therapist variability, such an implication may be misleading, both to practitioners and to potential patients. The absence of information about the necessary training and supervision of therapists may also lead to erroneous conclusions. If, for example, the efficacy of a treatment has been demonstrated only in studies in which therapists receive intensive supervision following training, it is important that the conclusion not be drawn that the treatment will be equally efficacious in the hands of newly trained therapists who do not receive additional supervision. Until we have studies that actually test such generalizability of outcome findings, it would seem crucial to qualify any pronouncements about treatment efficacy by at least indicating in whose hands, and under what conditions, the treatments have been found to be efficacious. As indicated earlier, there has been a resurgent interest by psychotherapy researchers in the therapist, including a focus on the role played by therapist effects, as well as on the relationship to outcome of specific therapist and therapist-patient interactive variables. There has also begun to be more discussion about the importance of considering the therapist in our statements about outcome findings. In the last two editions ofthe Handbook $Psychotherapy and Behavior Change, the chapter on Effectiveness of Psychotherapy (Lambert & Bergin, 1994; Lambert, Shapiro, & Bergin, 1986) has included a section on therapist variables in the general discussion of outcome. In the last edition, the authors comment that the therapist factor, as a contributor to outcome, is looming large in the assessment of outcomes (Lambert & Bergin, 1994, p. 182). In their concluding section, Lambert and Bergin (1 994) write: Psychologists, psychiatrists, social workers, and marriage and family therapists as well as patients can be assured that a broad range of therapies, when ofered by skilyil, wise and stable therapists, are likely to result in appreciable gains for the client (p. 180, emphasis added). Unfortunately, we do not yet know a great deal about how to evaluate skill of therapists, let alone wisdom. AS our knowledge about the role of the therapist grows, CLINICAL PSYCHOLOGY: SCIENCE AND PRACTICE * V6 Nl, SPRING

18 however, we should be able to make more specific and empirically grounded statements about the efficacy ofparticular treatments in the hands of certain types of therapists working in particular contexts. We may never be able to truly disentangle the effects due to the therapist from those due to the therapy, because they may often be inherently intertwined and also very interactive with particular patient attributes. Although there may, thus, not be any true resolution of the dilemma posed in this article, it is important that we acknowledge that the dilemma exists. In evaluating the efficacy of any treatment, we must always bear in mind that the therapist is an important player in the therapeutic enterprise (as is the patient) and that treatments are never disembodied entities, equally effective in the hands of different therapists. NOTES 1. The principal investigators and project coordinators at the three participating research sites were Stuart M. Sotsky and David R. Glass, George Washington University; Stanley D. Imber and Paul A. Pilkonis, University of Pittsburgh; and John T. Watkins (now at the Atlanta Center for Cognitive Therapy) and William R. Leber, University of Oklahoma. The principal investigators and project coordinators at the three sites responsible for training therapists were Myrna M. Weissman (now at Columbia University), Eve S. Chevron, and Bruce J. Rounsaville, Yale University; Brian F. Shaw (now at Hospital for Sick Children, Toronto) and T. Michael Vallis (now at Dalhousie University), Clarke Institute of Psychiatry; and Jan A. Fawcett and Phillip Epstein, Rush Presbyterian-St. Luke s Medical Center, Chicago. 2. I thank John Docherty and Susan Fiester for their participation in this process. 3. The percentage of red-lined tapes reported in Shaw et al. (1998) is 27%; we believe the discrepancy is due to our only including regularly scheduled evaluations here and not including follow-up evaluations after red-lining. Although the intent of red-line calls was to ensure adequate therapist performance within the treatment model being studied, we must consider the possibility that these calls may sometimes have been experienced as aversive by the therapists. 4. In all of these studies, questions may be raised about whether there are actually significant therapist effects. Where significant variance is found attributable to therapists across treatments, for example, it is important to consider the role that may be played by the treatments themselves, both in their conmbution to this variance and when relating other variables to therapist effectiveness. Where there is no significant variance due to therapists (or where this has not been tested), and findings are based on arbitrary divisions on an effectiveness dimension or on extreme groups of more and less effective therapists, it is especially important to replicate findings. 5. In regard to the issue of differences among therapists related to intrinsic differences in treatments, Bruce Rounsaville (personal communication, January 6, 1998) makes the following interesting point: This is particularly an important issue for treatment of substance use disorders, where therapists who perform some types of treatments, particularly 12 step treatments, typically have less formal education along with personal recovery experiences when compared to therapists who perform other treatments. The important issue is to match the therapist training and competence to the spirit of the treatment being delivered. 6. One must also keep in mind the fact that the exclusion of 21% of the patients originally meeting screening criteria in the TDCRP, due to their lack of maintaining the required diagnosis and symptomatology at rescreening, undoubtedly reduced the number of very early treatment responders. 7. Both Hollon and I, commenting on the lack ofa difference between CBT and medication for the more severely depressed patients in his study and at least the suggestion of a difference in the TDCRP, had both suggested that the difference in outcome might be due to the lack of intensive supervision of CBT therapists during the outcome phase of the TDCRP. In comparing the data ofthe two studies, however, we later noted (Elkin et al., 1996) that, while the CBT patients in the TDCRP did do slightly worse than those in Hollon s study, the imipramine patients in his study did quite a bit worse than those in the TDCRP IMI-CM condition. It seemed that this latter difference might, in hct, be most responsible for the different findings in the two studies. 8. Although this issue has not been raised in the literature, one might reasonably question whether the long-distance supervision and consultation in the TDCRP (videotapes mailed to the training site, followed by supervisory telephone calls) may have been less effective than on-site supervision. 9. For copies ofthe Collaborative Study Psychotherapy Rating scale and related materials prepared under NIMH Contract (ER), order System for Rating Psychotherapy Audiotapes &om U.S. Department of Commerce, National Technical Information Service, Springfield, VA Although Myrna Weissman and her colleagues developed the IPT measures, in the case of CBT, it was actually the site that had originally developed and piloted the training program, led by Aaron T. Beck and Jeffrey Young (not the training site, led by Brian Shaw) that developed the CBT competence measure. 11. It was not possible to retrieve information from the CBT training site regarding the exact source of these mean ratings, that is, whether they were based on trainen ratings, outside experts ratings, or a combination; however, the finding was the same whether these means were compared to the IPT mean rat- DISENTANGLING THERAPISTS FROM THERAPIES * ELKIN 27

19 ings based on trainers ratings, outside experts ratings, or a combination. 12. By separating, in this discussion, the measurement of the therapist s contribution to the therapeutic relationship from the therapist s implementation of strategies and use of techniques, I do not mean to suggest that these are independent factors in the therapy process. Any adequate measure of the therapist s competence in the use of strategies and techniques would, in fact, have to include some consideration of the context of the therapeutic relationship, and in relating such a measure to outcome, it would be important to consider the mutually interactive nature of the relationship and specific interventions. 13. I would like to acknowledge the contribution to some of these ideas by my colleagues at a colloquium at the School of Social Service Administration, University of Chicago, particularly Elsie Pinkston and Sharon Berlin. 14. A one-to-one relationship between competence and outcome is, of course, not expected since, no matter how competent the therapist, some patients will not respond to treatment. On the other hand, ifa competence measure is not related to patient change, one must question the meaning of the concept of competence that is being measured (or the measure itself). 15. I thank Jane Yamaguchi for her helpful input in regard to IRT methods. ACKNOWLEDGMENTS This work was supported in part by National Institute of Mental Health Research Scientist Award MH I express my deep appreciation to the many participants in the NIMH Treatment of Depression Collaborative Research Program (TDCRP), including the patients, who made the study possible. Special thanks to Tracie Shea and Moms Parloff. my main NIMH colleagues in the TDCRP. Moms Parloff also deserves credit for some of the improvements in this revision of an earlier draft of the manuscript, but none of the blame for any of its shortcomings. Thanks also to Jane Yamaguchi, Dale Faber, and Edward Henderson for their able assistance on the manuscript, to Robert Gibbons for the recent random regression analyses of therapist effects, and to Brian Shaw and Bruce Rounsaville for checking the accuracy of statements regarding CBT and IPT training and therapist performance. REFERENCES Addis, M. E. (1997). Evaluating the treatment manual as a means of disseminating empirically validated psychotherapies. Clinical Psychology: Science arid Practice, 4, American Psychiatric Association. (1993). Practice guideline for major depressive disorder in adults. AmerikanJournal ofpsychiatry, 150(Suppl.), Beck, A. T., Rush, A. J., Shaw, B. F., &Emery, G. (1979). Cognitive therapy ofdepression. New York Guilford Press. Bergin, A. E. (1971). The evaluation of therapeutic outcomes. In A. E. Bergm & S. L. Garfield (Eds.), Handbook ofpsychotherapy and behavior change: An empirical ). New York: Wiley. Beutler, L. E. (1997). The psychotherapist as a neglected variable in psychotherapy: An illustration by reference to the role of therapist experience and training. Clinical Psychology: Science and Practice, 4, Beutler, L. E., Machado, P. P. P., & Neufeldt, S. A. (1994). Therapist variables. In A. E. Bergin & S. L. Garfield (Eds.), Handbook of psychotherapy and behavior change (4th ed., pp ). New York: Wiley. Blatt, S. J., Sanislow, C. A., Zurof, D.C., & Pilkonis, F! A. (1996). Characteristics of effective therapists: Further analyses of data fiom the National Institute of Mental Health Treatment of Depression Collaborative Research Program. Journal of Consulting and Clinical Psychology, 64, Bordin, E. S. (1979). The generalizability of the psychoanalytic concept of the working alliance. Psychotherapy: 7 heory, Research and Practice, 16, Butler, S. F., & Strupp, H. H. (1986). Specific and nonspecific factors in psychotherapy: A problematic paradigm for psychotherapy research. Psychotherapy, 23, Castonguay, L. G., Goldfried, M. R., Wiser, S., Raue, P. J., & Hayes, A. M. (1996). Predicting the effect of cognitive therapy for depression: A study of unique and common factors. Journal ofconsulting and Clinical Psychology, 64, Chambless, D. L., Sanderson, W. C., Shoham, V., Johnson, S. B., Pope, K. S., Crits-Christoph, E., Baker, M., Johnson, B., Woody, S. R., Sue, S., Beutler, L., Williams, D. A., & McCurry, S. (1996). An update on empirically validated therapies. Clinical Psychologist, 49, Chang, C. H., & Cella, D. (1997). Equating health-related quality of life instruments in applied oncology settings. Physical Medicine and Rehabilitation: State of the Art Reviews, 11, Clarke, G. N. (1995). Improving the transition fiom basic efficacy research to effectiveness studies: Methodological issues and procedures. Journal of Consulting and Clinical Psychology, 63, Cries-Christoph, E., Baranackie, K., Kurcias, J. S.. Beck, A. T., Carroll, K., Perry, K., Lubonky, L., McLellan, A. T., Woody, G. E., Thompson, L., Gallagher, D., & Zitrin, C. (1991). Meta-analysis of therapist effects in psychotherapy outcome studies. Psychotherapy Research, 1, Crits-Christoph, P., Cooper, A., & Luborsky, L. (1988). The accuracy of therapists interpretations and the outcome of dynamic psychotherapy. Journal ofcomukng and Clinical Psycholqy, 56, CLINICAL PSYCHOLOGY: SCIENCE AND PRACTICE V6 N1. SPRING

20 Crits-Christoph, F!, & Mintz, J. (1991). Implications of therapist effects for the design and analysis of comparative studies of psychotherapies. Journal cf Consulting and Clinical Psychology, 59, Depression Guideline Panel. (1993, April). Depression in primary care: Gl. 2. Treatment oj major depression (Clinical Practice Guideline No. 5, AHCPR Publication No ). Rockville, MD: U.S. Department of Health and Human Services, Public Health Service, Agency for Health Care Policy and Research. Dobson, K. S., & Shaw, B. F. (1988). The use oftreatment manuals in cognitive therapy: Experience and issues. Journal of Consulting and Clinical Psychology, 56, Ellcin, I. (1988, June). The relationship of adherence to outcome. Paper presented at the meeting ofthe Society for Psychotherapy Research, Santa Fe, NM. Elkin, I. (1994). The NIMH Treatment of Depression Collaborative Research Program: Where we began and where we are. In A. E. Bergin & S. L. Garfield (Eds.), Handbook ofpsychotherapy and behavior change (4th ed., pp ). New York: Wiley. Elkin, I., Gibbons, R. D., Shea, M. T., & Shaw, B. F. (1996). Science is not a trial (but it can sometimes be a tribulation). Journal OfConsulting and Clinical Psychology, 64, Elkin, I., Gibbons, R. D., Shea, M. T., Sotsky, S. M., Watkins, J. T., Pilkonis, P. A., & Hedeker, D. (1995). Initial severity and differential treatment outcome in the National Institute of Mental Health Treatment of Depression Collaborative Research Program. Journal of Consulting and Clinical Psychology, 63, Elkin, I., Parloff, M. B., Hadley, S. W., & Autry, J. H. (1985). NIMH Treatment of Depression Collaborative Research Program: Background and research plan. Archives of General Psychiatry, 42, Elkin, I., Pillconis, F! A., Docherty, J. I?, & Sotsky, S. M. (1988). Conceptual and methodological issues in comparative studies of psychotherapy and pharmacotherapy, I: Active ingredients and mechanisms of change. American Journal of Psychiatry, 145, Elkin, I., Shea, M. T., Collins, J. F., Klett, C. J., Imber, S. D., Sotsky, S. M., Watkins, J. T., &Parlog M. B. (1990). NIMH Collaborative Research on Treatment of Depression [Letter to the editor]. Archives ofceneral Psychiatry, 47, Elkin, I., Shea, M. T., Watkins, J. T., Imber, S. D., Sotsky, S. M., Collins, J. F., Glass, D. R., Pilkonis, P. A., Leber, W. R., Docherty, J. P., Fiester, S. J., & Par106 M. B. (1989). National Insthe of Mental Health Treatment of Depression Collaborative Research Program: General effectiveness of treatments. Archives $General Psychiatry, 46, Fawcett, J., Epstein, P., Fiester, S. J., Elkin, I., & Autry, J. H. (1987). Clinical management-imipramine/placebo adminis- tration manual: NIMH Treatment of Depression Collaborative Research Program. Psychopharmacology Bulletin, 23, Foley, S. H., O Malley, S., Rounsaville, B., Prusoff, B. A., & Weissman, M. M. (1987). The relationship of patient difficulty to therapist performance in interpersonal psychotherapy of depression. Journal ofaflective Disorders, 12, Frank, E., Kupfer, D. J., Wagner, E. F., McEachran, A. B., & Cornes, C. (1991). Efficacy of interpersonal psychotherapy as a maintenance treatment of recurrent depression. Archives ofgenera1 Psychiatry, 48, Garfield, S. L. (1994). Eclecticism and integration in psychotherapy: Developments and issues. Clinical Psychology: Science and Practice, 1, Garfield, S. L. (1996). Some problems associated with validated forms of psychotherapy. Clinical Psychology: Science and Practice, 3, Garfield, S. L. (1997). The therapist as a neglected variable in psychotherapy research. Clinical Psychology: Science and Practice, 4, Gibbons, R. D., Hedeker, D., Elkin, I., Waternaux, C., Kraemer, H. C., Greenhouse, J. By, Shea, M. T., Imber, S. D., Sotsky, S. M., & Watkins, J. T. (1993). Some conceptual and statistical issues in analysis of longitudinal psychiatric data: Application to the NIMH Treatment of Depression Collaborative Research Program dataset. Archives of General Psychiatry, 90, Goldfiied, M. R. (1980). Toward the delineation of therapeutic change principles. American Psychologist, 35, Goldhed, M. R., & Newman, C. F. (1992). A history of psychotherapy integration. In J. C. Norcross & M. R. Goldfried (Eds.), Handbook ofpsychotherapy integration (pp ). New York Basic Books. Gum, A. S., & bin, A. M. (Eds.). (1977). Eftaivepsychotherapy. New York Pergamon Press. Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals ofitem response theory. Newbury Park, CA: Sage. Hartley, D. E., & Strupp, H. H. (1983). The therapeutic alliance: Its relationship to outcome in brief psychotherapy. In J. Masling (Ed.), Empirical studies oj psychoanalytical theories (Vol. 1, pp. 1-37). Hillsdale, NJ: Erlbaum. Henry, W. P., Schacht, T. E., & Stmpp, H. H. (1986). Structural analysis of social behavior: Application to a study of interpersonal process in differential psychotherapeutic outcome. Journal ofconsu1ting and Clinical Psychology, 54, Henry, W. P., Schacht, T. E., & Strupp, H. H. (1990). Patient and therapist introject, interpersonal process, and differential psychotherapy outcome. Journal of Consulting and Clinical Psychology, 58, Henry, W. P., Schacht, T. E., Strupp, H. H., Butler, S. F., & Binder, J. L. (1993). Effects of training in time-limited DISENTANGLING THERAPISTS FROM THERAPIES ELKIN 29