Client Feedback Form Manual

Transcription

1 On Track Outcomes Program Client Feedback Form Manual Prepared for Beacon Health Options by Jeb Brown, PhD, Center for Clinical Informatics February 2014 For more information about this manual contact Jeb Brown, Ph.D. at or call (801) For more information about the On Track Outcomes Program send to

2 Table of Contents Section 1: Overview of the Client Feedback Form...4 Administration... 4 Outcomes Questionnaires and Clinician Feedback... 5 Section 2: Development, Scoring, and Interpretation...6 Development... 6 Absenteeism/Presenteeism... 7 Alliance Scale... 7 Global Distress Scoring and Interpretation... 8 Client Feedback Form Score Ranges for the Three Severity Levels... 9 Monitoring Change over Time... 9 Case Mix Adjustment and Outcomes Benchmarking Reliability and Validity Section 4: CFF-Adult Specific Information...13 Reliability and Construct Validity CFF-ADULT Factor Analysis (N=86,185) IRT Analysis of Global Distress and Substance Abuse Scales Formulas to predict final score (benchmark score) for the episode of care Section 5: CFF Adolescent Specific Information...17 Reliability and Construct Validity Youth CFF Factor Analysis (Responder=Youth, N=9,647) Youth CFF Factor Analysis (Responder= Parent or other adult, N=3747) Formulas to predict final score (benchmark score) for the episode of care Section 6: CFF Child Specific Information...22 Reliability and Construct Validity Child CFF Factor Analysis (Responder=Youth, N=3,278) Child CFF Factor Analysis (Responder=Parent or other adult, N=6,114) Formulas to predict final score (benchmark score) for the episode of care References...26 Appendix A: Adult CFF Normative and Benchmarking Data...28 Results of GLM prediction - adding variables incrementally Beacon Health Options 2 Client Feedback Forms

3 Appendix B: Youth Self-Report CFF Normative and Benchmarking Data...30 Results of GLM prediction - adding variables incrementally Appendix C: Youth - Parent CFF Normative and Benchmarking Data...32 Results of GLM prediction - adding variables incrementally Appendix D: Child - Self CFF Normative and Benchmarking Data...34 Results of GLM prediction - adding variables incrementally Appendix E: Child - Parent CFF Normative and Benchmarking Data...36 Results of GLM prediction - adding variables incrementally Beacon Health Options 3 Client Feedback Forms

4 Section 1: Overview of the Client Feedback Form The Client Feedback Forms (CFFs) used in Beacon s On Track Outcomes Program are brief, reliable and valid client completed questionnaires designed to measure improvement in symptoms, quality of social relations, and functioning at work/school and other daily activities. Use of these or similar outcome questionnaires has been shown to significantly improve treatment outcomes across large sample of clinicians treating thousands of clients. The questionnaires measure symptom severity and improvement across a broad range of problems, and are not intended to be diagnostic or to substitute for clinical evaluation or other assessments that a clinician may routinely conduct. There are three versions of the CFF now available: Adult version (age 18 and older) Youth version (ages 13-17; completed by youth or adult) Child version (under age 13; completed by youth or adult) All versions of the questionnaires have high reliability (coefficient alpha =>.87). Extensive factor analyses demonstrate high construct validity, with items loading on the common factor found in the most commonly used measures of treatment outcomes. The Client Feedback Forms are unique among available outcome measures in that items asking for feedback on the working alliance are included on every questionnaire. The use of alliance items has been shown to reduce no shows and premature termination while contributing significantly to improved outcomes. Administration The questionnaires were designed for routine use in clinical practice and can be completed by most clients in less than two minutes, using a simple paper form. When first introducing the questionnaire to a client, it is best practice to provide a brief explanation of the reason for the questionnaire, and how it will be used as a routine part of treatment. At many sites, this explanation is provided by office staff when the client checks in. Following are a sample explanation scripts used by support staff: Beacon Health Options 4 Client Feedback Forms

5 Please take a moment to fill out the questionnaire. This will help you and your therapist talk about how treatment is going. We are really excited about the questionnaires. Research shows that therapists who use questionnaires like these get much better outcomes. Please answer as honestly as you can. This is important to your treatment, because it will help your therapist understand how to help you. Research asking clients to provide feedback on their experience with the questionnaires indicated a high level of honesty and willingness to complete the alliance items, especially if they perceived that the clinician was interested in their responses. Of this group, 95% agreed that the questionnaires were helpful in treatment. For this reason it is recommended that the clinician acknowledges and value the client s willingness to give feedback. Taking a few moments at the start of the session to review the questionnaire provides the clinician with a wealth of clinically relevant information while acknowledging the value of the client s time to complete the questionnaire. Research of the past several years has revealed that clinicians who use the questionnaires with a high percentage of their practice display significant gains in overall outcomes. Outcomes Questionnaires and Clinician Feedback A large body of research supports the proposition that routine use of outcomes questionnaires combined with feedback to the clinician results in significant improvement in treatment outcomes, as measured by pre-post change, percentage of patients improved, and reduced dropout rates (Lambert, 2009; Goodman et al., 2013). Among users of On Track Client Feedback Forms and related ACORN questionnaires, observation of frequency of measurement and use of the Decision Support Toolkit provides real world confirmation of results from clinical trials. Both the number of clients measured and the frequency at which the clinician views the data are strongly correlated with year-to-year improvement in treatment outcomes at the clinician level (r>.3; p<.0001; see Brown 2013 in references). Beacon Health Options 5 Client Feedback Forms

6 Section 2: Development, Scoring, and Interpretation Development The CFFs were developed for Beacon s On Track Outcomes program as part of a suite of forms developed cooperatively through A Cooperative Online Resource Network (ACORN). The network consists of statisticians and researchers affiliated with the Center for Clinical Informatics along with over 1,000 clinicians using client completed questionnaires in a wide variety of clinical settings across the country. The initial work on developing the ACORN questionnaire items was done in collaboration with Warren Lambert, Ph.D., at Vanderbilt University. Dr. Lambert was instrumental in the development of the Peabody Treatment Progress Battery (PTPB) for adolescents. Items from the PTPB were utilized with permission, and additional items for adults, adolescents and children were added using the same item format. Normative data on these items was collected through the ACORN network of clinicians. Takuya Minami, Ph.D., of the University of Massachusetts Boston, further assisted in the analyses of the psychometric properties of all of the items. The ACORN form development process takes advantage of the network s ability to collect large amounts of data and continuously test and refine items over time. Rather than focus on the development of static forms, this process centers on the development of reliable and valid items that can be combined flexibly. The psychometric properties of each item, as well as the unique combination, are carefully evaluated with items for a specific questionnaire selected based on the population to be measured and the needs of the various participating organization. The result is a large item inventory with data from over 200,000 clients. Evidence of the validity and reliability of the CFF was derived from data on tens of thousands of administrations in both clinical and community settings. The development process included item analyses to determine: Item frequencies and distributions Item correlations Factor structure Construct validity Scale reliability Sensitivity to change The CFFs consists of items well-suited for general use in outpatient settings. The questionnaires are designed to be as brief as possible while retaining excellent psychometric properties. The Beacon Health Options 6 Client Feedback Forms

7 Adolescent and Child versions may be completed either by the youth or by an adult who knows the youth well. Absenteeism/Presenteeism The two absenteeism/presenteeism items on the adult form are based on the format and wording of items in the Health and Productivity Questionnaire (HPQ), a questionnaire in the public domain developed by Kessler and colleagues at Harvard University. Because the CFF is designed for use by mental health professionals in a mental health setting, the HPQ wording has been revised to focus on absenteeism/presenteeism specifically due to mental health problems, rather than both mental and physical health problems encompassed by the original HPQ questions. Alliance Scale All ACORN questionnaires also include items asking the client to provide feedback on their experience of the prior session. The use of these so-called Alliance items ask for feedback on elements of the therapeutic working alliance, such as agreement on treatment goals, and the client s perception of the quality of the relationship. Use of alliance measure has been demonstrated to reduce treatment dropout and improve outcomes. Among the large sample of clinicians using the CFFs, about 80% of clients complete these items routinely. These clients have significantly better outcomes than those who fail to complete the items. The client s responses on these items may be easily influenced by their perception of the clinician s response. Clients may be reluctant to give anything other than perfect ratings in order to avoid hurting the clinician s feelings or out of fear of possible consequence to the clinician if Satisfaction ratings are high. It rests on the skill of the clinician to create a therapeutic environment that encourages honest feedback on the alliance items. Skilled clinicians are about to use the alliance items as tools to foster a strong collaborative working alliance with the client. Consistent ratings of near perfect alliance are NOT associated with the best outcomes. Rather, clients who provide meaningful feedback early in therapy are very likely to rate the alliance as improved over time. This pattern, displayed by approximately one third of clients, is associated with significantly greater improvement in treatment. The best outcomes are associated with improvement on the Alliance Scale over the course of the treatment episodes. This means that the patients with the best outcomes are also willing to give feedback that the treatment encounters early in the treatment episode are less than perfect, otherwise there is no room for improvement. Beacon Health Options 7 Client Feedback Forms

8 Failing to complete the alliance items is associated with less improvement. If the client leaves the items blank, this provides the clinician with an opportunity to initiate a discussion of how the client is experiencing the treatment process. Global Distress Scoring and Interpretation The core global distress scale (GDS) is scored as the mean of all non-missing items on the form. If more than 4 items are missing from the adult global distress scale or 6 items from the child or youth scales, the questionnaire is not scored. Scores can be divided into three severity ranges (Normal, Moderate, Severe) based on normative data from clinical and community samples. The cut-off scores for each range are included with the information on the individual questionnaires. In the case of the adult questionnaire, the cut-off score for the Normal Range was determined by collecting a sample of over 1,000 individuals from the community who had never sought mental health service combined with a sample of over 75,000 individuals receiving mental health services. The cut-off score was calculated using the method proposed by Jacobson & Truax (1991) as represented by this formula: C = (SD 1)(mean 2 )+(SD 2 )(mean 1 ) SD 1 +SD 2 A score in the normal range means that the score is in a range typical of respondents from a community sample that have never sought mental health services. Seventy-five percent (75%) of a community sample and 25% of a clinical sample will fall into this range. Mental health clients with scores in this range tend to not show improvement with treatment. A score in the moderate range of distress is characteristic of individual seeking mental health services. About 50% of a clinical sample will fall into this range. Mental health clients with scores in this range tend to show significant improvement within a few sessions of therapy, and most complete treatment with a good outcome in fewer than 8 sessions. Twenty five percent (25%) of a clinical sample will have scores in the severe range, while fewer than 10% of a community sample will fall in this range. Clients with scores in this range are highly likely to show rapid improvement with psychotherapy, but may need more sessions to realize the full benefit of treatment. Beacon Health Options 8 Client Feedback Forms

9 In the case of child and adolescent questionnaires, cut-off scores were estimated by the 25 th and 75 th percentile. Practicality prevented collection of large enough community samples. However, the cut-off scores are comparable to those reported for other similar measures with the OQ-45 for adults and YOQ-64 for children and youth. Items were selected for the Client Feedback Form in order to assure comparability to the OQ-45 and YOQ-64. Analysis of archival data for the OQ-45 and YOQ-64 confirms that the Client Feedback Forms produce results very similar to these OQ measures. Client Feedback Form Score Ranges for the Three Severity Levels Form Normal Range Moderate Range Severe Range Adult CFF 0 to to to 4.0 Adolescent CFF 0 to to to 4.0 (Youth completed) Adolescent CFF 0 to to to 4.0 (Parent/Adult completed) Child CFF 0 to to to 4 (Youth completed) Child CFF (Parent/Adult completed) 0 to to to 4.0 Monitoring Change over Time The ability for the clinician to monitor client change as the treatment unfolds is one of the features of On Track that leads to improved outcomes. Identification of clients who are off track assists the clinician in preventing early drop-out in treatment. The On Track Decision Support Toolkit provides graphs for each case. The actual client scores are compared to a predicted score at each assessment point. The predicted score is determined using a statistical prediction technique known as General Linear Modeling. The method takes advantage of all of the normative data for other clients completing multiple questionnaires at multiple points in treatment. The predicted score at each assessment is computed using the initial CFF Global Distress Score, the assessment number, and the number of weeks that have passed since the initial assessment to determine the expected assessment at each measurement point. The actual score can them be compared to the predicted score in order to determine the extent that the clients current score deviates from the expected score. Clients who scores are significantly higher than expected at classified as off track. Beacon Health Options 9 Client Feedback Forms

10 Case Mix Adjustment and Outcomes Benchmarking Simply measuring pre-post change on an outcome questionnaire provides little information without some basis for comparison or benchmarking. The On Track program is based on a long history of research using real world effectiveness data as well as meta-analyses of clinical trials to establish efficacy. At the most basic level, change scores on the CFF questionnaires are converted to effect size, based on dividing the pre-post global distress change score by the standard deviation of the global distress scores at intake. A simple effect size is calculated by dividing the pre-post change score by the standard deviation of the outcome measure at intake. An effect size of one means that the client improved one standard deviation on the measure. In order to make results comparable to results from clinical trials, effect size is only calculated for cases with intake scores above the clinical cutoff score. In a general outpatient population, 75% of cases will be in the clinical range. Another reason for excluding non-clinical range cases is that these who enter treatment reporting little distress do not show improvement on average. This would have the effect of artificially lowering effect sizes and making comparisons to results from published studies invalid. The second manner in which outcomes are benchmarked is by using a large and diverse normative sample of over 90,000 outpatient treated at hundreds sites around the country. The sample represents a wide range of ages and ethnic groups, includes those covered by commercial insurance, EAP programs, Medicaid/Medicare, as well as self-pay and other sources of funding. As such, the sample is highly representative of patients seen in outpatient general practice. In order to establish a benchmark, the statistical procedure known as General Linear Model was applied to establish which variables collected at intake predicted the final global distress score at the end of the episode. Employing this model, it us evident that the first global distress score is the strongest predictor of subsequent scores, including the final score. The intake score alone accounts for 30% to 50% of the variance in final scores, depending on the length of treatment. A second predictor in naturalistic data such as On Track is the session at which the first questionnaires were administered. If the first assessment is at intake or no later than the second appoint, measured change is larger than if the first assessment is later. In many cases, the session number for the first assessment is unknown. In order to take session number into account, each episode is classified as Early Assessment (session 1 or 2), Later Assessment, or Beacon Health Options 10 Client Feedback Forms

11 Unknown. These three categories are included in the General Linear Model as a class variable, but explain less than 1% of additional variance. A third potential predictor is diagnosis. Inclusion of diagnosis adds minimally to the prediction of final score, typically explaining less than.005% of additional variance. In some instances, such as the youth self-reported version of the CFF, diagnosis is a non-significant predictor. Diagnosis is not currently included in the On Track benchmarking models. The prediction of the final (benchmark) score for the On Track program is based on predictive formulas. These formulas take the form of a simple regression formula: Last Score = First Score * slope + intercept). The regression formulas are coded from each version of the questionnaire, with regression formulas differing based on session number at first assessment. Sections 4-6 contain form-specific analyses, including the specific formulas used for each CFF form and session number. The appendices provide detailed normative information for each questionnaire with regards to mean intake scores, final scores, change scores, number of assessments, and average time prepost (reported in weeks) for the entire sample, including a breakout by diagnostic group. A second set of normative information is provided for only those cases with intake scores in the clinical range. Each patient s actual final score is compared to the benchmark target to determine to what extent the patient reported more or less improvement than comparable patient in the complete normative sample. The results of the General Linear Model analyses from which the formulas are derived are provided in the appendices. The general methodology for benchmarking outcomes has been published in a series of peer reviewed journal articles, though in these publications the benchmark scores are calculated using a multivariate GLM. These publications are available upon request (Minami et al. 2007; Minami et al. 2008a and 2008b; Minami et al. 2011). Reliability and Validity Reliability of the global distress scale is measured using Cronbach s alpha, which is a measure of internal consistency. This is consistent with classical test theory, which seeks to develop single factor scales with internal consistency. Reliability for the CFFs are as follows: Adult version = 0.89 Youth version = 0.87 Child version = 0.90 Beacon Health Options 11 Client Feedback Forms

12 Validity of the scales is estimated primarily as construct validity, addressing whether the questionnaire measures a single construct or factor. Prior research indicates that items on most outcomes questionnaires used in behavioral health correlate highly with one another and measure a single construct, generally referred to as Global Distress. For example, Brophy et al. (1988) found that the SCL-90 subscales all load on a common factor, and likewise correlate highly with similar scales from other measures. Miller et al. (2003) found that the Outcome Rating Scale correlates highly with the OQ-45. Enns et al. (1998) performed factor analyses on the Beck Depression Inventory and the Beck Anxiety Inventory. To quote from this study: " [T]he parameter estimate was very high (0.784) and a unidimensional, single-factor model of negative affectivity approached the criteria for good fit. It was concluded that the Beck Anxiety and Depression Inventories assess distinct anxiety and depression phenomena to a limited extent when used in a clinically depressed sample." Recent research by the ACORN collaboration further investigates the relationship between items assessing states of emotional well-being and high life satisfaction with measures of psychiatric symptoms and lost productivity (Brown & Minami, 2013). This work demonstrates that measures of well-being and life satisfaction likewise correlate highly with the same common factor as symptoms and lost productivity. The existence of a global distress factor and the fact that multiple outcome questionnaires, including widely used measures of depression, are all found to be correlated with one another provide strong evidence of the construct validity of patient self-report outcome measures designed to measure and assess global subjective distress. The estimate of reliability and validity for the Alliance Scale is complicated by the fact that the responses are not normally distributed. Rather, approximately 50% of clients report the alliance is virtually perfect all of the time, with little variance from week to week. This pattern is the most common, but is not associated with better than average outcomes. Less than perfect Alliance Scores at the start of the treatment are associated with better outcomes if the client remains engaged in treatment. To a large extent the utility of the Alliance Scale is dependent on the skill of the clinician to elicit and utilize frank feedback from the client. The following sections describe the psychometric properties of each version of the Client Feedback Form, including detailed information on item analyses. Beacon Health Options 12 Client Feedback Forms

13 Section 4: CFF-Adult Specific Information Reliability and Construct Validity The Adult Client Feedback Form consists of 10 items which assess symptoms of depression, anxiety, social relationships, and functioning in work and other daily activities. Three additional items assess problems related to substance abuse. In addition, the forms contain three items which ask for feedback on the client s last session experience. Use of these Alliance items is associated with better outcomes. The adult form also includes items asking about prior treatment and presence of chronic illnesses. The variables may be used when performing calculations for case mix adjustment. The following assessment of the factor structure and reliability of the Adult questionnaire is based on a sample of 86,185 clients completing the questionnaire at the start of a treatment episode. Factor analyses (principal components and varimax rotation) reveal that these items related to depression, anxiety, interpersonal problems, and impaired functioning in work, school and other daily activities all load on a common factor labeled Global Distress. Table A displays the results of this analysis. Beacon Health Options 13 Client Feedback Forms

14 CFF-ADULT Factor Analysis (N=86,185) Factor 1 in principle components is the Global Distress factor. Note that all GDS items have positive factor loadings of.54 or greater. Varimax rotation does little to alter factor structure. Global Distress Scale contains a single factor. Factor Method: Principle Components Factor Method: Varimax Rotation Scale/Subscale Item wording Factor 1 Factor 2 Factor 1 Factor 2 GDS/Symptoms...feel unhappy or sad? GDS /Functioning feel unproductive at work or other daily activities? GDS /Symptoms have problems with sleep (too much or too little)? GDS /Symptoms feel tense or nervous? GDS /Symptoms have little or no energy? GDS /Symptoms feel hopeless about the future? GDS /Social have a hard time getting along with family, friends, or coworkers? GDS /Social feel lonely GDS /Symptoms think about harming yourself GDS /Symptoms Substance Abuse Substance Abuse Substance Abuse have a hard time paying attention? have someone express concerns about your alcohol or drug use? have had a problem at work, school or home because of alcohol or drug use?...consume five or more drinks on a single occasion Variance Explained by Each Factor Final Commonality Estimate Ten of the items load heavily on the Global Distress Factor. This results in high reliability for the Global Distress Scale. Reliability as estimated using Cronbach s coefficient alpha was The CFF-Adult can be scored with subscales for symptoms, interpersonal problems and functioning. However, these subscales cannot be identified using factor analysis. Beacon Health Options 14 Client Feedback Forms

15 The Substance Abuse items do form a separate factor, and are scored on a separate scale with a Cronbach s alpha of Correlation between the Global Distress Scale and Substance Abuse scale was.14 (p<.001). The question how often did you have five or more drinks of alcohol at one time? is based on items widely used in assessment of binge drinking behavior, such as those used in the National Household Survey on Drug Abuse (SAMHSA, 1998) and Harvard School of Public Health College Alcohol Study (Henry, 1997). The addition of a question that touches on binge drinking behavior is a valuable supplement. The evidence of a common factor for global distress demonstrates construct validity. Similar measures of common symptoms of depression and anxiety have likewise been shown to share a common factor. These include the Beck Depression Inventory, the Beck Anxiety Inventory, the OQ-45, and the PHQ9. Correlational studies between the Adult questionnaire and the both the PHQ9 and the Beck Depression Inventory show a concurrent validity of 0.8. IRT Analysis of Global Distress and Substance Abuse Scales The initial development of the Adult form was informed by an IRT analysis of the 10 global distress scale and 3 substance abuse scale items. These analyses were conducted on a smaller dataset available at the time of initial form development. Label N Mean MH (Global Distress) Variance Kurtosis Min Max Itemtotal MSA Std. Loading on One Factor Rasch Measure Score Unhappy or sad Little or no energy Getting along with family and friends Hard time paying attention Unproductive at work Problems with sleep Tense or nervous Lonely Hopeless Self-harm Infit Outfit Disc. Substance Abuse Someone expressed SA concerns SA problem at work/home Five or more drinks Beacon Health Options 15 Client Feedback Forms

16 The table below summarizes the overall scale characteristics based on this sample: Scale Items Cronbach's alpha MSA Bentler CFI 1 factor RMSEA 1 factor Rasch Person Reliability Rasch Item Separation Reliability MH (Global Distress) SA Formulas to predict final score (benchmark score) for the episode of care First assessment at session 1 or 2 Last GDS score=first GDS Score * (R-square=.30) First assessment at session 3 or later Last GDS score=first GDS Score * (R-square=.37) First assessment session unknown Last GDS score=first GDS Score * (R-square=.40) See Appendix A for detailed normative information and predictive models. Beacon Health Options 16 Client Feedback Forms

17 Section 5: CFF Adolescent Specific Information Reliability and Construct Validity The Adolescent version of the CFF can be completed by either the youth or a parent. The questionnaire contains a total of 15 items, seven of which assess symptoms of anxiety and depression, while eight assess symptoms related to attention and behavioral problems. A single item inquires about drug or alcohol use. Factor analysis (principal components) likewise reveals that all items load on a common factor, as evidenced by reliability (coefficient alpha) of 0.87 for the full scale. The sample size for this analysis was 13,394. The factor structure was the same whether the questionnaire was completed by the youth or an adult who knew the youth well. Factor analysis with varimax rotation indicates the existence of two factors, one containing symptoms of anxiety and depression (sometimes referred to as internalizing symptoms) while the second contains symptoms related to attention and behavioral problems (externalizing symptoms). The main difference is that the internalizing items accounted for slightly more variance than externalizing items with the youth report version, while the opposite was the case for the parent completed version. Beacon Health Options 17 Client Feedback Forms

18 Youth CFF Factor Analysis (Responder=Youth, N=9,647) Factor 1 in principle components is the Global Distress factor. Note that all GDS items have positive factor loadings of.45 or greater. Rotated Factor 1 reflects symptoms of depression and anxiety. This accounts for more of the variance with Youth completed forms than Factor 2, which reflects attention and behavioral problems. Factor 3 is substance abuse. Factor Method: Principle Components Factor Method: Varimax Rotation Scale GDS/Symptoms Item wording eat a lot more or a lot less than usual? Factor 1 Factor 2 Factor 3 Factor 1 Factor 2 Factor GDS/Social get into trouble? GDS/Social interrupt others? GDS/Social lie to get things you wanted? GDS/Symptoms have a hard time controlling your temper? GDS/Symptoms worry about a lot of things? GDS/Symptoms feel worthless? GDS/Symptoms have a hard time having fun? GDS/Social have a hard time waiting your turn? GDS/Social GDS/Social hang out with kids who get into trouble? feel nervous and/or shy around other people? GDS/Functioning lose things you need? GDS/Symptoms have a hard time sleeping because you were worried? GDS/Social annoy other people on purpose? GDS/Social Substance Abuse think that you don't have any friends? drink alcohol or use other substances Variance Explained by Each Factor Final Commonality Estimate Beacon Health Options 18 Client Feedback Forms

19 Youth CFF Factor Analysis (Responder= Parent or other adult, N=3747) Factor 1 in principle components is the Global Distress factor. Note that all GDS items have positive factor loadings of.45 or greater. (Note that Factors 1 and 2 in varimax rotation are reversed from Youth completed version) Factor 1 reflects attention and behavioral problems and accounts for more variance than Factor 2, symptoms of depression and anxiety. Factor Method: Principle Components Factor Method: Varimax Rotation Scale GDS/Symptoms Item wording eat a lot more or a lot less than usual? Factor 1 Factor 2 Factor 3 Factor 1 Factor 2 Factor GDS/Social get into trouble? GDS/Social interrupt others? GDS/Social lie to get things you wanted? GDS/Symptoms have a hard time controlling your temper? GDS/Symptoms worry about a lot of things? GDS/Symptoms feel worthless? GDS/Symptoms have a hard time having fun? GDS/Social have a hard time waiting your turn? GDS/Social hang out with kids who get into trouble? GDS/Social feel nervous and/or shy around other people? GDS/Functioning lose things you need? GDS/Symptoms GDS/Social GDS/Social Substance Abuse have a hard time sleeping because you were worried? annoy other people on purpose? think that you don't have any friends? drink alcohol or use other substances Variance Explained by Each Factor Final Commonality Estimate Beacon Health Options 19 Client Feedback Forms

20 The coefficient alpha for the internalizing factor was 0.84, and for the attention/behavior problems The correlation between the two factors was The correlation between the single substance abuse item and internalizing items was 0.15, while the correlation with externalizing items was All correlations were statistically significant (p<.001). As with the Adult questionnaire, the Adolescent version is scored with a Global Distress Scale with subscales for symptoms (anxiety/depression), interpersonal problems, and attention/functioning items, with the externalizing symptom items divided between interpersonal and attention/functioning problems. Despite the evidence for up to three factors as revealed by varimax rotation, the correlation between factors is strong such that they can be treated as a single factor measure. This is consistent with scoring instructions for the OQ-64. As with the Adult questionnaire, the factor analysis shows evidence of construct validity. The Youth measure will likewise show a strong correlation with other similar youth measures such as the Youth Outcome Questionnaire, the Child Behavior Checklist, and the Connors Rating Scale. Beacon Health Options 20 Client Feedback Forms

21 Formulas to predict final score (benchmark score) for the episode of care Youth self-completed First assessment at session 1 or 2 Last GDS score=first GDS Score * (R-square=.30) First assessment at session 3 or later Last GDS score=first GDS Score * (R-square=.37) First assessment session unknown Last GDS score=first GDS Score * (R-square=.40) Youth parent-completed First assessment at session 1 or 2 Last GDS Score = First GDS Score * (R-square=.28) First assessment at session 3 or later Last GDS Score = First GDS Score * (R-square=.41) First assessment session unknown Last GDS Score = First GDS Score * (R-square=.41) See Appendices B and C for detailed normative information and predictive models. Beacon Health Options 21 Client Feedback Forms

22 Section 6: CFF Child Specific Information Reliability and Construct Validity The Child version has 16 items, with similar content to the adolescent version. Some items are substituted to be more age appropriate. The sample size of the Child version was 8,948. As with the adolescent version, the factor structure was the same whether completed by the child or a parent. Factor analysis (principal components) likewise reveals that all items load on a common factor, as evidenced by reliability (coefficient alpha) of 0.90 for the full scale. Factor analysis with varimax rotation indicates the existence of three factors, one containing symptoms of anxiety and depression (sometimes referred to as internalizing symptoms), the second with items reflecting behavior problems, and the third with symptoms related to attention problems. The coefficient alpha for the anxiety/depression factor was 0.75, for the behavior problems 0.89, and for the attention problems The correlation between behavior problems and attention problems was 0.69, while the correlation between behavior problems and symptoms of depression/anxiety was The correlation between attention and depression/anxiety was All correlations were statistically significant (p<.001). The questionnaire can be scored as a single Global Distress Scale. Despite the evidence for up to three factors as revealed by varimax rotation, the correlation between factors is sufficiently strong such that the instrument can be treated as a single factor measure. As with the Youth questionnaire, the factor analysis is evidence of construct validity. The Child measure will likewise show a strong correlation with other similar youth measures such as the Youth Outcome Questionnaire, the Child Behavior Checklist, and the Connors Rating Scale. Beacon Health Options 22 Client Feedback Forms

23 Child CFF Factor Analysis (Responder=Youth, N=3,278) Factor 1 in principle components is the Global Distress Scale. Note that all but one GDS items have positive factor loadings of.49 or greater. A single item, shyness, is at.25 Rotated Factor 1 reflects behavioral problems. Rotated Factor 2 reflects internalizing symptoms of depression and anxiety. Rotated Factor 3 reflects problems with attention. Combined Rotated Factors 1 and 3 for measure of externalizing symptoms. Factor Method: Principle Components Factor Method: Varimax Rotation Factor Factor Factor Factor Factor Factor Scale Item wording GDS/Symptoms cry easily? GDS/Symptoms feel unhappy or sad? GDS/Social get into trouble? GDS/Social interrupt others? GDS/Social lie to get things you wanted? GDS/Symptoms have a hard time controlling your temper? GDS/Symptoms worry about a lot of things? GDS/Symptoms have a hard time sitting still? GDS/Symptoms GDS/Social GDS/Social GDS/Social have a hard time paying attention? have a hard time waiting your turn? get into fights with family members and/or friends? feel nervous and/or shy around other people? GDS/Functioning lose things you need? GDS/Social argue with adults? GDS/Social GDS/Social annoy other people on purpose? think that you don't have any friends? Variance Explained by Each Factor Final Commonality Estimate Beacon Health Options 23 Client Feedback Forms

24 Child CFF Factor Analysis (Responder=Parent or other adult, N=6,114) Factor 1 in principle components is the Global Distress Scale. Note that all but one GDS items have positive factor loadings of.49 or greater. A single item, shyness, is at.25 Factor structure is same as for Child completed, but with different order in factors. Rotated Factor 1 reflects behavioral problems. Rotated Factor 2 reflects problems with attention. Rotated Factor 3 reflects internalizing symptoms of depression and anxiety. Combined Rotated Factors 1 and 2 for measure of externalizing symptoms. Factor Method: Principle Components Factor Method: Varimax Rotation Scale Item wording Factor Factor Factor Factor Factor Factor GDS/Symptoms cry easily? GDS/Symptoms feel unhappy or sad? GDS/Social get into trouble? GDS/Social interrupt others? GDS/Social lie to get things you wanted? GDS/Symptoms have a hard time controlling your temper? GDS/Symptoms worry about a lot of things? GDS/Symptoms have a hard time sitting still? GDS/Symptoms GDS/Social GDS/Social GDS/Social have a hard time paying attention? have a hard time waiting your turn? get into fights with family members and/or friends? feel nervous and/or shy around other people? GDS/Functioning lose things you need? GDS/Social argue with adults? GDS/Social GDS/Social annoy other people on purpose? think that you don't have any friends? Variance Explained by Each Factor Final Commonality Estimate Beacon Health Options 24 Client Feedback Forms

25 Formulas to predict final score (benchmark score) for the episode of care Child self-completed First assessment at session 1 or 2 Last GDS Score = First GDS Score * (R-square=.31) First assessment at session 3 or later Last GDS Score = First GDS Score * (R-square=.33) First assessment session unknown Last GDS Score = First GDS Score * (R-square=.40) Youth parent-completed First assessment at session 1 or 2 Last GDS score = First GDS Score * (R-square=.31) First assessment at session 3 or later Last GDS Score = First GDS Score * (R-square=.36) First assessment session unknown Last GDS Score = First GDS Score * (R-square=.40) See Appendices D and E for detailed normative information and predictive models. Beacon Health Options 25 Client Feedback Forms

26 References Brophy CJ, Norvell NK, Kiluk DJ (1988) An Examination of the factor structure and convergent and discriminant validity for the SCL-90R in an outpatient clinic population. Journal of Personality Assessment 52(2) Brown (2013) Measurement Plus Feedback Equals Improved Outcomes: An evidence based practice. Brown J, Minami T (2013) Quality of Life and Well-being Questionnaires. Enns MW, Coxa BJ, Parker B JDA, & Guertinc JE (1998) Confirmatory factor analysis of the Beck Anxiety and Depression Inventories in patients with major depression Journal of Affective Disorders Goodman JD, McKay JR, DePhilippis D (2013) Progress Monitoring in Mental Health and Addiction Treatment: A Means of Improving Care. Professional Psychology: Research and Practice 44(4) Henry W. (1997) Harvard School of Public Health College Alcohol Study, Ann Arbor, MI: Interuniversity Consortium for Political and Social Research. Jacobson, NS., & Truax, P. (1991). Clinical significance: A statistical approach to defining meaningful change in psychotherapy research. Journal of Consulting and Clinical Psychology, 59, Lambert MJ (2009) Yes, It Is Time for Clinicians to Routinely Monitor Treatment Outcome. In Miller, S. & Hubble, (M. Eds.), Heart and Soul of Change (2nd ed.). American Psychological Association Press: Washington. Miller DD, Duncan Bl, Brown J et al. (2003) The Outcome Rating Scale: A preliminary study of the reliability, validity, and feasibility of a brief visual analog measure. Journal of Brief Therapy 2 (2), Beacon Health Options 26 Client Feedback Forms

27 Minami T, Brown GS, McCulloch J, Bolstrom B. (2011) Benchmarking therapists: Furthering the benchmarking method in its application to clinical practice. Quality & Quantity. 46: Minami T, Wampold BE, Serlin RC, Hamilton EG, Brown GS, & Kircher JC. (2008a) Benchmarking the effectiveness of psychotherapy treatment for adult depression in a managed care environment: A preliminary study. Journal of Consulting and Clinical Psychology. 76, Minami, T, Serlin, RC, Wampold, BE, Kircher, JC, & Brown, GS (2008b) Using clinical trials to benchmark effects produced in clinical practice, Quality and Quantity 42: Minami, T., Wampold, B. E., Serlin, R. C., Kircher, J. C., & Brown, G. S. (2007) Benchmarks for psychotherapy efficacy in adult major depression, Journal of Consulting and Clinical Psychology, 75, SAMHSA (1998) National Household Survey on Drug Abuse. Beacon Health Options 27 Client Feedback Forms

28 Appendix A: Adult CFF Normative and Benchmarking Data Adult CFF - Cases with pre-post change and with first assessment at session 1 or 2. First GDS Score Last GDS Score Pre-post change # CFF per episode Pre-post weeks N Mean (sd) Mean (sd) Mean (sd) Mean (sd) Mean (sd) All 23, (0.77) 1.64 (0.80) 0.50 (0.73) 5.37 (6.33) (18.25) ADHD (0.72) 1.59 (0.68) 0.45 (0.70) 4.66 (4.17) (14.84) Adjustment disorder 3, (0.70) 1.46 (0.72) 0.44 (0.70) 5.04 (5.21) (16.34) Anxiety 2, (0.69) 1.68 (0.75) 0.44 (0.71) 5.68 (6.40) (18.96) Behavior disorder (0.72) 1.34 (0.79) 0.38 (0.70) 4.12 (2.44) (8.70) Bipolar (0.76) 1.97 (0.82) 0.49 (0.82) 6.82 (9.64) (25.08) Depression 4, (0.69) 1.84 (0.79) 0.60 (0.75) 6.14 (7.00) (20.44) Eating disorder (0.77) 1.64 (0.81) 0.49 (0.69) 6.12 (4.98) (17.75) Not specified 10, (0.79) 1.54 (0.78) 0.51 (0.71) 4.82 (5.84) (16.39) Other (0.78) 1.74 (0.82) 0.42 (0.71) 5.54 (6.10) (17.29) PTSD (0.73) 1.94 (0.81) 0.44 (0.72) 6.69 (7.93) (21.33) Personality disorder (0.61) 1.83 (0.66) 0.67 (0.52) 4.30 (2.50) 8.37 (3.95) Psychosis (0.84) 1.86 (0.99) 0.37 (0.83) 6.51 (6.27) (25.21) Substance abuse (0.94) 1.35 (0.82) 0.38 (0.83) 5.73 (5.34) (15.00) Adult CFF - Cases with intake scores in the clinical range, multiple assessments within the episode and care, and the first assessment at session 1 or 2. First GDS Score Last GDS Score Pre-post change # CFF per episode Pre-post weeks N Mean (sd) Mean (sd) Mean (sd) Mean (sd) Mean (sd) All 19, (0.58) 1.80 (0.75) 0.61 (0.71) 5.56 (6.54) (18.79) ADHD (0.54) 1.72 (0.64) 0.58 (0.66) 4.81 (4.45) (15.52) Adjustment disorder 2, (0.52) 1.64 (0.68) 0.58 (0.69) 5.24 (5.45) (17.02) Anxiety 1, (0.55) 1.79 (0.73) 0.56 (0.69) 5.73 (6.60) (19.38) Behavior disorder (0.54) 1.58 (0.73) 0.52 (0.66) 3.82 (2.00) (8.73) Bipolar (0.62) 2.05 (0.80) 0.56 (0.79) 6.72 (8.38) (24.59) Depression 4, (0.58) 1.90 (0.77) 0.66 (0.73) 6.20 (7.05) (20.37) Eating disorder (0.54) 1.85 (0.71) 0.56 (0.70) 5.06 (4.24) (17.45) Not specified 8, (0.58) 1.73 (0.74) 0.63 (0.70) 5.04 (6.20) (17.15) Other (0.61) 1.88 (0.78) 0.53 (0.68) 5.77 (6.32) (16.90) PTSD (0.60) 2.04 (0.76) 0.50 (0.70) 6.70 (8.02) (21.45) Personality disorder (0.61) 1.83 (0.66) 0.67 (0.52) 4.30 (2.50) 8.37 (3.95) Psychosis (0.68) 2.02 (0.96) 0.46 (0.82) 6.89 (6.63) (24.82) Substance abuse (0.60) 1.65 (0.70) 0.68 (0.73) 6.03 (5.98) (17.13) Beacon Health Options 28 Client Feedback Forms